RightNow AI is a research lab and software company working on GPU programming tools, CUDA development workflows, model-hardware co-design, and inference infrastructure.

Which NVIDIA GPUs are supported by RightNow AI?

RightNow AI supports all NVIDIA GPUs with CUDA Toolkit 11.0-12.5, including GeForce RTX 40/30/20 series, GTX 16/10 series, Quadro RTX, Tesla, A100, and H100.

How much does RightNow AI cost?

RightNow AI is free to use with unlimited profiling and benchmarking. RightNow Pro costs $20 per month and adds GPU emulator access (50+ GPUs), multi-GPU comparison, and 1,000 AI credits per month.

What CUDA development workflow does RightNow AI support?

RightNow AI supports CUDA development workflows that combine editing, profiling, emulation, remote GPU execution, and benchmarked performance analysis.

Can I use RightNow AI on macOS?

Yes, RightNow AI is fully available on macOS (Apple Silicon and Intel). Mac users can use remote GPUs for free or our built-in GPU emulator for CUDA profiling.

Serve reasoning as a tree.

AutoTree forks KV cache mid-generation, prunes weak branches with instant memory reclaim, and holds every request to a hard token budget. Open source, Apache-2.0, adoption is one base_url change.

View on GitHub Contact Sales

Available on RunInfra to optimize and deploy any HuggingFace model.

8.79x

KV reuse, measured end to end

31.6x

kernel branch scaling at 32 branches

instant

page reclaim on branch prune

verified

token parity vs stock HuggingFace

Every number ships with its model, hardware, workload, and configs in the repository. No exceptions.

Branching is nearly free

The tree-attention kernel scales close to linearly with branch count: exploring 32 candidates costs about as much per candidate as exploring one.

What only a tree engine can do

Prefix caching shares prompts. It cannot share what has not been generated yet. Tree execution shares the reasoning itself.

Capability	AutoTree	vLLM	SGLang	TRT-LLM
OpenAI-compatible API	✓	✓	✓	✓
Prefix / radix caching	✓	✓	✓	✓
Continuous batching	✓	✓	✓	✓
Mid-generation KV fork (token granularity)	✓	-	-	-
Mid-flight prune with instant page reclaim	✓	-	-	-
Hard token budget across a branch family	✓	-	-	-
Tree policies (beam / best-first / MCTS) in the scheduler	✓	-	-	-
Branch-level value-scorer hook	✓	-	-	-
Native tree endpoint (/v1/tree/completions)	✓	-	-	-

Compare what is measured today

AutoTree internal validation and token ablations are shown with their exact regimes. Comparison numbers against vLLM and SGLang land only with raw logs and one-command repro, so those rows stay visibly pending until the evidence ships.

Migrate in one line

Keep the OpenAI client and sampling controls your application already uses. Point base_url at AutoTree, then opt into tree execution per request.

Measured, not projected

Every claim carries its regime. Full head-to-head benchmarks against stock vLLM and SGLang, with their best flags on, are running now and will be published here with raw logs and one-command repro.

Metric	Result	Regime
KV reuse ratio, end to end	8.79x	H100, Qwen3-8B bf16, MATH-500, beam-8, 3 seeds
Tree-attention kernel branch scaling, 1 to 32 branches	31.6x	A100, bf16, context 128, kernel microbenchmark
Triton kernel parity vs reference implementation	106 / 106 tests	A100, all dtypes and GQA ratios
Greedy token parity vs stock HuggingFace	exact	Qwen3-8B, CUDA bf16

We publish our negatives too: naive log-probability branch selection loses to sequential sampling on solvable tasks. That measured baseline is exactly what the pluggable value scorer exists to beat. Details in the repository.

FAQs

AutoTree is an open-source serving engine that executes LLM reasoning as a tree: one shared prefix, many candidate branches, weak branches pruned mid-flight with their memory reclaimed instantly. It is built for teams running reasoning workloads, best-of-n sampling, RL rollout generation, and agentic search.

Prefix caching shares the prompt across requests. It cannot share generated tokens, because the prefix does not exist until someone generates it. AutoTree forks the KV cache mid-generation, so branches share the reasoning computed so far, and pruned branches return their memory to the pool while the request is still running.

The reference engine is open source today with the full test suite, measured results, and an honest limitations list. A production-speed engine built on SGLang's runtime, inheriting its continuous batching and speed with tree execution added, is in active development in the open.

Running now. The protocol is public before the results: baselines run with prefix caching enabled and their best flags, full branch-count sweeps rather than cherry-picked points, ablations that isolate where wins come from, and two workloads where we expect parity or worse, published anyway. Every number will ship with configs, seeds, and raw logs.

The component that decides which branches live. AutoTree ships with a pluggable scoring interface: log-probability, self-consistency voting, or your own reward model. Our published measurements show naive scoring is not enough, which is precisely why the interface is open and the baseline is public.

Yes. Apache-2.0, no strings. The engine, the benchmark harness, the results data, and the paper-grade validation evidence are all in the repository.

Enterprise deployments, Helm charts, SLURM templates, and dashboards ship in the repository. For dedicated support, contact us.

Serve reasoning as a tree.

AutoTree forks KV cache mid-generation, prunes weak branches with instant memory reclaim, and holds every request to a hard token budget. Open source, Apache-2.0, adoption is one base_url change.

View on GitHub Contact Sales

Available on RunInfra to optimize and deploy any HuggingFace model.

fork

active pruned winner

8.79x

KV reuse, measured end to end

31.6x

kernel branch scaling at 32 branches

instant

page reclaim on branch prune

verified

token parity vs stock HuggingFace

Every number ships with its model, hardware, workload, and configs in the repository. No exceptions.

One prompt, many candidate answers

Each character is one token of KV cache. Stock serving computes every candidate to the end. AutoTree shares the prefix, prunes losers mid-flight, and reclaims their memory instantly.

Stock serving: sequential best-of-4

0 KV pages held|prompt stored per candidate

AutoTree: beam-8, one shared prefix

0 KV pages held|1.00x KV reuse|pruned pages return to the pool

Branching is nearly free

The tree-attention kernel scales close to linearly with branch count: exploring 32 candidates costs about as much per candidate as exploring one.

Decode throughput vs branch count, relative to 1 branch

hover to inspect

╌ ideal linear

Hover the chart: the second column is the cost of one candidate relative to running a single branch, at that branch count. Measured on A100 SXM4, bf16, tree-attention decode kernel microbenchmark, 200 iterations. Kernel-level result, not an end-to-end serving number. Raw data and repro command in the repository.

Decode tokens computed per task, same engine, same protocol

Sequential best-of-4 (stock serving pattern)2,560 tokens / task

AutoTree beam-8, 1,920 budget969 tokens / task

vLLM (prefix caching on, native n>1)measuring, publishing with raw logs

SGLang (radix cache on, native n>1)measuring, publishing with raw logs

Measured: H100, Qwen3-8B bf16, 25 numeric MATH-500 tasks, 3 seeds. At this configuration the tree spends 2.6x fewer decode tokens per task and reaches 8.79x KV reuse; accuracy without a value scorer is lower than sequential sampling, and that measured baseline ships in the repository. Head-to-head vLLM and SGLang rows land with their best flags on, full sweeps, seeds, and one-command repro.

What only a tree engine can do

Prefix caching shares prompts. It cannot share what has not been generated yet. Tree execution shares the reasoning itself.

Capability	AutoTree	vLLM	SGLang	TRT-LLM
OpenAI-compatible API	✓	✓	✓	✓
Prefix / radix caching	✓	✓	✓	✓
Continuous batching	✓	✓	✓	✓
Mid-generation KV fork (token granularity)	✓	-	-	-
Mid-flight prune with instant page reclaim	✓	-	-	-
Hard token budget across a branch family	✓	-	-	-
Tree policies (beam / best-first / MCTS) in the scheduler	✓	-	-	-
Branch-level value-scorer hook	✓	-	-	-
Native tree endpoint (/v1/tree/completions)	✓	-	-	-

Compare what is measured today

Engine

Arms

Accuracy per arm

hover to inspect, click to pin

Internal validation uses 12 math tasks on A10, so every result keeps its n visible. The 7B-AWQ tree result is reported as the observed 11-12/12 range; the bar uses 11/12 rather than inventing a midpoint.

Decode tokens per task and per correct answer

linked inspection

tokens/task tokens/correct not yet reported

The EMVPT ablation reports 725 tokens/task with pruning and 956 without it. Paired accuracy was not supplied for this ablation, so tokens/correct stays explicitly unreported.

vLLM + SGLang: measuring, publishing with raw logs. No cross-engine value is plotted until its regime, raw logs, and one-command repro are ready.