In brief
We built a proprietary inference engine to push the performance of GPT-OSS-120B beyond what we achieved with vLLM on the same NVIDIA A100 80GB GPU.
Our latest benchmark achieved 813 aggregate decode tokens per second across eight concurrent requests, delivering more than 100 tokens per second per stream.
By optimising both input processing and token generation, we substantially reduced the total time needed to complete inference workloads, without requiring additional GPUs.
The challenge
Modern inference frameworks provide powerful general-purpose serving capabilities. However, large mixture-of-experts models introduce execution challenges that cannot always be solved through hardware scaling alone.
With GPT-OSS-120B, we wanted to improve performance across two critical stages of inference.
Prefill: Processing incoming context efficiently, particularly for long prompts and large documents.
Decode: Generating tokens quickly while maintaining performance across concurrent requests.
Our goal was to improve both stages on the same hardware rather than optimise one at the expense of the other.
Our approach
We developed a Rust and CUDA inference engine with an execution architecture tailored to the model and GPU.
Our work focused on improving how inference computations are organised and executed.
More efficient expert execution
GPT-OSS-120B uses a mixture-of-experts architecture, where different experts are activated depending on the incoming tokens.
We improved how expert workloads are grouped, routed and executed, reducing unnecessary memory movement and improving GPU efficiency.
Faster input processing
We redesigned prefill execution to improve reuse across requests and reduce repeated expert computation.
This allows the engine to process larger input workloads more efficiently while retaining the original model's computation and attention history.
Long-context attention optimisation
We implemented GPU-specific attention execution paths designed to improve throughput when requests contain thousands of tokens.
These optimisations help reduce memory and synchronisation overhead during generation.
GPU-native execution
We also reduced unnecessary CPU/GPU handoffs, improved execution scheduling and used CUDA Graphs to lower repeated execution overhead.
Together, these improvements allow us to extract more useful performance from the same accelerator.
Benchmark results
Our comparison uses GPT-OSS-120B on a single NVIDIA A100 80GB with eight concurrent requests and LongBench-derived inputs.
| Metric | vLLM | Our engine |
|---|---|---|
| GPUs | 1× A100 | 1× A100 |
| Concurrent requests | 8 | 8 |
| Input tokens per request | 8,192 | 8,192 |
| Output tokens per request | 500 | 500 |
| Prefill throughput | 6,500 tok/s | 13,000 tok/s* |
| Aggregate decode throughput | 600 tok/s | 813 tok/s |
| Per-stream decode throughput | 75 tok/s | 101.67 tok/s |
13K prefill is the updated figure supplied for this comparison.
Improvements across both inference stages
Our prefill improvements enable faster processing of incoming context, while higher decode throughput allows more tokens to be generated per second.
Using the stage throughputs above, the combined 8,192-input/500-output workload takes approximately 16.75 seconds with the vLLM baseline and 9.96 seconds with our engine.
That represents 68.2% higher calculated combined throughput, or approximately 40.5% less time to process the same workload.
Improving both stages is important because optimising decoding alone does not eliminate the time spent processing large inputs.
Why this matters
For AI agents, reasoning workloads and applications processing long documents, inference performance depends on both how quickly context is processed and how efficiently tokens are generated.
Our results demonstrate the opportunity to improve performance through better execution strategies rather than simply adding GPU capacity.
We are continuing to build our inference technology around actual workload characteristics, allowing us to improve efficiency across different models, hardware and traffic patterns.
The outcome
Our GPT-OSS-120B optimisation demonstrates the potential of a workload-specific inference engine:
One GPU. Eight concurrent requests. Over 100 decode tokens per second per stream.
By improving GPU execution efficiency and reducing unnecessary overhead, we can deliver more inference capacity from existing hardware.
Our goal is straightforward: faster, more efficient inference for real AI workloads.