Qwen2.5-7B Inference Optimisation with Phala

Model: Qwen2.5-7B Hardware: 1× NVIDIA H200 Precision: FP8 Input length: 2,000 tokens Output length: 1,000 tokens Concurrency: 32 concurrent requests Metric: mean per-request generation rate, measured with 32 requests in flight Evaluation: Phala

The challenge

Phala was serving Qwen2.5-7B as an inference provider when Geodd began working on improving the model's inference performance.

The target was the rate each individual user sees while the server is under load. A single NVIDIA H200 had to hold 32 concurrent requests and still return a high per-request generation rate, rather than trading per-user speed for aggregate throughput.

Geodd kept the model, the precision and the workload shape fixed so that the effect of the optimisation could be measured against the vLLM deployment already serving the model.

The objective was not simply to produce a benchmark result. Geodd wanted to turn the optimisation into a deployable implementation that could be run and evaluated outside Geodd's own infrastructure.

The optimisation

Geodd used its AI optimisation agent, powered by the Mosaic LLM, to analyse the target workload and produce a faster inference implementation for Qwen2.5-7B on NVIDIA H200.

This is a question about execution path selection, not about which serving framework is better in general. For a given model shape, hardware generation, precision and workload, there is a faster way to execute the model and a slower one, and which is which changes from workload to workload. Finding that path is normally expert work done by hand, per model, per GPU, per traffic pattern. That search is what the agent automates.

For this workload the agent:

• read the workload shape: a 7B decoder in FP8, a 2,000-token prefill, a 1,000-token generation, 32 concurrent requests, on Hopper-class hardware with high memory bandwidth
• selected the execution path it predicted would be fastest for that shape: a TensorRT-LLM build using the FP8 GEMM plugin path, FP8 context attention and a paged KV cache
• modified the TensorRT-LLM GEMM plugin source for the target workload rather than only configuring what was already there
• rebuilt the implementation, benchmarked it, and packaged it for deployment

Geodd then benchmarked the implementation the agent produced against the vLLM FP8 deployment in place, on the same class of NVIDIA H200 hardware and the same workload shape.

The benchmark used:

• 1× NVIDIA H200
• FP8 precision
• 2,000-token input
• 1,000-token output
• 32 concurrent requests

The implementation the agent produced included a modified GEMM plugin together with the following settings:

--gemm_plugin fp8 \
--tokens_per_block 64 \
--use_fp8_context_fmha enable \
--use_paged_context_fmha enable

These settings control several different parts of model execution:

• --gemm_plugin fp8 selects the FP8 GEMM plugin path containing Geodd's source-level changes for eligible matrix multiplications, using the FP8 calibration stored with the checkpoint.
• --tokens_per_block 64 configures the paged KV cache to store 64 tokens per block.
• --use_fp8_context_fmha enable enables FP8 fused multi-head attention during the context, or prefill, phase.
• --use_paged_context_fmha enable enables fused context attention over the paged KV cache.

On the NVIDIA H200's Hopper architecture, this combination allows TensorRT-LLM to use FP8 execution for eligible GEMM operations and the context-attention path while reducing intermediate memory movement during prefill. At 32 concurrent requests, prefill and generation compete for the same device, so the prefill path and the KV-cache layout affect the rate each user sees, not just the time to the first token.

Mosaic's role in this iteration went beyond selecting build flags. The agent changed the GEMM plugin source, integrated the modified plugin into the TensorRT-LLM build and combined it with the FP8 context-attention and paged KV-cache configuration used for the workload.

The model weights were not retrained or changed. The measured result reflects the complete inference implementation the agent produced, including the GEMM plugin changes, the runtime, precision path, attention path and KV-cache configuration, rather than attributing the gain to one flag alone.

Performance results

51.8% higher generation rate

The implementation the agent produced increased mean generation performance from 128.57 to 195.16 tokens per second per request, with 32 requests running concurrently on the same single H200.

This is the rate an individual user sees while the server is loaded, so the gain is in per-user speed under concurrency rather than in aggregate throughput alone.

The improvement was not limited to generation performance.

23.8% lower mean time to first token

Mean time to first token was 23.8% lower in the Geodd run compared with the vLLM deployment in place.

The context-attention configuration the agent selected is particularly relevant to this result because time to first token includes processing the 2,000-token input before generation begins, and at 32 concurrent requests that prefill work is repeated across every request in flight.

Together, the two measurements showed improvement in both the model's sustained generation rate and the time required to begin returning a response.

From optimisation to Phala evaluation

Following the optimisation work, Geodd packaged the Qwen2.5-7B implementation into a Docker image and provided it to Phala's engineering team for technical evaluation.

The teams worked directly on bringing the deployment into Phala's environment.

During the evaluation, Phala's engineering team ran the supplied Qwen2.5-7B deployment and confirmed that it was returning inference outputs. The team subsequently conducted initial performance testing of the implementation.

Geodd also provided technical support around deployment and runtime compatibility during the evaluation process.

Beyond an internal benchmark

For Geodd, the significance of the project was not only the 51.8% increase in mean generation rate.

The optimisation progressed through the complete engineering path:

Benchmark → AI-guided optimisation → Containerisation → External deployment → Provider-side testing

What the agent produced was not a recommendation or a report. It was a build: an execution path chosen for one workload, a source-level change made to support it, and a container that another team could run in their own environment. Geodd's system did the selection and the modification; the result left Geodd's benchmark environment and was tested by an external inference infrastructure team.

Result

Test configuration

The baseline measurement used the vLLM FP8 deployment in place on a single NVIDIA H200. The Geodd measurement used an FP8 TensorRT-LLM implementation whose GEMM plugin source was modified by the Mosaic-powered optimisation agent.

The benchmark workload used a 2,000-token input, a 1,000-token output and 32 concurrent requests. Reported generation rates are per request under that concurrency.

This is an implementation-level comparison between the deployment in place and the implementation the agent produced. Both are complete serving implementations, and the figures describe the implementations as a whole. They do not isolate the contribution of the GEMM plugin source changes from the other execution settings the agent selected.

The result is specific to this model, this hardware and this workload shape. A different shape can select a different execution path.