The challenge
We were extending Tenstorrent’s existing TT-Transformers implementation to support Ministral 8B and UnslopNemo 12B. This required adapting tokenizer handling and matrix-multiplication configurations to the models while working within Tenstorrent’s tiled execution model.
During testing, UnslopNemo 12B ran at batch size 1 with both temperature settings. It also ran at batch size 32 with temperature 0. But switching that batch to temperature 1 produced a configuration that did not work.
The model could generate tokens, and batching worked with greedy decoding. The remaining problem was enabling sampled generation at the same batch size.
What the AI agent changed
Geodd Autopilot produced source-level changes to the runtime rather than only adjusting generation settings. The update addressed three parts of the implementation.
Host-side sampling
The agent replaced a full-vocabulary sorting step with a torch.topk candidate-selection path, capped at 1,000 tokens, before probability filtering and sampling. This avoided sorting the entire vocabulary on that path.
Batch handling
The agent updated input splitting and decode-output collection to use the actual batch size assigned to each model instance. Previously, the implementation collected maximum-sized outputs and trimmed them only after concatenation.
Model-specific attention dimensions
The agent corrected the attention-output projection’s input width to use n_heads * head_dim, accommodating models where that width differs from the hidden size.
These changes were integrated with the model-support work while reusing TTNN’s existing transformer operations.
The results
The before-and-after runs covered four configurations on Tenstorrent P150b, all at a reported clock of 1400 MHz.
| Temperature | Batch size | Before | After |
|---|---|---|---|
| 0 | 1 | 42 TPS | 42 TPS |
| 1 | 1 | 38 TPS | 38 TPS |
| 0 | 32 | 34 TPS | 34 TPS |
| 1 | 32 | Not working | 29 TPS |
TPS means tokens per second. The run reports per user per tps and test was done on concurrent request as per batch size
The main result was a previously failing batch-32 sampling configuration becoming operational, rather than a speed increase in configurations that already worked. The three working configurations recorded the same generation rates after the update.
These results describe the complete implementation change. They do not isolate the contribution of sampling, batch handling or model-configuration corrections individually.
Beyond getting the model to load
For this workload, supporting the model meant more than producing an output from a single prompt. The runtime also needed to handle the tested batch sizes and sampling settings.
Geodd’s AI agent changed the serving implementation to extend that tested operating range. The model weights were unchanged; the work was in model integration, host-side sampling and batch handling.
On Tenstorrent P150b, that moved UnslopNemo 12B from a failing temperature-1, batch-32 run to a working configuration reporting 29 TPS.
Sampling tradeoff
The bounded candidate-selection path approximates full-vocabulary top-p sampling and can change the sampling distribution. For top_p >= 0.99, the implementation samples directly from the full distribution.
The result therefore demonstrates an operational configuration, not identical sampling behaviour across every setting.