Bringing Arcee Trinity to TensorRT-LLM - Part One

Bringing Trinity Mini to TensorRT-LLM — Part One

We brought Arcee AI’s Trinity Mini to TensorRT-LLM and used a Geodd AI agent to identify and fix a major inference bottleneck.

Trinity uses the AFMoE architecture, which combines sliding-window and global attention layers. This reduces the work required to process long contexts while preserving access to information across the full sequence. (Arcee AI, AFMoE configuration)

The problem

The model ran, but TensorRT-LLM was not receiving the attention-window size for each layer.

Without that configuration, the runtime treated every layer as full attention. The output remained correct, but prefill became unnecessarily expensive. In our reproduction, time to first token increased to approximately 15 seconds instead of the expected sub-second range.

What the AI agent found

The agent traced the bottleneck to max_attention_window_size.

Trinity follows a repeating pattern of three sliding-window attention layers followed by one global attention layer. TensorRT-LLM therefore needs a per-layer configuration:

max_attention_window:
  - 2048
  - 2048
  - 2048
  - 2147483647

Here, 2048 limits local attention to the configured window, while 2147483647 represents unrestricted global attention. This pattern is repeated across the model’s layers.

The fix

The agent produced a runtime fix that:

1. Reads the AFMoE configuration from the TensorRT-LLM engine.
2. Reconstructs the correct local/global attention pattern.
3. Passes the resulting window list to ModelRunner or ModelRunnerCpp.
4. Enables paged context FMHA for more efficient prefill.

For trtllm-serve, the required options can be supplied through a YAML file:

kv_cache_config:
  max_attention_window: [2048, 2048, 2048, 2147483647]
  paged_context_fmha: true

The server is then started with:

trtllm-serve serve engine \
  --tokenizer /path/to/trinity-mini \
  --backend trt \
  --max_batch_size 32 \
  --trust_remote_code \
  --host 0.0.0.0 \
  --extra_llm_api_options afmoe_config.yaml

The result

TensorRT-LLM now respects Trinity’s intended hybrid-attention architecture instead of applying full attention to every layer.

This was not a modification to Trinity’s weights. It was a serving-path correction discovered and implemented by our AI agent to make the model run as its architecture intended.

In Part Two, we will publish the hardware configuration, benchmark methodology, and measured performance improvement.