Agent Economics

InferenceForge: Agents Optimizing Agentic Serving

Published

InferenceForge is our recursive self-improvement loop for inference: a collection of a custom environment, harness, and evaluation suite that lets an off-the-shelf coding agent do SLA-aware inference optimization. The agent works like a senior engineer pushing serving performance. It measures key metrics such as cache hit rates and per-kernel time, writes and runs its own test scripts and microbenchmarks, identifies the most significant bottleneck, implements a targeted optimization, and does all the necessary testing and validation.

Under a 30 tok/s per user latency SLA, InferenceForge pushed SGLang Kimi K3 serving to 12.8k tok/s per GPU, 43% above the best published AgentX reference meeting the same bar (8.9k tok/s per GPU). It took 9 iterations over 3 days to optimize the serving config.

Best throughput at the 30 tok/s per user SLA after each of nine loop iterations on Kimi K3, stepping from the published 8.9k up to 12.8k tok/s per GPU, with fixes 1 and 2 at iteration 5 and fix 3 at iteration 6
Figure 1. Best one-hour AgentX throughput after each optimization iteration, at ≥30 tok/s per user on a single 8×B300 node. Iteration 0 represents the best published SemiAnalysis InferenceX AgentX result for Kimi K3 under the same SLA, as of November 10, 2026.

The biggest gains came from fixes specific to agent traffic, where a few very long contexts sit beside many short ones. Our serving stack is built on a fork of SGLang, and each fix landed there as a pull request. Three fixes contributed most of the gains:

  1. Splitting long attention reads across more of the GPU. [PR] The attention decode kernel chose how many pieces to split each request's context into from the batch size alone, so at 40 or more concurrent requests each one ran on a single thread block, and the longest context in the batch set the time of every decode step: 417 µs per launch under AgentX load, against 55 µs with uniform lengths. The agent forced at least 8 splits per request, cutting a step with one 900K-token request from 522 to 126 µs. Together with fix 2, it made iteration 5 on the chart: +7.5% at the SLA, from 10.9k to 11.8k tok/s per GPU.

    Schematic of decode thread blocks: before, one 900K-token request on a single block sets a 522 µs step while other blocks sit idle; after, every request is split at least 8 ways and the step takes 126 µs
    Figure 2. Kernel times from our microbenchmark at 40 requests on B300.
  2. Turning a sequential recurrence into a parallel scan. [PR] K3's linear-attention layers update their state 64 tokens at a time, 512 steps in a row for a 32K-token prefill chunk, on only 48 of the GPU's 148 SMs. The agent noticed the update is affine, so it computes 8 segments in parallel and chains their states: 2.8x faster on a 32K chunk (1,199 to 426 µs).

    Schematic of the linear-attention state update: before, 512 dependent 64-token steps take 1,199 µs per 32K chunk; after, a three-stage parallel scan takes 426 µs
    Figure 3. Kernel times for one 32K-token prefill chunk on B300.
  3. Packing the cached prefix to FP8 in one pass. [PR] With a long cached prefix, each attention layer rebuilt K and V with strided copies and then cast them to FP8 in separate steps. The agent wrote one kernel that packs and casts together, bit-exact with the original, cutting 889 to 180 µs per layer on a 100K-token prefix. It made iteration 6, alongside a smaller MoE routing change: +4.4% at the SLA, from 11.8k to 12.3k tok/s per GPU.

    Schematic of building FP8 K and V from a cached prefix: before, strided copies then separate casts take 889 µs per layer; after, one fused pack-and-cast kernel takes 180 µs
    Figure 4. Kernel times per attention layer on a 100K-token cached prefix, B300.

We then ran InferenceForge on two more models: Kimi K3.1, a newer version of K3, and GLM-5.3, a different architecture. Below are the full serving frontiers for all three models.

Results

Each chart below is a serving frontier built from one-hour AgentX runs. Serving more users at once raises throughput (total tokens per second per GPU) but lowers interactivity (the median decode speed each user sees), so each run is one point, and the frontier is the best throughput reached at each speed. Ours is the colored line. The gray line is the full frontier of SemiAnalysis's published results for that model on B300, pulled from InferenceX on November 10, and each comparison below links to the published run. We start with Kimi K3.

Kimi K3

Kimi K3 serving frontier on B300: our one-hour runs above the best published vLLM line, 43% above at the 30 tok/s per user SLA and 29% at peak
Figure 5. Ours: one-hour AgentX runs, Oct 30 to Nov 4, one 8×B300 node. Best published: the SemiAnalysis InferenceX AgentX frontier for Kimi K3 on B300 (vLLM, Aug 16), as of Nov 10.

On one 8×B300 node, InferenceForge pushed Kimi K3 serving to 16.2k tok/s per GPU, 29% above the best published AgentX result. Our SLA point alone, 12.8k, already beats the best published system's peak with no SLA at all.

Kimi K3.1

Kimi K3.1 serving frontier on B300: our one-hour runs above the best published SGLang line, 5% above at 56 tok/s per user
Figure 6. Ours: one-hour AgentX runs, Nov 7 to 10, one 8×B300 node. Best published: the SemiAnalysis InferenceX AgentX frontier for Kimi K3.1 Pro on B300 (SGLang, Nov 5), as of Nov 10.

Our best K3.1 point clears the SLA with room to spare: 23.7k tok/s per GPU at 56 tok/s per user, 5% above the best published result (SGLang, 22.6k at 53 tok/s per user), while also being faster for each user. This lead holds across the frontier.

GLM-5.3

GLM-5.3 is a different model family, and here we tuned mainly for peak throughput. SemiAnalysis hasn't listed GLM-5.3 yet, so we compare against the best published results for GLM-5.2, which shares its architecture.

GLM-5.3 serving frontier on B300: our one-hour runs 12% above the best published GLM-5.2 peak and ahead of the published line up to 160 tok/s per user
Figure 7. Ours: one-hour AgentX runs, Oct 30 to Nov 10, one 4×B300 engine. Best published: the SemiAnalysis InferenceX AgentX frontier for GLM-5.2 on B300, which shares GLM-5.3's architecture (Dynamo, Oct 14; SGLang, Oct 22), as of Nov 10.

At the best published peak (Dynamo) we serve 12% more per GPU, and the first token arrives much sooner: 4.5 s at p90 against 13.2 s.

What's Next

InferenceForge is our first step in this direction. Next, we will put our own models in the loop: FireFlash, the agentic model from our open RL recipe, and models post-trained specifically for inference optimization.

Acknowledgments

Thanks to the SGLang team, whose engine our serving stack is built on, and to the vLLM, SGLang, NVIDIA Dynamo and TensorRT-LLM teams, whose published results set the bar we measured against. Thanks to SemiAnalysis for InferenceX and its AgentX benchmark, which made a public, reproducible comparison possible; to the FlashInfer team, whose kernels the agent built on; and to Moonshot AI and Z.ai for releasing Kimi and GLM as open-weight models.

Authors

Jesse Cai, Huimin Zeng, Yuxuan Lu, Dakuo Wang, Kai-Wei Chang