System One Decision Models: Optimizing Routing Latency in Multi-Agent Architectures

Within modern multi-agent systems, one of the most frustrating bottlenecks undermining user experience and inflating cloud infrastructure cost is routing latency. Traditionally, when an incoming prompt arrives, architectures dispatch requests to a large generative language model merely to generate natural language explanations or JSON payloads to decide which downstream worker should handle the task.

The emergence of System One models (also recognized as Decision Models) completely restructures this workflow. Rather than generating autoregressive natural language text, these architectures are specifically tuned to output raw probability vectors, discrete categorical classifications, and calibrated confidence scores in single-digit milliseconds.

1. Generative LLMs vs. Decision Models

Standard generative models are architected for sequential token prediction. When leveraged as multi-agent routers:

• They generate textual responses (e.g., {"route": "database_engineer"}), incurring latency penalties between 200ms and 1,500ms before initial dispatch.
• They introduce parsing fragility, susceptible to formatting hallucinations or extraneous conversational prose.
• Decision Models: Consume raw text representations but terminate in classification prediction heads, directly exporting calibrated numeric arrays in 5ms to 25ms with 90% reduced compute overhead.

2. Production Applications in Multi-Agent Pipelines

In distributed agent orchestration (such as the architecture detailed in the 4-Agent Pipeline in Claude Code), Decision Models fulfill three high-throughput mandates:

1. Ultra-Low-Latency Intent Dispatch: Deterministically routing user prompts between retrieval engines (RAG), code generation pipelines, or administrative tools.
2. Deterministic Safety Guardrails: Evaluating prompt-injection and policy compliance in near-zero runtime before engaging upstream frontier models.
3. Outcome Evaluation (Reward Scoring): Rapidly evaluating and scoring candidate generations across multi-agent deliberations to select optimal responses.

# High-throughput System One Decision Routing
def route_request(user_prompt):
    # Predict continuous confidence array directly without generating tokens
    scores = decision_model.predict_categories(user_prompt)
    
    # scores: {"coding": 0.94, "security_risk": 0.02, "rag": 0.04}
    if scores["security_risk"] > 0.8:
        return "BLOCKED_BY_GUARDRAIL"
    if scores["coding"] > 0.7:
        return "DISPATCH_TO_CODER_AGENT"
    return "DISPATCH_TO_STANDARD_RAG"

Read next: Integrating gstack with Claude Code: Complete Installation and Optimization Guide

Conclusion

Pairing rapid System 1 Decision Models for millisecond-level routing with deep System 2 reasoning engines for complex execution defines the frontier architecture for highly responsive, cost-effective enterprise AI platforms.