gRPC Pipeline¶

When workers communicate via gRPC, SMG becomes a complete OpenAI-compatible server with a sophisticated request processing pipeline for reasoning extraction, tool call parsing, and MCP execution.

Overview¶

Pipeline Architecture¶

Responsibility Comparison¶

Reasoning Parsers¶

Reasoning parsers extract chain-of-thought content from model outputs. Essential for models that produce thinking tokens before their final response.

Configuration¶

Supported Parsers¶

smg --reasoning-parser deepseek_r1

smg --reasoning-parser qwen3

smg --reasoning-parser kimi

smg --reasoning-parser glm45

Complete Parser Reference¶

Output Format¶

When separate_reasoning: true is set in the request:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "The answer is 42.",
      "reasoning_content": "Let me think step by step..."
    }
  }]
}

Tool Call Parsers¶

Tool call parsers extract function calls from model output and validate arguments against schemas.

Configuration¶

Supported Parsers¶

<|python_tag|>{"name": "get_weather", "parameters": {"location": "NYC"}}

<tool_call>
get_weather(location="NYC")
</tool_call>

{"name": "get_weather", "arguments": {"location": "NYC"}}

<tool_call><function=get_weather><parameter=location>NYC</parameter></function></tool_call>

Complete Parser Reference¶

Tool Execution Flow¶

1. Parse: Extract tool calls from model output
2. Validate: Check arguments against tool schema
3. Execute: Run MCP tools or return to client
4. Inject: Add tool results back to conversation
5. Continue: Resume generation if needed

Constrained Decoding (XGrammar)¶

When a request declares tools or a structured response_format, SMG builds a constraint during chat preparation and sends it with the gRPC request. The backend engine compiles it with XGrammar and enforces it token-by-token during decoding, so tool-call framing and JSON arguments are guaranteed to conform to the declared schemas before SMG's own parsers ever see the output.

Where It Runs in the Pipeline¶

In the regular gRPC pipeline, the constraint is generated after chat template rendering and tokenization (model_gateway/src/routers/grpc/regular/stages/chat/preparation.rs):

1. Filter tools by tool_choice
2. Apply chat template
3. Tokenize
4. Build tool constraint (generate_tool_constraint)
5. Build stop decoder → worker selection → dispatch

The Harmony pipeline (GPT-OSS) differs: it generates the constraint before encoding (model_gateway/src/routers/grpc/harmony/stages/preparation.rs). The architecture diagrams follow the regular pipeline order.

Constraint Types¶

Enforcement¶

The constraint travels on GenerateRequest's constraint oneof (crates/grpc_client/proto/), which also carries structured-output controls (json_schema, regex, grammar, json_object, choice). Engines compile these through XGrammar-guided decoding, and SMG's tool/reasoning parsers then extract the already well-formed calls from the response.

Configuration¶

Parser CLI Options¶

MCP Integration¶

When MCP is configured, tool calls can be executed automatically:

smg \
  --mcp-config-path /path/to/mcp.json \
  --tool-call-parser llama

See the MCP Guide for detailed configuration.

Recommended Configurations¶

smg \
  --model-path deepseek-ai/DeepSeek-R1 \
  --reasoning-parser deepseek_r1 \
  --worker-urls grpc://worker1:50051

smg \
  --model-path meta-llama/Llama-3.2-70B-Instruct \
  --tool-call-parser llama \
  --mcp-config-path /config/mcp.json

smg \
  --model-path Qwen/Qwen2.5-72B-Instruct \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen \
  --mcp-config-path /config/mcp.json \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-enable-l1 \
  --worker-urls grpc://worker:50051

Monitoring¶

Pipeline Metrics¶

Debug Logging¶

# Enable pipeline debug logging
RUST_LOG=smg::pipeline=debug smg ...

# Enable parser debug logging
RUST_LOG=smg::parsers=debug smg ...

Troubleshooting¶

What's Next?¶