Use Case 030: LLM Request Dispatch

Overview

Implementation status (agent-identity release, June 2026). Implemented as LlmGatewayDispatcher (not LlmDispatchService): resolve task type → deployment, classification check, pre-flight cap/plan reservation, provider dispatch, and reconciliation into the append-only ledger — plus the streaming passthrough path (UC-031). The governed entry points are POST /api/llm/complete and POST /api/llm/embed, with GET /api/llm/models and GET /api/llm/health. Gaps vs. this spec: the SDK PII pattern sweep (step 4) and response-schema validation with retry/fallback (step 9) are not yet implemented, and routing resolves a single deployment rather than walking a chain.

Description

This use case describes the end-to-end LLM request lifecycle hosted inside Application Manager (AM) by the LlmDispatchService. An agent calls POST /api/llm/complete (or /embed); AM resolves the task type to a model chain, applies classification and PII constraints, reserves and reconciles spend against the cap lattice and active plan, translates the request to provider-native format via the appropriate IProviderAdapter, dispatches, validates the response against a schema with retry/fallback, and writes an append-only LlmCallLedger row. AM holds all provider credentials; agents never see them.

This is part of the agent-identity release. The SDK master design is riptide-sdk/docs/plans/AGENT-IDENTITY-ARCHITECTURE.md; the AM-side plan is docs/internal/agent-identity-plan.md (§2). The gateway absorbs the RAF-SVC-LLM specification — this dispatch lifecycle is RAF-SVC-LLM §4.1 hosted inside AM, so RAF components migrate by changing only the base URL. The request envelope adds a planId (active plan context for cost enforcement) alongside the RAF-SVC-LLM contract fields. Budget reserve/reconcile detail lives in UC-034 and cap-lattice resolution in UC-032; this UC focuses on the dispatch path itself.

Actors

Preconditions

1. Application Manager is running; provider accounts, model catalog, deployments (UC-028), and task types (UC-029) are configured and routable.
2. The agent is authenticated (X-Api-Key for SDK callers; agent credential where applicable) and has an AllowedTaskTypes set with a resolvable DefaultTaskType.
3. An active AgentPlan exists (explicit, or an implicit per-turn plan from the agent's PerTurnBudget) with budget headroom.
4. The relevant ModelDeployment(s) are Healthy or Degraded (not Unreachable).
5. ASP.NET Core Data Protection holds the ProviderAccountCredentials key ring so the gateway can decrypt provider credentials at dispatch.

Postconditions

Success Postconditions

1. The request is dispatched to a viable ModelDeployment and a valid response is returned with metadata.
2. Spend is reserved pre-flight and reconciled post-call against the cap lattice and plan; an LlmCallLedger row is written.
3. Response metadata (model used, token counts, computed cost, reservation id) is assembled and returned to the agent.
4. Reservation outcome and dispatch are auditable.

Failure Postconditions

1. A request whose model has no catalog row, or whose task type resolves to no viable deployment, is rejected (fail-closed) — never priced or guessed.
2. A cloud-bound request with detected PII is rejected before dispatch; no provider call is made.
3. A request without plan budget headroom returns BudgetExhausted; no provider call is made.
4. A response that fails schema validation after exhausting maxRetries and model-chain fallback returns a terminal failure; the ledger still records actual spend (Source = Reconciled or ReconciledAfterExpiry).
5. An adapter bug is isolated — it fails its own request without crashing the gateway or affecting other adapters.

Primary Flow Sequence

sequenceDiagram
    actor Agent as AI Agent
    participant Ctl as LlmController
    participant Disp as LlmDispatchService
    participant Cap as CapLatticeResolver / Reservation
    participant Adapter as IProviderAdapter
    participant Prov as LLM Provider
    participant Ledger as LlmCallLedger

    Agent->>Ctl: POST /api/llm/complete (planId, taskType, input)
    Ctl->>Disp: Dispatch(request)
    Disp->>Disp: 1. Validate; resolve task type -> model chain
    Disp->>Disp: 2. Preference cascade -> first viable ModelDeployment
    Disp->>Disp: 3. Data-classification constraint
    Disp->>Disp: 4. SDK PII pattern sweep (cloud-bound only)
    Disp->>Cap: 5. Check active AgentPlan budget headroom
    Cap->>Cap: 6. Reserve spend across cap lattice
    Cap-->>Disp: Reservation (Held)
    Disp->>Adapter: 7. Translate to provider-native format
    Adapter->>Prov: 8. Dispatch
    Prov-->>Adapter: Provider response
    Adapter-->>Disp: Normalized response
    Disp->>Disp: 9. Validate vs schema; retry/fallback per maxRetries
    Disp->>Cap: 10. Reconcile spend (actual token counts)
    Cap->>Ledger: Write LlmCallLedger row
    Disp->>Disp: 11. Assemble response metadata
    Disp-->>Ctl: Response + metadata
    Ctl-->>Agent: Completion + metadata

UC-030a: Synchronous Completion (Full Lifecycle)

Triggers

• An agent calls POST /api/llm/complete with a planId, task type (or relies on DefaultTaskType), and input.

Basic Flow

1. Validate request, resolve task type → model chain (ResolveTaskType, UC-029b).
2. Preference cascade → select the first viable ModelDeployment.
3. Data-classification constraint — drop deployments below the request's classification.
4. SDK PII pattern sweep (cloud-bound only) — reject on detection.
5. Check active AgentPlan budget headroom (UC-034).
6. Reserve spend across the cap lattice (UC-032/UC-034).
7. Translate to provider-native format via the appropriate IProviderAdapter.
8. Dispatch to the provider.
9. Validate response against the task's responseSchema; retry/fallback per maxRetries → model-chain fallback → terminal failure.
10. Reconcile spend with actual token counts; write the LlmCallLedger row.
11. Assemble response metadata; return to the caller.

Alternative Flows

• A1: Unknown model — no catalog row; rejected (fail-closed) at step 1/2.
• A2: No viable deployment — cascade + classification filter yield empty; Unroutable, fail closed.
• A3: Budget exhausted — no plan headroom at step 5; BudgetExhausted; no provider call.
• A4: Provider error — adapter surfaces a provider failure; retry per maxRetries, then model-chain fallback, then terminal failure; reconcile records any partial spend.

UC-030b: Embedding Generation

Triggers

• An agent calls POST /api/llm/embed with input text(s) and a planId.

Basic Flow

1. Validate; resolve the embedding task type (e.g. embedding.search) to its chain.
2. Preference cascade selects an embedding-capable deployment; classification filter applies.
3. PII sweep on cloud-bound embedding input; reject on detection.
4. Check plan headroom; reserve spend.
5. Translate via the adapter's embedding path; dispatch.
6. Reconcile spend with actual token counts; write the ledger row.
7. Return the embedding vector(s) and metadata.

Alternative Flows

• A1: Non-embedding deployment in chain — filtered out; only embedding-capable deployments are eligible.
• A2: Batch input — token counts summed across inputs for reservation and reconciliation.
• A3: Empty input — rejected by validation; no reservation.

UC-030c: Structured-Output Enforcement & Retry/Fallback

Triggers

• A completion request whose task type carries a responseSchema and a maxRetries > 0.

Basic Flow

1. Steps 1–8 proceed as in UC-030a; the adapter requests structured output where the deployment supportsStructuredOutput.
2. The response is validated against the responseSchema.
3. On validation failure, the dispatcher retries on the same deployment up to maxRetries.
4. If retries are exhausted, the dispatcher falls back to the next deployment in the model chain and repeats.
5. If the whole chain is exhausted, a terminal failure is returned.
6. Reconciliation records spend for every provider attempt (each call consumed tokens), tagged to the reservation.

Alternative Flows

• A1: Deployment lacks structured-output support — the adapter falls back to prompt-guided JSON; schema validation still gates acceptance.
• A2: First retry succeeds — earlier failed attempts' tokens are still reconciled and ledgered.
• A3: Chain fallback crosses classification — a fallback deployment below the request classification is skipped, not used.

sequenceDiagram
    participant Disp as LlmDispatchService
    participant Adapter as IProviderAdapter
    participant Prov as LLM Provider
    participant Cap as Reservation/Ledger

    Disp->>Adapter: Request (structured output, schema)
    Adapter->>Prov: Dispatch
    Prov-->>Adapter: Response
    Adapter-->>Disp: Normalized response
    Disp->>Disp: Validate vs responseSchema
    alt Valid
        Disp->>Cap: Reconcile + ledger
        Disp-->>Disp: Return result
    else Invalid, retries remain
        Disp->>Adapter: Retry (same deployment)
    else Retries exhausted, chain remains
        Disp->>Adapter: Fallback to next deployment
    else Chain exhausted
        Disp->>Cap: Reconcile partial spend + ledger
        Disp-->>Disp: Terminal failure
    end

UC-030d: Classification + PII Rejection

Triggers

• A request whose data classification or content triggers the classification constraint or the PII sweep.

Basic Flow

1. The dispatcher computes the effective classification (max(requestClassification, task.classificationFloor)).
2. Deployments below that classification are dropped during the cascade.
3. For a cloud-bound (non-local) target, the SDK PII pattern sweep runs over the request content.
4. On PII detection, the request is rejected before any provider call; an audit entry records the rejection (without storing the PII payload).
5. Local-target dispatches skip the cloud-bound PII rejection (the sweep is cloud-bound only) but still honor the classification constraint.

Alternative Flows

• A1: Classification leaves no viable deployment — Unroutable; fail closed.
• A2: PII detected on cloud target — rejected; no provider call; no spend reserved/charged.
• A3: Local target with PII — not rejected by the cloud-bound sweep; classification ceiling still governs which local deployment is eligible.

UC-030e: Deployment Health & Config Reload

Triggers

• GET /api/llm/health (per-deployment, sub-1s SLA) or POST /api/llm/admin/reload (atomic full-set config reload).

Basic Flow

1. GET /api/llm/health returns per-deployment health within the sub-1s SLA, used by the cascade to skip Unreachable deployments.
2. POST /api/llm/admin/reload loads all registries (providers, models, deployments, task types), validates cross-references atomically, and swaps the active configuration.
3. In-flight requests complete against the previous configuration; new requests use the reloaded set.
4. Tokenizers resolve provider-native where the adapter supports it, tokenizerRef for local models, Tiktoken as a last-resort fallback.

Alternative Flows

• A1: Reload validation fails — cross-reference validation rejects the new set atomically; the previous configuration stays active; the admin sees the validation error.
• A2: Health probe times out — deployment marked Degraded/Unreachable (UC-028e); cascade adjusts on next dispatch.
• A3: Adapter isolation — an adapter throwing during dispatch fails only that request; other adapters and the gateway stay up.

API Endpoints

// POST /api/llm/complete
// Request
{
  "planId": "01J8ZB10PLAN0000000000001",
  "taskType": "developer.code-generation",
  "classification": "Internal",
  "messages": [
    { "role": "user", "content": "Write a function to reverse a linked list." }
  ],
  "maxOutputTokens": 800
}

// Response 200
{
  "model": "claude-sonnet-4-5",
  "modelDeploymentId": "01J8Z9C0DEPLOYMENT0001",
  "output": "public ListNode Reverse(ListNode head) { /* ... */ }",
  "usage": {
    "inputTokens": 142,
    "outputTokens": 318,
    "cacheReadTokens": 0,
    "cacheWriteTokens": 0
  },
  "computedCost": 0.00214,
  "pricingVersion": "2026-05-anthropic-v3",
  "reservationId": "01J8ZB10RESERVATION00001",
  "ledgerSource": "Reconciled",
  "attempts": 1
}

Business Rules

Data Requirements

LlmCallLedger (append-only)

Dispatch request envelope (additions over RAF-SVC-LLM contract)

Security Considerations

• Authentication: all dispatch routes require X-Api-Key (agent-bound); admin reload additionally requires llm.gateway.config.
• Authorization / capabilities: dispatch is bounded by the agent's AllowedTaskTypes; admin reload by llm.gateway.config.
• Data protection: provider credentials are decrypted at dispatch via IDataProtector purpose ProviderAccountCredentials and never returned to the agent; the agent sees only model output and metadata.
• Audit: reservation outcomes and dispatch events fold into AuditLog on the actor-chain pipeline; PII rejections are audited without storing the PII payload.
• PII handling: the SDK PII pattern sweep runs cloud-bound only and rejects on detection before any provider call; detected PII content is never persisted in the ledger or audit.

Testing Scenarios

Related Use Cases

• UC-029: Task-Type Routing & Agent Provisioning — supplies ResolveTaskType output as step 1–2.
• UC-028: LLM Provider Account & Deployment Management — the deployments, adapters, and credentials dispatch uses.
• UC-031: Compatibility Passthrough Endpoints — OpenAI/Anthropic-shaped front doors that call this dispatch path.
• UC-032: Budget Cap Lattice — the cap lattice spend reservation walks.
• UC-034: Plan Budget Reserve / Reconcile — the reserve/reconcile detail behind steps 5, 6, and 10.

Revision History