Use Case 031: Compatibility Passthrough Endpoints

Overview

Implementation status (agent-identity release, June 2026). Implemented: POST /v1/chat/completions (OpenAI) and POST /v1/messages (Anthropic) with SSE streaming, agent-bearer auth, task-type routing, and a mid-stream budget hard-stop that emits a structured error event and reconciles spend even on caller disconnect (LlmPassthroughGateway). Gap: the /v1/embeddings passthrough (UC-031d) is not yet implemented.

Description

This use case describes the OpenAI- and Anthropic-shaped passthrough endpoints on the Application Manager (AM) LLM Gateway. They let external tools that already speak the OpenAI Chat Completions or Anthropic Messages wire format route through AM with no custom client work — only a base-URL and API-key change. Each endpoint accepts the standard request shape, extracts agent identity from Authorization: Bearer <agent-credential>, looks up the agent's DefaultTaskType, calls the platform-neutral LlmDispatchService (UC-030), and translates the response back to the expected wire shape. Streaming endpoints use Server-Sent Events (SSE) via ASP.NET Core Results.Stream + IAsyncEnumerable<string> — no third-party SSE library.

This is part of the agent-identity release. The SDK master design is riptide-sdk/docs/plans/AGENT-IDENTITY-ARCHITECTURE.md; the AM-side plan is docs/internal/agent-identity-plan.md (§2 Compatibility passthrough). The gateway absorbs the RAF-SVC-LLM specification — these passthrough endpoints front the same dispatch lifecycle. A strict mid-stream hard-stop applies: tokens emitted count against the reservation in real time, and the SSE stream aborts with a structured BudgetExhausted final event the instant the reservation is exhausted. The MVP audience is MCP server developers (documentation in docs/integration/mcp-server-setup.md); polished coding-agent onboarding (VS Code Copilot BYOK, Cursor, Continue.dev, Claude Code CLI) is deferred to Phase 5.

Actors

Preconditions

1. Application Manager is running; provider accounts, deployments (UC-028), task types (UC-029), and the dispatch path (UC-030) are configured.
2. The calling agent has a valid, non-revoked credential presented as Authorization: Bearer <agent-credential> (UC-026).
3. The agent has a resolvable DefaultTaskType.
4. An active AgentPlan with budget headroom exists (UC-034).
5. The compatibility controllers are registered in the Api project with SSE streaming enabled.

Postconditions

Success Postconditions

1. A request in OpenAI/Anthropic wire format is translated, dispatched through LlmDispatchService, and the response is translated back to the expected wire shape.
2. For streaming, SSE chunks are emitted in the native wire format until completion or budget exhaustion.
3. Spend is reserved and reconciled; an LlmCallLedger row is written (via the underlying dispatch).
4. Mid-stream abort, when it occurs, emits a structured BudgetExhausted final event and stops emission immediately.

Failure Postconditions

1. A missing or invalid Bearer credential returns 401; no dispatch.
2. An agent with no resolvable DefaultTaskType fails closed (Unroutable); no provider call.
3. Budget exhaustion before first token returns a non-streaming BudgetExhausted error (or, for SSE, a BudgetExhausted event before content).
4. A mid-stream reservation exhaustion aborts the stream with partial content and a BudgetExhausted event; no automatic continuation.

Primary Flow Sequence

sequenceDiagram
    actor Agent as Agent / MCP Server
    participant Compat as OpenAiCompatibilityController
    participant Disp as LlmDispatchService
    participant Cap as Reservation
    participant Prov as LLM Provider

    Agent->>Compat: POST /v1/chat/completions (Bearer agent-credential)
    Compat->>Compat: Extract agent identity; look up DefaultTaskType
    Compat->>Disp: Dispatch(neutral request)
    Disp->>Cap: Reserve spend (UC-034)
    Cap-->>Disp: Reservation (Held)
    Disp->>Prov: Translate + dispatch
    Prov-->>Disp: Response
    Disp->>Cap: Reconcile + ledger
    Disp-->>Compat: Neutral response + metadata
    Compat->>Compat: Translate to OpenAI wire shape
    Compat-->>Agent: 200 OpenAI-shaped completion

UC-031a: OpenAI-Shaped Chat Completion (Non-Streaming)

Triggers

• An MCP server or tool calls POST /api/llm/openai/v1/chat/completions with stream: false and a Bearer agent credential.

Basic Flow

1. The controller extracts the agent identity from Authorization: Bearer <agent-credential> (UC-026).
2. It looks up the agent's DefaultTaskType.
3. It maps the OpenAI messages, max_tokens, temperature, and response_format fields to the platform-neutral request.
4. It calls LlmDispatchService (UC-030), which runs the full lifecycle (resolution, classification, PII sweep, reserve, dispatch, reconcile, ledger).
5. The neutral response is translated back to the OpenAI Chat Completions shape (choices, usage, model).
6. The OpenAI-shaped response is returned with HTTP 200.

Alternative Flows

• A1: Missing / invalid Bearer — 401; no dispatch.
• A2: No resolvable DefaultTaskType — fail closed (Unroutable), surfaced as an OpenAI-shaped error.
• A3: PII detected (cloud-bound) — request rejected before dispatch; OpenAI-shaped error returned.
• A4: Budget exhausted before dispatch — BudgetExhausted returned in OpenAI error shape.

UC-031b: OpenAI-Shaped Chat Completion with SSE Streaming

Triggers

• A tool calls POST /api/llm/openai/v1/chat/completions with stream: true.

Basic Flow

1. The controller authenticates, resolves DefaultTaskType, and maps the request as in UC-031a.
2. It calls LlmDispatchService in streaming mode and returns Results.Stream over an IAsyncEnumerable<string> of SSE chunks.
3. Each provider token is translated into an OpenAI chat.completion.chunk SSE event and flushed.
4. Tokens emitted count against the reservation in real time (UC-034).
5. On normal completion, a final data: [DONE] sentinel is emitted.

Alternative Flows

• A1: Mid-stream budget exhaustion — see UC-031e; the stream aborts with a BudgetExhausted event.
• A2: Provider stream error — the controller emits an error SSE event and closes; partial spend is reconciled.
• A3: Client disconnect — the gateway stops emission; spend up to the disconnect is reconciled.

sequenceDiagram
    actor Agent
    participant Compat as OpenAiCompatibilityController
    participant Disp as LlmDispatchService
    participant Cap as Reservation
    participant Prov as LLM Provider

    Agent->>Compat: POST /v1/chat/completions (stream:true)
    Compat->>Disp: Dispatch(stream=true)
    Disp->>Cap: Reserve spend
    Disp->>Prov: Stream request
    loop Per token
        Prov-->>Disp: token
        Disp->>Cap: Count token vs reservation
        alt Reservation has headroom
            Disp-->>Compat: chunk
            Compat-->>Agent: SSE chat.completion.chunk
        else Reservation exhausted
            Disp-->>Compat: BudgetExhausted
            Compat-->>Agent: SSE BudgetExhausted (final)
            Compat->>Disp: Abort stream
        end
    end
    Disp->>Cap: Reconcile + ledger
    Compat-->>Agent: SSE [DONE]

UC-031c: Anthropic-Shaped Messages with SSE

Triggers

• A tool calls POST /api/llm/anthropic/v1/messages (optionally stream: true) with a Bearer agent credential.

Basic Flow

1. The AnthropicCompatibilityController authenticates and resolves DefaultTaskType.
2. It maps the Anthropic messages, system, max_tokens, and temperature fields to the neutral request.
3. It calls LlmDispatchService; for streaming, it emits Anthropic-style SSE events (message_start, content_block_delta, message_delta, message_stop).
4. Tokens count against the reservation in real time.
5. The response (or stream) is returned in the Anthropic Messages wire shape.

Alternative Flows

• A1: Non-streaming — a single Anthropic message object is returned.
• A2: Mid-stream budget exhaustion — a BudgetExhausted final event is emitted; the stream aborts (UC-031e).
• A3: System prompt handling — the Anthropic system field is mapped into the neutral request's system role.

UC-031d: Embeddings Passthrough

Triggers

• A tool calls POST /api/llm/openai/v1/embeddings with input text(s) and a Bearer agent credential.

Basic Flow

1. The controller authenticates and resolves DefaultTaskType (mapped to an embedding-capable task type where the agent allows it).
2. It maps the OpenAI input (string or array) to the neutral embedding request.
3. It calls LlmDispatchService embedding path (UC-030b).
4. The response is translated back to the OpenAI embeddings shape (data[].embedding, usage).

Alternative Flows

• A1: Agent's default task type not embedding-capable — fail closed; OpenAI-shaped error.
• A2: Batch input — token counts summed for reservation and reconciliation.
• A3: Empty input — rejected; OpenAI-shaped validation error.

UC-031e: Mid-Stream Budget Exhaustion Abort

Triggers

• During an SSE stream (UC-031b/UC-031c), the reservation is exhausted as tokens are emitted.

Basic Flow

1. Each emitted token decrements the reservation in real time.
2. The instant the reservation is exhausted, the gateway stops emitting content tokens.
3. A structured BudgetExhausted final SSE event is emitted, carrying the partial content already streamed and the exhaustion reason.
4. The stream is closed; there is no automatic continuation (agents' own retry logic picks up on next attempt).
5. Spend up to the abort point is reconciled and written to the LlmCallLedger.

Alternative Flows

• A1: Lenient mid-stream mode on the governing cap — where a cap is configured MidStreamMode = Lenient (UC-032), the in-flight stream is allowed to complete against its reservation rather than aborting; strict is the default.
• A2: Exhaustion exactly at completion — if the last token coincides with exhaustion, the normal [DONE]/message_stop is emitted and no BudgetExhausted event is needed.
• A3: Reservation expiry mid-stream — late reconciliation writes the ledger row tagged ReconciledAfterExpiry (UC-034); the stream still aborts on exhaustion.

API Endpoints

// POST /api/llm/openai/v1/chat/completions
// Request (Authorization: Bearer <agent-credential>)
{
  "model": "gpt-4o",
  "stream": false,
  "messages": [
    { "role": "user", "content": "Summarize the attached policy in two sentences." }
  ],
  "max_tokens": 200
}

// Response 200 (OpenAI Chat Completions shape)
{
  "id": "chatcmpl-01J8ZC10",
  "object": "chat.completion",
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "The policy ... . It ... ." },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 612, "completion_tokens": 41, "total_tokens": 653 }
}

// Streaming abort (SSE) — final event on mid-stream exhaustion
data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":" The policy applies"},"finish_reason":null}]}

data: {"object":"error","error":{"type":"BudgetExhausted","message":"Reservation exhausted mid-stream; stream aborted.","partial":true}}

data: [DONE]

Business Rules

Data Requirements

The passthrough endpoints introduce no new persisted entities. They translate wire formats around the existing dispatch path; the LlmCallLedger and LlmSpendReservation (UC-030/UC-034) carry the durable records.

Passthrough request mapping (transient, not persisted)

BudgetExhausted SSE final event (transient)

Security Considerations

• Authentication: Bearer agent credential on every passthrough call; missing/invalid/revoked → 401. The credential is resolved to an agent identity (UC-026), not a static API key.
• Authorization / capabilities: dispatch is bounded by the agent's AllowedTaskTypes/DefaultTaskType; the passthrough cannot escape the agent's allowed routing.
• Data protection: provider credentials stay inside AM (decrypted via ProviderAccountCredentials at dispatch); the passthrough caller never sees them.
• Audit: dispatch and mid-stream abort events fold into AuditLog; PII rejections are recorded without storing the PII payload.
• PII handling: the underlying dispatch runs the cloud-bound SDK PII sweep; passthrough requests get the same classification and PII enforcement as native dispatch — no bypass.

Testing Scenarios

Related Use Cases

• UC-026: Agent Credential Issuance & Bearer Authentication — the Bearer agent credential these endpoints authenticate.
• UC-030: LLM Request Dispatch — the dispatch lifecycle the passthrough endpoints front.
• UC-034: Plan Budget Reserve / Reconcile — the reservation that mid-stream emission counts against.
• UC-029: Task-Type Routing & Agent Provisioning — the DefaultTaskType the passthrough resolves.

Revision History