Use Case 037: Spend Anomaly Detection

Overview

Implementation status (agent-identity release, June 2026). Implemented: AnomalyEvent detection on a trailing 7-day baseline with a 7-day cold-start floor, severity levels, scope split (per agent-template and per application), and three delivery channels (email / webhook / dashboard); detection runs hourly (detect-anomalies, at :15). Admin UI at /anomalies with acknowledgement.

Description

This use case describes spend anomaly detection and alerting for the agent-identity release. AM watches LLM spend for unusual surges and emits anomaly events when the observed spend in a trailing window deviates materially from an expected baseline. Detection uses dual baselines — one per (tenant, templateId) and one per (tenant, applicationId) — each computed from 14–30 days of SpendRollup history and adjusted for day-of-week. A 7-day cold-start floor suppresses all alerts for a (tenant, …) pair until enough history exists, so a freshly onboarded tenant or a brand-new application never triggers spurious alarms.

Detection runs as two Hangfire jobs: a nightly baseline computation and a 10–15 minute trailing-window comparison. Anomalies are delivered through three channels — email via IEmailService, a tenant-configured webhook, and the always-on dashboard panel — selectable per tenant and severity. Each anomaly is persisted as an AnomalyEvent and surfaced on the insights dashboard's Anomalies panel (see UC-036) and via GET /api/insights/anomalies. The SDK master design is riptide-sdk/docs/plans/AGENT-IDENTITY-ARCHITECTURE.md; the Application Manager plan is docs/internal/agent-identity-plan.md (Section 5). These are planned features and not yet built.

Actors

Preconditions

1. The cost-governance plane is deployed; SpendRollup is populated hourly by SpendRollupJob (see UC-034).
2. The AnomalyEvent table exists (EF migration applied).
3. Both Hangfire jobs are registered: nightly baseline computation and the 10–15 minute trailing-window comparison.
4. Per-tenant delivery configuration exists (channels and severity mapping); defaults apply if unconfigured.
5. (For webhook delivery) the tenant has configured a reachable webhook URL.

Postconditions

Success Postconditions

1. Baselines per (tenant, templateId) and (tenant, applicationId) are recomputed nightly from 14–30 days of day-of-week-adjusted history.
2. The trailing-window job emits an AnomalyEvent when observed spend deviates beyond the configured threshold.
3. Each emitted anomaly is delivered to the configured channels for its severity; per-channel delivery status is recorded on the event.
4. Anomalies are visible on the dashboard Anomalies panel and via GET /api/insights/anomalies.
5. An administrator can acknowledge/dismiss an anomaly; the change is audited.

Failure Postconditions

1. No anomaly fires for any (tenant, …) pair with fewer than 7 days of history (cold-start suppression).
2. If a webhook delivery fails, the dashboard channel still records the anomaly; the failed channel's status reflects the failure (retry per policy).
3. A baseline that cannot be computed (no qualifying history) is skipped, not defaulted to zero.

Primary Flow Sequence

sequenceDiagram
    participant HF as Hangfire (delta job)
    participant Det as AnomalyDetector
    participant Roll as SpendRollup
    participant Base as Baselines
    participant Repo as AnomalyEvent repo
    participant Email as IEmailService
    participant Hook as Tenant Webhook
    participant Dash as Dashboard panel

    HF->>Det: Trigger trailing-window comparison
    Det->>Roll: Read trailing window (per tenant/template & tenant/app)
    Det->>Base: Load nightly baseline (day-of-week adjusted)
    Det->>Det: Check >=7 days history (cold-start floor)
    Det->>Det: Compute deviation vs baseline
    alt deviation beyond threshold
        Det->>Repo: Persist AnomalyEvent (severity, deviation)
        Det->>Dash: Surface on Anomalies panel (always)
        opt medium severity
            Det->>Email: Send anomaly email
        end
        opt high severity
            Det->>Hook: POST anomaly to tenant webhook
        end
        Det->>Repo: Update per-channel deliveryStatus
    else within threshold or cold start
        Det->>Det: No event emitted
    end

UC-037a: Nightly Baseline Computation

Triggers

• The nightly Hangfire BaselineComputationJob fires on schedule.

Basic Flow

1. The nightly job enumerates active (tenant, templateId) and (tenant, applicationId) pairs from SpendRollup.
2. For each pair, it reads 14–30 days of rollup history.
3. It computes a day-of-week-adjusted baseline (e.g., Mondays compared to Mondays) so weekly cyclicality doesn't read as anomalous.
4. It persists/updates the baseline (expected spend per window, with the dispersion the threshold uses).
5. Pairs with fewer than 7 days of history are marked cold-start and excluded from emission until they qualify.

Alternative Flows

• A1: Insufficient history for a pair — baseline skipped; pair stays cold-start (no zero-default baseline).
• A2: Sparse / bursty history — dispersion captured so a wide-variance pair has a correspondingly wide threshold (fewer false positives).
• A3: Configurable window — the 14–30 day window and day-of-week adjustment are configuration values, tunable per deployment.

UC-037b: Trailing-Window Comparison & Anomaly Emission

Triggers

• The 10–15 minute Hangfire delta job fires.

Basic Flow

1. The delta job reads the current trailing window of spend per (tenant, templateId) and (tenant, applicationId) from SpendRollup.
2. For each pair, it loads the latest nightly baseline.
3. It confirms the pair has cleared the 7-day cold-start floor; if not, it is skipped (UC-037c).
4. It computes the deviation of observed spend from the baseline.
5. If the deviation exceeds the configured threshold, it assigns a severity (medium/high per configurable thresholds) and persists an AnomalyEvent (scope, observed vs baseline spend, deviation, window bounds, detectedAt).
6. It hands the event to multi-channel delivery (UC-037d).

Alternative Flows

• A1: Within threshold — no event; the pair is simply re-evaluated on the next run.
• A2: Both baselines flag the same surge — the template-scoped and application-scoped baselines may each emit; events are distinct (different scope) and de-duplicated only within their own scope.
• A3: Threshold configurable — severity thresholds are deployment/tenant configuration, not hardcoded.

sequenceDiagram
    participant HF as Hangfire (delta)
    participant Det as AnomalyDetector
    participant Roll as SpendRollup
    participant Base as Baseline store
    participant Repo as AnomalyEvent repo

    HF->>Det: Run (every 10-15 min)
    loop per (tenant, template) & (tenant, app)
        Det->>Roll: Read trailing-window spend
        Det->>Base: Load baseline
        alt < 7 days history
            Det->>Det: Skip (cold-start)
        else baseline available
            Det->>Det: deviation = observed vs baseline
            alt deviation > threshold
                Det->>Det: Assign severity
                Det->>Repo: Persist AnomalyEvent
            else
                Det->>Det: No emission
            end
        end
    end

UC-037c: Cold-Start Suppression (<7 Days History)

Triggers

• A (tenant, templateId) or (tenant, applicationId) pair has fewer than 7 days of SpendRollup history (new tenant, new application, newly used template).

Basic Flow

1. During both the nightly baseline job and the delta comparison, AM checks the available history length for the pair.
2. If history is below the 7-day floor, the pair is flagged cold-start.
3. No anomaly event is emitted for that pair, regardless of how large the observed spend is.
4. Once the pair accumulates 7+ days, normal baseline computation and emission resume automatically.

Alternative Flows

• A1: Gap in history — if a pair went idle and resumed, the floor counts qualifying days of activity, not just calendar age.
• A2: Cold-start floor configurable — the 7-day value is configuration; deployments may tighten or loosen it.

UC-037d: Multi-Channel Delivery (Email / Webhook / Dashboard by Severity)

Triggers

• An AnomalyEvent has been emitted by the delta job.

Basic Flow

1. AM reads the tenant's channel/severity configuration (or defaults).
2. Dashboard panel always records the anomaly (no opt-out).
3. Email via IEmailService is the default channel for medium-severity anomalies.
4. Webhook to the tenant-configured URL is the default channel for high-severity anomalies.
5. Each delivery attempt updates the per-channel deliveryStatus on the AnomalyEvent.
6. The viewer sees the anomaly on the dashboard regardless of email/webhook outcome.

Alternative Flows

• A1: Webhook unreachable — delivery marked failed for that channel; dashboard channel unaffected; retry per configured policy.
• A2: Email send failure — IEmailService failure recorded (consistent with existing email failure logging); dashboard still shows the anomaly.
• A3: Channels overridden per tenant — a tenant may, e.g., route all severities to email; the configuration overrides the defaults.
• A4: No webhook configured for high severity — falls back to dashboard (always) and any other enabled channel; the missing webhook is noted, not an error.

UC-037e: Acknowledge / Dismiss an Anomaly

Triggers

• A tenant administrator reviews an anomaly on the dashboard or via the API and acts on it.

Basic Flow

1. The administrator opens the Anomalies panel (or queries GET /api/insights/anomalies).
2. They select an anomaly and issue AcknowledgeAnomaly (acknowledge or dismiss).
3. AM marks the event acknowledged/dismissed with the acting principal and timestamp.
4. The change is recorded in AuditLog on the actor-chain pipeline.
5. Acknowledged anomalies drop out of the active-alert view but remain queryable for history.

Alternative Flows

• A1: Already acknowledged — operation is idempotent; no error.
• A2: Missing capability — a user without the required governance capability (see UC-038) cannot acknowledge; 403.

API Endpoints

// GET /api/insights/anomalies?tenantId=5f7a1c2e-...&since=2026-06-01
// Response 200
{
  "anomalies": [
    {
      "id": "9b2e4f10-7c3a-4d61-8a02-1e2f3a4b5c6d",
      "tenantId": "5f7a1c2e-0b3d-4e6f-9a01-2c3d4e5f6a7b",
      "scope": { "templateId": "analysis.financial", "applicationId": null },
      "severity": "High",
      "observedSpend": 188.42,
      "baselineSpend": 41.10,
      "deviation": 3.58,
      "windowStart": "2026-06-09T13:00:00Z",
      "windowEnd": "2026-06-09T13:15:00Z",
      "detectedAt": "2026-06-09T13:16:02Z",
      "deliveryStatus": {
        "dashboard": "Delivered",
        "email": "NotAttempted",
        "webhook": "Delivered"
      },
      "acknowledged": false
    }
  ]
}

Business Rules

Data Requirements

AnomalyEvent (new entity, IdentityDbContext, EF migration):

Exactly one of TemplateId / ApplicationId is set per event (scope discriminator).

Security Considerations

• Authentication: the anomaly API endpoints use the same auth surfaces as the insights dashboard (OIDC viewer roles) and the admin governance capabilities; jobs run inside the trusted Hangfire host.
• Authorization / capabilities: listing anomalies requires cost.insights.view.tenant/.agency; acknowledging requires the appropriate governance capability (see UC-038). Tenant viewers see only their tenant's anomalies.
• Data protection: anomaly events carry aggregated spend figures and scope ids only — no provider credentials, no raw ledger rows, no agent ids. Webhook payloads carry the same minimal shape.
• Audit: acknowledge/dismiss actions are written to AuditLog with the acting principal; delivery outcomes are recorded per channel on the event.

Testing Scenarios

Related Use Cases

• UC-034 — SpendRollup Aggregation (history source for baselines and comparison)
• UC-036 — Cost Insights Dashboard & Billing Report (Anomalies panel; GET /api/insights/anomalies)
• UC-032 — Budget Cap Lattice Enforcement (caps and burn rate; complementary to anomaly surges)
• UC-038 — Agent Cost Governance RBAC (capabilities gating anomaly view and acknowledgement)
• UC-012 — Email Verification Process (shares IEmailService delivery substrate)

Revision History