A model-risk function asks five questions about any AI in claims: what does it decide, what does it see, can we reproduce what it did, what stops it going wrong, and what does it cost. This post answers them for InsightCW's agents and copilot, with where each answer lives in the product.

1. What does it decide?

Nothing on its own, out of the box. Agents prepare decisions; the ceiling ships at “ask first”; six actions are capped there permanently. Where a carrier raises the ceiling, actions at L3 open a pull-back window. The model never sets a figure: reserves, recoveries, value ranges and authority gaps come from the ledger and the methods. Coverage positions can only move towards caution, and the supervisor's recommendation can only tighten.

2. What does it see?

  • PII is redacted before anything reaches a prompt.
  • The copilot works in the asker's scope: it opens a claim only if it is in their book, and every lookup re-checks scope and runs as the asker's organisation.
  • Privileged notes and privileged or confidential document content are never returned to the copilot.
  • News is fetched by InsightPulse about commercial insureds only, never claimants.

3. Can we reproduce what it did?

  • The decision record holds every run's hits, the facts and rules behind them, the agents' findings and the verdict.
  • Every agent step records its model, tokens, cost, confidence, prompt version and whether the model's answer was kept, rejected, invalid, failed or not run.
  • Lense traces a span per agent step and per copilot answer when tracing is configured.
  • Prompts are versioned in InsightPrompts, with local defaults when no release exists.
  • Audit rows mark an agent's actions as the agent's, with the level, who it acted for and who approved.
The coverage agent's page with recent runs and one run's trace, on demo data
The coverage agent's page (demo data): recent runs with model, tokens and cost, and one run's trace — what the file and the engines say, what the model reasoned, and what was prepared for a person.

4. What stops it going wrong?

flowchart TB A["Model output"] --> B["Shape check:
valid JSON with
the expected fields"] B --> C["Grounding: every number
is a fact"] C --> D["Citations: every finding
cites a fact"] D --> E["Merge rules: caution only,
tighten only, templates kept"] E --> F["Authority: level, hard cap,
ceiling; same code path
as a person"] F --> G["Payment controls and
authority matrix"] style C fill:#efeafe,stroke:#7c6cf0 style F fill:#e8f8f4,stroke:#22c1a4

Each layer can stop an answer on its own. An invalid or ungrounded answer falls back to the rules' text; an action above the effective level comes back as prepared work; a payment an agent requests is held by the same controls as a person's.

5. What does it cost, and what stops a runaway?

  • A daily budget per agent. Over budget, the fast model answers.
  • Retries behind a circuit breaker per model, so a failing provider is not hammered.
  • Autorun is debounced to once a minute per claim and never triggered by an agent's own action.
  • Cost per step is priced and shown on the trace.

And the tests

The test suites never call a real model. Model calls in tests use injected stand-ins, and a guard at the AWS client refuses any Bedrock call made during a test run and reports it. Grounding, invalid output, budgets, PII redaction, dry runs that write nothing, autorun debounce and the L2 caps are each tested.

Keep reading

Adopting it Build versus buy for a claims workbench