Triage v2 — algorithm flow

v2 is v1's pydantic-ai agent locked to a single Olly-owned pathway, with two narrow additions before the agent runs: a universal red-flag short circuit and a back-pain purity gate. Continuation turns run through a stripped walker agent (two tools only: advance + red_flag_check) on gemini-3.5-flash.

End-to-end flow

flowchart TD
    REQ(["POST /v2/chat/completions<br/>body: messages[], chat_id"])
    AUTH[/"Resolve user<br/>PERSONAS_V2['olly-mvp'] = grant_35"/]
    LASTMSG["last_user_msg = last message in req.messages"]

    CHATID{"chat_id
supplied?"}
    LOOKUP[("SELECT status<br/>FROM triage.sessions<br/>WHERE id = chat_id")]
    BUG2{"status =
'completed'?"}
    MINT["mint fresh<br/>session_id = uuid4()"]
    REUSE["session_id = chat_id"]

    ENSURE[("_ensure_session_row
INSERT IF NOT EXISTS")]
    RF_SIDE[("rf_sidecar.observe_turn
fire-and-forget, async")]

    PINNED{"_has_pathway_pinned?<br/>pathway_id ≠ NULL<br/>AND status = 'active'"}

    %% --- continuation walker ---
    WALK["_v2_walk_continuation"]
    LOAD_HIST[("Valkey GET
triage:v2_walk_history:
<user>:<session>")]
    AGENT_RUN[/"walker agent.run(last_user_msg)<br/>model: gemini-3.5-flash<br/>tools: advance, red_flag_check"/]
    ADVANCE[("tools.advance<br/>_match_option (strict)<br/>UPDATE current_state_id")]
    RFC_MID[/"red_flag_check<br/>volunteered new red flag?"/]
    NEXT{"is_terminal?"}
    RENDER_Q["render: question + numbered options"]
    RENDER_T["render: **Title** + advice + safety net"]
    SAVE_HIST[("Valkey SET history<br/>TTL 24h")]
    REC_ASST[("_record_assistant_turn")]
    COMPLETE[("_complete_session
status='completed'
disposition jsonb")]

    %% --- first turn ---
    L0[/"Layer 0 — red_flag_check<br/>substring scan on last_user_msg"/]
    L0_URG{"urgent?"}
    T_999["999 terminal<br/>_set_session_state + _complete_session"]

    L2[/"Layer 2 — purity classifier<br/>gemini-flash, A/B/C"/]
    L2_VER{"verdict?"}
    REFUSE["HTTP 501<br/>out_of_scope_v2 / _mixed"]

    PIN[("tools.start_session
pw_code='olly_back_mvp'
set pathway_id + current_state_id")]
    WARM["_first_turn_warm_emit<br/>synthetic history → agent.run"]
    SAVE_FT[("Valkey SET history<br/>(per-session key)")]

    RESP(["200 OpenAI envelope<br/>+ X-Triage-Session-Id<br/>+ X-Triage-V2-State-Id"])

    REQ --> AUTH --> LASTMSG --> CHATID
    CHATID -- yes --> LOOKUP --> BUG2
    BUG2 -- yes --> MINT
    BUG2 -- no --> REUSE
    CHATID -- no --> MINT
    MINT --> ENSURE
    REUSE --> ENSURE
    ENSURE --> RF_SIDE --> PINNED

    PINNED -- yes --> WALK
    WALK --> LOAD_HIST --> AGENT_RUN
    AGENT_RUN -. tool .-> ADVANCE
    AGENT_RUN -. tool .-> RFC_MID
    ADVANCE --> NEXT
    NEXT -- no --> RENDER_Q --> SAVE_HIST --> REC_ASST --> RESP
    NEXT -- yes --> RENDER_T --> SAVE_HIST --> COMPLETE --> REC_ASST --> RESP

    PINNED -- no --> L0
    L0 --> L0_URG
    L0_URG -- yes --> T_999 --> REC_ASST
    L0_URG -- no --> L2
    L2 --> L2_VER
    L2_VER -- B or C --> REFUSE
    L2_VER -- A --> PIN --> WARM --> SAVE_FT --> REC_ASST

    classDef dec fill:#dbeafe,stroke:#93c5fd,color:#1a1a1a;
    classDef gate fill:#fef3c7,stroke:#fcd34d,color:#1a1a1a;
    classDef walk fill:#dcfce7,stroke:#86efac,color:#1a1a1a;
    classDef refuse fill:#fee2e2,stroke:#fca5a5,color:#1a1a1a;
    classDef side fill:#e5e7eb,stroke:#cbd5e1,color:#1a1a1a;

    class CHATID,BUG2,PINNED,L0_URG,L2_VER,NEXT dec;
    class L0,L2,AGENT_RUN,RFC_MID gate;
    class WALK,ADVANCE,WARM,PIN,RENDER_Q,RENDER_T walk;
    class REFUSE,T_999 refuse;
    class LOOKUP,ENSURE,RF_SIDE,LOAD_HIST,SAVE_HIST,SAVE_FT,REC_ASST,COMPLETE side;

Back-pain tree it walks

Pre-pinned at first turn via tools.start_session(pw_code='olly_back_mvp'). Six NICE-cited gates, each pulled from postgres pathway_options. The "Yes" branch on every gate is a red-flag exit; the "No" branch advances. STarT Back stratification (Q7) is deferred — Q6 "No" routes directly to the physio terminal.

flowchart LR
    Q1{"Q1 — Cauda equina?<br/>saddle / bladder / bowel<br/>NICE NG59"}
    Q2{"Q2 — Progressive weakness?<br/>NICE NG59"}
    Q3{"Q3 — Cancer red flags?<br/>weight loss / night pain<br/>NICE NG12"}
    Q4{"Q4 — Inflammatory?<br/>AM stiffness ≥30 min<br/>NICE NG65"}
    Q5{"Q5 — Significant trauma?<br/>NICE NG41"}
    Q6{"Q6 — Systemic illness?<br/>fever / IVDU<br/>NICE NG59"}

    T_999["T_999_cauda_equina<br/>Call 999 / A&E"]
    T_GP_W["T_samed_gp_weakness<br/>Same-day GP"]
    T_2WW["T_gp_2ww_cancer<br/>GP — 2-week wait"]
    T_RHEUM["T_gp_rheum_inflam<br/>GP — rheumatology"]
    T_AE["T_ae_trauma_imaging<br/>A&E — imaging"]
    T_INF["T_samed_gp_infection<br/>Same-day GP"]
    T_PHYSIO["T_olly_physio_routine<br/>Book Olly physiotherapist"]

    Q1 -- "1-3 (Yes)" --> T_999
    Q1 -- "4 None" --> Q2
    Q2 -- "1-2 (Yes)" --> T_GP_W
    Q2 -- "3 No" --> Q3
    Q3 -- "1-4 (Yes)" --> T_2WW
    Q3 -- "5 None" --> Q4
    Q4 -- "1 AM stiff" --> T_RHEUM
    Q4 -- "2-3 No" --> Q5
    Q5 -- "1 Significant" --> T_AE
    Q5 -- "2-3 No" --> Q6
    Q6 -- "1-2 (Yes)" --> T_INF
    Q6 -- "3 No" --> T_PHYSIO

    classDef gate fill:#fef3c7,stroke:#fcd34d,color:#1a1a1a;
    classDef urgent fill:#fee2e2,stroke:#fca5a5,color:#1a1a1a;
    classDef gp fill:#fef3c7,stroke:#fcd34d,color:#1a1a1a;
    classDef routine fill:#dcfce7,stroke:#86efac,color:#1a1a1a;

    class Q1,Q2,Q3,Q4,Q5,Q6 gate;
    class T_999,T_AE urgent;
    class T_GP_W,T_2WW,T_RHEUM,T_INF gp;
    class T_PHYSIO routine;

What changed in the latest commit

Known residual

Even with the stripped agent the LLM can emit final_result with hallucinated options instead of calling advance first. We tried gemini-2.5-pro (slower, reliable in test) and then dropped to gemini-3.5-flash (faster, also reliable in test — 8/8 curl turns + Playwright pass with no observed hallucinations). If drift recurs in the wild, the structural fix is the deterministic walker: call tools.advance ourselves and put an LLM-only-for-option-mapping step in front of it (port v3_narrow's _llm_map_freetext_to_index).

Prompts

Three prompts shape v2's behaviour. Each runs at a different point in the request lifecycle and on a different model.

Walker system prompt

You are Olly's back-pain triage walker.

Context:
  - The patient's session is already pinned to the `olly_back_mvp`
    pathway. Q1's cauda-equina question has already been emitted on
    the previous turn.
  - You have exactly TWO tools: `advance` and `red_flag_check`.
  - The patient's latest reply is the chosen option (or a paraphrase
    of one) for the question they just saw. Your job is to walk the
    tree.

What you MUST do on every turn:
  1. If the patient's reply mentions a NEW acute red-flag symptom that
     isn't already captured by the current state's options (e.g.
     "actually my chest hurts now", "I lost vision in one eye"), call
     `red_flag_check(message=)`. If `urgent=True`, emit
     a terminal TriageOutput pointing them to 999 / A&E with a short
     safety net. Otherwise continue.
  2. Call `advance(chosen_option_text=)`. Pass the option's verbatim text as it appears in the
     state's options list — paraphrase the patient's reply into one of
     them. Do NOT call `advance` with the patient's literal text if it
     doesn't match an option label cleanly.
  3. Use the AdvanceResult to compose the next message:
     - If `is_terminal=True`: render the disposition (title + advice)
       as a warm Olly message, set `is_terminal=true`, attach the
       disposition object.
     - Else: render the next state's question + options as the user-
       facing message. Acknowledge the patient briefly in one short
       line ("Thank you.", "Got it.", "Okay."), then the question.

What you MUST NOT do:
  - Do not call any tool other than `advance` and `red_flag_check`.
  - Do not propose a different pathway — the pathway is pinned.
  - Do not ask the patient an unrelated clarifying question. If you
    can't paraphrase their reply to an option, call `advance` with
    your best guess and let the matcher decide.

Style:
  - British English. No em dashes. No diagnostic naming. Warm, short,
    sentence-case acknowledgements.

Layer 2 purity classifier prompt

The user's message describes a health problem. Decide which
category fits BEST:
A. "back_pain_only" — complaint centred on back pain
(lower / upper / mid back), possibly with related leg symptoms.
B. "back_pain_mixed" — back pain alongside another major
symptom (chest pain, abdominal pain, neuro beyond legs, fever
as the primary concern, etc.)
C. "not_back_pain" — not primarily about back pain.

Reply with EXACTLY one of: A, B, C

Message: "<text>"

A → proceed to start_session pin. B → HTTP 501 out_of_scope_v2_mixed. C → HTTP 501 out_of_scope_v2. The BFF then falls back to v1's chat completions which handles non-back-pain complaints freely.

First-turn warm-emit nudge

Now produce the first warm reply: acknowledge what the patient
said in one short line, then ask the question from the current
state with all its options listed. British English, no em
dashes, no diagnostic naming.

The full v1 system prompt lives at services/triage/src/triage/prompts/olly_system_prompt.md and gets re-fetched from Langfuse (key olly_system_prompt) at agent build time with the .md as the offline fallback. It's the same document v1 uses for any non-MSK complaint — see docs/CLINICAL_TRIAGE_BLUEPRINT.md for the layered model it encodes.

Agent functions

The triage service registers 12 pydantic-ai tools across its agents. v2's stripped walker uses only 2 of them (advance + red_flag_check). v1's full agent has access to all 12 — the broader surface is why v1 can drift and why we trimmed for v2. Every tool wraps a function in services/triage/src/triage/tools.py, so behaviour is consistent across endpoints.

The strict matcher inside advance

tools._match_option is the deterministic gate. Strategy, strict → permissive:

1. Exact lower-case match — "None of these apply" == option.text
2. Numeric prefix — "4" picks option 4 (1-indexed)
3. Strict prefix — "No" matches "No - I feel well…"
4. Unambiguous substring — only when exactly one winner; "No" against a state with multiple "No-…" options returns None

If none of the four strategies finds a match, advance raises ValueError. Today the walker propagates that to a 500 — we rely on the agent paraphrasing well enough that this never fires in practice. The structural fallback (when we get to it) is to port v3_narrow._llm_map_freetext_to_index in as a second chance: same flash model, structured-JSON output of {"index": int}, called only when strict misses.

Why the walker trimmed 10 tools

Each tool the agent can call is a place the LLM can drift. v1's agent reaches for list_pathways / propose_pathway_candidates / extract_facts on continuation turns when the message history shape confuses it — none of which advance the pinned olly_back_mvp tree. Gateway rules block some chains after start_session, but the agent can still emit a final_result with a clarifying text question instead of calling advance. The walker exists because the only correct continuation action for a pinned olly_back_mvp session is advance; removing the choice removes the drift.

Tool-level test coverage

Each tool wraps a function in services/triage/src/triage/tools.py (or a sibling clinical module). Tests sit in three layers — unit tests over the underlying function, clinical-rule tests over the data tables a tool reads, and e2e scenarios that exercise the tool end-to-end through a real agent.

Totals at a glance

Three test layers, three different jobs

Unit tests verify the function's contract — given these args, return this shape. Fast (no postgres / no Valkey, mocked pools where needed), no LLM. Lives in services/triage/tests/test_*.py.

Clinical-rule tests verify the data tables and the pure-Python matchers over them. The 51 Layer-0 tests are the most valuable safety net in the codebase — every red-flag rule has a "should match" and "should NOT match" vignette. The PF / MH / Layer 1 suites verify decision-tree branches (eligibility thresholds, age cutoffs, scoring bands). No LLM, no DB, sub-second to run the lot. Lives in services/triage/tests/clinical/.

E2E scenarios exercise tools through a real agent against the running stack. Persona + answer-rules drive the multi-turn conversation; scenario_evaluators assert on the resulting postgres row, Langfuse trace, disposition shape, and tone. Self- describing (every scenario carries its own evaluators inline). Run via tests/e2e/scripts/run_v*_wave.py. This is the only layer where LLM non-determinism shows up — failures here usually point at agent drift, not tool bugs.

What's NOT well covered

• _match_option has no dedicated unit test — its four strategies are only exercised through advance's 2 tests. A focused suite would catch regressions on the substring/prefix edge cases (ambiguity rejection) cheaply.
• The walker itself has no unit test yet — only the Playwright smoke covers it end-to-end. A test that hits _v2_walk_continuation with a fixed Valkey history blob + a fixture user would lock the no-drift behaviour without paying for an LLM call. On the to-do list.
• Layer 2 purity classifier has no clinical test suite — only the substring fast-path and the LLM call. A 30-vignette A/B/C bucket test would harden the gate against prompt-engineering drift.

Test cases

v2 ships with 14 e2e scenarios under services/triage/tests/e2e/scenarios/MVP-BP-*.v2.json. Every scenario is self-describing: persona + answer rules driven by question-text matching, plus a list of evaluators that run post-session against postgres / Langfuse / disposition shape.

Scenario shape

Excerpted from MVP-BP-RF-002_progressive_weakness_samed_gp.v2.json:

{
  "name": "MVP-BP-RF-002_progressive_weakness_samed_gp.v2",
  "description": "Adult male with 2 weeks of back pain and progressive leg
                  weakness. Denies saddle/bladder/bowel (Q1) but affirms
                  worsening leg weakness (Q2). Expected: T_samed_gp_weakness.",
  "fixture_user": "grant_35",
  "model": "olly-mvp",
  "persona": {
    "intro": "I've had back pain for two weeks and now I'm noticing my legs
              feel weak when I try to walk.",
    "answer_rules": [
      { "if_question_contains": ["urgent attention", "saddle area"],
        "respond": "None of these apply" },
      { "if_question_contains": ["weakness in your legs", "getting worse"],
        "respond": "Yes, my legs feel weak and it is getting worse" },
      ...
    ],
    "max_turns": 8,
    "expected_pathway": "olly_back_mvp"
  },
  "scenario_evaluators": [
    { "type": "db_query",  "sql": "SELECT COUNT(*) FROM triage.sessions
                                    WHERE id = '${session_id}'
                                    AND disposition IS NOT NULL ...",
      "expected_min": 1 },
    { "type": "disposition_band_strict",
      "expected": "urgent", "expected_care_type": "gp" },
    { "type": "contains_any",
      "field": "assistant_message",
      "values": ["gp", "general practitioner", "doctor"] },
    { "type": "safety_net_present", "min_chars": 40 },
    { "type": "no_diagnostic_naming" }
  ]
}

Cross-cutting evaluators (auto-injected)

Every v2 scenario receives three defaults at runtime via runner_v2.py step 4b, even if the scenario JSON doesn't declare them:

A scenario can tighten any default by declaring its own copy in scenario_evaluators[] — the auto-inject only fires when the type isn't already declared.

Running the wave

# From /root/olly/services/triage with the triage container running.
# Median-of-3 for variance filtering.
uv run python tests/e2e/scripts/run_v2_wave.py 'MVP-BP-*.v2.json' \
  --parallel 3 --runs 3

Frontend smoke (Playwright)

/root/triage-rn/e2e/grant-backpain.spec.ts drives the live URL end-to-end: greeting hero → "I have a back pain." → CARE tile → 3 clips → Q1 → "None of these apply" → Q2 progressive weakness. Authenticates as Grant by injecting a synthetic JWT with preferred_username=grant on every /api/chat POST, so v1's agent (when it runs) and v2's pinned session both resolve to grant_35.

cd /root/triage-rn
npx playwright test --config=e2e/playwright.config.ts

Try it

curl -sX POST 'https://triage-rn.dev.hiolly.com/api/chat?json=true' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <keycloak JWT for grant>' \
  -d '{"messages":[{"role":"user","content":"I have a back pain."}],
       "chat_id":"<new-uuid>"}'