AI/LLM Engineering

• Live incident-driven model/provider fallback

  When a third-party inference vendor silently retired the pinned extraction model, the extraction stage, a Python service, was re-routed to a different provider through an OpenAI-compatible bridge within the same release cycle. Along the way, the fix uncovered and worked around an undocumented schema-strictness incompatibility. (Medical Claim SaaS Platform, the GPU worker service's LLM-extraction stage)

• Deterministic-first design as a stated safety invariant

  The eligibility and routing decision engine runs as pure rule-based Python code with zero LLM input by design. A dedicated pytest test proves the resulting decision object is never included in anything sent to the model. (Medical Claim SaaS Platform, the pathway-resolution engine and the reasoning service's rule-output construction)

• Hallucination guardrails on every LLM call

  Every response from Anthropic's Claude API is checked against the exact set of source citations and required output items sent in the prompt. It rejects phantom citations and incomplete or added items, and failure reasons are restricted to a closed enum instead of free text. (Medical Claim SaaS Platform, the reasoning service's output-validation step)

• Two independent eval harnesses, both scored against the live model

  One Python harness scores structured-extraction accuracy, covering triage and extraction field correctness at pass@3. A separate Python harness scores tool-use correctness, covering answer accuracy plus cache-hit proof and zero-unauthorized-write assertions. The two are kept as documented separate systems because they are easy to conflate. (Payor Rate Data Platform, the messaging-channel and web-chat eval suites)

• Built row-level entitlement enforcement as SQL rewriting rather than prompting

  The model may name any allowed resource in generated SQL. But a TypeScript rewriting layer injects the actual access predicate before execution, so a compliant answer never depends on the model choosing to behave. (Payor Rate Data Platform, the web layer's AST predicate-injection fence)

• Sub-agent-orchestrated design synthesis with a verification pass

  A Terraform infrastructure design came from collecting multiple independently generated draft proposals. A judging pass fact-checked every load-bearing technical claim against primary sources, then scored and merged the strongest elements of each into a final design. (Medical Claim SaaS Platform, the worker scale-to-zero design process)

• Eval harness spanning PR-time and nightly cadence

  A curated gold-set of denial-classification cases runs on every PR via GitHub Actions, without calling an LLM. A smaller LLM-backed eval subset using Anthropic's Claude API also runs on PRs, and a full Claude API eval runs nightly. The system skips automatically and gracefully when no eval credentials are configured. (Medical Claim SaaS Platform, the gold-set test suite and CI / nightly-eval workflows)

• TDD applied to AI-authored/AI-adjacent code

  CI-enforced pytest tests, run through GitHub Actions, pin exact hallucination-guardrail behaviors, exact PHI-firewall exclusions, and exact fallback taxonomies. These tests are written before or alongside the corresponding production code paths. (Medical Claim SaaS Platform, the reasoning service's test suite)

• Confidence-scored classification with an audited fallback path

  The denial classifier prefers a corpus-matched code. It only falls back to a category inferred by Anthropic's Claude API, and when it does, the confidence score is deliberately lowered to just under the human-review threshold. The system tracks which path produced each result for audit. (Medical Claim SaaS Platform, the reasoning service's denial-classification step)

• Licensing-aware content guardrail

  Member-facing procedure descriptions are restricted to a curated, human-reviewed, checksum-pinned glossary stored in PostgreSQL. This avoids any verbatim licensed descriptor text the model might otherwise echo. (Medical Claim SaaS Platform, the reasoning service's curated procedure-translation table)

• A streaming tool-use loop with two distinct prompt-cache breakpoints

  One prompt-cache breakpoint in the calls to Anthropic's Claude API sits at a byte-static system prefix, and the other moves per round. An explicit rule says the static prefix may never interpolate a date or id, because doing so would silently invalidate the cache for every user. (Payor Rate Data Platform, the web layer's chat engine)

• A cheap-model-gates-expensive-model extraction pipeline

  A fast triage pass, built in Python, filters most incoming content before an expensive structured-extraction pass using Anthropic's Claude API ever runs. A separate dedupe-judgment pass avoids creating duplicate structured records from the same underlying event. (Payor Rate Data Platform, the messaging-channel ingestion pipeline)

• Guardrail against a real hallucination-adjacent failure mode

  A React charting component refuses to draw, rather than mislabel, when the underlying data mixes incompatible units. This was added after a live incident where a percentage value was drawn on the same axis as a dollar figure. (Payor Rate Data Platform, the web layer's chart-rendering component)

• A dual-persona voice system implemented entirely through prompt text

  The two-persona voice runs with no routing code path at all. It deliberately relies on self-selection by Anthropic's Claude API instead of an explicit branch, so the single cached system prompt stays intact. (Payor Rate Data Platform, the persona-refresh and cadence design)

• Diagnosed a model-behavior defect distinct from a backend defect

  After an entitlement changed mid-conversation, every backend layer correctly reflected the new state. But Anthropic's Claude API kept asserting the old state, because it anchored on its own prior turns. The fix added an explicit directive that the current context is authoritative, engineered as a pure function with zero extra PostgreSQL reads. (Payor Rate Data Platform, the web layer's entitlement-and-chat integration)

• Designed the teaser answer path to be computed by deterministic SQL rather than by the model

  The teaser path is computed directly in PostgreSQL. A pytest test asserts it contains none of the values that would make an incomplete answer look complete, which keeps unentitled or rejected interactions cheap relative to full tool-using answers. (Payor Rate Data Platform, the product's locked-turn response pipeline)