Loading the changelog…
Loading the changelog…
What changed in Larkspur's agents, what it did for customers and revenue, and what's waiting on you.
Each change is judged twice: Evals (did it pass its tests?) and Customers (did it help them and revenue?).
Waiting on you
7
decisions to make
Revenue at risk
$495,000
on launches with a problem left
Won from agent fixes
$310,000
from Email claims intake · $790,000 across all launches
Held before release
2
failing their evals
Decide before it ships.
The change: Read UK broker dates day first
Evals
FixedCustomers
Not live yetRecommended
Approve it, then merge it
Approve and merge it. It passes 98.8% of test cases, including all 24 UK date cases. It's the fix Kestrel Ridge Mutual asked for before their December renewal ($275,000 at risk).
The change descriptions, authors, who proposed each change, statuses, results and next steps are example content. Your decisions are saved for this sandbox session only; Reset sandbox clears them. Changes still in evals are judged by the eval suite; live changes by whether a known problem remains, because the example logs hold too few runs to compare rates. “Outcome” is the verdict on the launch the change belongs to. Version dates are when each version first appears in the agent logs. Post-launch evaluation measures how each launch moved revenue, usage and customer satisfaction.
The change: Don't approve when the deductible is missing
Evals
RegressionCustomers
Not live yetRecommended
Don't ship it
Don't ship. The auto-fix raised the pass rate but now denies valid claims with a $0 deductible. Have Decisioning rewrite the rule by hand.
Can't approve yet: It now fails a test case that used to pass.
The change: Read approval limits written with a comma
Evals
Below thresholdCustomers
Not live yetRecommended
Send it back to Decisioning
Read limits written like "$5k", then run the tests again. It needs 97% to ship.
Can't approve yet: It passes 94.8% of test cases and needs 97%.
Another agent fix won't solve it. Add the change to the roadmap.
The change: Explain claim statuses in plain English
Revenue since it went live
No revenue tied to it yet. Customers are still reporting the problem.
Evals
Partly fixedCustomers
Outcome partly metRecommended
Add “Proactive claim status updates” to the roadmap
Another wording change won't solve this. Customers still call because nobody tells them what happens next. Build proactive status updates.
The change: Hand off after three failed uploads instead of asking again
Revenue since it went live
No revenue tied to it yet. Customers are still reporting the problem.
Evals
Partly fixedCustomers
Outcome partly metRecommended
Add “Voice and chat claim intake” to the roadmap
Another agent fix won't solve this. The person taking over needs a summary of the chat, which is a change to the handoff screen.
Ask the team that owns the agent to fix what's left.
The change: Send failed pushes to a person instead of marking them done
Revenue since it went live
Evals
Partly fixedCustomers
Outcome partly metRecommended
Ask Claims intake to fix what's left
Teach it UK date formats before Kestrel Ridge Mutual renews in December ($275,000 at risk).
The change: New layout model for faxes and low-resolution scans
Revenue since it went live
Evals
Partly fixedCustomers
Outcome partly metRecommended
Ask Document AI to fix what's left
Detect sideways pages and read the letter O as zero. Fernhollow won't move its commercial book ($220,000) until faxes read cleanly.
Showing 4 changes proposed by an observability auto-fixClear filters
Auto-fixProposed by an observability tool, reviewed by Marcus Lindqvist, Decisioningwaiting for your sign-off
Prompt
The decision prompt now tells the model to send a claim to a person when the policy's deductible is blank, instead of paying the full invoice.
Auto-fixProposed by an observability tool, reviewed by Marcus Lindqvist, Decisioningwaiting for your sign-off
Auto-fixProposed by an observability tool, reviewed by Sofia Ruiz, Conversational channels
Auto-fixProposed by an observability tool, reviewed by Aiko Brennan, Document AI
Customer outcome Not live yet
No customer has seen it. It ships only once it passes its tests and is signed off.
Evals and logs Regression
Passes 95.8% of test cases (was 94.8%), but now fails a case that used to pass.
Still wrong
It now also denies a valid claim whose deductible is set to 0 (case_090).
Don't ship. The auto-fix raised the pass rate but now denies valid claims with a $0 deductible. Have Decisioning rewrite the rule by hand.
Can't approve yet: It now fails a test case that used to pass.
auto-decision@0.9.1-rc5 · changed from 0.9.1-rc4 · Marcus Lindqvist, Decisioning · Claims decisioning
Approve only when the estimate is within the account's auto-approve limit.+ If the policy's deductible is missing or blank, do not approve.+ Return decision "refer" with reason "missing_deductible".
Before · 0.9.1-rc4
Test suite of 400 cases
After · 0.9.1-rc5
Test suite of 400 cases
| Failed test cases, by check | 0.9.1-rc4 | 0.9.1-rc5 |
|---|---|---|
| over limit approved | 2 | 2 |
| missing deductible paid full | 11 | 9 |
| excluded peril approved | 6 | 6 |
| wrong line of business | 2 | 0 |
A version needs 97% to ship.
Newly failing: case_090 now denies a valid claim (deductible=0 explicitly set, treated as missing)
Runs before
TRC-010 · 0.9.1-rc4 · Larkspur internal · 4 Oct 2026 · eval failed
EVAL nightly auto-decision@0.9.1-rc4 suite=simple_claims_v4 n=400 02:10:00 passed=379 failed=21 (94.8%) threshold=97% -> FAIL failures by rule: over_limit_approved ............ 2 (limit parsing fixed for "5,000" but not "$5k") missing_deductible_paid_full ... 11 (unchanged) excluded_peril_approved ........ 6 wrong_line_of_business .......... 2 note: llm still returns approve when deductible missing; guard is advisory not blocking ERROR release gate blocked
Runs after
TRC-028 · 0.9.1-rc5 · Larkspur internal · 11 Oct 2026 · eval failed
EVAL nightly auto-decision@0.9.1-rc5 suite=simple_claims_v4 n=400 02:10:00 passed=383 failed=17 (95.8%) threshold=97% -> FAIL failures by rule: missing_deductible_paid_full ... 9 excluded_peril_approved ........ 6 over_limit_approved ............ 2 (case_311 limit="$5k") regression vs rc4: case_090 now denies a valid claim (deductible=0 explicitly set, treated as missing) ERROR release gate blocked; 3rd consecutive nightly fail
Tool
The limit check now reads "5,000" as 5000. Before, it read it as 5, so every claim looked over the limit and the check was ignored.
Customer outcome Not live yet
No customer has seen it. It ships only once it passes its tests and is signed off.
Evals and logs Below threshold
Passes 94.8% of test cases (was 90.3%). It needs 97% to ship.
Still wrong
Limits written as "$5k" are still misread.
Read limits written like "$5k", then run the tests again. It needs 97% to ship.
Can't approve yet: It passes 94.8% of test cases and needs 97%.
auto-decision@0.9.1-rc4 · changed from 0.9.1-rc2 · Marcus Lindqvist, Decisioning · Claims decisioning
- const limit = parseFloat(rules.auto_approve_limit);+ const limit = Number(rules.auto_approve_limit.replace(/,/g, ""));
Before · 0.9.1-rc2
Test suite of 400 cases
After · 0.9.1-rc4
Test suite of 400 cases
| Failed test cases, by check | 0.9.1-rc2 | 0.9.1-rc4 |
|---|---|---|
| over limit approved | 14 | 2 |
| missing deductible paid full | 11 | 11 |
| excluded peril approved | 8 | 6 |
| wrong line of business | 6 | 2 |
A version needs 97% to ship.
Runs before
TRC-009 · 0.9.1-rc2 · Larkspur internal · 27 Sept 2026 · eval failed
EVAL nightly auto-decision@0.9.1-rc2 suite=simple_claims_v4 n=400 02:10:00 passed=361 failed=39 (90.3%) threshold=97% -> FAIL failures by rule: over_limit_approved ............ 14 e.g. case_118 limit="2,500" est=3110 -> approve missing_deductible_paid_full ... 11 e.g. case_207 deductible=null -> payout=full excluded_peril_approved ........ 8 e.g. case_033 cause=flood policy excl flood -> approve wrong_line_of_business .......... 6 ERROR release gate blocked: 0.9.1-rc2 not promoted
Runs after
TRC-010 · 0.9.1-rc4 · Larkspur internal · 4 Oct 2026 · eval failed
EVAL nightly auto-decision@0.9.1-rc4 suite=simple_claims_v4 n=400 02:10:00 passed=379 failed=21 (94.8%) threshold=97% -> FAIL failures by rule: over_limit_approved ............ 2 (limit parsing fixed for "5,000" but not "$5k") missing_deductible_paid_full ... 11 (unchanged) excluded_peril_approved ........ 6 wrong_line_of_business .......... 2 note: llm still returns approve when deductible missing; guard is advisory not blocking ERROR release gate blocked
System prompt
The status assistant's instructions now tell it to describe a status in everyday words instead of showing internal queue names such as PENDING_REVIEW_L2.
Customer outcome Outcome partly met
2 measures improved; 2 complaints or failed runs since launch.
Revenue, usage and satisfaction →Evals and logs Partly fixed
Live, and the main problem is fixed. One part still goes wrong.
Still wrong
"In progress" doesn't say what happens next or who it's waiting on, and the status was served from a cache 19 days old (TRC-029).
Another wording change won't solve this. Customers still call because nobody tells them what happens next. Build proactive status updates.
claim-status@1.3.2 · changed from 1.3.0 · Sofia Ruiz, Conversational channels · Claims servicing
- Show the claim's status field.+ Describe the status in plain English. Never show internal queue names+ such as PENDING_REVIEW_L2.
Before · 1.3.0
Live 1 Sept 2026 to 14 Oct 2026
Running for Tallwater Home & Auto
After · 1.3.2
Live 14 Oct 2026 to 12 Nov 2026
Running for Saltmarsh Casualty
Too few runs in the example logs to compare rates. Read the runs below.
Runs before
TRC-018 · 1.3.0 · Tallwater Home & Auto · 1 Sept 2026 · handoff
run_8c0f4471 agent=claim-status@1.3.0 account=tallwater env=prod
10:15:02.110 tool_call get_claim_status(claim="CLM-TW-61822") -> {status:"PENDING_REVIEW_L2", updated:"2026-08-15", next_action_owner:"CLAIMANT", missing:["repair_estimate"]}
10:15:04.902 render status_card -> "Status: PENDING_REVIEW_L2" (fields next_action_owner, missing not rendered)
10:15:20.441 user: "what does that mean. do i need to do something"
10:15:27.003 llm.turn -> intent=talk_to_agent
10:15:27.010 handoff -> queue=contact_centre reason=user_confusedRuns after
TRC-029 · 1.3.2 · Saltmarsh Casualty · 14 Oct 2026 · handoff
run_2af5067d agent=claim-status@1.3.2 account=saltmarsh env=prod 16:30:11.003 tool_call get_claim_status(claim="CLM-SM-78004") -> served from cache age=19d 16:30:11.004 WARN cache ttl misconfigured (expected 1h) 16:30:12.550 render status_card -> "In progress" 16:30:40.118 user: "its been 3 weeks, is anyone looking at this" 16:30:52.002 handoff -> queue=contact_centre reason=user_requested
Tool
When OCR confidence is low, the extraction tool now retries the page with enhancement, then again with 2x upscaling.
Customer outcome Not measured
Reverted before customers could judge it. The launch is judged on “New layout model for faxes and low-resolution scans”.
Revenue, usage and satisfaction →Evals and logs Reverted
Taken back out in 4.3.0.
Why it was undone
The retries pushed a 14-page scan past the 1,200-second limit, and nothing was saved (TRC-013).
Nothing more to do here: the new layout model in 4.3.0 replaced it.
doc-extract@4.2.1 · changed from 4.2.0 · Aiko Brennan, Document AI · Document extraction · rolled back in 4.3.0
+ if (page.meanConf < 0.6) retry({ enhance: true });+ if (page.meanConf < 0.6) retry({ enhance: true, upscale: 2 });
Before · 4.2.0
Live 18 Aug 2026 to 3 Sept 2026
Running for Fernhollow Insurance Group
After · 4.2.1
Live 3 Sept 2026 to 1 Oct 2026
Running for Saltmarsh Casualty
Too few runs in the example logs to compare rates. Read the runs below.
Runs before
TRC-011 · 4.2.0 · Fernhollow Insurance Group · 18 Aug 2026 · completed low confidence
run_1a6b8c77 agent=doc-extract@4.2.0 account=fernhollow env=prod doc=police_report_fax.tif pages=3 12:58:10.400 preprocess: dpi=98 skew=6.8deg (deskew threshold 10deg -> skipped) 12:58:16.902 ocr page1 mean_conf=0.61 12:58:21.115 field damage_estimate raw="$7,200" conf=0.48 (source region shows "$1,200") 12:58:21.116 field incident_date raw="06/14/2062" conf=0.52 12:58:22.001 WARN 2 fields below 0.6 conf; no human-review route configured -> accepted 12:58:22.010 run status=completed
Runs after
TRC-013 · 4.2.1 · Saltmarsh Casualty · 14 Sept 2026 · failed
run_33d1f620 agent=doc-extract@4.2.1 account=saltmarsh env=prod doc=estimate_scan_multi.pdf pages=14 09:27:02.001 preprocess -> 14 pages, 9 flagged low-quality 09:27:04.330 ocr pass 1 ... 09:33:10.118 ocr low conf -> retry with enhance=true 09:39:51.774 retry 2 with enhance=true, upscale=2x 09:47:05.002 ERROR job timeout after 1200s (limit 1200s) 09:47:05.003 run status=failed; no partial results saved