Loading the changelog…
Loading the changelog…
What changed in Larkspur's agents, what it did for customers and revenue, and what's waiting on you.
Each change is judged twice: Evals (did it pass its tests?) and Customers (did it help them and revenue?).
Waiting on you
7
decisions to make
Revenue at risk
$495,000
on launches with a problem left
Won from agent fixes
$310,000
from Email claims intake · $790,000 across all launches
Held before release
2
failing their evals
Decide before it ships.
The change: Read UK broker dates day first
Evals
FixedCustomers
Not live yetRecommended
Approve it, then merge it
Approve and merge it. It passes 98.8% of test cases, including all 24 UK date cases. It's the fix Kestrel Ridge Mutual asked for before their December renewal ($275,000 at risk).
The change descriptions, authors, who proposed each change, statuses, results and next steps are example content. Your decisions are saved for this sandbox session only; Reset sandbox clears them. Changes still in evals are judged by the eval suite; live changes by whether a known problem remains, because the example logs hold too few runs to compare rates. “Outcome” is the verdict on the launch the change belongs to. Version dates are when each version first appears in the agent logs. Post-launch evaluation measures how each launch moved revenue, usage and customer satisfaction.
The change: Don't approve when the deductible is missing
Evals
RegressionCustomers
Not live yetRecommended
Don't ship it
Don't ship. The auto-fix raised the pass rate but now denies valid claims with a $0 deductible. Have Decisioning rewrite the rule by hand.
Can't approve yet: It now fails a test case that used to pass.
The change: Read approval limits written with a comma
Evals
Below thresholdCustomers
Not live yetRecommended
Send it back to Decisioning
Read limits written like "$5k", then run the tests again. It needs 97% to ship.
Can't approve yet: It passes 94.8% of test cases and needs 97%.
Another agent fix won't solve it. Add the change to the roadmap.
The change: Explain claim statuses in plain English
Revenue since it went live
No revenue tied to it yet. Customers are still reporting the problem.
Evals
Partly fixedCustomers
Outcome partly metRecommended
Add “Proactive claim status updates” to the roadmap
Another wording change won't solve this. Customers still call because nobody tells them what happens next. Build proactive status updates.
The change: Hand off after three failed uploads instead of asking again
Revenue since it went live
No revenue tied to it yet. Customers are still reporting the problem.
Evals
Partly fixedCustomers
Outcome partly metRecommended
Add “Voice and chat claim intake” to the roadmap
Another agent fix won't solve this. The person taking over needs a summary of the chat, which is a change to the handoff screen.
Ask the team that owns the agent to fix what's left.
The change: Send failed pushes to a person instead of marking them done
Revenue since it went live
Evals
Partly fixedCustomers
Outcome partly metRecommended
Ask Claims intake to fix what's left
Teach it UK date formats before Kestrel Ridge Mutual renews in December ($275,000 at risk).
The change: New layout model for faxes and low-resolution scans
Revenue since it went live
Evals
Partly fixedCustomers
Outcome partly metRecommended
Ask Document AI to fix what's left
Detect sideways pages and read the letter O as zero. Fernhollow won't move its commercial book ($220,000) until faxes read cleanly.
Showing 4 changes that passed their evals but haven't yet met their customer outcomeClear filters
Auto-fixProposed by an observability tool, reviewed by Sofia Ruiz, Conversational channels
Written by Sofia Ruiz, Conversational channels
Written by Dana Okafor, Claims intake
Written by Aiko Brennan, Document AI
System prompt
The status assistant's instructions now tell it to describe a status in everyday words instead of showing internal queue names such as PENDING_REVIEW_L2.
Customer outcome Outcome partly met
2 measures improved; 2 complaints or failed runs since launch.
Revenue, usage and satisfaction →Evals and logs Partly fixed
Live, and the main problem is fixed. One part still goes wrong.
Still wrong
"In progress" doesn't say what happens next or who it's waiting on, and the status was served from a cache 19 days old (TRC-029).
Another wording change won't solve this. Customers still call because nobody tells them what happens next. Build proactive status updates.
claim-status@1.3.2 · changed from 1.3.0 · Sofia Ruiz, Conversational channels · Claims servicing
- Show the claim's status field.+ Describe the status in plain English. Never show internal queue names+ such as PENDING_REVIEW_L2.
Before · 1.3.0
Live 1 Sept 2026 to 14 Oct 2026
Running for Tallwater Home & Auto
After · 1.3.2
Live 14 Oct 2026 to 12 Nov 2026
Running for Saltmarsh Casualty
Too few runs in the example logs to compare rates. Read the runs below.
Runs before
TRC-018 · 1.3.0 · Tallwater Home & Auto · 1 Sept 2026 · handoff
run_8c0f4471 agent=claim-status@1.3.0 account=tallwater env=prod
10:15:02.110 tool_call get_claim_status(claim="CLM-TW-61822") -> {status:"PENDING_REVIEW_L2", updated:"2026-08-15", next_action_owner:"CLAIMANT", missing:["repair_estimate"]}
10:15:04.902 render status_card -> "Status: PENDING_REVIEW_L2" (fields next_action_owner, missing not rendered)
10:15:20.441 user: "what does that mean. do i need to do something"
10:15:27.003 llm.turn -> intent=talk_to_agent
10:15:27.010 handoff -> queue=contact_centre reason=user_confusedRuns after
TRC-029 · 1.3.2 · Saltmarsh Casualty · 14 Oct 2026 · handoff
run_2af5067d agent=claim-status@1.3.2 account=saltmarsh env=prod 16:30:11.003 tool_call get_claim_status(claim="CLM-SM-78004") -> served from cache age=19d 16:30:11.004 WARN cache ttl misconfigured (expected 1h) 16:30:12.550 render status_card -> "In progress" 16:30:40.118 user: "its been 3 weeks, is anyone looking at this" 16:30:52.002 handoff -> queue=contact_centre reason=user_requested
System prompt
The chat agent's instructions now stop it asking for photos after three failed uploads and pass the customer to a person instead of looping until they leave.
Customer outcome Outcome partly met
2 measures improved; chat CSAT got worse; 3 complaints or failed runs since launch.
Revenue, usage and satisfaction →Evals and logs Partly fixed
Live, and the main problem is fixed. One part still goes wrong.
Still wrong
The person receives the customer with no summary of the conversation (TRC-017). Large photos are still rejected.
Another agent fix won't solve this. The person taking over needs a summary of the chat, which is a change to the handoff screen.
claims-intake-chat@2.2.1 · changed from 2.2.0 · Sofia Ruiz, Conversational channels · Claims intake
- If an upload fails, apologise and ask the customer to try again.+ If an upload fails three times, stop asking. Tell the customer a person+ will help, and hand off to the claims_live queue.
Before · 2.2.0
Live 23 Aug 2026 to 16 Sept 2026
Running for Tallwater Home & Auto
After · 2.2.1
Live 16 Sept 2026 to 12 Nov 2026
Running for Kestrel Ridge Mutual, Tallwater Home & Auto
Too few runs in the example logs to compare rates. Read the runs below.
Runs before
TRC-016 · 2.2.0 · Tallwater Home & Auto · 23 Aug 2026 · abandoned
run_6a91d3e0 agent=claims-intake-chat@2.2.0 account=tallwater env=prod device=mobile_ios 19:28:01.330 step 1/6 collect_policy -> ok 19:29:40.118 step 2/6 collect_incident -> ok 19:30:02.004 step 3/6 request_photos 19:31:15.551 tool_call upload_media(file=IMG_4410.HEIC) -> 415 Unsupported Media Type 19:31:15.600 bot: "Sorry, that upload failed. Please try again." 19:32:48.209 tool_call upload_media(file=IMG_4410.HEIC) -> 415 19:32:48.250 bot: "Sorry, that upload failed. Please try again." 19:34:12.000 session idle 90s -> session.end status=abandoned last_step=request_photos
Runs after
TRC-017 · 2.2.1 · Kestrel Ridge Mutual · 16 Sept 2026 · handoff
run_7bd25e18 agent=claims-intake-chat@2.2.1 account=kestrel-ridge env=prod device=desktop 16:38:10.002 step 3/6 request_photos 16:39:01.443 upload_media -> 413 Payload Too Large (14.2MB, limit 10MB) 16:40:20.871 upload_media -> 413 16:41:30.006 upload_media -> 413 16:41:30.010 max upload attempts reached -> handoff 16:42:55.300 tool_call create_handoff(queue="claims_live", context=null) 16:42:55.301 WARN handoff context empty: summary builder not invoked for handoff_reason=upload_failed
Tool
When the push to ClaimsCore fails, the run now goes to the manual-entry queue instead of being marked complete. Runs also save a checkpoint after each step, so a deploy no longer kills them midway.
Customer outcome Outcome partly met
$310,000 won or kept; 3 measures improved; 1 complaint or failed run since launch; $275,000 still at risk.
Revenue, usage and satisfaction →Evals and logs Partly fixed
Live, and the main problem is fixed. One part still goes wrong.
Still wrong
A UK broker's date "03/10/2026" was read as 10 March, not 3 October, so ClaimsCore rejected the claim (TRC-006).
Teach it UK date formats before Kestrel Ridge Mutual renews in December ($275,000 at risk).
claims-intake-email@2.10.0 · changed from 2.9.2 · Dana Okafor, Claims intake · Claims intake
- if (push.failed) { log.warn("push_failed"); return run.complete(); }+ if (push.failed) return handoff({ queue: "manual_entry", reason: "push_failed" });+ checkpoint: "after_each_step"
Before · 2.9.2
Live 5 Sept 2026 to 3 Oct 2026
Running for Saltmarsh Casualty
After · 2.10.0
Live 3 Oct 2026 to 12 Nov 2026
Running for Kestrel Ridge Mutual, Tallwater Home & Auto
Too few runs in the example logs to compare rates. Read the runs below.
Runs before
TRC-004 · 2.9.2 · Saltmarsh Casualty · 5 Sept 2026 · failed
run_c88d2e70 agent=claims-intake-email@2.9.2 account=saltmarsh env=prod 16:41:28.009 step extract_fnol -> ok 16:41:29.551 step map_to_claimscore -> ok 16:41:29.560 tool_call claimscore.create_claim -> ... 16:41:59.561 ERROR timeout after 30000ms (no response) 16:42:00.100 retry 1/2 -> timeout after 30000ms 16:42:30.700 retry 2/2 -> timeout after 30000ms 16:42:30.701 ERROR push_failed reason=timeout; queued_for_retry=false 16:42:30.702 run status=completed (push failure logged as warning only)
Runs after
TRC-006 · 2.10.0 · Kestrel Ridge Mutual · 3 Oct 2026 · failed
run_e5512ab8 agent=claims-intake-email@2.10.0 account=kestrel-ridge env=prod
11:08:15.004 step extract_fnol -> ok dateOfLoss_raw="03/10/2026"
11:08:15.210 normalize date -> "2026-03-10" (assumed MM/DD; email from UK-based broker)
11:08:18.771 tool_call claimscore.create_claim -> 422 {"detail":"dateOfLoss 2026-03-10 is before policy effective date 2026-09-01"}
11:08:18.772 ERROR push_failed; no retry for 4xx
11:08:19.001 handoff -> queue=manual_entry reason=push_failedTRC-025 · 2.10.0 · Tallwater Home & Auto · 8 Oct 2026 · completed
run_f7c2d34a agent=claims-intake-email@2.10.0 account=tallwater env=prod 22:00:58.004 worker received SIGTERM (scheduled deploy) 22:00:58.010 checkpoint saved at step extract_fnol 22:01:09.880 resumed on new worker from checkpoint 22:01:15.002 run status=completed (resumed=1)
AI model
Document extraction moved from layout model v4 to v5, which is tuned on faxed and scanned pages. This replaced the enhancement retries from 4.2.1.
Customer outcome Outcome partly met
3 measures improved; 3 complaints or failed runs since launch; $220,000 still at risk.
Revenue, usage and satisfaction →Evals and logs Partly fixed
Live, and the main problem is fixed. One part still goes wrong.
Still wrong
Sideways pages aren't detected, and faxed totals with the letter O for zero still can't be read.
Detect sideways pages and read the letter O as zero. Fernhollow won't move its commercial book ($220,000) until faxes read cleanly.
doc-extract@4.3.0 · changed from 4.2.1 · Aiko Brennan, Document AI · Document extraction
- layout_model: "v4"+ layout_model: "v5"- on_low_conf: [retry({ enhance: true }), retry({ enhance: true, upscale: 2 })]
Before · 4.2.1
Live 3 Sept 2026 to 1 Oct 2026
Running for Saltmarsh Casualty
After · 4.3.0
Live 1 Oct 2026 to 12 Nov 2026
Running for Fernhollow Insurance Group, Kestrel Ridge Mutual
Too few runs in the example logs to compare rates. Read the runs below.
Runs before
TRC-011 · 4.2.0 · Fernhollow Insurance Group · 18 Aug 2026 · completed low confidence
run_1a6b8c77 agent=doc-extract@4.2.0 account=fernhollow env=prod doc=police_report_fax.tif pages=3 12:58:10.400 preprocess: dpi=98 skew=6.8deg (deskew threshold 10deg -> skipped) 12:58:16.902 ocr page1 mean_conf=0.61 12:58:21.115 field damage_estimate raw="$7,200" conf=0.48 (source region shows "$1,200") 12:58:21.116 field incident_date raw="06/14/2062" conf=0.52 12:58:22.001 WARN 2 fields below 0.6 conf; no human-review route configured -> accepted 12:58:22.010 run status=completed
TRC-013 · 4.2.1 · Saltmarsh Casualty · 14 Sept 2026 · failed
run_33d1f620 agent=doc-extract@4.2.1 account=saltmarsh env=prod doc=estimate_scan_multi.pdf pages=14 09:27:02.001 preprocess -> 14 pages, 9 flagged low-quality 09:27:04.330 ocr pass 1 ... 09:33:10.118 ocr low conf -> retry with enhance=true 09:39:51.774 retry 2 with enhance=true, upscale=2x 09:47:05.002 ERROR job timeout after 1200s (limit 1200s) 09:47:05.003 run status=failed; no partial results saved
Runs after
TRC-014 · 4.3.0 · Kestrel Ridge Mutual · 1 Oct 2026 · completed low confidence
run_4e870aa2 agent=doc-extract@4.3.0 account=kestrel-ridge env=prod doc=proof_of_loss_scan.pdf 13:12:30.200 preprocess: page 2 rotation=90deg detected=false 13:12:34.552 ocr page2 mean_conf=0.22 (text sideways) 13:12:36.004 fields from page2: claimed_amount=null signature_date=null 13:12:36.005 WARN required fields missing; marked complete anyway (validation=lenient)
TRC-015 · 4.3.0 · Fernhollow Insurance Group · 9 Oct 2026 · completed low confidence
run_5f0a2c39 agent=doc-extract@4.3.0 account=fernhollow env=prod doc=medical_bill_fax_p1-4.tif 10:03:51.101 preprocess: dpi=96 noise=high (fax header lines) 10:03:57.440 table extraction: 11 line items found, expected 14 (3 rows merged) 10:03:57.998 field total_billed="$4,18O.OO" -> parse error (letter O vs zero) -> null 10:03:58.001 ERROR total_billed unparseable; sum(line_items)=3,412.00 mismatch 10:03:58.010 run status=completed_low_confidence