Loading the changelog…
Loading the changelog…
What changed in Larkspur's agents, what it did for customers and revenue, and what's waiting on you.
Each change is judged twice: Evals (did it pass its tests?) and Customers (did it help them and revenue?).
Waiting on you
7
decisions to make
Revenue at risk
$495,000
on launches with a problem left
Won from agent fixes
$310,000
from Email claims intake · $790,000 across all launches
Held before release
2
failing their evals
Decide before it ships.
The change: Read UK broker dates day first
Evals
FixedCustomers
Not live yetRecommended
Approve it, then merge it
Approve and merge it. It passes 98.8% of test cases, including all 24 UK date cases. It's the fix Kestrel Ridge Mutual asked for before their December renewal ($275,000 at risk).
The change descriptions, authors, who proposed each change, statuses, results and next steps are example content. Your decisions are saved for this sandbox session only; Reset sandbox clears them. Changes still in evals are judged by the eval suite; live changes by whether a known problem remains, because the example logs hold too few runs to compare rates. “Outcome” is the verdict on the launch the change belongs to. Version dates are when each version first appears in the agent logs. Post-launch evaluation measures how each launch moved revenue, usage and customer satisfaction.
The change: Don't approve when the deductible is missing
Evals
RegressionCustomers
Not live yetRecommended
Don't ship it
Don't ship. The auto-fix raised the pass rate but now denies valid claims with a $0 deductible. Have Decisioning rewrite the rule by hand.
Can't approve yet: It now fails a test case that used to pass.
The change: Read approval limits written with a comma
Evals
Below thresholdCustomers
Not live yetRecommended
Send it back to Decisioning
Read limits written like "$5k", then run the tests again. It needs 97% to ship.
Can't approve yet: It passes 94.8% of test cases and needs 97%.
Another agent fix won't solve it. Add the change to the roadmap.
The change: Explain claim statuses in plain English
Revenue since it went live
No revenue tied to it yet. Customers are still reporting the problem.
Evals
Partly fixedCustomers
Outcome partly metRecommended
Add “Proactive claim status updates” to the roadmap
Another wording change won't solve this. Customers still call because nobody tells them what happens next. Build proactive status updates.
The change: Hand off after three failed uploads instead of asking again
Revenue since it went live
No revenue tied to it yet. Customers are still reporting the problem.
Evals
Partly fixedCustomers
Outcome partly metRecommended
Add “Voice and chat claim intake” to the roadmap
Another agent fix won't solve this. The person taking over needs a summary of the chat, which is a change to the handoff screen.
Ask the team that owns the agent to fix what's left.
The change: Send failed pushes to a person instead of marking them done
Revenue since it went live
Evals
Partly fixedCustomers
Outcome partly metRecommended
Ask Claims intake to fix what's left
Teach it UK date formats before Kestrel Ridge Mutual renews in December ($275,000 at risk).
The change: New layout model for faxes and low-resolution scans
Revenue since it went live
Evals
Partly fixedCustomers
Outcome partly metRecommended
Ask Document AI to fix what's left
Detect sideways pages and read the letter O as zero. Fernhollow won't move its commercial book ($220,000) until faxes read cleanly.
Showing 2 changes where the real fix is a product changeClear filters
Auto-fixProposed by an observability tool, reviewed by Sofia Ruiz, Conversational channels
Written by Sofia Ruiz, Conversational channels
System prompt
The status assistant's instructions now tell it to describe a status in everyday words instead of showing internal queue names such as PENDING_REVIEW_L2.
Customer outcome Outcome partly met
2 measures improved; 2 complaints or failed runs since launch.
Revenue, usage and satisfaction →Evals and logs Partly fixed
Live, and the main problem is fixed. One part still goes wrong.
Still wrong
"In progress" doesn't say what happens next or who it's waiting on, and the status was served from a cache 19 days old (TRC-029).
Another wording change won't solve this. Customers still call because nobody tells them what happens next. Build proactive status updates.
claim-status@1.3.2 · changed from 1.3.0 · Sofia Ruiz, Conversational channels · Claims servicing
- Show the claim's status field.+ Describe the status in plain English. Never show internal queue names+ such as PENDING_REVIEW_L2.
Before · 1.3.0
Live 1 Sept 2026 to 14 Oct 2026
Running for Tallwater Home & Auto
After · 1.3.2
Live 14 Oct 2026 to 12 Nov 2026
Running for Saltmarsh Casualty
Too few runs in the example logs to compare rates. Read the runs below.
Runs before
TRC-018 · 1.3.0 · Tallwater Home & Auto · 1 Sept 2026 · handoff
run_8c0f4471 agent=claim-status@1.3.0 account=tallwater env=prod
10:15:02.110 tool_call get_claim_status(claim="CLM-TW-61822") -> {status:"PENDING_REVIEW_L2", updated:"2026-08-15", next_action_owner:"CLAIMANT", missing:["repair_estimate"]}
10:15:04.902 render status_card -> "Status: PENDING_REVIEW_L2" (fields next_action_owner, missing not rendered)
10:15:20.441 user: "what does that mean. do i need to do something"
10:15:27.003 llm.turn -> intent=talk_to_agent
10:15:27.010 handoff -> queue=contact_centre reason=user_confusedRuns after
TRC-029 · 1.3.2 · Saltmarsh Casualty · 14 Oct 2026 · handoff
run_2af5067d agent=claim-status@1.3.2 account=saltmarsh env=prod 16:30:11.003 tool_call get_claim_status(claim="CLM-SM-78004") -> served from cache age=19d 16:30:11.004 WARN cache ttl misconfigured (expected 1h) 16:30:12.550 render status_card -> "In progress" 16:30:40.118 user: "its been 3 weeks, is anyone looking at this" 16:30:52.002 handoff -> queue=contact_centre reason=user_requested
System prompt
The chat agent's instructions now stop it asking for photos after three failed uploads and pass the customer to a person instead of looping until they leave.
Customer outcome Outcome partly met
2 measures improved; chat CSAT got worse; 3 complaints or failed runs since launch.
Revenue, usage and satisfaction →Evals and logs Partly fixed
Live, and the main problem is fixed. One part still goes wrong.
Still wrong
The person receives the customer with no summary of the conversation (TRC-017). Large photos are still rejected.
Another agent fix won't solve this. The person taking over needs a summary of the chat, which is a change to the handoff screen.
claims-intake-chat@2.2.1 · changed from 2.2.0 · Sofia Ruiz, Conversational channels · Claims intake
- If an upload fails, apologise and ask the customer to try again.+ If an upload fails three times, stop asking. Tell the customer a person+ will help, and hand off to the claims_live queue.
Before · 2.2.0
Live 23 Aug 2026 to 16 Sept 2026
Running for Tallwater Home & Auto
After · 2.2.1
Live 16 Sept 2026 to 12 Nov 2026
Running for Kestrel Ridge Mutual, Tallwater Home & Auto
Too few runs in the example logs to compare rates. Read the runs below.
Runs before
TRC-016 · 2.2.0 · Tallwater Home & Auto · 23 Aug 2026 · abandoned
run_6a91d3e0 agent=claims-intake-chat@2.2.0 account=tallwater env=prod device=mobile_ios 19:28:01.330 step 1/6 collect_policy -> ok 19:29:40.118 step 2/6 collect_incident -> ok 19:30:02.004 step 3/6 request_photos 19:31:15.551 tool_call upload_media(file=IMG_4410.HEIC) -> 415 Unsupported Media Type 19:31:15.600 bot: "Sorry, that upload failed. Please try again." 19:32:48.209 tool_call upload_media(file=IMG_4410.HEIC) -> 415 19:32:48.250 bot: "Sorry, that upload failed. Please try again." 19:34:12.000 session idle 90s -> session.end status=abandoned last_step=request_photos
Runs after
TRC-017 · 2.2.1 · Kestrel Ridge Mutual · 16 Sept 2026 · handoff
run_7bd25e18 agent=claims-intake-chat@2.2.1 account=kestrel-ridge env=prod device=desktop 16:38:10.002 step 3/6 request_photos 16:39:01.443 upload_media -> 413 Payload Too Large (14.2MB, limit 10MB) 16:40:20.871 upload_media -> 413 16:41:30.006 upload_media -> 413 16:41:30.010 max upload attempts reached -> handoff 16:42:55.300 tool_call create_handoff(queue="claims_live", context=null) 16:42:55.301 WARN handoff context empty: summary builder not invoked for handoff_reason=upload_failed