Post-launch evaluation
Did each launch pay off? How every shipped feature, agent fix and new use case moved revenue, product usage and customer satisfaction, compared with before launch on Expected Impact. A fix can pass its evals and still not help customers, so each one is judged on both, with what to do next.
- Revenue won or kept
- $790,000
- Revenue still at risk
- $495,000
- Launches that met their outcome
- 2 of 6
- Fixes that passed evals, missed outcome
- 4 of 4
Revenue won or kept by week
New deals, expansions and renewals after each launch
- –Week of
14 Sept - –Week of
21 Sept - –Week of
28 Sept - $60kWeek of
5 Oct - $240kWeek of
12 Oct - $490kWeek of
19 Oct - –Week of
26 Oct - –Week of
2 Nov - –Week of
9 Nov
Agentic fixNew use caseNew feature
- Outcome partly metUI designLaunched 14 Oct 2026
Claim statuses in plain English
2 complaints or failed runs since launch.
Revenue
No deals tied to this
Product usage
67.2% →61%better
Status chats followed by a call or chat within 48h (lower is better)
Customer satisfaction
2.9/5 →3.2/5better
Status agent CSAT
Passed its evals, but missed its customer outcome
- Evals and logsPartly fixed
- Explain claim statuses in plain EnglishAuto-fixLive, and the main problem is fixed. One part still goes wrong.
- Customer outcomeOutcome partly met
- 2 measures improved; 2 complaints or failed runs since launch.
Next stepNeeds a product change, not a prompt fixAnother wording change won't solve this. Customers still call because nobody tells them what happens next. Build proactive status updates.
- Outcome partly metAgentic fixLaunched 3 Oct 2026
Claims reach ClaimsCore, or a person, every time
1 complaint or failed run since launch.
Revenue
$310,000
won or kept
$275,000 at risk
Product usage
82% →95%better
Email claims reaching ClaimsCore without re-keying
Customer satisfaction
3.4/5 →4.1/5better
Adjuster rating of email intake
Passed its evals, but missed its customer outcome
- Evals and logsPartly fixed
- Send failed pushes to a person instead of marking them doneLive, and the main problem is fixed. One part still goes wrong.
- Customer outcomeOutcome partly met
- $310,000 won or kept; 3 measures improved; 1 complaint or failed run since launch; $275,000 still at risk.
Next stepIterate on the agentTeach it UK date formats before Kestrel Ridge Mutual renews in December ($275,000 at risk).
- Outcome partly metAgentic fixLaunched 1 Oct 2026
Read faxed and low-resolution documents
3 complaints or failed runs since launch.
Revenue
No deals tied to this
$220,000 at risk
Product usage
74% →83%better
Fields extracted with high confidence
Customer satisfaction
3.1/5 →3.4/5better
Adjuster rating of extracted data
Passed its evals, but missed its customer outcome
- Evals and logsPartly fixed
- New layout model for faxes and low-resolution scansLive, and the main problem is fixed. One part still goes wrong.
- Customer outcomeOutcome partly met
- 3 measures improved; 3 complaints or failed runs since launch; $220,000 still at risk.
Next stepIterate on the agentDetect sideways pages and read the letter O as zero. Fernhollow won't move its commercial book ($220,000) until faxes read cleanly.
- Outcome partly metUI designLaunched 16 Sept 2026
Hand chat customers to a person when photos fail
Chat CSAT got worse; 3 complaints or failed runs since launch.
Revenue
No deals tied to this
Product usage
26.1% →29.3%better
Chats that end in a submitted claim
Customer satisfaction
3.6/5 →3.5/5worse
Chat CSAT
Passed its evals, but missed its customer outcome
- Evals and logsPartly fixed
- Hand off after three failed uploads instead of asking againLive, and the main problem is fixed. One part still goes wrong.
- Customer outcomeOutcome partly met
- 2 measures improved; chat CSAT got worse; 3 complaints or failed runs since launch.
Next stepNeeds a product change, not a prompt fixAnother agent fix won't solve this. The person taking over needs a summary of the chat, which is a change to the handoff screen.
- Outcome metNew use caseLaunched 1 Oct 2026
Underwriting submission triage (pilot)
Revenue
$180,000
won or kept
Product usage
412new
Broker submissions triaged by the agent
Customer satisfaction
3.9/5new
Pilot underwriters' rating
Next stepIterate on the agentCheck loss runs against the ACORD form, so underwriters open one document, not two.
- Outcome metNew featureLaunched 21 Sept 2026
Red-flag detection at claims intake
Revenue
$300,000
won or kept
Product usage
0 →3better
Accounts with red flags turned on
Customer satisfaction
4.2/5new
SIU rating of the flags
Next stepIterate on the agentAdd an allow list of fleet tow partners, as Pellbrook asked.
Example data, including who proposed each fix and the next steps. Outcome met: every measure improved and nothing is at risk. Too early to tell: live under 14 days. “Evals and logs” is the result on the agent changelog.