PO
Example data

Post-launch evaluation

Did each launch pay off? How every shipped feature, agent fix and new use case moved revenue, product usage and customer satisfaction, compared with before launch on Expected Impact. A fix can pass its evals and still not help customers, so each one is judged on both, with what to do next.

Revenue won or kept
$790,000
Revenue still at risk
$495,000
Launches that met their outcome
2 of 6
Fixes that passed evals, missed outcome
4 of 4

Revenue won or kept by week

New deals, expansions and renewals after each launch

  1. –
    Week of
    14 Sept
  2. –
    Week of
    21 Sept
  3. –
    Week of
    28 Sept
  4. $60k
    Week of
    5 Oct
  5. $240k
    Week of
    12 Oct
  6. $490k
    Week of
    19 Oct
  7. –
    Week of
    26 Oct
  8. –
    Week of
    2 Nov
  9. –
    Week of
    9 Nov

Agentic fixNew use caseNew feature

  • Outcome partly metUI designLaunched 14 Oct 2026

    Claim statuses in plain English

    2 complaints or failed runs since launch.

    Revenue

    No deals tied to this

    Product usage

    67.2% →61%better

    Status chats followed by a call or chat within 48h (lower is better)

    Customer satisfaction

    2.9/5 →3.2/5better

    Status agent CSAT

    Passed its evals, but missed its customer outcome

    Evals and logsPartly fixed
    Explain claim statuses in plain EnglishAuto-fixLive, and the main problem is fixed. One part still goes wrong.
    Customer outcomeOutcome partly met
    2 measures improved; 2 complaints or failed runs since launch.
    Next stepNeeds a product change, not a prompt fix

    Another wording change won't solve this. Customers still call because nobody tells them what happens next. Build proactive status updates.

  • Outcome partly metAgentic fixLaunched 3 Oct 2026

    Claims reach ClaimsCore, or a person, every time

    1 complaint or failed run since launch.

    Revenue

    $310,000

    won or kept

    $275,000 at risk

    Product usage

    82% →95%better

    Email claims reaching ClaimsCore without re-keying

    Customer satisfaction

    3.4/5 →4.1/5better

    Adjuster rating of email intake

    Passed its evals, but missed its customer outcome

    Evals and logsPartly fixed
    Send failed pushes to a person instead of marking them doneLive, and the main problem is fixed. One part still goes wrong.
    Customer outcomeOutcome partly met
    $310,000 won or kept; 3 measures improved; 1 complaint or failed run since launch; $275,000 still at risk.
    Next stepIterate on the agent

    Teach it UK date formats before Kestrel Ridge Mutual renews in December ($275,000 at risk).

  • Outcome partly metAgentic fixLaunched 1 Oct 2026

    Read faxed and low-resolution documents

    3 complaints or failed runs since launch.

    Revenue

    No deals tied to this

    $220,000 at risk

    Product usage

    74% →83%better

    Fields extracted with high confidence

    Customer satisfaction

    3.1/5 →3.4/5better

    Adjuster rating of extracted data

    Passed its evals, but missed its customer outcome

    Evals and logsPartly fixed
    New layout model for faxes and low-resolution scansLive, and the main problem is fixed. One part still goes wrong.
    Customer outcomeOutcome partly met
    3 measures improved; 3 complaints or failed runs since launch; $220,000 still at risk.
    Next stepIterate on the agent

    Detect sideways pages and read the letter O as zero. Fernhollow won't move its commercial book ($220,000) until faxes read cleanly.

  • Outcome partly metUI designLaunched 16 Sept 2026

    Hand chat customers to a person when photos fail

    Chat CSAT got worse; 3 complaints or failed runs since launch.

    Revenue

    No deals tied to this

    Product usage

    26.1% →29.3%better

    Chats that end in a submitted claim

    Customer satisfaction

    3.6/5 →3.5/5worse

    Chat CSAT

    Passed its evals, but missed its customer outcome

    Evals and logsPartly fixed
    Hand off after three failed uploads instead of asking againLive, and the main problem is fixed. One part still goes wrong.
    Customer outcomeOutcome partly met
    2 measures improved; chat CSAT got worse; 3 complaints or failed runs since launch.
    Next stepNeeds a product change, not a prompt fix

    Another agent fix won't solve this. The person taking over needs a summary of the chat, which is a change to the handoff screen.

  • Outcome metNew use caseLaunched 1 Oct 2026

    Underwriting submission triage (pilot)

    Revenue

    $180,000

    won or kept

    Product usage

    412new

    Broker submissions triaged by the agent

    Customer satisfaction

    3.9/5new

    Pilot underwriters' rating

    Next stepIterate on the agent

    Check loss runs against the ACORD form, so underwriters open one document, not two.

  • Outcome metNew featureLaunched 21 Sept 2026

    Red-flag detection at claims intake

    Revenue

    $300,000

    won or kept

    Product usage

    0 →3better

    Accounts with red flags turned on

    Customer satisfaction

    4.2/5new

    SIU rating of the flags

    Next stepIterate on the agent

    Add an allow list of fleet tow partners, as Pellbrook asked.

Example data, including who proposed each fix and the next steps. Outcome met: every measure improved and nothing is at risk. Too early to tell: live under 14 days. “Evals and logs” is the result on the agent changelog.