Measuring trust in AI products
Last Saturday, I had the chance to listen to Waymo's Head of Insights during my mentoring group. Her talk about cross-touchpoint trust within AI services got me thinking about: How do we actually measure trust in AI products?
In most AI products I've worked on, we never measured user trust effectively (some failed because of it…but that's another story). We used the standard playbook: Likert scales, trust scores, cross-product surveys...and:
→ Users report 7/10 trust, but manually verify every recommendation
→ Scores stay stable, but users churn
→ One bad experience poisons trust across the entire product suite
This is what led me to adopt a triangulation approach to AI trust measurement
Vulnerability-Based Behavioral Metrics
People don't trust what they say they trust. They trust what they're willing to risk. Instead of asking "Do you trust XYZ?” measure what users actually put at stake.
Example: Financial Planning App:
→ Low trust: User tests with $100
→ Medium trust: User links secondary account
→ High trust: User authorizes automatic $10K investments
Trust Metric: Average $ amount users let AI manage without manual approval.
Event-tagged trust trajectories
Trust isn't static. It breaks fast and rebuilds slowly. Track exactly how trust changes after specific incidents.
Traditional surveys ask "How much do you trust this AI?". But they miss the critical question: what happens to trust right after something goes wrong?
Example: Autonomous Ride-Sharing (Inspired by that Waymo Head of Insights talk, see Part 1 if you missed the backstory)
Here's what one rider's trust timeline looks like:
Week 1: Trust baseline = 6/10 User books rides but only for non-critical trips
↓
Week 2: [EVENT: Smooth ride in heavy rain]
↓
Week 3: Trust = 7/10 (+1) User now books rides to work
↓
Week 4: [EVENT: Car braked suddenly, felt jerky]
↓
Week 5: Trust = 4/10 (-3) User switches back to manual rideshare
↓
Week 6-8: [Recovery period: 3 consecutive smooth rides]
↓
Week 9: Trust = 6/10 (recovered +2) User cautiously returns to autonomous rides
The pattern that shows up consistently:
→ Negative events hit 3x harder than positive ones boost trust (lost 3 points from one bad ride, gained only 1 from one good ride)
→ Recovery is slow: took 3 good experiences over 4 weeks to recover just 2 points
→ Trust recovery slope = 0.5 points/week—your key metric for how "forgiving" your product is to users who hit errors
Why this matters:
Most teams track "monthly active users" or "overall trust trend." But those metrics hide what's actually happening: one AI error can erase weeks of trust-building.
If your trust recovery slope is too shallow, users who hit one bad experience may never come back.
(This also depends on the space your product operates in, but that's Part 3: Context-Dependent Trust, coming next week.)
Your metrics to track:
→ How fast trust drops after errors
→ How many positive experiences are needed to recover
→ What trust level triggers churn
Where does trust actually live in your product?
Asking "Do you trust our AI?" is like asking "Do you like food?" The answer tells you nothing without context.
Think about Music Player's AI recommendations: Most users happily auto-play Discover Weekly (routine, low stakes, easily skipped). But would you trust that same AI to:
→ Choose music for your wedding? Probably want manual control.
→ Delete songs from your library permanently? Absolutely not.
Same AI. Same recommendation engine. Different trust levels based on context.
The framework: Map trust across three dimensions:
1. Task Stakes → Low stakes (suggest next song): High trust, auto-play enabled → High stakes (curate wedding playlist): Low trust, manual selection
2. Reversibility
→ Reversible (play a song, can skip): High trust → Irreversible (delete from library): Zero trust, never automate
3. Complexity → Routine pattern matching (similar songs): High trust → Judgment call (what fits the "mood"): Lower trust, wants control
Real-world example: email's smart compose
The AI writes email suggestions as you type. Trust varies by context:
Casual email to colleague: 80%+ accept AI suggestions → Low stakes, reversible, routine phrasing
Formal email to executive: 40% accept rate → Medium stakes, users want more control
Legal/sensitive communication: 10% accept → High stakes, users review every word
Email with financial details: 5% accept → Very high stakes, nearly always manual
You wouldn’t try to "increase trust to 80% everywhere." Accept low adoption in high-stakes contexts
How to implement this in your product:
Stop asking: "Do you trust our AI?"
Start measuring: "How comfortable are you having AI [specific action] without your review?"
Then map your features:
Feature A: Routine + Low stakes + Reversible = High trust expected → Design: Auto-apply, minimal friction
Feature B: Complex + High stakes + Irreversible = Low trust expected
→ Design: Preview required, confirmation needed, "are you sure?"
Feature C: Judgment + Medium stakes = Variable trust → Design: Suggest but don't apply, easy to modify
The complete triangulation method:
Measure what users risk (vulnerability-based: $ amounts, time committed, personal data shared)
Track trust after specific events (event-tagged: how much trust drops, how long recovery takes)
Map trust by context (which tasks deserve high vs low trust)