2025 UX Connect Aarhus - garbage in garbage out.pptx

Garbage in, garbage out?
Why measure and how to measure error rates as we head into AI
UX Connect, Aarhus, June 2025
Caroline Jarrett
@cjforms.bsky.social

Caroline Jarrett Effortmark.co.uk @cjforms (CC) BY SA-4.0
2
Martin Jordan announced error rate as a metric
https://bsky.app/profile/martinjordan.com/post/3kzmgtkdcuz2k

3
I got interested in better forms because of errors
Understanding the costs of data capture: paper, automatic and with the internet - Effortmar
k

4
For “AI”, data quality matters
https://bsky.app/profile/carnage4life.bsky.social/post/3liouuj3phs2z
The “garbage in, garbage out” principle also applies to
- Large language models (LLMs)
- Image recognition
- And everything else really

5
Let’s meet
Write a sticky note for the work that you do
Write another sticky note for what brought you to that work
Compare your answers with your neighbour
I’ll be collecting all the sticky notes later, so please keep them
2 minutes

6
I’ve observed that people “fudge” through forms
To fudge a form:
answer one or more questions
in a way that’s not entirely truthful
Photo by Phil Hearing on Unsplash
FUDGE | definition in the Cambridge Learner’s Dictionary

7
I admit that I fudged my bus pass photo
The application form said that I had to submit a passport photo

8
I chose one that doesn’t comply with passport rules

9
If you’ve fudged a form, share your story
Write a sticky note with an example of fudging a form
Share with your neighbour
2 minutes

10
Takeaway There are many ways in which
the data we collect isn’t
entirely accurate

11
Agenda
Share what we know about our error rates (if anything)
Try using “Six ways to think about errors”
Meet the data quality framework
Think about what we might do differently

12
Share what we know about
our error rates (if anything)

13
Some errors hit
the headlines
Disabled Ipswich woman told to rep
ay £5,000 of universal credit - BBC
News

14
Some people don’t
get to the end of
their application
People on low incomes or no
income in California can apply
for government money to buy
fresh food (CalFresh).
https://www.calsaws.org/wp-content/uploads/2025/02/BenefitsCal-Usage-
Metrics-Report-Nov-Dec-2024.pdf

15
The result matters more than the journey
People rarely fill in forms for fun
We’re aiming to achieve some result
Photo by Marcos Ramírez on Unsplash

16
Let's learn from our experiences and ideas about
errors
Write on a sticky note:
• a brief description of a service or product related to your work
• what you consider as an error in that service or product
• whether you know or don’t know the error rates
If you have different types of error, please write one sticky note for each type
Then compare your note(s) with your neighbour
5 minutes

17
Agenda

18
I’ve been thinking about ways to think about errors
Thanks to the team at HMRC who inspired this
Errors Examples
Problems along the way Tried to buy 2Kg carrots, ordered 2 carrots
Wrong result Ordered pork pie, got lettuce
Unnecessary action No delivery arrives, called to find out why
Delayed-impact problem Credit card OK when placed order, failed later
Non-uptake Decide to use a different supermarket
Over-uptake Accidentally placed order twice
Technology issue Supermarket goes offline due to cyber-attack
Something else ?
https://bit.ly/5wayserrors

19
For example,
“non-uptake” and
“over-uptake” gets
complex quickly
Week 39: OK Arrrr - Frankie Roberto

20
Did you find any examples of these errors?
Add a sticky note that says whether
the errors that you noted
are in any of these categories:
• Problems along the way
• Wrong result
• Unnecessary action
• Delayed-impact problem
• Non-uptake / over-uptake
• Technology problem
• Something else?
3 minutes
https://bit.ly/5wayserrors

21
What about error rates?

22
UK government services
must publish completion
rates
Measuring completion rate - Service Manual - GOV.UK

23
Completion rate is a relatively simple calculation
Completion rate =
Number of people who complete
Number of people who start

24
Error rate might also be simple
Error rate =
Number of errors

25
Other error rate calculations are available
Error rate =
Number of attempts with an error
complete
are eligible
other?

26
Example: there are at least 3 definitions of turnout
In an election, a failure to vote is an error
Turnout =
Number of votes counted
Number of people who
1. Send in postal votes or go in person
2. Are on the electoral roll
3. Are in the voting-eligible population
Thanks to Whitney Quesenbery for this example

27
In the error rate, what do we divide by?
For the service or product you are thinking about,
what do you divide by?
If there’s more than one sensible thing to measure or divide by,
write extra sticky notes
3 minutes

28
Agenda

29
Data quality can deteriorate because of errors
National Audit Office: Improving family court services for children

30
But some problems occur purely because of delay
National Audit Office: Improving family court services for children

31
Delay adds to errors in Denmark, too
Rigsrevisionen's report on case processing in the system of family law 2024

32
Data quality can deteriorate or change all by itself
People change
• Move house
• Change jobs
• Lose their phones
• And many others
Organisations change
• Add a new product
• Delete an old one
• Are affected by energy costs
• And many others
Photo by Michal Balog on Unsplash

33
Previous
takeaway
There are many ways in which
the data we collect isn’t
entirely accurate

34
Reframed There are many ways in which
the data we use isn’t
entirely accurate

35
What is the longer-term life of our data?
Are there data quality issues that might arise over time?
Write one sticky note for each data quality issue.
Compare with your neighbour.
3 minutes

36
There is a UK Government Data Quality Framework
The Government Data Quality Framework - GOV.UK

37
I asked folks at another workshop to have a look
It looks a bit like
WCAG – I think there
are probably useful
ideas in it, but right
now I can’t see how
to use it myself
The Government Data Quality Framework - GOV.UK

38
Takeaway Data quality is important but
not easy

39
Agenda

40
Reflect on what we’ve shared - EasyRetro
Current practice Share your best practice, tip, or
suggestion for others to try
Next step Anything that you yourself plan
to find out or to do differently
Something else Any other observations or
thoughts you’d like to share
5 minutes
The EasyRetro we
used is now closed .
Contact Caroline for
feedback.

41
Takeaway AI may help to start useful
conversations about error
rates and data quality

42
I’m starting to collect some resources
General advice on measurement
Steve Messer Metrics, measures and indicators: a few things to bear in mind
Official / government guidance
ONS Quality in official statistics - Office for National Statistics
Headline stories
The Guardian Windrush victims could have compensation reconsidered
Disability NewsDWP helped cause mental distress of claimant who took her own life
Academic research
The King’s Fund Lost In The System: The Need For Better NHS Admin

43
You need to see
me in person for a
sticker
There is a poster for each sticker on
my website
Effortmark posters - Effortmark

44
Keep up the conversation: Caroline Jarrett
BlueSky @cjforms.bsky.social
caroline.jarrett@effortmark.co.uk
www.effortmark.co.uk

2025 UX Connect Aarhus - garbage in garbage out.pptx

More Related Content

Similar to 2025 UX Connect Aarhus - garbage in garbage out.pptx

More from Caroline Jarrett

Recently uploaded

2025 UX Connect Aarhus - garbage in garbage out.pptx