The confidence shortcut
When organizations assess GenAI readiness, they often start with surveys: "How confident are you using GenAI tools?" The logic seems sound — people who feel confident are probably further along.
But confidence and competence are not the same thing. And the gap between them is where operational risk hides.
What the data shows
When self-rated confidence is plotted against scenario-based competence scores, respondents split into four distinct quadrants. The most concerning is the top-left.
23.5%
Overconfident
n=36
High self-confidence, low actual capability. Highest risk — most Verification and Data Handling errors.
33.3%
Capable
n=51
Confident and competent. Lowest error rates. The benchmark group.
25.5%
Emerging
n=39
Low confidence, low capability. Needs foundational enablement.
17.6%
Underconfident
n=27
Low confidence, but reasonable capability. May underuse GenAI.
Nearly 1 in 4 respondents (23.5%) are Overconfident — they rate their GenAI skills highly but perform poorly on realistic workplace scenarios. This group will never self-identify as needing support. They believe they are already capable.
Why it matters
Overconfident users are invisible to self-report surveys and training sign-up lists. They do not raise their hand for help because they do not believe they need it. Meanwhile, they are the group most likely to produce unverified outputs, share sensitive data with AI tools, and make decisions based on unchecked AI responses.
The risk compounds with usage frequency. An overconfident daily user does not make one mistake — they make the same mistake across dozens of workflows each week. Objective diagnostics like the GenAI Capability Pulse can identify this miscalibration before it scales.
What to do about it
- Don't rely on self-report: Confidence surveys and voluntary training sign-ups will systematically miss the highest-risk group.
- Use objective diagnostics: A short scenario-based assessment that measures actual decision-making — not self-perception — is the only reliable way to surface miscalibration.
- Intervene with workflow checkpoints: For users identified as overconfident, embed verification steps and review gates where AI outputs enter decisions or customer-facing work.
If you rely on self-report to find who needs help, you will miss the riskiest quarter of your users.
These findings are drawn from the GenAI Capability Pulse — a scenario-based assessment that measures what non-technical teams actually do with GenAI, not what they think they can do. If your organization is scaling GenAI adoption, start with a baseline.
Source: AGASI GenAI Capability Pulse (N=153). Quadrants use median split on SJT competence and mean split on self-rated confidence. Percentages may not total 100% due to rounding.