When the numbers and the stories disagree

A mixed-methods study where the quantitative signal pointed one way and the qualitative story pointed the opposite — a working example of why a single method can quietly mislead a decision.

My role

Researcher (solo)

Timeline

2023 · Purdue

Org

Graduate research

Methods

Survey analysis (GSS, n=1,820) + 20 in-depth interviews

Why this is a UX research story, not just a school project

Swap 'education and trust' for any product metric and a behavioral driver, and this is a problem teams hit constantly: a clean quantitative signal that points to a confident — and wrong — conclusion. I treated it as a test of method, not a sociology paper.

Working with 2021 General Social Survey data (n=1,820 after cleaning), the cross-tab was tidy: higher education, higher trust. The tempting move is to stop there. Instead I asked whether real experiences would corroborate the number — and designed the qual phase specifically to try to break the finding.

Why explanatory mixed-methods, in this order

Method I chose

Quant first to establish the pattern → qual interviews to explain (and stress-test) it

The survey could show whether a relationship existed but never why. I ran 20 in-depth interviews (ages 20–69), asked the same trust question as the GSS so I could compare like-for-like, then used inductive coding to find what the number hid.

Constraints I balanced: A solo researcher on a course timeline — so I prioritized depth (45-min interviews, transcribed and coded by hand) over a larger sample, and was explicit about that limit in the conclusions.

Alternatives I considered

Report the survey result alone

Why not: Fast and 'clean' — and exactly the trap. It would have shipped a confident, wrong conclusion.

Qual-only interviews

Why not: Rich, but with no baseline pattern to explain or contradict; the tension between methods is the whole insight.

A bigger survey

Why not: More precision on the same misleading signal; precision isn't the same as truth.

What the two methods, together, revealed

The qualitative data refuted the quantitative conclusion

Evidence: 17 of 20 participants reported their trust had changed over time; 14 said it declined — including all high-school graduates, even those who rated their trust as high in the survey-style question.

Education's effect ran both directions

Evidence: Of the 11 who tied trust change to education, 8 became less trusting (exposure to social diversity, identity theft, betrayal), 1 more trusting, 2 ambiguous — the opposite of the tidy 'more education → more trust' line.

The real drivers sat outside the model

Evidence: Early-life environment, personality, and lived disadvantage outweighed the degree itself — variables the survey never captured.

Why it matters for product work

A method demonstration with a transferable lesson: a single clean metric can be confidently wrong; triangulate before you decide.

Modeled disciplined, decision-grade synthesis: same instrument across methods, inductive coding, honest contradiction.

Shows I'll stress-test a flattering result rather than ship the convenient narrative.

Directly relevant to AI/product work where dashboards look decisive but mask the human 'why'.

Reflection & self-critique

What I'd change: I'd add a short follow-up survey to the interview sample to quantify the qualitative themes — closing the loop from quant→qual→quant rather than stopping at the contradiction.

What I'd keep: Designing the qual phase to actively challenge the quant finding. Setting out to break a result is the fastest way to learn whether to trust it.

Honest limit: n=20 is directional, not generalizable — and I said so. Naming the boundary is part of the rigor.