Simpson's Paradox

The dashboard says the new method works. Pass rate up, green arrow, already pasted into the deck. Then someone splits the same students by class, and the arrow flips in both rooms. Morning class, the new method did worse. Afternoon class, worse again. Both rooms down, the building up. Simpson's paradox is that quarrel between the building and the rooms. The building number is a blend. Give one method the easy room and the other the hard room, and the blend can crown the method that lost every fair comparison.

A grade book small enough to check on a napkin: easy class, the new method passes 81 of 100 and the old method passes 18 of 20. Hard class, the new method passes 2 of 10 and the old method passes 30 of 100. Inside each class the old way wins, 90 percent against 81, and 30 percent against 20. Add the classes and it reverses. The new method passes 83 of 110, about three quarters. The old method passes 48 of 120, which is 40 percent. Nothing mystical happened. The new method got almost all of the easy work.

The total is not a liar. It answers the question "what happened to this whole intake," and that question is sometimes the one you meant. It is a bad stand-in for "what happens to a student in a class like mine." Hospitals, hiring channels, and citywide rates pull the same trick when the piles are not the same kind of case. You can feel the single number settle the argument while the mix that produced it never gets a sentence.

• "Overall we're improving. Inside every team we're not. Both can be true."
• "The hospital with the higher death rate may be the one taking the sicker patients."
• "The average salary rose because the senior people stayed and the juniors left."
• "The ad won on the total and lost in every city we actually sell in."
• "Women's success rate is higher in each department and lower for the campus. Look at where they applied."
• "The trend reversed when I colored the dots by group. I had been reading the uncolored cloud."
• "We declared a winner from a table that mixed easy cases with hard ones."

The tutoring total: a new program posts a better pass rate. Split by course, the old program wins in each course. The new program was assigned to the introductory course, where most students pass anyway. The company renews the contract on the basis of the mix, and calls it evidence.

The clinic: Clinic A loses more patients than Clinic B. Clinic A also takes the emergencies Clinic B refers away. A family choosing on the headline rate chooses the place that looks safe because it sees easier cases. The rate answered a different question than "where should a sick person go."

The hiring funnel: one source has a better offer rate overall. Inside each role, the other source does. The first source mostly sent candidates for the role that was easy to fill. The total praised the recruiting channel for a choice the channel did not make.

A contract gets renewed because the green arrow was easy to paste, and the napkin never makes the deck. People get blamed, or cleared, by a building-wide rate their own room does not match. Once a simple winner has been said out loud, the person with the split table sounds like they are complicating a settled thing. The simple sentence keeps the audience. It is simple because it threw away the mix.

A single number is easier to carry than a table. Dashboards, headlines, and "the data shows" all prefer the single number. The mix hides because it is not a column anyone plotted. Cases are not randomly assigned to groups in ordinary life: harder patients find some hospitals, harder courses attract some students, harder roles attract some applicants. Comparing totals without asking "who is in each pile" treats those routing decisions as if they were part of the treatment. The brain also likes a cause it can point at. "The method works" is a cause. "The weights changed" feels like an accountant's objection.

Edward Simpson described the reversal in 1951, in a paper on interpreting interaction in contingency tables. The surprise was mathematical: associations in subgroups can vanish or flip when the subgroups are combined, if a third factor is tied to both the group and the outcome. The famous public example came later. In 1975 Peter Bickel, Eugene Hammel, and William O'Connell examined graduate admissions at Berkeley. The campus-level rate looked biased against women. Department by department, the picture was different, because women and men were applying in different proportions to departments with very different admission rates. The aggregate had folded the application mix into the admission result.

The rooms are not automatically the truth and the building automatically the lie. If you are funding a whole campus, the combined rate may be exactly the fact you need, and the split may be a distraction. What goes wrong is announcing a cause from one level of the table because that level arrived first. A method for the next student is a within-class question. A budget for the intake is a whole-pool question. Simpson's paradox is those two sentences wearing each other's number.

You are about to ship, hire, or blame from one rate, and you cannot say who was over-represented in the winning pile. Annoyance at the person who "complicated" it is a clue. So is a success story that is mostly a change in who showed up.

Write the question in the subject line of the chart: "for a student in this class" or "for the whole intake." If you only have the building number, say you only have the building number. The colleague with the split table is not stalling the meeting. I have been that colleague and I have also been the person who sighed. The sigh was wrong.

Which number is the real one?

Both totals can be arithmetically correct. The real number is the one that matches the decision. If you are choosing a treatment for the next patient, a rate that mixes easy and hard cases is the wrong instrument. If you are describing what happened to an entire incoming class, the combined rate may be exactly the fact you need. State the unit, person or pool, before you state the winner.

Is this the same as a base rate error?

They meet, and they are not the same move. Base rate neglect ignores how common the groups are when you judge a new case. Simpson's paradox is a reversal between a combined table and the tables inside it. You can respect base rates and still be fooled by an aggregate that hides a changing mix.

Ask what was combined before you let the average pick a winner.

The dashboard

The clinic choice

The accusation

Practice this pattern in the Reframing App - capture the trigger, label it (like Simpson's Paradox), check evidence, and write a more balanced thought.