Growing up, some part of me always wanted to fill in the bubbles carelessly, just once. Not on the SAT or anything else with high stakes, but maybe on one of the surveys we were administered — and there were many of them along the way — that simply didn’t seem to matter. Just race through and select C for every item, and then lay my head down on my desk and reflect on what suckers my classmates were. Alas, the Goody Two-shoes part of me always won, and I always filled out those surveys honestly. I was one of those suckers.
Dr. Collin Hitt of the University of Arkansas was curious about the kids who do fill in the bubbles randomly. For a working paper published in July (that Andrew Gelman just linked to), he looked at a segment of a big, national survey, correlating it with later educational outcomes to see what, if anything, it means when kids fill in the bubbles carelessly.
To define carelessly, he used the fact that within a given survey, different items tend to be correlated with one another. To take an oversimplified example, let’s say a survey of high-schoolers had one item that asked, “Are you above 6’6” tall?” and another that asked, “Are you taller than most of the kids in your class?” In a normal population of high-schoolers, you’d expect yes answers on these two items to be highly correlated — yes on one should almost certainly mean yes on another. So if you saw that one respondent had yes for one of those questions and no for another, and a lot of other, similarly confusing answer combinations, you could infer that they were filling in the bubbles randomly. Hitt adopted a much more sophisticated version of this basic methodology, one in which he only considered students to be filling out the bubbles carelessly if “unpredictable answers persist[ed] across many items[.]”
Here’s what he found:
Using two national longitudinal datasets, I show that careless answer patterns from adolescent respondents are negatively predictive of later educational attainment, independent of cognitive ability and other traditionally-measured noncognitive skills. I posit that careless answers, as I have quantified them, proxy as a behavioral measure of a negative noncognitive trait.
In other words: Even when you control for more important stuff like how smart a kid is, if that kid fills out a survey in a sloppy manner, then all things being equal they’re less likely to achieve a high level of education. Hitt has a simple hypothesis for this: “Perhaps students who put little careful effort into completing a survey also put little careful effort into the paperwork that impacts future success, like homework or financial aid applications.” Sure, but on the other hand, at least they are recognizing, at an early age, that life is mostly just the drudgery of filling out pointless paperwork.
