Skewing the Curves

One of the news stories that's making the rounds in the past couple of days is the result of a survey that indicates that 10% (or more) of US college graduates think that Judge Judy is a Supreme Court Justice.

The problem is, the headline is a bit misleading because the question asked in the survey is misleading.  The question lists four names and asks which one is a Supreme Court Justice.  Judge Judy is not one of the choices as such.  She is listed under her real name, Judith Sheindlin.

While I have not seen the actual results from the survey, it is entirely conceivable that 10% of the respondents did tick Judith Sheindlin as an answer to that question.  I know her real name, so I would have known that was an incorrect answer, but I may not have been able to pick the correct answer from the other choices.

This is a lesson, but not necessarily in the way the survey designers think.  It is much more telling about the way survey questions are written.  When I took statistics, there was a section on the fact that how the questions were worded can influence the answers, and thus the outcome of the survey.  It is entirely possible to phrase the question to produce desired results, rather than independent, objective answers.  The way the questions are phrased can change the results of the survey.  This is one of those cases.

Had "Judge Judy" been a choice, I am sure most people would have known that was not the right answer.  So I am not sure how much faith we can put in the results of the survey as a whole, considering that one question was worded in such a way as to influence the answers.

Much has been made of the survey here in Australia, mostly laughing about how stupid Americans are.  But I wonder, how many Australians would be able to pick out the names of any of the High Court Judges?  I know there is far more coverage of the US Supreme Court and the Supreme Court Justices here in Australia than there is of the Australian High Court and High Court Judges.

While it's fun to laugh about the results, it's more important to take away the lesson that how questions are worded is just as important as the statistical results.