Meta-analyses and pollster.com

The race for Minnesota's US Senate seat, US data on hypothetical McCain-Obama matchup for the general presidential election, and Pennsylvania Democratic primary polling, as shown by Pollster.com

I have a new addiction.

Pollster.com is the website political junkies have been jonesing for even before we knew what it was. It clusters the results of polls that ask the same question--like, Who are you going to vote for? or, Do you think the country is on the right track? Then it puts them together into a single graph with a unifying trend line. It's imperfect--I can't satisfy myself that the trend line weights for sample size--but it's a lot better than reading the polls one by one.

The medicine parallel is in what we call meta-analyses--when we try to figure out a medical question by combining a number of studies that try to answer that question. Even if a bunch of smaller studies contradict each other, the idea is that by combining a number of studies you get the effect of having one huge study, and in this kind of data (with simple results like "worked better" vs "worked the same"), sample size is all. Thus, if you can create a meta-analysis that has the effect of creating one very large set of data, the answer those data give may be more reliable.

Like any kind of statistics, the problems get more complex as you try to get around the simplest problems. For instance, even when you weight for sample size, you can't throw less-reliable studies and more-reliable studies together and act like they're equivalent--so an ideal meta-analysis gives the data from more reliable studies more weight in the final result. But judging quality, and the appropriate weight given to difference between studies, begins to become a bit more subjective the more you try to fine-tune this problem. (What is quality? And how much weight does which measure of quality get?)

Pollster.com doesn't seem to do any weighting for sample size or other aspects of reliability, but even their relatively straightforward trend line is better than a lot of nonsense political handicapping you hear on TV politics talk shows. By giving more raw data and by combining large sets of data, these graphs and datasets allow you to begin cutting through some of the worst excesses of data-mining by stupid or biased pundits. In other words, you can be your own pundit.

I am aware, of course, that one of the ways that I manage to avoid coherent political action is by being a political junkie--an observer rather than a participant. Another thing that I need to change a little bit in the coming years.

PS:
Wikipedia on meta-analysis
Pollster.com on their trend-line method