In Which I Get Debunked at Fivethirtyeight and Set Off A Minor Media Firestorm

Well, well. So my post comparing HDI scores of US states to foreign countries has gone a bit viral. It got picked up by Catherine Rampell at Economix, a New York Times blog; then by Richard Florida, of all people, posting at Andrew Sullivan's blog; and then a bunch of other places. I have to say, though, most folks seem less interested in my insightful analysis that Mississippi is kinda like Albania than in the Wikipedia map I used for the post:



Well, so Andrew Gelman, posting at fivethirtyeight.com, got ahold of the map, and it seems to have irked him. Says Gelman:

Is Alaska really so developed as all that? And whassup with D.C., which, according to the table, is #4, behind only Connecticut, Massachusetts, and New Jersey? I know about gentrification and all that, but can D.C. really be #4 on any Human Development Index worth its name?

Time to look behind the numbers.

Gelman goes on to bring his statistical chops to bear on the data, which came from this table from Wikipedia, and his analysis is frankly a bit beyond my pay grade. The upshot is, he's an expert, and he's dubious about the data. So I decided to look into the matter a bit more.

The Wikipedia page that lists HDI by state claims that the data come from the American Human Development Project. So I contacted them to find out if the numbers from the Wikipedia page were reliable. Kristen Lewis, a Co-Director of the project, emphasized that the AHDP's HDI index - though calaculated based on measurements of education, health, and income, just like the UN's index - uses, as she put it, "different indicators that serve as more reliable and meaningful proxies in the U.S. context." She went on to say:

To avoid creating the impression that our index was comparable to the UNDP global HDI published every year that ranks all the world's countries, we used a different scale. Rather than 0 to 1, we used 1 - 10. In addition, we said in several places that our index was not directly comparable to the UN index.

We still wanted to make international comparisons, but we did so in our book by using more discrete indicators. So for instance, we compared our incarceration rate to those of other countries, noting that ours was higher than those of China or Russia; we compared our infant mortality rate, noting that in parts of Mississippi, the infant death rate was on par with those of Libya and Thailand, etc.

I have no idea who created that table in Wikepedia and what methodology they used to convert our scale to the UN scale. We have data tables on our website where the person could have gotten the LIEX by state; I'm not sure what he or she used for income, but if they used median personal earnings, it's not comparable to the UN scale and if they used state GDP, they would run into the problems described above; and in terms of education, he or she may well have used school enrollment, which we have, but I don't know what they would combine it with as we don't have literacy by state and, again, the educational attainment figures would not be comparable.

So there you have it. Who knows where the numbers on which the above map is based came from? It seems, to the extent that anyone drew their data from the AHDP report, their methodology must necessarily have been flawed, since the AHDP data are incompatible with the data the UN uses for their HDI, and the table used for the map above used values based on the UN's scale.

However, the AHDP does have its own maps of their data, to wit:



Note that the HDI presented here is, as Lewis points out, on a 1-10 scale, rather than the UN's 0-1 scale. Note also that this map closely mirrors the Wikipedia map; indeed, the ordering of the states is nearly identical. But the ratios of HDI between the states are not. For instance, look at the bottom 15 or so states on the AHDP list (pdf):

36. Missouri - 4.54
36. Nevada - 4.54
38. South Dakota - 4.53
38. Wyoming - 4.53
40. New Mexico - 4.49
41. Idaho - 4.37
42. Montana - 4.34
43. South Carolina - 4.27
44. Kentucky - 4.12
45. Tennessee - 4.10
46. Oklahoma - 4.02
47. Alabama - 3.98
48. Arkansas - 3.86
49. Louisiana - 3.85
50. West Virginia - 3.84
51. Mississippi - 3.58

There does seem to be something of a "long tail" - several states seem to be pretty significant outliers from the national median. But the conclusion I had drawn based on the Wikipedia table was that there was a core of eight states - the bottom eight on this list - that were relatively close to each other on the development scale but which were collectively far below not just the other states in the US, but just about anywhere else in the developed world. But based on the AHDP table, there is not such a clear break - more of a gentle downward slope as you get towards the tail end of the distribution. And while it may or may not be the case that the level of development of these states are well outside the mainstream of other developed countries, there's no way to tell based on this data alone.

For Gelman's part, he tongue-in-cheekily proposes another metric for levels of development of US states:



Gelman reflects:

Why do I have such strong feelings about this? It's probably a simple case of envy, that this little bit of index-averaging has probably received more publicity than all of my life's research put together, envy that it has received so much funding. I'm sure they all have had good intentions, but I think something went wrong, at least with this part of the project.

But maybe I'm thinking about this all wrong: these folks are clearly doing well, so maybe I should emulate them. I'll start by making maps of everything ranked by state, and we'll see how that goes.

To that I can only say: map away, sir!

So, to sum up, I have a couple points. First: it is really danged difficult to find a measure of HDI by US state that can be compared to other countries. But it's such an inherently interesting - even important - question. The US is a huge country, diverse in every way. To really understand our place in the world, we need more fine-grained data than national scale measurements provide. That's what makes the AHDP cool. And it's nice to be able to make intra-US comparisons between states. But it would be fascinating to be able to compare states, in as close to an apples-to-apples way as possible, to other countries. It was curiosity about such comparisons that led me to write the original post, and it was probably a similar sort of curiosity that led whoever drew up the map on Wikipedia to do so. But in all the vast, vast internets, I can find no such comparisons. So, social scientists, what's the hold-up? Is West Virginia more developed than Serbia? The people want to know!

Second: good lord, but things do get a life of their own on the internets, don't they? Please note that the American Human Development Project has done some really nice work. I mean, their report has a foreword by Amartya Sen, for crissakes! The greenish map above is not their responsibility, but the brownish one IS theirs. And I encourage everyone to go check it out at their site; there is a ton of interesting information there.