Here are some remarks on this latest pearl of wisdom from the Harvard School of Public Health. I'm trying to evaluate it on the merits of the science as opposed to how I feel about the authors.
Miller, Hemenway, & Azrael (2007). State-level homicide victimization rates in the US in relation to survey measures of household firearm ownership, 2001-2003. Social Science and Medicine, 64, 656-664.
Warning: If you don't like statistics you might want to skip this one. Seriously.
The short version:
Miller et. al. used rates of household firearm ownership from the Behavioral Risk Factor Surveillance System (a large scale survey conducted by the Centers for Disease Control), aggregated across the 50 US states, and compared these with homicide rates and a number of other variables from a multitude of sources such as the US census. They report several analysis showing a relationship between rates of firearm ownership and homicide, and conclude that "...our findings suggest that the household may be an important source of firearms used to kill men, women, and children in the United States."
Issues of concern:
The article reports an extensive and complex series of analyses using a data set with 50 observations. It requires a lot of work to determine what specifically was done. My first concern is that if this relationship in fact exists in a meaningful way, it should be possible to describe it in simpler terms.
The BRFSS survey data are about as credible as survey data get, so this part of the study should not be dismissed out of hand. However, any survey estimates of household firearms ownership should be expected to involve a substantial amount of error due to likely relationships between survey nonresponse, firearms ownership, item nonresponse (willingness to reveal owning firearms), and geography. The measurement issues are not seriously discussed in the paper, and they represent a major gap in the development of credible research on the relationship between firearms ownership and crime.
There are several specific statistical issues to be concerned about.
1. Reducing individual-level data to aggregate figures as practiced here is an example of what's called the "ecological fallacy", in which inferences are made about individuals based only on information on the group level. The authors admit this, but are clearly arguing that firearms ownership represents a risk factor on the individual level.
2. Some of the models involve regression analysis with a substantial number (11) of independent variables, based on only 50 data points. This is technically possible, but is rarely practiced because the results obtained often unstable. Nearly any textbook describing multiple regression methods will confirm this, usually recommending a minimum of 10 - 20 cases per variable as a reasonable "rule of thumb". The "degrees of freedom" used for statistical significance testing is not reported.
3. There is no report of an effect size or "proportion of variance explained" associated with the models. Very few journals will report regression analysis without including this statistic (e.g., “R-squared”), because it is the most easily understood information that allows a reader to evaluate the overall usefulness of the model. Leaving it out usually implies that the model explains very little of the variation in the measure being studied, and it also tells you something about the journal's standards.
4. "To derive estimates of the association between household firearm prevalence and homicide, we used negative binomial regression models and generalized estimating equations to estimate regression parameters. Negative binomial regression is appropriate for estimating models for count data that are overdispersed (i.e. the variance is greater than the mean) (Lawless, 1987), as is the case with state-level homicide data." This all sounds very technical and impressive. However the investigators were not analyzing raw count data, but rates. I don't have time to look up whether it makes a difference, but it seems likely to me that there is a difference. Perhaps there is someone here who is familiar with this method. Additionally, the description of the rates as "overdispersed" should raise a red flag. It suggests that these data may be distributed in ways that are not conducive to effective regression analysis.
5. There is a scatterplot of homicide victimization rates by the percent of state populations living in households with firearms, which shows an apparent relationship based on regression lines. However, there is an obvious outlier in the graph, the state with the highest homicide rate (nearly 10 per 100,000). This state also has one of the higher rates of firearms ownership at just over 40%. It is very clear that this single state is likely to have a strong influence on the regression line.
A hint that the authors are aware of this is in their description of their sensitivity analysis. They found equivalent results when removing the top 5 or bottom 5 states in firearm ownership. But they did not consider whether their results might have been different if they had removed the top 5 or bottom 5 states for homicide rates. The slopes are not very strong in the first place (.1), so perhaps they would not have been found to be significant in the absence of the apparent outlier case.
6. There is very little information on base rates in the article. Almost every cited statistic reflects a relative rate or transformation that is difficult to understand. But there is a clue; firearm homicide rates by state are stated to have ranged from 16 to 5,181, (mean = 693, SD = 909). There is a lot of variability here. These rates are broken down by age and gender, so that means there are a lot of states with either zero or very low numbers in some of the categories. This information is not provided in the tables of rates.
A few other odds and ends:
The authors make a great deal about finding a relationship between rates of firearms ownership and rates of firearm homicide, but no relationship between rates of firearms ownership and rates of non-firearm homicide. We would expect to see the latter as well if the relationship reflects a response to crime rather than a cause of it. It's hard to say for sure whether this argument has merit, as I'm not fully convinced that the relationship between firearms ownership and firearm homicide has been adequately demonstrated. It's worth thinking about it though, what if the relationship is real, would that pattern mean what they say it means?
It's worth remembering that not all homicides should be considered "victimization". Some of them represent cases of legitimate self-defense. If a state has one homicide in which a gun owner fatally shoots an intruder bent on mayhem we have a perfect correlation between firearms ownership and "victimization". But who's the victim?
It's also worth remembering that not all shooting incidents result in death. It's like trying to understand highway safety by only looking at fatalities. You need to look at the total number of accidents. In this case, if the authors are correct there should be an identical pattern among non-fatal, intentional shootings (if such data exist, and they very well might). This would be more convincing than the statistical gymnastics reported in the article.
On balance, I don't see any single "fatal flaw" that obviously invalidates the results described by Miller et. al. However, there is a great deal of ambiguity in their very complex analysis. Even if this were a relationship that I was motivated to see, I don't think I would find the argument to be convincing. The sophistication of the methods used stands in contrast to the types of omissions in the report (such as the lack of any measures of effect size and the very limited information about the base numbers behind the models for age/sex subgroups).
In the conclusions, a statement is made in reference to “our cross-sectional finding that household firearm prevalence is a risk factor for homicide victimization of Americans…”, but the aggregated results do not really permit this type of individual-level interpretation.
Finally, consider the closing statement: "Our findings that household firearm ownership rates are related to firearm and overall homicide rates, but not to non-firearm homicide rates, for women, children, and men of all ages, even after controlling for several potential confounders previously identified in the literature, suggests that household firearms are a direct and an indirect source of firearms used to kill Americans both in their homes and on their streets."
Given that virtually anybody who is not living in an institutional setting can be considered to be living in a "household", the conclusion of this report can be alternatively stated as Firearms used in homicides are owned by people, who often live in households.”
Quick, someone call the MacArthur Foundation, it's time to nominate a "genius" award...
John
