Data from the U.S. Census Bureau is an indispensible source of demographic information that helps businesses, non-profits, and even the smallest mom-and-pop store plan, market, and ultimately reach their target audience. But there are limitations to the data, particularly for smaller geographies, that should be considered before making any critical demographics-related decisions.
Since participation rates for census surveys are never 100%, there will always be some error associated with demographics estimates. For example, the American Community Survey samples about 2% of American households every year, and the accuracy of the estimates can vary significantly, especially for geographic areas where populations are small and sampling error can be large. Here are some results from the latest 5-year data set from the ACS covering the years 2005-2009:
The error results for the American Community Survey are all for 90% margin of error, meaning that there is 90% confidence that the true result lies within the margin of error around the estimate. For instance, for Albany County, there is 90% confidence that the median income is $55,350 plus or minus $1,061; this means the actual 90% confidence range is $54,289 - $56,411. Even with this level of error, however, the median income estimates for Albany County, Albany city, and Census Tract 2 are significantly different from a statistical point of view, since there is no overlap when including the margin of error for each estimate.
In the case below, involving all small census tract geographies, the picture is not so clear:
Since the margin of error is so large for Census Tract 4.01, we cannot be confident that the true median income is statistically different from Census Tract 1. The estimate ranges, including the margin of error, overlap for Census Tracts 1 and 4.01, even though the estimate values ($37,619 versus $62,039) are quite different. We can say with confidence, however, that the median income for Census Tract 2 is statistically different from Census Tract 4.01 since the estimate ranges do not overlap. It certainly helps in this case that the margin of error for Census tract 2 is relatively small for this geography.
It's easy to cherry-pick data like this to show how the margin of error effects how we interpret the results, but the fact remains that survey error can be an important part of data interpretation particularly for cases where the error is large. When important decisions need to be made based on demographics, margin of error should always be part of the analysis.