Articles / Bad stats / Good research

My comment on transgender data collection

Comment on Notice of Availability of Proposed Data Collection Standards for Race, Ethnicity, Primary Language, Sex, and Disability Status Required by Section 4302 of the Affordable Care Act (Document ID HHS-OMH-2011-0013-0001)

The deadline to submit your comment is Monday, August 1, 2011!

As a transgender person and a social scientist, I am excited to hear that HHS will be collecting information relating to transgender phenomena. These activities have the potential to bring us valuable information about the prevalence of transgender feelings, thoughts, beliefs and actions in the general population, beyond an often self-selected community that identifies as transgender and participates in the existing surveys. As a social scientist I have some longstanding concerns about the collection and presentation of survey data about transgender individuals, and I hope that your work will improve the situation. Here are some recommendations that I have, for the process of deciding what data to collect and how, and for the data collection itself.

In my experience, many organizations and agencies working with transgender communities repeatedly and consistently make generalizations about transgender populations that are unsupported by any data. For example, the Transgender Law Center found 194 transpeople through unrepresentative “convenience” techniques, of whom 114 reported annual incomes of less than $15,333. A cover article in the San Francisco Bay Guardian summarized it as, “In other words, more than half of local transgender people live in poverty” – an incorrect characterization that was not disputed by the study authors. As any introductory statistics textbook will tell you, prevalence in a convenience sample tells you nothing about prevalence in the general population. No one knows if the sample was representative of “local transgender people.” Presenting it as representative is misleading to the public and can lead to inappropriate funding allocations and badly targeted health initiatives, and possibly even a backlash against transgender people.

I believe that convenience samples can be very useful, for example to show the existence of job discrimination, poverty and prostitution in our community. There is a limit to their usefulness, however, and they are consistently used beyond that limit by social service providers and community advocates. The result is to spread unreliable information, and quite probably to waste taxpayer money and charitable contributions.

Reports like this are often accompanied by a disclaimer; the Bay Guardian article said, “TLC doesn’t claim the study is strictly scientific — all respondents were identified through trans organizations or outreach workers.” Unfortunately, they almost always go on to report the data as if the disclaimer were meaningless: the next sentence reads, “But the data give a fairly good picture of how hard it is for transgender people to find and keep decent jobs, even in the city that is supposed to be most accepting of them.” The reporting of percentages invites this kind of lip service to sampling procedure. Percentages are meaningless in these situations, but they are always reported, and the effect is to dismiss the disclaimer as a formality, encouraging media reporters to do the same.

On your website I see that you anticipate that the Williams Institute and the Fenway Institute will play a strong role in helping you formulate procedures for collecting information on transgender communities. I agree that they have done a lot of good work, and I support their inclusion in any round tables that you convene. However, both institutes have a history of presenting convenience samples as representative. I strongly recommend that you balance their participation with people who are knowledgeable about the appropriate use of sampling.

I am a strong advocate of qualitative research as a means of finding out problems that exist in the world. There are several advocates from the transgender community who have done quality ethnographic and autoethnographic work. One that I know personally is Gail Kramer, who has written the books My Husband Betty and She’s Not the Man I Married under the pseudonym Helen Boyd. I urge you to include in your Roundtables at least one qualitative researcher like Helen.

To my knowledge, only one researcher has done a representative sample of any segment of the transgender community. That is Niklas Långström of the Karolinska Institutet in Sweden. I strongly recommend that Långström, or someone familiar with his survey, be part of your Roundtables. I am also willing to participate, as a transgender person interested in these issues and as a social scientist who has used representative sampling in my professional work.