My comment on transgender data collection

Comment on Notice of Availability of Proposed Data Collection Standards for Race, Ethnicity, Primary Language, Sex, and Disability Status Required by Section 4302 of the Affordable Care Act (Document ID HHS-OMH-2011-0013-0001)

The deadline to submit your comment is Monday, August 1, 2011!

As a transgender person and a social scientist, I am excited to hear that HHS will be collecting information relating to transgender phenomena. These activities have the potential to bring us valuable information about the prevalence of transgender feelings, thoughts, beliefs and actions in the general population, beyond an often self-selected community that identifies as transgender and participates in the existing surveys. As a social scientist I have some longstanding concerns about the collection and presentation of survey data about transgender individuals, and I hope that your work will improve the situation. Here are some recommendations that I have, for the process of deciding what data to collect and how, and for the data collection itself.

In my experience, many organizations and agencies working with transgender communities repeatedly and consistently make generalizations about transgender populations that are unsupported by any data. For example, the Transgender Law Center found 194 transpeople through unrepresentative “convenience” techniques, of whom 114 reported annual incomes of less than $15,333. A cover article in the San Francisco Bay Guardian summarized it as, “In other words, more than half of local transgender people live in poverty” – an incorrect characterization that was not disputed by the study authors. As any introductory statistics textbook will tell you, prevalence in a convenience sample tells you nothing about prevalence in the general population. No one knows if the sample was representative of “local transgender people.” Presenting it as representative is misleading to the public and can lead to inappropriate funding allocations and badly targeted health initiatives, and possibly even a backlash against transgender people.

I believe that convenience samples can be very useful, for example to show the existence of job discrimination, poverty and prostitution in our community. There is a limit to their usefulness, however, and they are consistently used beyond that limit by social service providers and community advocates. The result is to spread unreliable information, and quite probably to waste taxpayer money and charitable contributions.

Reports like this are often accompanied by a disclaimer; the Bay Guardian article said, “TLC doesn’t claim the study is strictly scientific — all respondents were identified through trans organizations or outreach workers.” Unfortunately, they almost always go on to report the data as if the disclaimer were meaningless: the next sentence reads, “But the data give a fairly good picture of how hard it is for transgender people to find and keep decent jobs, even in the city that is supposed to be most accepting of them.” The reporting of percentages invites this kind of lip service to sampling procedure. Percentages are meaningless in these situations, but they are always reported, and the effect is to dismiss the disclaimer as a formality, encouraging media reporters to do the same.

On your website I see that you anticipate that the Williams Institute and the Fenway Institute will play a strong role in helping you formulate procedures for collecting information on transgender communities. I agree that they have done a lot of good work, and I support their inclusion in any round tables that you convene. However, both institutes have a history of presenting convenience samples as representative. I strongly recommend that you balance their participation with people who are knowledgeable about the appropriate use of sampling.

I am a strong advocate of qualitative research as a means of finding out problems that exist in the world. There are several advocates from the transgender community who have done quality ethnographic and autoethnographic work. One that I know personally is Gail Kramer, who has written the books My Husband Betty and She’s Not the Man I Married under the pseudonym Helen Boyd. I urge you to include in your Roundtables at least one qualitative researcher like Helen.

To my knowledge, only one researcher has done a representative sample of any segment of the transgender community. That is Niklas Långström of the Karolinska Institutet in Sweden. I strongly recommend that Långström, or someone familiar with his survey, be part of your Roundtables. I am also willing to participate, as a transgender person interested in these issues and as a social scientist who has used representative sampling in my professional work.

A critical opportunity in transgender research

The Department of Health and Human Services has just made a big announcement: they will begin collecting data on LGBT issues, including transgender issues. The goal is to document disparities in health care, as well as plain old disparities in health, so that they can be addressed in the future. The plan is to have two roundtables on “gender identity data collection” with “key experts” this summer and fall, and then the “Data Council” will present a strategy next spring.  The department will also collect public comment in various ways, one being through a website called regulations.gov, which is currently down.

If done right, this could be a tremendous help to understanding transgender issues.  “The first step is to make sure we are asking the right questions,” HHS Secretary Kathleen Sebelius told the Washington Post. “Sound data collection takes careful planning to ensure that accurate and actionable data is being recorded.”  As I’ve written before, current research on transgender feelings and actions is severely hampered by the lack of any kind of representative sample.  Just to give you a quick sense, here are ten very basic questions that nobody knows the answer to:

  • How many transgender people are there?
  • How common are the various transgender thoughts, feelings and beliefs?
  • How common are transgender actions like cross-dressing, body modifications, and “soft mods” like shaving?
  • How common are transgender name and pronoun changes?
  • How common are part-time cross-living and full time transition?
  • How often are sexual activities part of transgender activities?
  • How common are diseases and destructive habits among transgender populations?
  • How many transgender people are in long-term relationships?
  • How often are various subgroups targeted by violence and discrimination?
  • How satisfied are transsexuals twenty, thirty or forty years post-transition?

Unfortunately, transgender research is dominated by two camps, the pathologists who make unfounded generalizations based on case studies of their own patients, and the social service providers who make unfounded generalizations based on service recipients, Internet surveys and word of mouth.  Neither of them seem to have understood the idea that while convenience samples can provide the basis for many useful existential statements, prevalence statements based on unrepresentative samples are worthless.

At this point it looks like the roundtables will be heavily influenced by social service providers who only pay lip service to the limitations of their research.  The Plan says, “While HHS is in the beginning stages of developing data collection on gender identity, many researchers (e.g., Williams Institute at the University of California Los Angeles and the Center for Population Research in LGBT Health at the Fenway Institute) have been working on such data collection for several years.”  The Williams Institute produces reports like “Bias in the Workplace” (PDF), an important summary of numerous studies investigating workplace discrimination that repeatedly acknowledges that the studies are based on convenience samples – and then goes ahead and repeats percentage results as though they meant something.

The Fenway Institute employs transgender health advocate Emilia Dunham as a Program Associate, and she also hosted a webinar on the issue.  It seems quite likely that she will be one of the experts at the roundtables.  But in an otherwise solid article for Bay Windows presenting these changes at Health and Human Services, Dunham uncritically repeated several of these unsupported percentages.

There is a very short list of Experts who I think should be on these Roundtables.  The strongest research into transgender issues has been qualitative research: listening, reading and introspection, finding existential statements but not making unsupported claims of prevalence.  I’ve said before that the best qualitative researcher in the transgender community is Helen Boyd, author of My Husband Betty.  At Helen’s recommendation I’ve also read a few things by Raven Kaldera that have been pretty good, particularly his post on female transvestites (which has somehow disappeared from his website).

There is only one person out there who has ever collected data from a representative sample of any transgender community, and that’s Niklas Långström of the Karolinska Institutet in Sweden.  He’s not focused on the transgender community, and he’s associated with the pathologist Kenneth Zucker (who is not someone we want involved), but he does know how to do a national survey, and it would be worth every penny for HHS to fly him over from Stockholm for the Roundtables.

If they can’t get Långström, then I want to be on that Roundtable.  I don’t have a degree in psychology or public health, but I did take an elementary course in statistics, and I learned what you have to do in order to make a generalization.  But what matters more than any qualification is that I care about doing this right.  If they can’t find anyone else who does, I want to be there.

Am I missing anyone? Are you doing quality quantitative research? Please let me know.

Blinding me with Neuroscience

Perusing Mark Liberman’s Language Log, I came across a post about an awesome study (PDF).  Yale psychology student Deena Skolnick Weisberg and her colleagues noticed that people seemed to like psychological explanations that contained a certain amount of neuroscience.

Weisberg and her colleagues came up with a list of interesting psychological phenomena.  For each one, they created two explanations: a “good” one corresponding to the usual explanation given, and a “bad” one, which was usually circular.  They found that their subjects (who were not particularly knowledgeable about psychology) were capable of distinguishing the bad explanations from the good ones.

The researchers changed the explanations, adding a few words of neuroscience that was consistent with the explanation.  They were very careful to ensure that it was identical to the explanation given, so that it added no new information.  The subjects who received the explanations with neuroscience were much less capable of distinguishing the bad explanations from the good ones.  Specifically, the researchers write, “the addition of such neuroscience information encouraged them to judge the explanations more favorably, particularly the bad explanations.  That is, extraneous neuroscience information makes explanations look more satisfying than they actually are, or at least more satisfying than they otherwise would be judged to be.”

The study was repeated with students taking an introductory neuroscience course; unfortunately, “a semester’s worth of instruction is not enough to dispel the effect of neuroscience information on judgments of explanations.”  It was repeated again with experts in neuroscience, who were found to be immune to this effect.

The authors conclude, “Since it is unlikely that the popularity of neuroscience findings in the public sphere will wane any time soon, we see in the current results more reasons for caution when applying neuroscientific findings to social issues.”  In other words, be skeptical and try to compensate for this effect.

Doing Transgender Research: Recognizing and Compensating for Limitations

(Thanks to the members of the My Husband Betty message boards for helpful feedback.)

Over many years of contemplating transgenderism, I’ve come up with a few principles that I’ve encountered over and over again. Principle One is “No one really knows what’s going on.” With so many closeted cross-dressers, stealth transsexuals, and people from all over the transgender spectra lying to themselves and to others, and lots of people who’d rather not hear about us, it’s very difficult to make any statements with confidence, or to believe any statements from anyone else.

Recently, there’s some good news and some bad news regarding Principle One. The good news is that there’s been an increase in funding for transgender research over the past few years. Much of this funding is in the form of locally based studies in the context of research on the transmission of AIDS. (I could write a whole article on the problems involved in that idea.) Social service organizations around the country have gotten grants to study their local transgender populations, and have gone out, found transgender people, asked them questions, and published the results. With the sometimes-generous support of the government and philanthropies, hard-working teams of investigators have collected large amounts of data, sometimes on only a small salary or even on a volunteer basis. Just do an Internet search for transgender study and you’ll find a bunch of them.

So what’s the bad news? Unfortunately, a lot of this time, effort and money have been wasted due to poor methodology. It really upsets me every time I read about one of these studies, because I want the same information that the study organizers want, and I know how caring and dedicated they are. I hate the idea that they could have taken all that money and left it in a pile for the clients of their walk-in clinic and probably done more good. Continue reading “Doing Transgender Research: Recognizing and Compensating for Limitations”