Articles / Bad stats / Good research

Doing Transgender Research: Recognizing and Compensating for Limitations

(Thanks to the members of the My Husband Betty message boards for helpful feedback.)

Over many years of contemplating transgenderism, I’ve come up with a few principles that I’ve encountered over and over again. Principle One is “No one really knows what’s going on.” With so many closeted cross-dressers, stealth transsexuals, and people from all over the transgender spectra lying to themselves and to others, and lots of people who’d rather not hear about us, it’s very difficult to make any statements with confidence, or to believe any statements from anyone else.

Recently, there’s some good news and some bad news regarding Principle One. The good news is that there’s been an increase in funding for transgender research over the past few years. Much of this funding is in the form of locally based studies in the context of research on the transmission of AIDS. (I could write a whole article on the problems involved in that idea.) Social service organizations around the country have gotten grants to study their local transgender populations, and have gone out, found transgender people, asked them questions, and published the results. With the sometimes-generous support of the government and philanthropies, hard-working teams of investigators have collected large amounts of data, sometimes on only a small salary or even on a volunteer basis. Just do an Internet search for transgender study and you’ll find a bunch of them.

So what’s the bad news? Unfortunately, a lot of this time, effort and money have been wasted due to poor methodology. It really upsets me every time I read about one of these studies, because I want the same information that the study organizers want, and I know how caring and dedicated they are. I hate the idea that they could have taken all that money and left it in a pile for the clients of their walk-in clinic and probably done more good.

What’s the problem with the methodology? Basically, it’s that they use quantitative methods – specifically percentages – when it’s not appropriate. To understand why that is, we need to differentiate between two kinds of statements: existential and distributional. Existential statements are ones that can be paraphrased as “X exists.” Some examples are, “There are at least 50 self-identified transgender people in the city of X,” “Some male-to-female transsexuals are attracted to women,” and “Barry says he has had a recurrent desire to be a woman since the age of ten.” Distributional statements can be paraphrased as “X% of category Y are Z.” Some examples are “Candidate X received Y votes for Mayor of Z, out of W votes cast in the 2005 general election,” “X percent of respondents agreed with Statement Y,” and “Most of the population of X identify as Y.”

Existential statements are relatively straightforward: if you observe the existence of something, then you’re justified in reporting it. Distributional statements are more problematic, and the problems have to do with population sampling and representativeness. If you’ve counted the whole population (as in an election or a census) then you’re free and clear. However, most social scientists don’t have the means to count an entire population. Instead, they use the principle of sampling: if a distribution exists in a representative sample of the population, then it’s very likely that the same distribution exists in the population as a whole. The key is the word representative. Random samples are often used, but there are other techniques.

There are plenty of sampling techniques that are not random, like posting a survey on a World Wide Web discussion group, standing on a street corner with a clipboard, or asking people to refer their friends to you. Non-representative samples bring in the possibility of biases. A famous example mentioned in Wikipedia is when a Literary Digest poll forecast that Alfred M. Landon would defeat Franklin Delano Roosevelt in the 1936 Presidential election. That poll was biased because the sample was taken from lists of people who owned telephones and automobiles, and those people were not representative of the voters overall. The editors of the Literary Digest were not justified in generalizing those distributional statements to the electorate as a whole, and thus failed to predict Roosevelt’s re-election.

Unfortunately, almost every transgender study I’ve seen has made distributional statements based on a non-representative sample of their target population. The usual pattern is that a social services organization will survey as many transgender people as they can find, ask them lots of short questions, count up their answers and trumpet the results as valid for all transgender people in the area, if not for all transgender people in the country or the world. Since their primary contacts are usually with people who are in the process of a gender transition, poor, homeless, unemployed, mentally ill, promiscuous or involved in prostitution, they often wind up claiming that a large percentage of transgender people are transitioning, poor, homeless, unemployed, mentally ill, involved in prostitution, suffering from a sexually transmitted disease, or some combination of the above. Every time I read those kinds of statements I wonder if the non-transitioning, middle-class, well-paid, monogamous, healthy transgender people I have regular contact with are some kind of anomaly. I don’t think we are; I just think that neither I nor the surveyors are getting the whole picture.

I feel bad calling out a specific study, but examples seem to really help when discussing this issue. In March, the San Francisco Bay Guardian published the results of a study done with the Transgender Law Center, leading with this figure:

What’s more, 59 percent of respondents reported an annual salary of less than $15,333. Only 4 percent reported making more than $61,200, which is about the median income in the Bay Area.

This runs counter to my experience interacting as a out transperson; the only jobs I’ve had where I made less than $15,333 have been part-time student jobs. Among my out transgender friends and acquaintances are some very well-paid computer technicians, and I’ve read plenty of articles about high-income transsexuals who haven’t lost their jobs. This includes my experiences in New York and Albuquerque, but I know of some transpeople in the Bay Area who make more than $61,200 (and probably didn’t participate in the survey). If you read the survey’s methodology, the sample of 194 was recruited from social service agencies and the Internet, with not even a thought of representativeness. I know that the sample that constitutes my personal acquaintances is biased, but so is their sample. Which is closer to reality? Either one? At this point we have no way of knowing.

But we want to study the transgender community. We want to do surveys, and some people are even offering grants for them. So how do we do it right, so that we get reliable results? Well, it is possible to get a representative sample – if you have enough time and money. One example of this is a large study funded by the Swedish Public Health Institute (Långström and Zucker 2005).

A total of 5,250 randomly selected 18-74 year-olds from the general population of Sweden (N = 6,200,000) were contacted by mail as part of a large interview study of sexual attitudes, behaviors, and health in Sweden. … After subjects with language problems, severe visual or hearing impairment, long-term illness, or who had emigrated had been excluded, the remaining 4,781 individuals were invited to participate. A total of 59% (n = 2,810) of these subjects chose to participate in the survey. … Comparisons revealed no social, or geographic differences between participants and non-participants. In addition, cross-validation of interview data did not find general social desirability bias or untruthfulness in answer patterns, not even for sensitive sexuality-related data (Lewin et al., 1998). However, since elder individuals, particularly females, were underrepresented among responders, we included only subjects up to age 60. This yielded a final sample of 2,450 individuals (1,279 males and 1,171 females).

This study has yielded very valuable data about the prevalence and correlations of transgender-related behaviors, even though it only contained one question relating specifically to transgender issues. Of course, that question (“Have you ever dressed in clothes pertaining to the opposite sex and become sexually aroused by this?”) is itself problematic, and the responses to it have to be analyzed with that in mind, but the sampling is sound.

What if your organization doesn’t have the budget of the Swedish Public Health Institute? You could piggyback on a similar large-scale study of sexuality, preferably one that offers enough confidentiality to reach stealth and closeted transgender people. Another possibility is to commission some kind of telephone poll, perhaps in combination with another study. Official name change records, if used properly, may yield some information about people who undergo gender transitions.

There’s no question that it’s hard to get a representative sample of transgender people, and it may take years before somebody’s willing to pay the money to do it, or until the world becomes safe enough for all the anonymous transgender people to come out of the closet and identify themselves. We may just have to forego our desire to make distributional statements for a while.

In the meantime, however, there’s plenty of interesting information to be found in the world of existential statements. These methods, known broadly as qualitative research, can give you all kinds of insights into what people think, feel and do. They just can’t tell you how many, beyond the number that you actually counted. So if you find ten HIV-positive transgender people in your city, you can say that there’s at least ten HIV-positive people in your city. You can’t give percentages or anything like that.

In some ways, dropping the quantification requirements can actually be quite freeing. It lets researchers actually get to know the participants as people, with stories, and the researchers. connection to the subject is usually an asset, if properly disclosed. It leaves it to the reader to decide how typical these stories are. There are two examples of qualitative transgender studies that I’d like to recommend. One, The Man in the Red Velvet Dress (Allen 1996), is by a closeted cross-dresser. The other, My Husband Betty (Boyd 2003), is by the wife of a trans person. In both books, there is no attempt to make claims about the prevalence of various behaviors or feelings; the focus is on individuals and their stories. (I liked My Husband Betty so much that I joined the message board that Betty set up on the book’s website, and got to know Helen and Betty and some of the other people whose stories are told in the book.) If you want short examples before going out and buying a book, much of the work of Raven Kaldera, a female-to-male transsexual, is qualitative.

Quantitative research can be very valuable even when it is possible to get a representative sample of a population. One of its most important uses is to get a basic understanding of the issues involved in order to properly formulate questions for quantitative studies. If you don’t know that there are women who get sexually aroused by wearing men’s clothes, how do you know to ask them about it when you’re doing that grand survey? Agar (1996) has suggested that instead of two contradictory, or even complementary, approaches, qualitative and quantitative methods exist on a continuum, from the very formal, very broad, multiple-choice survey conducted by a distant, disinterested researcher to the informal, in-depth, free-form discussions conducted by an ethnographer who becomes a member of the community, if he or she wasn’t one before the study. The qualitative, ethnographic investigations make it possible to have successful quantitative investigations later on.

Maybe you’re thinking that nobody’s going to want to read a bunch of stories instead of some snappy statistics. And nobody’s going to pay for you to go and hang out at Ina’s taking notes. But it’s not necessarily true. There’s no denying that percentages have a certain prestige, or that certain people involved in public health funding like to see quantitative work. That doesn’t mean there’s no funding, though. There’s a large number of social scientists who support qualitative research and are willing to recommend funding it. My Husband Betty is currently in its fifth printing, and Helen’s presentations at transgender conferences are quite popular.

So please, keep studying the transgender community. And if you’ve got the money or the influence, please do a representative sample of the transgender population; it would be incredibly useful. But if you can’t, rather than wasting time and money on more useless statistics, I hope that you’ll throw away your questionnaires and go do some in-depth interviews of interesting transgender people. Or just observe.

If you’re wondering how to do this kind of thing, read the writings of Helen Boyd, J.J. Allen and Raven Kaldera. More generally, the linguist Melissa Axelrod has recommended the books by Agar (1996), Jackson and Ives (1996), and Rubin and Rubin (2004), and I found all three to be extremely helpful. There are sample chapters on the website for Rubin and Rubin (2004). My Husband Betty message board member Switchme has also recommended the book Quick Ethnography (2001) by W. Penn Handwerker. Judging by Handwerker’s “Sampling Design” article (2005), he seems to follow the principles I’ve used in this article. Switchme also recommends Analyzing Social Settings, by Lofland et al. (2005).

Good luck, and if you do some good research, drop me a line.

References

Agar, Michael. 1996. The Professional Stranger: An Informal Introduction to Ethnography. New York: Academic Press.

Allen, J.J. 1996. The Man in the Red Velvet Dress: Inside the World of Cross-Dressing. New York: Carol Press.

Boyd, Helen. 2003. My Husband Betty: Love, Sex and Life with a Cross-Dresser. New York: Thunder’s Mouth Press.

Handwerker, W. Penn. 2001. Quick Ethnography. Alta Mira, CA: Walnut Creek.

– 2005. Sample Design. In Kempf-Leonard, Kimberley, ed. Encyclopedia of Social Measurement. Amsterdam: Elsevier.

Jackson, Bruce, and Edward D. Ives, eds. 1996. The World Observed: Reflections on the Fieldwork Process. Champaign: University of Illinois Press.

Långström, Niklas, and Kenneth J. Zucker. 2005. Transvestic Fetishism in the General Population. Journal of Sex and Marital Therapy 31: 87-95.

Lofland, John, David A. Snow, Leon Anderson and Lyn H. Lofland. 2005. Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Belmont: Wadsworth.

Rubin, Herbert J., and Irene S. Rubin. 2004. Qualitative Interviewing: The Art of Hearing Data. Thousand Oaks: SAGE Publications.