Articles / Bad stats

Bad stat of the month: Trans regret

I don’t want to do this. I want to be a happy trans. I don’t like being negative, and I don’t like singling out people. I’ve held back for years. Unfortunately, people keep putting in hours, months, years of work producing these bad statistics, and then advocating questionable policies based on them. This latest one could be really damaging. It’s about trans regret, and it comes from a British grad student named Maeve Regan:

The recent Trans Mental Health Study was the largest study of its kind ever undertaken in Europe, with almost 900 respondents. The study asked specifically about individuals’ feelings of regret following social and/or medical transition. These are the results:

In terms of social changes made during transition (coming out to friends and family, changing name, living full or part time in a gender not assigned at birth), 34% of respondents had minimal regrets and 9% had significant regrets. A small majority, 53% had no regrets.

Specific regrets given included: not having the body they had wanted from birth, not transitioning earlier, losing friends and family, and the impact of transition on others.

In terms of physical changes made during transition (resulting from hormone therapy and surgical interventions), the vast majority, 86%, had no regrets. Of the remainder, 10% had minor regrets and 2% had major regrets.

The specific regrets given include complications relating to surgery (especially loss of sensitivity), and the choice of surgeon (if surgery resulted in complications or required revisions and repairs).

Regan, who identifies herself as “cisgendered (not trans),” wrote the above as part of a summary of findings of the Trans Mental Health Study, for which she was “part of the Advisory Board.” [Edited to add]: She concludes, “It is clear from the actual reported regrets, along with the conclusions that transition is associated with a broad range of positive indicators, that most cases of regret around transition are not related to being a different gender to the one assigned at birth, or undergoing transition, but rather due to poor surgical outcomes, social stigma and huge barriers to medical care.”

In other words, trans people never regret transitioning. We only regret the way we’re treated, or the way the surgery turns out.

This statistic was used in an online discussion where I brought up trans regret. [Edited]: One of Regan’s co-authors, Jay McNeil, wrote, “we found that where people did have regrets, they were largely related to their experiences of how they were treated rather than the choices they’d made around their bodies. Where people felt like de-transitioning it was mainly because they lacked support to cope wit the social stigma and losses.”

How are these percentages bad statistics? They are taken from a non-representative sample and generalized to all transgender people in the United Kingdom. Specifically, this survey was collected using the ubiquitous “snowball” method:

Participants were encouraged to take part mainly through a process of snowballing. Trans support groups, online forums and mailing lists with UK members were contacted and given information about the study and asked to share the survey as widely as possible. Other equality and health groups, and professional networks with potential links to the trans population (e.g. LGBT networks; professionals whose work might bring them into contact with trans people) were also contacted and asked to distribute information about the survey. The survey spread primarily through word-of-mouth, and the researchers attended a number of trans groups in person to discuss the project and encourage participation. The survey was open for approximately 3 months (mid-April 2012 to mid-July 2012), during which time reminders were posted online, and the survey continued to be publicised.

The problem is that “trans people” are typically defined by whether they intend to transition. Regretters by definition withdraw that intent to transition, and are thus considered by many to be “not really trans.” Some regretters, like non-transitioners, accept that classification and affirm, “I guess I wasn’t trans after all.” Others may still consider themselves trans, but not want to hang around with the happy transitioning crowd. Would you?

Regretters who don’t believe they’re trans wouldn’t take a “trans mental health survey.” Regretters who are alienated from “the trans community” might not know about such a survey, and might not want to fill one out if they did know about it. So this survey is highly unlikely to have received responses from any serious regretters. That’s not evidence that regretters don’t exist, it’s evidence of bad sampling.

These numbers could be really damaging because they give a false impression that regret doesn’t exist. That could set up any number of trans people for serious disappointment down the road. Of course, with this “no true Scotsman” (ahem) reasoning, the authors may never find out.

As I said, this snowball method is all over the place. I don’t want to single out the authors of this study for doing what practically every other trans researcher is doing. I felt bad for focusing on Emilia Dunham for publishing a bunch of unfounded generalizations back in 2011. But you need to start somewhere. And the conclusions that Regan draws are bad, bad, bad.

I get the feeling that Regan and the other authors of the Trans Mental Health study are kind, caring people, and want to help all trans people. I think if they looked into the eyes of a regretter and saw the disappointment and frustration that I’ve seen, they would care about them too, and stop this denial. I wonder if they will ever allow themselves to do it.

25 Comments

  1. Hi there,

    You make an interesting and valuable point, and one which we widely acknowledged in the TMH study at the time – that only folks who had some trans* identity or history, and who had access to the areas of support we sampled in would have filled out the survey. Indeed we were very clear that the sample was unrepresentative, not just in terms of transition or gender identity or mental health, but in terms of ethnicity and other factors too, which meant that the stats we reported could only truly be said to represent the situation of those who took part. This is, sadly, a factor in all research – hence the use of p values when reporting statistics – because no sample can ever be said to be truly representative. We were much more explicit about this, and these limitations, than in most papers. We also spoke about the issues with our sampling method, as snowballing, although common, is widely recognised as being purposive.

    Due to the sample being unrepresentative of the population in the UK as a whole, we then ensured that we reported all findings as being from that sample. For example, X% of the respondents were X, rather than X % of trans people were X. We were therefore clear and unambiguous in who the figure represented, as indeed was Maeve Regan (a co-author) in her post – referring to respondents and the sample, not trans people.

    Having said this, the findings do represent the valid feelings and situations of those who took part in the study, and should not be readily dismissed because they are not representative (indeed if we dismissed all research findings that were not representative, we’d have little research, or science, or medicine left).

    Therefore, I do understand why you think the findings are ‘bad statistics’, but this is an issue endemic in all research and one we’ve been explicit about. One could argue that using a sentence like “It’s about trans regret, and it comes from a British grad student named Maeve Regan:” followed by a large statement saying “0% of trans people regret transitioning”, as you have done in your blog post, is poor reporting as it looks like you are saying that this is what Maeve stated, which is certainly not the case.

    In relation to the point you raise about trans people being defined by transition, we were quite clear in our sampling that we wanted views from as many trans people as possible, and the people who responded were varied and many, having a range of identities and with many having no intention of transitioning in any way.

    You also mention a very brief response I made to a posting of yours online in which you state that I “made a comment to the effect that trans regret doesn’t exist”. This is not the case – I was directing you to this research because it is an interesting piece of work. As someone who was writing about regret, I thought you’d be interested to know what we’d found, which is far from being an argument against your views. I have never claimed that regret does not exist. Indeed even in the stats you have posted from Maeve Regan’s blog, we are clear that a certain percentage of people did have regrets.

    Our study was not perfect for many reasons, but it was a good starting point for this type of work in the UK. With no funding, (or even with all the funding available), it is impossible for one study to encompass all possible facets of trans* mental health and wellbeing. This was a pilot, a beginning on which others can build, who will inevitable add credence to some of our conclusions and probably refute others. It was never an in-depth look at regret as an issue, but an overview. Work with people who do de-transition is necessary and essential in informing health services, however that was not the remit we had for this piece of work. Essential work, and it sounds like you have the contacts to carry it out. If you conducted that research I for one would welcome it and be very interested in reading your findings, as it is important to have balance and to describe all aspects of something you’re researching.

    Jay McNeil
    Lead researcher, Trans Mental Health Study (2012)

  2. Thanks for your thoughtful response, Jay. I’ve edited my post to include Regan’s conclusion and the statement that you made in our discussion, which despite the nuances were clearly intended to suggest that nobody regrets a decision to transition.

    “Representative” isn’t just an empty word. It means that you’re not justified in generalizing outside the sample, but that was just what you and Regan did when you claimed that your percentages were relevant to other cases.

    I hope you will have the time to read my 2006 post on the limitations of transgender research. It is possible to conduct a representative survey. People do it all the time, and it’s even been done in transgender research once. It’s incredibly difficult with a population that is still largely closeted, but the solution is not to do unrepresentative surveys and treat them as representative.

    The solution is this:

    (a) Get the institutional support necessary to do a truly representative study.
    (b) In the meantime, do more ethnographic, qualitative work.
    (c) Don’t report unrepresentative percentages.

    The last is why I was so disappointed to find that the heart of your report is a series of unrepresentative percentages. They sideline the useful results, which are the individual quotes.

    Again, I don’t want to single you out, because I know you’re doing the same thing that you read all the time. But it has serious consequences. I hope you’ll eventually come to see the danger, and change your methods.

  3. Now let me get this straight: you think an over 900 person study is somehow not representative?
    http://www.nla.gov.au/libraries/help/guide.html#sl
    “There are occasions when the quantity of titles to be examined is so large as to discourage individual handling eg to determine the median age of a set of books, or to determine language codes. In such cases, it is easier to select a sample and work on that portion only.

    The figure provides information on sample sizes for different quantities of titles.” The figure gives sample some serious and the population. At sample sizes of about 300 the population size it represents would be over 1000000.
    Also should I assume that people like yourself who don’t transition are just crisscrossed? Because that’s the attitude you give from your statement of “The problem is that “trans people” are typically defined by whether they intend to transition.” Which silly considering I have met trans people who don’t consider transition but still consider themselves to be transgender.

    “In terms of physical changes made during transition (resulting from hormone therapy and surgical interventions), the vast majority, 86%, had no regrets. Of the remainder, 10% had minor regrets and 2% had major regrets.”
    But you claim “Regretters who are alienated from “the trans community” might not know about such a survey, and might not want to fill one out if they did know about it”
    Even when the author of the study points out otherwise and you have no evidence to prove otherwise.

  4. Why yes, Jacob, sample sizes mean nothing if the sample is unrepresentative. Look in any stats textbook if you don’t believe me.

    I said trans people are typically defined by whether they intend to transition. I myself have decided not to transition, and I’ve had my transgender status questioned for it. More importantly, I’ve read lots of claims that regretters “were never really transgender,” and often the regretters seem to echo those claims.

    The authors do not contradict my claims, and it doesn’t matter whether I have evidence. They are the ones making claims about regret, and that means the burden of proof is on them to show that they have taken a proper sample of the population.

  5. Except a sample size of 900 is representative: http://www.nla.gov.au/libraries/help/guide.html#sl
    “I myself have decided not to transition, and I’ve had my transgender status questioned for it. More importantly, I’ve read lots of claims that regretters “were never really transgender,” and often the regretters seem to echo those claims.” So you debubnk the claims here…because of what other people say? And off course you don’t transition and still consider yourself transexual…something the study does as well… I don’t get it.

  6. Let me put it this way you aren’t transitioning yet are still transgender. The study agrees and thus by it’s standards you would qualify.

  7. Sorry for responding again, I am very stream of conscious at the moment.

    “The authors do not contradict my claims, and it doesn’t matter whether I have evidence.” Except you claim that they haven’t taken a representative sample of the population because they use a definition of transsexual you seem to agree with…and don’t show any evidence that the people who feel regret aren’t transexual, which is your claim.

  8. Jacob, that page (which talks about books, not people) doesn’t say what you think it says. It assumes that the sample is representative already.

    I do not consider myself transsexual, but I do consider myself transgender. The study authors pay lip service to that, but that doesn’t matter if they fail to reach an important population of regretters.

  9. Finally snowballing is considered to be non-probability sampling, not ‘non-representative’. . This sampling technique is often used in hidden populations which are difficult for researchers to access; example populations would be drug users or sex workers. Or transsexuals in this case. So there isn’t anything wrong with using this.

  10. No, my criticism is not about their definitions, it is about their methods. They do not use a representative selection method, and that is why their conclusions are unsupported. Your asserting that snowballing is representative does not make it so.

  11. I point out that you consider snowball to be non-representative. That is wrong termage.
    Second you say
    “I do not consider myself transsexual, but I do consider myself transgender. The study authors pay lip service to that” It’s more than lip service:
    By interest in or stage of transition
    The data has been separated using the question “Do you consider ‘gender
    reassignment’ or ‘transition’ to be relevant to you? (Any part of a personal, social
    and sometimes medical or surgical, process by which you have changed the way you
    express your gender)”? The possible answers were:
    ? No, I have not undergone and do not propose to undergo any part of a process
    of gender reassignment or transition
    ? Yes, I am proposing to undergo a process (or part of a process) of gender
    reassignment or transition
    ? Yes, I am currently undergoing a process (or part of a process) of gender
    reassignment or transition
    ? Yes, I have undergone a process (or part of a process) of gender reassignment or
    transition
    ? Unsure
    ? Other
    This question was used as it was felt that it might be possible that people who had
    no desire to undergo any transition, who wished to or had started a process, who
    were uncertain, and people who had undergone their transition, may have different
    outcomes. The question specifically stated that transition could mean some sort of
    social or personal transition rather than simply a medical process, to represent the
    diversity of people’s identities and that transition is multi-faceted, meaning different
    things to different people. Although the question was worded in a manner that was
    as inclusive as possible, this does mean that it is open to different interpretations,
    which must be considered when interrogating the data using this as a filter question.”
    See they control for whether or not the transgenders in question were transitioning.

  12. No, Jacob, it is not the wrong word. Snowball sampling is not representative. That is the core of the issue. The content of the survey is irrelevant to that fact.

  13. Sorry, I forgot to mention something: “I point out that you consider snowball to be non-representative. That is wrong termage.” as the name states non probability implies that there is non known chance of selecting a given sample size into whole population under your study. The only important point to keep in mind is that ” referring to your study, the size of the pop. and the access to the sample you need to choose a reasonably sample which may help to extract all sort of knowledge you want to know or to come up with after sampling.
    900 seems like a good number to me and the results match others here: http://www.researchgate.net/publication/262734734_An_Analysis_of_All_Applications_for_Sex_Reassignment_Surgery_in_Sweden_1960-2010_Prevalence_Incidence_and_Regrets
    http://www.cakeworld.info/transsexualism/what-helps/srs
    When results of improved mental health happens over and over again, some in sample sizes of hundreds, I think it tells you something.

  14. “No, Jacob, it is not the wrong word. Snowball sampling is not representative.” It is used to find out things involving hidden populations, which fit the transgender community well.

  15. Aghhhh, why do I keep forgetting to put my thoughts down!?!?!
    Sorry, sorry last one:
    Snowball sampling is hardly likely to lead a representative sample, but there are times when it may be the best or only method available. For instance, if you are studying the homeless, you are not likely to find a list of all the homeless people in your city. However, if you identify one or two homeless individuals that are willing to participate in your study, it is likely that they know other homeless individuals in their area and can help you locate them. The same goes for underground subcultures, or any population that might want to keep their identity hidden, such as undocumented immigrants or ex-convicts.
    Replace homeless with transgender and you get this.
    Yes it is for exploritory purposes to help future research, no one denies that, least of all the authors as they point out to you. But, the data they do have is interesting.

  16. Snowball sampling can be used to find out existential facts. It is not representative, and thus inappropriate as the basis for percentages or universal statements. Do people use it for that? Yes, but against the advice of statisticians.

    Again, the size of the sample only matters if it is a representative sample. If you keep posting about the sample size I will delete your comments.

  17. And please put quotation marks around your quotes, so that I can check them if necessary.

  18. “Snowball sampling can be used to find out existential facts. It is not representative, and thus inappropriate as the basis for percentages or universal statements.”
    OK but when the data keeps repeating over and over again as seen here: http://www.cakeworld.info/transsexualism/what-helps/srs
    wouldn’t you say that there is a convergence of evidence
    Also sometimes representative samples of a minority, especially those that suffer discrimination, are really hard to get, so random sampling shouldn’t be end all be all, or else you would end up with results like this: http://wakingupnow.com/blog/regnerus-admits-he-lacks-the-data-to-critique-same-sex-parenting-so-why-is-he-doing-it

  19. There is a convergence of evidence, yes, but it could be (and I believe it is) simply that they’re all making the same mistake in their research. If you don’t believe it happens, I recommend you read this book:

    http://thebigfatsurprise.com/

    It sounds like a nutrition book, but it’s actually a cautionary tale about the dangers of groupthink and ego in science.

  20. Sorry, but I am leaving this message…to essentially inform you that I am done…I know it’s silly to say that, but I have given you the impression that I am done only to come back, so I am trying to minimize confusion…sorry 🙁

  21. “It sounds like a nutrition book, but it’s actually a cautionary tale about the dangers of groupthink and ego in science.” Interesting, but proving that it is happening here is the big thing.

  22. Jacob, please don’t say you’re done if you’re not!

    As I said before, I don’t have to prove anything. It is the responsibility of the people doing the study to show that they’ve collected the proper data. And they have not. If you read those studies, they all have a little disclaimer that says, “this data may not be representative of the overall population.” You can’t get a steak by eating lots of hamburgers.

  23. Point. However when taken with other studies it paints a clear picture, so a representative sample might not be necessary. And like in said probably not possible to get, so you probably shouldn’t consider it to be end all be all. But that’s just me.

  24. As I said, no matter how many hamburgers you make, you don’t have a steak. No matter how many unrepresentative studies you have, you still don’t have a clear picture.

    When you say “the end all be all,” you’re implying that an unrepresentative study is better than nothing. I’m saying that it’s worse than nothing, because it is probably misleading a lot of people. Sometimes we just have to say that we don’t know.

    In this case, we know that there are people who say they’re satisfied with transitioning, and people who say they regret it. We don’t know what percentage each group is of the total.

Comments are closed.