A missed opportunity: what the GAO report could have told us about DTC genetic testing

The recent United States Government Accountability Office report on direct-to-consumer (DTC) genetic tests was the star attraction of a bruising Congressional hearing into the DTC industry, and sparked widespread headlines about “bogus” results from the genetic testing industry.

The report is prefaced by the statement:

GAO did not conduct a scientific study but instead documented observations that could be made by any consumer.

While it is the GAO’s prerogative to conduct their study as they see fit, we believe that they missed a valuable opportunity to survey the DTC genetic testing industry and systematically evaluate what is and isn’t being done well. In this post, we discuss how the discoveries that the GAO reported were already largely known, and assess the opportunities that the GAO missed to provide genuine insight. What could the data they collected have told us if they had decided to add a little more scientific rigour to their investigation?  


Overview

The GAO report is based on two key sources of information. Firstly, samples from five donors were submitted anonymously and in duplicate to four different personal genomics companies, with the two samples from each individual being labelled with different ages and ethnic backgrounds (one factual and a second fictitious); undercover telephone calls were then made to the companies by these “customers” seeking additional medical advice based on genetic data. All four of these companies are high-profile companies offering genome scans; they were selected as being “frequently cited as being credible by the media and in scientific publications”.

Secondly, the GAO approached 15 companies (including the original four) from a much broader spectrum of the DTC market to investigate their marketing approaches. These companies were approached “both by phone and in person” by GAO employees posing as potential customers, and asked a series of questions about “the reliability and usefulness of test results, privacy policies regarding consumers’ genetic information, and the sale of supplements or other products”. These conversations were recorded.

In addition to the report itself, the GAO released highly edited excerpts of taped conversations as a YouTube video.

The names of all 15 of the companies investigated were obscured in the report, being referred to only by numbers, but in the Congress hearing the first four “credible” companies were revealed as (1) 23andMe, (2)deCODEme, (3) Pathway Genomics and (4) Navigenics, while the remaining 11 remain nameless. The report takes pains to note that the companies it sampled were “nonrepresentative selections” from the industry.

Risk prediction comparisons

The GAO report ordered 40 tests for 5 individuals; two for each individual from each of four “credible” companies. This is not a cheap investigation; the retail price of the tests alone would have cost in the region of $30k (£19k). They compared the risk profiles for 15 diseases, and highlighted a number of cases where the predicted disease risk differed markedly between companies, or differed from the actual medical history of the individuals.

This result is not new: discrepancies in the predictions reported by different DTC companies for the same individual have been reported since the industry’s inception, and their basis was explored in a meeting between three major personal genomics companies in July 2008. In October 2009, a paper was published in Nature by Craig Venter and colleagues that took a similar approach to the GAO (purchasing kits from multiple DTC companies and comparing the results) but included a far more thorough analysis. As well as validating the accuracy of the raw genotyping data, the authors compared disease risk from the two companies, and found that they often disagreed, largely due to differences in which markers are included.

Thus the main message of the GAO report was already well known. The report fails to acknowledge this, despite citing the Nature paper. It also fails to explore the actual basis of these discrepancies, an investigation that would have been of far greater value than simply highlighting previously reported findings.

The larger and more complete dataset collected by the GAO could have been used to performed a number of useful assessments of the industry; they could, for instance, have used their replicate samples to get an independent assessment of genotyping error (both random and systematic) for each company. They also could generated quantitative measures of risk agreement, as the Pathway Genomics response did, showing that the companies agree around 77% of the time, and give opposite predictions around 8% of the time.

Most importantly, the investigation could have moved beyond a simple comparison of numbers to explore the actual basis for the discrepancies: to what extent are these due to differences in the markers used, the risk prediction algorithms employed, or the epidemiological literature used to define population risk estimates? The report makes much of its extensive consultation with experts in the field (led by Dr James Evans at the University of North Carolina); such an analysis would (hopefully!) not have been beyond their capabilities. If the report is to be used as the basis for increased regulation of the industry – and that was the clear subtext throughout the Congressional hearing – it would be ideal if this was based on a detailed understanding of where companies need to improve their analysis, rather than a simplistic pointing out of differences.

Advertising evaluation

The purview of the GAO investigation was all companies selling DTC genetic tests (not just those providing disease risk prediction), and for their assessment of privacy policies, test reliability, and product sales they surveyed a total of 15 companies.  Unfortunately, they did not separate their results by the type of product each company was marketing.  Thus companies providing information on ancestry and risk prediction based upon scientific research were lumped together with the much more dubious companies hawking “personalised” dietary supplements or promising to predict children’s athletic prowess, which have little or no scientific basis.

The GAO report does not explicitly state the criteria against which advertising claims were evaluated, and selectively reports negative assessments. It’s not clear from the report whether all companies were asked the same set of questions (e.g. whether they would accept a DNA sample submitted on someone else’s behalf, or whether their tests are “diagnostic”).  Nor is it clear whether (or how) the GAO validated all of the advertising claims each company makes about its product.

Again, a more thorough investigation could have provided far more useful outcomes. The GAO could have generated a census of the marketing claims being made by these companies, both justified and unjustified, and systematically recorded the accuracy of responses made by customer service representatives. This could have provided an initial framework for appraising the claims of DTC genetics companies.  Instead, we have only a series of cherry-picked statements selected by the GAO as misleading or deceptive, and no way of knowing how widespread such statements were within the industry as a whole.

Conclusions

In terms of scope of data collection, the GAO’s investigation was admirable: 4 major products, 5 individuals, with replicates, recorded follow-up inteviews, investigation of the advertising, a head-to-toe assessment of the industry. But frustratingly, the only conclusions drawn were simple facts that were already well-known:

  • companies give differing risk predictions;
  • predictions are not diagnoses;
  • some snake-oil companies sell dodgy “genetics” products.

The one new result, the two observed instances of major, otherwise reputable companies making clearly problematic statements in phone interviews, is rendered difficult to interpret due to the lack of full transcripts and information on questions asked. It is disappointing that the GAO decided to forsake a constructive investigation for an antagonistic attack, given that, with a little extra work and greater transparency, something far more useful could have been produced.

However, the data still, presumably, exists. Third parties can investigate the risk prediction for the 15 diseases the GAO looked at (as Pathway Genomics did); however, more useful analyses would require the GAO publicly release the product descriptions for each company, full telephone transcripts, information on all the websites surveyed and the criteria used to check the validity of claims. This information could then be used as a resource by the community as a whole to make fine detailed assessments of each aspect of each company and product, rather than broad brush denouncements of the entire industry.

The GAO logo is taken from Wikimedia Common. It is for illustration purposes only, and should not be taken to imply that this content has the endorsement, support or consent of the GAO.

  • Digg
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Twitter
  • Google Bookmarks
  • FriendFeed
  • Reddit

5 Responses to “A missed opportunity: what the GAO report could have told us about DTC genetic testing”


  • I think as scientists, there’s a tendency to look at genetic testing results as fairly innocuous and empowering. We take for granted the jargon associated with genetics and risk. But it’s important to remember that most consumers of this information do not come from a pool of scientists (and that the average reading level is 6th-8th grade). So I think it’s very appropriate that the GAO did not view their assessment as a scientific research endeavor. You make a good point about the lack of transparency of the study, however.

    IMHO, the issue is not whether the public should be able to access their own genetic information, it’s whether the informed consent consumers provide by signing a piece of paper when they send a swab is adequate. Physicians treat signatures on a form as the last step of an informed consent process. The first step is a conversation. They verify that patients really understand the benefits, risks, and limitations of testing. They verify that patients understand how genetic information can affect their psychological well-being and their relationships with family members, and that they understand that genetic counseling is an available option when results come back. This is what I see as lacking in DTC gene tests available “over the counter.” Obviously there are good players with good intentions out there but I submit that good intentions aren’t really enough. Companies may claim that all the information consumers need is in a piece of paper they sign and for the good players, likely that’s true, but it’s how consumers receive this information that’s most important. It would be interesting to survey patients who order DTC genetic tests to gauge their actual understanding of what their tests can provide, what the test results mean, and what services are available for them post-testing. That would have been an additional worthy pursuit by the GAO.

  • Will FitzHugh

    Thanks for the post. One comment, however. You say, “the main message of the GAO report was already well known” (that being that disease risk results from differ between DTC genetics vendors). I agree that it is known but not that is it widely, or well, known. These are direct to _consumer_ tests and I doubt that most of the general population was aware of this issue. Clearly the well-informed scientific community, including you, was.

    My opinion is that the biggest problem these companies face is that the complexity of their product is not well-tailored to their potential customers, the general public. And the U.S. Congress and the GAO are being surrogates for that public, in that they are not totally understanding the results and the issues. If they don’t understand these things, then I suppose that’s partly their fault, but it has to be partly the vendors’ fault, too.

  • @Will – at risk of answering for the authors. What you say is correct but my take is that the GAO could have explained or highlighted this without wasting time and money creating an “undercover sting”. There was already plenty of published work for them to refer to – but it would have been less dramatic.

    The GAO missed an opportunity, no doubt about that, unfortunately, as in 2006, I don’t think that they wanted to take it in the first place – they did exactly what they set out to do. The difference this time round is that most of the on-line reporting has been objective and critical and not even all of the main stream press has just rolled over and uncritically regurgitated the GAO press release.

    If the GAO were to make all of their material available for scrutiny that would be a good service to the cause of understanding more about the state of the market

  • The linked Pathway Genomics response (PDF) is well worth a look. It provides context that I had missed, from reading the GAO report and press reports of the Congressional hearings.

    Of note in the “surprise my fiance” incident, Pathway claims that their customer-service agent thought that the undercover GAO agent was planning to present the *kit* as the surprise, rather than surreptitiously having his saliva analyzed. They say that they added training as a result of that call, implying that they initiated the training process *before* the GAO went public. If borne out, that would be strong evidence of good faith on their part. As noted in the post, without the release of the entire phone transcript, there’s no way to know.

  • I agree with AMac, Pathway did a very good job. They also point point out that it’s very hard to secretly get 2 ml of saliva from another person, which backs up what they say.

Comments are currently closed.

Page optimized by WP Minify WordPress Plugin