Notes
2 Unsung Contributors to the Human Genome Project
NIH Staff and Advisors
Mark Guyer, Kris A. Wetterstrand, and Jane Peterson
In response to ongoing debates within the biological research community in the late 1980s about a proposal to map and sequence the human genome, the Board on Basic Biology of the Commission on Life Sciences of the National Research Council (NRC) appointed a special committee “to examine the desirability and feasibility of [such a project] and to suggest options for implementing the project if it were deemed feasible” (NRC 1988). In 1988, the committee issued a report that began “A special effort . . . should be organized and funded specifically” to acquire “a map, a sequence, and an increased understanding of the human genome . . .” (NRC 1988).
The report’s Executive Summary began by noting that “Humans have long been intrigued by the forces that shape them and other organisms”; that a century of biological studies had led to the conclusion that, in the humans as well as all other animals and plants, inherited traits are controlled by genes; that genes are composed of DNA; and that “as more of our genes are mapped and their DNA sequenced, we will have an increasingly useful resource [emphasis added]—an essential data base that will facilitate research in biochemistry, physiology, and medicine.” To that end, the committee concluded that a “special effort” should begin immediately to construct detailed genetic and physical maps of the human genome and the genomes of important model organisms, and to mount a “focused effort” to develop the “required advanced DNA technologies” to analyze genomic DNA. The committee recommended that this be supported with additional funds, “not diverted from the federal research budget for biomedical sciences,” that would rapidly scale up to $200M per year. Significantly, the report noted that this “human genome project should differ from present ongoing research inasmuch as the component subprojects should” have quantitative goals and progress be assessed quantitatively, and “should have the potential to improve by 5- to 10-fold increments the scale or efficiency of mapping, sequencing, analyzing, or interpreting the biological significance of the information in the human genome.” It was predicted that the special effort could be carried out “in the next two decades.”
While there was a wide-ranging set of responses to the NRC’s recommendations, from vehement disagreement to enthusiastic support, the National Institutes of Health (NIH) and the Department of Energy (DOE) in the United States, and agencies in several other nations, accepted the recommendations and the Human Genome Project (HGP) officially began on October 1, 1990.1 Less than thirteen years later, the International Human Genome Consortium announced (International Human Genome Sequencing Consortium 2004) that a highly accurate, highly continuous, “essentially complete” human genome sequence had been finished. In the course of the last decade of the twentieth century and the first three years of the twenty-first century, the public human genome effort generated:
- a sequence of “2.85 billion nucleotides interrupted by only 341 gaps [covering] ca. 99% of the euchromatic genome [at an accuracy of] ca. 1 error in 100,000 bases (99.99%).”
- The total cost of this sequence was less than $0.09 per finished base, a reduction in cost of at least five orders of magnitude.
- Other accomplishments included a 1 centimorgan human genetic map (Gyapay et al. 1994), and maps of 3.7 million human single-nucleotide polymorphisms (International HapMap Consortium 2005) and 15,000 full-length human cDNAs (Mammalian Gene Collection [MGC] Program Team) (Mammalian Gene Collection 2002), plus the complete DNA sequences of E. coli (Blattner 1997), S. cerevisiae (baker’s yeast) (Mewes et al. 1997), C. elegans (a roundworm) (C. elegans Sequencing Consortium 1998), D. melanogaster (the fruit fly) (Adams et al. 2000), and a draft sequence of M. musculus (the laboratory mouse) (Mouse Genome Sequencing Consortium 2002).
- The project also developed many disruptive technologies, in addition to DNA sequencing methodology.
- The project’s embedded program in the Ethical, Legal, and Social Implications (ELSI) of genomic research became a major funder of bioethical studies in the United States.
The fact that the HGP was completed at all, not to mention completed at a cost less than originally estimated and in a time frame faster than originally predicted, was a stunning achievement, even to those of us who had been deeply involved from the beginning. It was an event of enormous satisfaction and tremendous pride to all the participants (as well as of great exhaustion to many), and one that was universally recognized as being a true game changer for all aspects of biological and biomedical research. So, how did the participants manage to pull it off?
The data and analyses comprising this amazing historical achievement were, of course, generated through a massive, sustained effort of scores of laboratories and hundreds, if not thousands, of individual researchers, whose approaches, failures, and successes have been well documented (Davies 2002). Less well documented are the contributions of the many scientist-administrators who guided the participating funding agencies’ efforts and, very importantly, coordinated each agency’s activities with those of the other funders. Another critical set of contributions came from the above-the-call-of-duty participation of the scientific advisors upon whose wisdom and judgment the active participants relied; this critical outside input started with the NRC report and continued throughout the entire thirteen years of the project. The purpose of this paper is to describe the activities of those participants and some of the lessons learned in the course of bringing the HGP to its successful completion.
From its outset as Elke Jordan vividly describes in her contribution to the volume, the HGP posed many challenges, the primary one being that it was an audacious effort, of a scale that had never previously been attempted in biological research. And it had to be started in the face of a significant amount of controversy in the scientific community (Weinberg 1988; Angier 1990) Some scientists, while recognizing the value of the planned resource, did not think it could be done in the planned time frame or for the proposed cost. Others thought that, even if it could be completed, the resource that would be generated would be of little scientific value and would not add significantly to the efforts of their own research activities. The critics saw the proposed HGP as a serious misdirection of the government’s funds for support of scientific research (Angier 1990).
While acutely aware of those objections and in some cases taking them into account, the participating scientists and involved NIH staff committed to achieving the long-term goals and pressed on into the challenging unknown, betting the future of their careers on the value of what they were trying to achieve. Implementation was conceived in both the short (first five years) and long (fifteen-year) terms, and was implemented according to plans that were, throughout the project, developed in series of discussions and meetings, which very consciously included skeptics among the attendees (see for example Collins et al. 1998)
Even in the face of warnings against participating in the HGP at such a vulnerable point in their careers, given the high-risk nature of the project and unsure future funding, a disciplinarily diverse set of young scientists, who had not previously worked together, was recruited to begin the work. The required scale, novelty, and rapid pace of the project presented many challenges, both conceptual and technological, to them as well as to the NIH2 staff. For the latter, one of the challenges was to develop ways to support high-risk/high-payoff projects, which meant that failures of a large number of the individual efforts needed to be anticipated.
Staff had to develop mechanisms (discussed below) that would allow rapid response both to such failures and, probably more importantly, to significant successes or changes in technologies. A particular challenge was disruptive developments, innovations that fundamentally changed the ways in which the project was being carried out. Both disruptive successes and significant failures required individuals and the overall project to redirect efforts and/or funding more rapidly than the NIH grant system usually allowed. Dealing with such issues regularly could have, over time, discouraged the scientists and staff working on the project. However, they continued to be motivated by their continuing belief in the ultimate value of the project and their conviction that it was worth trying to push to do it in the time frame and with the budget proposed. They were confident that once data production began and the data became widely available, the criticisms would disappear as it became clear to the scientific community that the HGP was a value-generating activity, and that the resource generated would transform biology.
The HGP can be thought of as (1) a large data production effort consisting of two phases, mapping (both genetic and physical) in the first and sequencing in the second and (2) associated efforts in technology development, bioinformatics, pilot sequencing projects to inform the strategies and methods for the human sequencing effort, and a program to support research about the ethical, legal, and social issues raised genomics. To deal with the complexity attendant in initiating and maintaining these various components, and providing the necessary degree of integration, the NIH relied heavily on input from outside experts. For example, in 1990, the official start of the HGP, the NIH and DOE presented their joint plan for the first five years, “Understanding Our Genetic Inheritance. The U.S. Human Genome Project: The First Five Years, FY 1991–1995” (National Center for Human Genome Research 1990). This was written by the agencies’ staff, with input from three outside committees—a Joint Mapping Working Group (six members); a Joint Informatics Task Force (fourteen members plus ten additional contributors); a Joint Working Group on Ethical, Legal, and Social Issues (six members); plus six other senior scientists. The involvement of these forty-plus outside scientists set a precedent for the three subsequent five-year plans 1993, 1998, and 2003 (see Collins et al. 2003), each of which had input from a couple of hundred workshop and meeting attendees.
The NCHGR, and later the NHGRI, also got required input from the National Advisory Council for Human Genome Research (NACHGR), whose formal approval was required by law before any grant solicitation could be issued or any funding decision could be made. The Council, which met regularly three times per year, also provided advice about other agency activities, and the overall progress of the NIH’s contribution to the Human Genome Project. The DOE got similar input from its Health and Environmental Advisory Committee (HERAC). The two agencies’ interactions were overseen by a Joint DOE-NIH Subcommittee on the Human Genome. Finally, as the program at NIH progressed, a number of individual committees of external scientists were convened to provide more specific, detailed guidance for many of the individual programs, including sequencing.
A more specific example of how the NCHGR/NHGRI used outside advice in 1998 concerns events in 1998 when an updated version of the NIH/DOE strategic plan was needed. A meeting of about 200 participants was held at Airlie, a conference center in Warrenton, Virginia, to discuss the new plan. NIH staff developed a draft based on input obtained from several individual workshops on specific issues, as the basis for the meeting’s discussions. Upon its presentation in the opening session, however, many of the attendees found serious fault with the draft, and the agenda of the meeting was revised on the fly to allow consideration of their concerns. By the end of the three-day meeting, a restructured, much-improved draft was produced, which went on to be approved by both the NACHGR and the HERAC. Over the thirteen-year lifetime of the HGP at NIH, the advice that was solicited and obtained from several hundred outside experts was critical to the advancement and ultimate success of the program.
At the NIH, the unique activities (and controversial nature) of the human genome effort demanded some innovative administrative measures. A new funding agency, the National Center for Human Genome Research (NCHGR) (which later became the National Human Genome Institute [NHGRI]),3 was created to ensure that the Congress was able to appropriate funds that were (1) specifically used for genomic research and (2) clearly not taken from other research activities at the NIH. Because of the nature of the research that would be required to carry out the HGP (e.g., research designed to meet specific production goals, high-risk/high-payoff research, hypothesis-generating research as opposed to hypothesis-testing research), new review committees had to be established, and new review criteria had to be developed. Program staff found that they needed to use existing funding approaches in innovative, non-standard ways, which often required obtaining permission from the appropriate NIH offices (see later discussion of cooperative agreements).
An early example of the way in which new thinking had to be applied to implement the HGP at NIH came with the strategic decision to support large-scale genetic and physical mapping on the basis of individual human chromosomes. Staff first proposed to use a Centers mechanism (P30 and P50 in technical terms), but several Council members did not initially sanction that approach. They were concerned about committing large amounts of funding in a small number of places. Those concerns had to do with the potential difficulty of terminating such large grants or otherwise holding investigators accountable if they failed to deliver, and were based on the perception of some of the Council members that the funding for some NIH Centers in other fields had continued long after their useful lifetime. After debate about other funding mechanisms, including contracts, the initial mapping efforts were funded as program projects (P01).
But early experience revealed problems with using the P01 mechanism for these purposes. Program projects consist of related, but quasi-independent, research projects led by different investigators, each of whom has the inherent autonomy to pursue their projects in different directions. The HGP mapping efforts, however, needed much more integration and singleness of purpose. Accordingly, the Council subsequently approved the use of P30 and P50 grants to fund Genome Science and Technology Centers (GESTECs), but only after obtaining assurance from staff that these awards would be very closely monitored, and renewals would only be funded with Council approval after a clear and convincing demonstration of a continuing need. Staff achieved the close monitoring expected by exercising greater than normal interactions with the investigators, expecting more frequent reporting, having regular telephone calls to discuss progress and conducting yearly site visits. Over the course of the first few years of the HGP at the NIH, eighteen GESTECs were supported for a wide range of genomic activities, including integrated genetic and physical maps of human chromosomes 3, 5, 6, 7, 11, 12, 13, 15, 17, 22, and X; development of three different sequencing technologies; sequencing of the nematode and fruit fly genomes, as well as S. cerevisiae chromosomes IV (half) and V; genome-wide mapping of the human and mouse genomes; and new technology for identifying disease genes.
The rapid development of human genetic maps is a good illustration of the rapidity with which progress was made in the HGP and the flexibility required on the part of NIH staff to respond to that progress. The first complete human linkage map was published in 1987 (Donis-Keller et al. 1987) before the HGP began, and comprised 403 restriction fragment length polymorphism (RFLP) markers. Initial NIH genome funding included support for efforts to improve the maps of individual chromosomes. However, RFLP mapping was relatively slow and laborious, and by 1991, questions were already being raised about whether the HGP human genetic mapping goal of a 1 centimorgan (cM) map would be achieved. In response, NCHGR created an “index mapping project” to rapidly construct relatively low-resolution (10–15 cM) maps composed of very highly polymorphic markers to provide an interim resource for the human genetics community, and to create scaffolds for building the higher resolution maps called for by the HGP. Shortly thereafter, however, the index mapping effort was overtaken by a whole-genome mapping effort made possible by the development of a new type of genetic marker, the microsatellite repeat (or short tandem repeat polymorphism, or simple sequence length polymorphism). The advantages of microsatellites over RFLPs were quickly recognized and the genetic mapping field, including NHGRI-supported investigators, largely abandoned RFLP mapping and switched to microsatellite-based mapping. In 1991, NHGRI distributed a list of 283 highly informative human genetic markers, of which 163 were microsatellites. In 1994, the French research group at Genethon published a linkage map of 2,066 microsatellites, which had been constructed on a genome-wide, rather than individual chromosome, basis (Gyapay et al. 1994). Finally, a 0.7-cM human linkage map, exceeding the HGP goal, was developed by the Genethon group in collaboration with three NHGRI-supported laboratories and one other French group. This map comprised 5,826 loci (3,617 microsatellites, 427 genes, and 1,788 other polymorphisms.) The NHGRI then pulled back funding for genetic mapping and allowed (and encouraged) its grantees to turn their attention to physical mapping (Murray et al. 1994).
The human genetic mapping effort laid the groundwork for future genome activities in terms of being collaboratively generated, international in scope, based on a new technology that supplanted existing technology, and integrating all available data into a new, useful resource available to the entire scientific community. The new genetic map effectively replaced the gene-by-gene or trait-by-trait approach that had previously been the norm in human genetics. Investigators now had a better than 99 percent chance of localizing a gene of interest near a mapped marker, within a region of interest less than 0.2 percent of the human genome. This did mean that some investigators had to abandon the methods they had used for some time and learn to take advantage of the new resource, but it clearly demonstrated the overall value of the first product of the HGP to the larger research community, and exemplified the promise of the other resources, which the HGP intended to generate.
There were important lessons for both investigators and the NIH staff in the success of the HGP’s linkage mapping phase. In place of the intense competition that had characterized the field of human genetics up to that time, the cooperation that was encouraged and that developed among the HGP’s participating scientists was productive as well as rewarding. Staff, too, had to make changes. Having come from an administrative background that emphasized the success of individual grants and grantees, we had to learn new ways of encouraging and managing what came, in time, to be recognized as “team science.” From the beginning of the HGP, the investigators and administrators had to learn on the fly to develop powerful new approaches to achieve goals that had seemed to many to be futuristic, even impossible. One of us likened the atmosphere to that of a start-up biotech company in that the effort was very much in the public eye, that every decision seemed to be absolutely critical for the future of the enterprise, and that it was in the interest of each participant for the others to be successful. And for both the investigators and the administrators, it was critical that they develop new modes of working with others to further the overall project, but to do so in a way that would still allow the investigators to be competitive at the next grant renewal. The line between cooperation and competition was a fine one that required constant attention throughout the HGP.
The remaking of the field of human linkage mapping by the invention of a type of genetic marker that could be assayed by a common method, polymerase chain reaction (PCR), and was easily transferable from the lab that developed it to all other labs was not lost on those engaged in the physical mapping efforts of the HGP. Prior to starting the HGP, physical mapping was a broad term encompassing different types of methods for ordering pieces of DNA, such as in situ hybridization mapping, deletion mapping, long-range restriction mapping, and clone overlapping. In 1989, Olson et al. (1989) introduced the Sequence-Tagged Site (STS) as a “common language” for physical mapping. Since this element was defined only by its unique DNA sequence and could be assayed by PCR, STSs could be ordered in any of several ways, and individual STSs provided a connection between physical maps constructed by different techniques (polymorphic STSs also allowed connections to be made between the physical and genetic maps of a given region). Although, once again, there was considerable skepticism about its value when the STS concept was first described, it was quickly adopted by the HGP, which then defined its physical mapping goal as STSs spaced at intervals of approximately 100 kilobases (100,000 bases) and overlapping clone sets that represented physically contiguous regions (contigs) of the genome.
The physical mapping component of the HGP was, in some ways, the most difficult and most problematic. While clone/STS maps of some human chromosomes were published in the early 1990s (see for example Green and Green 1991; Collins et al. 1992), most if not all of the physical mapping strategies had some significant problems that had to be dealt with. Overall progress in physical mapping was more difficult to track than for either genetic mapping or sequencing. The capacity of some cloning vectors, such as cosmids, was too small to be practical to use in the construction of a clone-based map of the entire human genome. Other vectors, such as yeast artificial chromosomes, that could incorporate very large pieces of chromosomal DNA (up to a few megabases), yielded libraries that were frequently unstable (the inserted DNA suffered deletions as the clones were grown) or chimeric (DNA regions from different parts of the human genome were artefactually joined). Some regions of the human genome (e.g., highly repeated regions, centromeres, telomeres) did not seem possible to clone, at least in the available vectors. Because of these and other problems, quality control became a significant issue in physical mapping. In 1994, an international group of scientists, including several NHGRI grantees, proposed a set of standards for reporting physical maps (National Center for Human Genome Research 1994). With the advent of sequencing, physical mapping became integrated into the sequencing pipeline, with the sequencers collaborating with mappers to obtain mapped clones or building the physical maps as they went along. However, whatever the deficiencies were in the effort to provide quality control for physical mapping, the need for quality control was established and then became a critical feature of the sequencing phase of the HGP.
Genomic DNA sequencing began at the NIH, as recommended in the 1988 National Academy of Sciences report, with efforts to sequence smaller genomes. The international public effort focused on the sequencing of the genomes of a small number of important model organisms. Each of these involved a different organizational and technical strategy. The E. coli sequencing effort (Blattner et al. 1997) was pursued by a single laboratory using rather standard, gel-based sequencing methods, and was funded entirely by the NHGRI. The S. cerevisiae genome was sequenced in a distributed, chromosome-by-chromosome basis by a large number of investigators separately funded primarily by European sources, with some (<15 percent) participation by the NHGRI (Engel et al. 2014). The C. elegans genome (C. elegans Sequence Consortium 1998) was sequenced by a two-laboratory collaboration that used and, in the course of their efforts, dramatically improved capillary-based sequencing; that project was jointly funded by the NHGRI and the UK Biomedical Research Council. This effort was widely supported by the C. elegans research community, which had a tradition of openness and sharing. The sequencing of the D. melanogaster genome (Adams et al. 2000), which was carried out in a single laboratory and supported by the NIH, and the DOE in contrast did not have widespread support in the fruit fly research community until the data were generated and released, when the utility of the sequence data was recognized by most researchers. The sequences of other organisms were determined by others and also made an important contribution to the dawning age of genomic sequencing. A whole genome shotgun sequencing strategy, for example, was used in determining the first sequence of a microbial genome, that of Hemophilus influenzae (Fleischmann et al. 1995)
The effort to sequence the human genome began sooner than almost everyone, especially the NHGRI staff, thought that it could, would, or should. In 1994, John Sulston and Bob Waterston, the principal investigators of the two labs (at the Sanger Center and Washington University, respectively) sequencing the C. elegans genome, approached James Watson (Ferry and Sulston 2010) and then NIH with a proposal to start large-scale sequencing of human DNA. This was at least two or three years before NIH had expected to start funding such sequencing efforts.
The NIH response to the proposal is a good illustration of how the agency staff responded to such disruptive developments. A meeting of DNA sequencing experts was rapidly organized. After intensive discussion, the attendees of that meeting agreed to support the proposal and urged the NIH to start funding human sequencing at a higher funding level. Based on advice from advisory committees, the NIH, however, did not immediately go full tilt into human sequencing and instead pursued a middle ground, probably to the dismay of those investigators who had advocated for a larger scale effort to begin. In 1995, the staff wrote and NHGRI issued a solicitation for proposals for pilot projects to sequence megabase regions of the human genome, as well as to address the difficult issues of obtaining an adequate supply of mapped clones for the sequencing, new technology, sequence quality, sequencing costs, and an ability to scale up sequencing during the three-year period of the grants.4 It had become clear to staff by this time that questions pertaining to all of these issues would need to be answered in order for the staff and advisors to consider scaling up the project in 1999 after the end of the pilot project period.
Several important administrative changes occurred during this period. At the request of staff, quantitative sequencing goals were set by investigators for each year of their awards. NHGRI staff began to actively plan how to move more funds into the large-scale sequencing program by 1999 to enable the program, if successful, to scale up dramatically at that time (described below). It then turned out that the investigators counted and reported completed sequence in many different ways. So, NHGRI had to develop a standardized, quantitative progress report, which was implemented along with a strong urging that investigators use it. A policy for data release was developed and timelines for deposition of data in the public databases were established. Yearly administrative reviews, including site visits by advisors and staff, were conducted. A management plan for the overall sequencing program was developed, and individual management plans, which had to be approved by staff, were requested from each sequencing center. Quality control exercises were started among the centers to assess the accuracy of sequence data and assemblies. And a new, activist sequencing advisory committee was assembled; this committee regularly reviewed progress reports and production figures from, and made regular site visits to, each sequencing center. NHGRI staff took the lead in all of these initiatives, as more explicitly described below, for some.
One interesting, potentially confounding issue that occurred late in the pilot project period was the realization by NHGRI staff that the donors of the DNA being used in the sequencing of cloned libraries had not been properly consented. Staff quickly examined the ethical issues of the problem, were able to get interim approval for a few of the libraries to be used by the sequencers while new, properly consented donors were recruited, and new libraries were made.
Even in this pilot period, the rate of mapping fell behind the rate of sequencing, and clone supply was a continual issue. One consequence of this was there were problems with mappers and sequencers “claiming” genomic territories to work on which others were discouraged from working on. Staff had to help address this by developing a new set of rules about “claiming” territory and ceding previously claimed territory if overall progress was being slowed. During the subsequent production phase, the sequencers changed to a whole genome shotgun sequencing strategy, which did not require mapping. This critical development is discussed extensively in Ferry and Sulston (2010).
Before that change, however, the public sector HGP received a serious challenge. By the end of the second year of the pilot projects, 22.8 MB of finished DNA sequence had been deposited in the GenBank. This was a substantial amount of new data compared to what had been available previously, but still was only about 5 percent of the total human genome sequence. In spring of 1998, representatives of a for-profit biotechnology company met with the sequencing center investigators and NHGRI staff to discuss its plans to sequence the entire human genome with a different approach than the clone-by-clone, chromosome-based, finished sequence approach being pursued by the public sector effort. The company’s plan was to perform random “shotgun” sequencing of the entire genome (all the chromosomes at once) and rely on reassembly algorithms to put the huge jigsaw puzzle of sequences back together and generate the sequence of the entire genome. Further, the company planned to sell the sequence and assembled genome data to the scientific community, bypassing public deposition. The company estimated that it would be able to offer a complete genome to the scientific community by 2001.
This announcement caught the public project off guard and stimulated many discussions and proposals as to how it should react. The first order of business was to assess the quality, contiguity and value to the community of a sequence assembled by the two strategies. A “Summer Sequencing Project” was thus conducted by the sequencing centers, coordinated by the NHGRI staff. It involved each of the sequencing centers sequencing a genomic region of its choice by both strategies and comparing the results. At the end of the project, the participating investigators agreed that a substantial increase in the rate of sequencing the human genome could be accomplished by accelerating raw sequencing capacity, but the product would only be a “high quality” draft version. Biologists who tested the draft-level sequence data found them to be relatively useful for their studies, but not as useful as the sequence that the HGP had promised (see NHGRI Internal History Archive. Scanned Box List 6, File 23 PI Meeting 9-3-98). Almost all participants concluded that having a draft quality sequence sooner would be valuable, as long as the commitment to a fully finished sequence was maintained. Accordingly, the public project published a new five-year plan in October 1998 announcing the sequencing goals for the project were being modified to accomplish a “working draft” of the genome by 2001 and a finished, high-quality genome sequence by 2003, two years earlier than the original goal. There would be no change in the public effort’s data release policy; it would continue to rapidly deposit all its data in the public domain, where it would be “totally and freely accessible.”
This modification of the HGP plan came at the same time as the NHGRI sequencing pilot projects were coming to the end of their grants. The complexities of navigating this period in the HGP can be seen in the difficulties that program staff had in writing the RFA for the full-production phase of the sequencing effort. The initial RFA soliciting applications for scale-up projects was finally published in January 1998. But after the announcement by the private company and modification of the public effort’s approach, NHGRI staff had to revise the scope of the RFA to accommodate the new strategy and produce a working draft by 2001. Then, the submission dates had to be changed twice for technical reasons, first to June 1998 and then to September of that year. The intricacies of the scale-up process culminated when staff presented the applications to the NACHGR for its second-level review. The Advisory Council spent an unusually long time, eight hours over two days, discussing with staff how to structure, fund, and implement the scale-up program; this was far longer than the Council had ever spent on such a funding plan, before or since. In the end, the NHGRI decided to fund three large-scale sequencing centers to generate the public version of the working draft.
This outcome led to several new administrative changes. First, and most importantly, a different funding mechanism, the cooperative agreement, was used. As discussed below, this change gave NHGRI staff new tools for closer management of the new centers. Second, quantitative goals were made a condition of each award, and use of the cooperative agreement mechanism gave staff more ability to adjust funding levels if those goals were not met or were vastly exceeded. Third, the centers were reviewed yearly by staff and advisors to assess progress toward those goals. Raw sequence data was to be rapidly (every twenty-four hours) deposited in the public databases and only the publicly available data was to be considered in evaluating how well a center had met its goals. Sequence data had to meet specific quality criteria. Investigators were also expected to closely collaborate and coordinate with each other. Finally, staff continued to develop ways to increase the amount of funding available to the centers so that it would be enough to meet the project’s goals. For instance, utilizing the advisory process, larger-than-normal supplements could be provided to those centers that were making outstanding, better-than-expected progress toward meeting goals.
The next few years of the HGP were most demanding for the investigators producing the data, as they had to work under an extremely short deadline and intense public scrutiny, but also for the advisors and NHGRI staff. Weekly conference calls were held with the largest sequencers and their funders. There were also regular, although less frequent, calls with the entire international consortium. Semi-annual face-to-face meetings of the investigators of the largest efforts and staff were held, as was a yearly convening of the entire international consortium. Staff and advisors conducted frequent (at least yearly) site visits to the NHGRI grantees. The sequencing progress reports were submitted biannually and reviewed by staff and the sequencing advisory committee. Staff provided status reports to the Council at each of its three yearly meetings. The director and staff were frequently called on to give progress updates across the government.
Between the funding of these cooperative agreements in 1999 and 2001 (National Human Genome Research Institute) there was much public attention to the HGP, in part because of the perceived competition (significantly media-driven) between the public and private efforts. The intensity of the public production efforts and oversight paid off in 2001 when the international public effort and the private effort simultaneously announced completion of their working drafts. The announcement was made in the United States at the White House by the president and in a coordinated companion announcement by the UK prime minister in London (NHGRI Internal History Archive. Scanned Francis Collins Files, Box 0115, file 012).
While most of the sequencing centers’ efforts during this time were devoted to completing the working draft sequence, they were also able to pay some attention to the development of new technologies to make finishing faster and more efficient. With completion of the working draft sequence, the public sector effort returned its full attention to finishing a high-quality sequence. About 20 percent of the genome had been finished at the time of the working draft announcement; the final 80 percent was finished just two years later in 2003, a full two years earlier than the original goal.
The large-scale sequencing program of the NHGRI was continued beyond 2003 with a series of three-year awards, turning its attention to the sequencing of more human genomes for purposes of discovering the diversity in the human population, of the genomes of many additional organisms, and to medically important sequencing projects. During this time, while sequence production continued to increase, continuing reductions in the cost of sequencing through ongoing technological and strategic improvements allowed the NHGRI staff to shift funds out of the sequencing program and into other programs, reversing the funding increases that had been made at the start of sequencing.
1. Lessons Learned
During the HGP, there were many lessons learned (see also Galas et al. 2017). Our focus is on the NHGRI’s perspective of those lessons. Some of these were alluded to above but will be now be detailed further. Some were relevant to all phases of the project and some only to specific phases. It is important to note that the NHGRI is embedded in a bureaucratic management system that has served the NIH well for more than seventy-five years. Staff had to learn how to adapt that system to enable the HGP, a novel, fast-moving, very large, very expensive, very focused, high-risk/high-payoff project that continually challenged the norms of the NIH funding and management practices and frequently tested the boundaries of the NIH rules. In order to do so, NHGRI staff worked closely with staff in the NIH offices where the rules were made and interpreted, to ensure the project was being managed within appropriate limits but was not unnecessarily constrained by them. The creative thinking of these administrators means that they should also be included among the unsung contributors to the success of the HGP at the NIH.
1.1. Cooperation Was Essential
It was clear from the beginning of the project that no individual lab, agency, or nation could complete the HGP by itself; the effort was just too large and demanding. Therefore, it was in the interest of each of the HGP participants for the other participants to be successful (at least most of the time; during funding competitions, collaboration tended to be less intensive). Throughout the HGP, NHGRI staff’s management activities were intended to aid and foster the essential cooperation by encouraging and facilitating active communication among grantees. Over the course of the HGP, the most successful participants were those who figured out how to align their own laboratories’ interests and goals with those of the overall HGP.
Cooperation was also highly necessary on the agency and international levels. The interactions between NIH and the DOE are discussed elsewhere in this volume. International cooperation began almost as soon as the genome project began. There was an initial attempt at multilateral interaction at the scientist level with the formation of the Human Genome Organization (HUGO). This did have a certain amount of success with the organization of a series of single chromosome workshops in the late 1980s/early 1990s. But HUGO’s influence did not last long, as the organization was not able to develop a firm financial basis and had no direct funding authority or direct influence over the funding agencies. Cooperation was much more successful on a bilateral, funding agency level. There was a great deal of communication between the NIH and several European governmental funding agencies, including the UK Medical Research Council, the Centre National de la Recherche Scientifique in France, and agencies in Germany, Japan, and the European Commission, and later Canada, as well as a number of private funding agencies, including the Wellcome Trust (United Kingdom), the Centre d’Etudes des Polymorphismes Humaine (France), and the Howard Hughes Medical Institute (United States). Strong agency cooperation continued throughout the HGP, with the NHGRI—Wellcome Trust collaboration being particularly key, as these two agencies funded the majority of large-scale human genome sequencing.
It is probably worth spending a few words here on the topic of the “leadership” of the HGP. Very simply, the HGP was a true large international and interagency federated collaboration. Each agency followed its own rules, policies, and practices, and made its own decisions. No agency’s action was dependent on any other agency’s decision. At the same time, there was strong intellectual leadership from the United States, starting with the early organizational efforts at the University of California, Santa Cruz, the DOE, and then most importantly from the committee that wrote the 1988 National Academy report (see NHGRI Internal History Archive. Scanned Box 1 of 3, File 21 Director’s Report 1987–1988; NHGRI Internal History Archive. Scanned Francis Collins Files, Box 1210, folder 023). Once the project actually got underway, staff from NHGRI and other agencies worked closely with each other and with the leading participants in the mapping and sequencing programs as a consortium to determine the direction of the overall project. Some in the public may have seen Francis Collins as the leader of the public genome research effort, particularly during the late 1990s’ so-called “competition” with the private effort of Celera, Inc., but this was really a creation of the press and other media. As the director of NHGRI, Collins actually only had authority with regard to NHGRI’s activities, but in reality, he acted as part of an informal leadership group comprised of the leaders of the major sequencing laboratories.
1.2. Flexibility Was Critical
The HGP was not “business as usual” in many ways. NIH grants in the early 1990s were typically awarded for three to five years, with an emphasis on longer grants to provide more stability for productive researchers. Most NIH institutes made five-year awards unless there was justification for a shorter award period. The HGP at NIH, however, turned this approach on its head. Because many more of the HGP grants, particularly in the area of technology development, were to support high-risk/high-payoff proposals than typical NIH grants, and because progress toward achievement of the HGP’s goals was increasingly rapid, staff decided that it would be advantageous to be able to turn over grant funds more frequently. Thus, for HGP grants, the NHGRI adopted a default of three years, unless a longer grant period was justified (e.g., in the case of new investigators).
With most investigator-initiated NIH grants, goals or expectations are established at the beginning of the grant period, and any alterations of those goals that are made (e.g., a change in direction in response to an unexpected finding that makes a different approach or objective preferable) can normally be made by the investigator on her own, with the judgment of the success of such change being made by the standard NIH review process at the time an application for renewal is considered. Also, most NIH grants support hypothesis-driven research, that is, work to test a specific scientific idea, and do not necessarily have clear, quantifiable end points. At the end of the grant period, the investigator may or may not decide to submit a renewal application, and the review process will evaluate the quality of the results in advancing biological knowledge.
One of the early criticisms of the HGP was that it was not primarily a hypothesis-testing effort. This is true; the HGP was, instead, designed to collect very large datasets, the value of which would be as a resource for hypothesis-generation activities by the entire biological and biomedical research communities. Staff’s keen awareness of this essential feature of the HGP led to the decision that most grant applications, including renewals, would not be investigator-initiated, but were to be submitted in response to specific solicitations, Program Announcements, or RFA. In responding to these solicitations, investigators had to deal with specific issues and to propose quantitative goals that specifically addressed the focused NIH–DOE five-year plans. For many of the NHGRI awards, the proposed goals were actually made a formal condition at the time of the award, which allowed staff to cite progress toward the goals as a justification for adjusting the award level during the award period (e.g., increasing funding in response to significantly exceeding goals, or reducing funding or even terminating an award in response to significant lack of progress). Grantees were expected to be prepared for, and to take into account, the rapid developments in technologies and changes in strategies (see below) on an ongoing basis. They were expected to address the continuing relevance of their initial set of goals on an annual basis and to reset those goals as necessary.
These grant-making policies (i.e., default of three-year awards, solicited grant applications rather than investigator-initiated applications, emphasis on quantitative goals whenever possible, annual review of objectives) were among the tools that the NIH HGP staff used to maintain as much flexibility as possible in the program’s research activities and to keep the program’s progress on course to meet the HGP objectives.
But, to be sure, that course was not a linear one. Although the “basic dogma” for the HGP (linkage mapping followed by physical mapping followed by sequencing) remained the basic route, there were many twists and turns along the way, from adapting better technologies on the fly to responding to others’ successes by giving up particular efforts and moving on to new ones.
1.3. Close Management of Supported Research Activities Was Crucial
For the major sequencing efforts, the NIH HGP program used a funding mechanism known as the cooperative agreement much more frequently than was typical in most NIH extramural portfolios. The cooperative agreement was commonly misunderstood as a mechanism to support cooperative activities among the HGP participating laboratories. But this is not the actual purpose of the mechanism (even though it was an important outcome of its use). Formally, the “cooperation” referred to is between an awardee and a government official. Thus, the cooperative agreement is usually used to allow collaboration between a government scientist and an awardee in a joint laboratory activity. The NHGRI staff used this mechanism differently; by awarding several cooperative agreements for the scaled-up sequencing program, the awardees (e.g., the sequencers) could be organized as a research consortium in which NIH staff were active members. The cooperative agreement allowed the NIH HGP program staff to take a much more participatory role, as scientific managers of the funded activity, than would have been allowed with grant-funded projects, such as the grants used to fund the mapping parts of the HGP.
Because of the enhanced authorities allowed to staff under the cooperative agreement funding mechanism, its use must be specifically justified, and approval was required from the NIH Director’s Office for each use of the mechanism. Typically, the NHGRI request would state that “The program requires a considerable degree of integration among the multiple . . . awardees as well as with other domestic and international groups that engage in large-scale genomic [activities]. Staff involvement is also anticipated to be needed to coordinate the on-going identification and distribution of [e.g., sequencing targets], and to ensure the proper consideration of the many ethical, legal, and social questions that will need to be addressed as genomic [analysis] capacity increase[s]. . . .”
Thus, under the cooperative agreement as used by NHGRI, the program staff’s involvement was at the consortium level, not with individual investigators, as the term “collaboration” is generally understood. Staff’s activities in the consortium involved scientific management of the coordinated effort, to ensure that all of the pieces were fitting together all of the time. The role of individual program staff was described as “Scientific Management” in publication author lists (authorship implies a substantial contribution to a publication, and NIH staff are usually not coauthors on grantee papers). Notably, because the staff participation was with the group as a whole and not with individual projects, staff were determined not to have a conflict of interest with individual investigators (as would be normal among coauthors), and this was key to enabling necessary close management.
To make clear the relationship between the participants in this unusual funding situation, the cooperative agreement award statement always included a detailed description of the roles and responsibilities of (1) the investigator (“The Principal Investigator is the person who assembles the project . . . and is responsible for the performance of the project. The Principal Investigator will coordinate project activities scientifically and administratively. The Principal Investigator will have the primary responsibility for defining the details . . . and for performing the scientific activities”), and (2) of the NIH staff member (“. . . beyond the normal stewardship role in awards, through technical assistance, advice and coordination. However, the role of NHGRI Project Scientist will be to facilitate and not to direct the activities”). The award statement also described joint responsibilities. NHGRI cooperative agreements also typically included provision for the assembly of a committee of outside experts to provide advice and guidance to the consortium. Such expert committees were put together for all large-scale data production projects and some of the technology development programs in the NIH HGP program.
Another useful authority afforded to staff under the cooperative agreement mechanism came from a provision in the award statement that allowed staff to decrease or increase funding levels during the term of the award. This allowed funds, on an ongoing basis, to be directed to the most successful laboratories, even at the expense of others who, while successful in absolute terms, were comparatively not contributing as much to the overall effort.
The NHGRI staff effected close management in other ways. It was optimal to synchronize the review and funding of the independent projects engaged in the common activity (the complete human genome sequence in this case). This allowed all funding decisions to be made at the same time and avoided the possibility that a better application would be submitted after the awards for a particular project were made. The need for this was apparent in the sequencing program. To achieve this in appropriate cases, staff was able (again with specific permission from NIH) to make awards for an initial grant period of less than twelve months, allowing the later-funded awards to be synchronized with, and have the same annual anniversary date, as awards that had been made earlier. The time frames of the awards were also changed so that each competition for funding the large sequencing centers was done with a single solicitation.
A third device that staff was able to use in managing the HGP was administrative supplements. Additional funds could be awarded during the course of an award in response to outstanding progress, an unanticipated need, or to help in synchronizing awards. All administrative supplements had to be brought to the attention of the Advisory Council, and Council’s approval was necessary to make supplements above a certain size.
1.4. Establishing Standards for Quality, Cost Accounting, and Reporting Were Vital, as Was Tracking Data Production and Costs
As the purpose of the HGP was to achieve a defined, quantitative goal—accurately determining the sequence of the three billion base pairs of the human DNA genome—it was very important to know at all times what the project had attained and where it stood with respect to the remainder of the task. It was relatively easy for a sequencing lab to report the number of bases it had generated in a given time period. The problem was that different groups counted generated bases differently. In 1998, an important statistic, the Phred score, describing the accuracy of a sequenced base, was developed by Phil Green (Ewing et al. 1998). With staff’s strong encouragement and enablement (organizing the necessary discussions and meetings), the genomic sequencing centers rapidly adopted the Phred score as a standard, allowing confidence that the accuracy of a base reported from one lab was equivalent to the accuracy of a base with the same Phred score from another lab. For purposes of tracking progress, staff decided that only bases with a Phred score equivalent to 104, which were deposited in a public sequence database, would be credited toward a laboratory’s total number of bases sequenced. It is worth noting that this decision was made in concert with the staff of the UK’s Wellcome Trust, and while each of those agencies only had authority over the laboratories that it funded, the two supported the majority of large-scale human genomic sequencing, and so this became an international standard for the entire HGP.
In an early attempt to evaluate the quality of sequence data being generated by NHGRI grantees, staff coordinated a Quality Assessment exercise in the spring of 1997. This involved the electronic exchange of sequence data files between sequencing laboratories. Two completed large-insert clones from each of the human sequencing pilot projects were identified for at random. For each clone, the data files generated by the automated sequencing instruments were sent, along with other pertinent information, to each of two other laboratories (the “checkers”). Using that electronic information, the checkers attempted to determine how reproducibly the sequence could be reassembled. The number of discrepancies between the reassembled sequence and the sequence that had been submitted to the public databases by the producer laboratories was taken to indicate the level of quality of the submitted sequence. In summary, although the results from the two checkers were not identical (either to each other or to the data as represented in the database), in most cases, it was clear that poor data (many discrepancies) could be distinguished from high-quality data (few discrepancies). In those cases, in which the data were found to be of high quality, the rate of discrepancy was low enough to meet the project’s accuracy standards. The participants agreed that this initial experiment suggested that an effective approach to evaluating the quality of DNA sequence data could be developed.
A second analysis was then done in the same way but augmented with actual resequencing in the checking labs to resolve the discrepancies between the sequence submitted by the producing laboratory and the analyses of the checkers. The results showed that most of the sequencers were attaining the quality set for finished clones. The total number of single-base discrepancies in that exercise was 120 in 1.59 Mb.
Another standard that the NIH needed to define to track the progress of the HGP was the marginal sequencing cost of per base pair. Because the NIH had a defined budget at any one time, and because budget projections through the end of the project had to be made, the true cost of data production had to be known. Again, early in the sequencing phase of the project, the costs reported by different laboratories were not comparable because different labs included different cost components in their calculations. From the NIH’s point of view, we needed to know the actual “total” cost because NIH awards are made in total dollars (direct costs plus indirect costs) and only with a true total cost per base could we project how much money would be needed to finish the project. Thus, staff proposed, and the consortium agreed, that what would be reported was the total (fully loaded) cost per base, including the costs of reagents, labor, sequencing equipment, and indirect costs. Knowledge of the true cost of sequencing at any one time was a very strong driver in the amazing reduction of sequencing costs over the course of the HGP.
As noted above, once large-scale sequencing expanded in 1999, the NHGRI’s budget for sequencing had to be expanded significantly and rapidly. NHGRI leadership was successful in obtaining an increased budget for the institute at this phase. At the same time, staff was able to stretch the funds within the existing budget. This was accomplished by slowly adjusting dates on which each project received its yearly renewal until the dates were all the same (this synchronization of start dates was also important, as noted above so all the large-scale sequencing centers could be reviewed together). This was gradually accomplished in a few years, during which each center received its committed twelve-month amount of funds over a shorter award period, increasing the funds available to spend per month. This synchronization of projects allowed the sequencing centers to increase their monthly sequence production rates during the 1999 to 2001 period.
1.5. Ancillary Activities Can Be Important
The NIH included research programs in technology development, bioinformatics, and the ELSI of genomic research as integral parts of its HGP research effort because improvements and increased knowledge in each of those areas was going to be necessary for the success of the HGP. At the time the project began, none of these areas were typically supported by NIH, except as a component of a hypothesis-driven grant. From the outset, NHGRI funds dedicated to each of these areas were tracked so the program could ensure that they were being appropriately funded (soon after the project began, ELSI had a Congressionally mandated set-aside budget). The technology development and ELSI programs are specifically addressed in accompanying articles in this volume (see contributions by Jeff Schloss as well as Joy Boyer and Jean McEwen in chapters 6 and 10, respectively).
1.6. Planning Was Fundamental
While the HGP was initiated as a long-term (fifteen-year) program, there were so many unknowns that it was not possible to even think about a fifteen-year plan except in the broadest terms (e.g., start with genetic mapping and proceed to physical mapping and then sequencing; start with sequencing model organism genomes and proceed to the human). On the other hand, year-by-year planning could not accommodate the aggressive goals that had to be set and pursued. NIH and DOE jointly decided to initiate their components of the HGP with a five-year plan (in spite of the totalitarian overtones of that term), but we were specifically cautioned by advisors that the plan should not specify the approaches or technologies that had to be used. In practice, the initial plan became outmoded within three years, and a new one had to be developed in 1993. This one remained useful for its full five years and another was written in 1998. Importantly, each plan included accommodation of new objectives as they became attractive (e.g., draft sequence, whole genome approaches to the genetic map and the genomic sequence). A 2003 plan for the HGP was not needed because the project was completed in that year. However, the NHGRI’s experience that the process of developing a planning document with wide-ranging input from the scientific community was so positive, and having such a plan to operate against was so useful, that it developed and published a five-year plan for genomics beyond the HGP in 2003 and has since continued to present five-year plans regularly. (For all key strategic plans since 1990, and for key past planning meetings, see [NHGRI Background Information 2020].)
1.7. Outside Advice Is Indispensable
As described throughout this paper, NIH solicited and received input and advice from external experts at many levels throughout the HGP. Some of this was mandated by law (Initial Review Groups and Advisory Councils), some by the NIH’s grant-making policies (justification based on specially convened meetings before issuing solicitations), and some by NHGRI practice (Independent Expert Committees enabled under cooperative agreements). The Sequencing Advisory Committee in particular played a singular role in the sequencing project by providing extraordinarily useful advice to both the NIH and the awardees, as well as providing influential backup to staff (assuming the sequencing advisors agreed with staff) when the investigators were reluctant to cooperate with staff requests. Much of the management would have been less effective without their support. Planning input was wide-ranging. Panels, workshops, and meetings included expert users of the data, expert data generators, and skeptics or critics. It is fair to say that the HGP would not have succeeded as it did without the contributions of all the experts we consulted and who volunteered their time to do so.
1.8. The Concept of the Human Genome Sequence as a Public Resource Had to Be Taken into Account at All Times
The generation of the human genome sequence was recommended in the 1988 NRC Report precisely because it would be “an increasingly useful resource—an essential data base that will facilitate research in biochemistry, physiology, and medicine.” The committee recognized that, beyond the science involved in generating a human genome sequence and the data themselves, the value of the resource would be dependent on unrestrained access to the data, and the understanding that the application of the information generated would have consequential effects on society. Thus, the HGP adopted both a policy of rapid data release and included a program of research in the ethical, legal, and social implications (ELSI) of human genome research. The topic of release of genomic data has been extensively discussed recently (see Maxson et al. 2018). The history of the ELSI research program is discussed elsewhere in this book.
2. Conclusion
The HGP was an historic achievement and a shining example of what large-scale science can achieve in biology for the benefit of humankind. It generated both an invaluable resource for biomedical research and for biological research of all kinds, and it achieved technological advances beyond almost everyone’s expectations. It had to be multinational, multidisciplinary, and multiagency in order to succeed. It depended on the contributions from thousands of scientists as well as, as we hope we have shown in this paper, from the scientific staff and scientific advisors who devoted themselves to trying to make sure the predictions and hopes of its originators were realized.
Notes
1. See US Department of Energy, Human Genome Project, https://web.ornl.gov/sci/techresources/Human_Genome/project/5yrplan/index.shtml.
2. Organization of the Human Genome Project at NIH: The Office of Human Genome Research, in the Office of the NIH Director, was created in 1988 (the OHGR had no funding authority, so the 1988 appropriation for the Human Genome Project was spent by making grants through the National Institute of General Medical Sciences). In 1989, the National Center of Human Genome Research (NCHGR) was created as a funding unit to carry out the NIH’s component of the HGP. The National Human Genome Research Institute (NHGRI) was administratively established in 1997 by the Secretary, Department of Health and Human Services. For purposes of this article, staff are referred to as “NCHGR staff” prior to 1997 and as “NHGRI staff” for post-1997 activities.
3. To avoid confusion, in this paper, actions of the NIH funding unit for genomics prior to 1997 will be referred to as NCHGR actions while post-1997 actions will be referred to as NHGRI actions.
4. Request for Applications, see https://grants.nih.gov/grants/guide/rfa-files/RFA-HG-95-005.html.
Archival Sources
- NHGRI Internal History Archive. Scanned Box List 6, File 23 PI Meeting 9-3-98.
- NHGRI Internal History Archive. Scanned Francis Collins Files, Box 0115, file 012.
- NHGRI Internal History Archive. Scanned Box 1 of 3, File 21 Director’s Report 1987–1988.
- NHGRI Internal History Archive. Scanned Francis Collins Files, Box 1210, folder 023.
References
- Adams, M. D, S. E. Celniker, R. A. Holt, et al. 2000. “The Genome Sequence of Drosophila melanogaster.” Science 287 (5461): 2185–95.
- Angier, N. 1990. “Vast, 15-Year Effort to Decipher Genes Stirs Opposition.” New York Times, June 5, 1990.
- Blattner, F. R., G. Plunkett, C. A. Bloch, et al. 1997. “The Complete Genome Sequence of Escherichia coli K-12.” Science 277 (5331): 1453–62.
- C. elegans Sequencing Consortium. 1998. “Genome Sequence of the Nematode C. elegans: A Platform for Investigating Biology.” Science 282 (5396): 2012–18.
- Collins, F. S., E. D. Green, A. E. Guttmacher, et al. 2003. “A Vision for the Future of Genomics Research.” Nature 422 (6934): 835–47.
- Collins, F. S., A. Patrinos, E. Jordan, et al. 1998. “New Goals for the U.S. Human Genome Project: 1998–2003.” Science 282 (5389): 682–89.
- Collins, J. E., L. A. Everett, D. R. Bentley, et al. 1992. “A Panel of Human Chromosome 22-Specific Sequence Tagged Sites.” Genomics 14 (4): 1098–103.
- Davies, K. 2002. Cracking the Genome: Inside the Race to Unlock Human DNA. Johns Hopkins University Press.
- Donis-Keller, H., P. Green, C. Helms, et al. 1987. “A Genetic Linkage Map of the Human Genome.” Cell 51 (2): 319–37.
- Engel, S. R., F. S. Dietrich, D. G. Fisk, et al. 2014. “The Reference Genome Sequence of Saccharomyces cerevisiae: Then and Now.” G3 (Bethesda) 4 (3): 389–98.
- Ewing, B., L. D. Hillier, M. C. Wendl, et al. 1998. “Base-Calling of Automated Sequencer Traces Using Phred. I. Accuracy Assessment.” Genome Research 8 (3): 175–85.
- Ferry, G., and J. Sulston. 2010. The Common Thread. Transworld.
- Fleischmann, R. D., M. D. Adams, O. White, et al. 1995. “Whole-Genome Random Sequencing and Assembly of Haemophilus influenzae Rd.” Science 269 (5223): 496–512.
- Galas, D. J., A. Patrinos, and C. Delisi. 2017. “Notes from a Revolution Lessons from the Human Genome Project.” Issues in Science and Technology 33 (3): 57–62.
- Green, E. D., and P. Green. 1991. “Sequence-Tagged Site (STS) Content Mapping of Human Chromosomes: Theoretical Considerations and Early Experiences.” PCR Methods and Applications 1 (2): 77–90.
- Gyapay, G., J. Morissette, A. Vignal, et al. 1994. “The 1993–94 Genethon Human Genetic Linkage Map.” Nature Genetics 7 (Suppl no. 2): 246–339.
- International HapMap Consortium. 2005. “A Haplotype Map of the Human Genome.” Nature 437 (7063): 1299–320.
- International Human Genome Sequencing Consortium. 2004. “Finishing the Euchromatic Sequence of the Human Genome.” Nature 431:931.
- Mammalian Gene Collection (MGC) Program Team. 2002. “Generation and Initial Analysis of More Than 15,000 Full-Length Human and Mouse cDNA Sequences.” Proceedings of the National Academy of Sciences 99 (26): 16899.
- Maxson, Jones, K., R. A. Ankeny, and R. Cook-Deegan. 2018. “The Bermuda Triangle: The Pragmatics, Policies, and Principles for Data Sharing in the History of the Human Genome Project.” Journal of the History of Biology 51 (4): 693–805.
- Mewes, H. W., K. Albermann, M. Bähr, et al. 1997. “Overview of the Yeast Genome.” Nature 387 (6632 Suppl): 7–65.
- Mouse Genome Sequence Consortium. 2002. “Initial Sequencing and Comparative Analysis of the Mouse Genome” Nature 420:520–62.
- Murray, J. C., K. H. Buetow, J. L. Weber, et al. 1994. “A Comprehensive Human Linkage Map with Centimorgan Density.” Science 265 (5181): 2049–54.
- National Center for Human Genome Research. 1994. “Report of the Meeting of the Working Group on Physical Mapping Standards.” Accessed November 18, 2020, https://www.genome.gov/10001402/september-1994-nachgr-meeting-summary.
- National Center for Human Genome Research, 1990. Understanding Our Genetic Inheritance. The U.S. Human Genome Project: The First Five Years, FY 1991–1995, US Department of Health and Human Services, Public Health Service, National Institutes of Health, National Center for Human Genome Research.
- National Human Genome Research Institute. “Background Information on NHGRI’s Current Strategic Planning Process.” Retrieved November 19, 2020. https://www.genome.gov/about-nhgri/strategic-plan/overview.
- National Research Council. 1988. Mapping and Sequencing the Human Genome. The National Academies Press.
- Olson, M., L. Hood, C. Cantor, et al. 1989. “A Common Language for Physical Mapping of the Human Genome.” Science 245 (4925): 1434–35.
- Weinberg, R. A. 1988. “The Human Genome Sequence: What Will It Do for Us?” BioEssays 9 (2‐3): 91–92.