3. The Web Gets a Memory

While the political foundations of surveillance advertising were being secured, the companies of the marketing complex still needed to hash out how to make advertising work online. This involved not only building technical capacities to deliver ads but also establishing a set of practices and norms for conducting business. The data collection elements of targeted advertising often receive the most attention, but it was the intersection of data collection and networked ad distribution that enabled internet advertising to make its first generational leap. This operational structure was prototyped in the mid-1990s by companies that created so-called ad networks to distribute banner ads to disparate sites across the web, which provided unique opportunities to gather and combine user data. Many of the large-scale surveillance advertising operations conducted today by companies like Google and Facebook descend from this basic ad network model. Indeed, the most important ad network of the 1990s, DoubleClick, was acquired by Google in 2007 and now makes up a major division of its advertising business.

In this chapter, I trace the development of ad networks in the context of web advertising’s first three years, from 1994 to 1997. The story of ad networks, and DoubleClick in particular, demonstrates how scattered commercial practices and flexible technologies were brought together to form a particular business model that rendered data collection and ad distribution one in the same. By systematically coupling banner ad delivery with data gathering, and by working to ensure that such practices were enabled as default web browser settings, ad networks set the stage for surveillance advertising’s subsequent growth. The early success of the ad network business model, premised on the integration of tracking technologies like the HTTP cookie, catalyzed an important shift in the web’s configuration of user privacy. As companies adapted the web’s open protocols to make them more suitable for advertising, the interactive media landscape, originally designed to keep users anonymous, became increasingly organized around tracking and identification.

By the mid-1990s, it was increasingly clear that the personal computer, not the television, would be the gateway to the interactive media future.1 The cable industry’s vision for interactive television had failed to materialize, but the growth of commercial online services like AOL catapulted the internet to the forefront of new media. The World Wide Web emerged as the predominant internet application due in no small part to Netscape’s “killer app,” the graphical web browser.2 Although AOL and the like were originally walled-garden services, a burgeoning market of dial-up internet service providers offered direct access to the web. Facing this competition, commercial online services were compelled to incorporate web browser functionality, thereby opening up their walled gardens and giving millions of their subscribers access to the public web.3

Even though the web was interactive media’s breakout star, it remained a troublesome advertising platform at this early stage. Online services already had several years of experimentation with advertising and the results were underwhelming. Prodigy, a partnership between IBM and Sears, had been among the first online services to implement ads, but it was forced to scale back its program after “slow-drawing, screen-hogging ads gave subscribers fits.”4 Responding to the backlash, Ted Leonsis, president of AOL, pledged never to put ads on his company’s network. “Here we have this beautiful new interactive medium and we’re trying to put a 200-year-old idea on top of it,” he said. “It doesn’t work.”5

Though Leonsis would not keep his promise for long, the sentiment was animated by a mythos of anticommercialism ascribed to the denizens of cyberspace. “Academic, international, and apparently free, the internet developed into an almost militantly egalitarian and cooperative community,” recalled Netscape’s cofounder, Marc Andreessen. “Virtually nobody made any money from it directly.”6 According to the chairman of Delphi Internet Services, another commercial online service, the internet was nothing less than “the last bastion of socialism.”7 It is difficult to gauge the impact of these perceived cultural barriers to business development. What is more certain is that advertisers were flummoxed by the web’s hodgepodge of often clunky technologies and general lack of polish. Even with the enhancements of the graphical web browser, computing power and bandwidth were limited, which introduced all manner of technical and creative constraints for would-be advertisers. Although the entire enterprise was characterized by trial and error, early adopters looked for points of familiarity where they could.

The first web advertisements appeared in the fall of 1994. The most prevalent format was the banner ad, known in the industry as display advertising because it mixed text and graphical elements in a manner similar to print and outdoor media. These electronic billboards were largely static images placed on high-traffic websites, with the objective of attracting users’ attention as they navigated through content. But unlike other forms of display advertising, web banners offered a layer of interactivity. They could be configured as hyperlinks, enabling users to click through to visit a new website, dictated by the ad’s sponsor. One of the first banner campaigns foregrounded this element of interactivity. AT&T sponsored a series of ads on the website HotWired.com that read: “Have you ever clicked your mouse right here? You will.”8 People who clicked were transported to AT&T’s website, which featured information about long-distance telephone services and links to a handful of experimental sites created by fine art museums. The campaign didn’t make much sense, but it was clear that web advertising’s early adopters were leaning into the novel concept of interactivity.

Print publishers were among the first media outlets to test out interactive spaces, and they struggled with decisions about whether to pursue advertising, subscription funding, or both. Some partnered with commercial online services like AOL, but many began creating their own websites, as did increasing numbers of marketers themselves.9 Still, the bulk of the Madison Avenue ad industry establishment moved online fairly slowly in this early period. Many traditional marketers were willing to bide their time, letting others make the first forays into a challenging and unproven new medium. In 1995, web ad spending barely registered on the scale compared to more established media, but rapid growth was just around the corner. “The promise of [interactive media] is here,” said Jeffrey Katzenberg, chairman of Walt Disney Studios. “The delivery and demand are a bit further over the horizon.”10

The marketing complex’s aggressive political mobilization around the advertising funding model discussed in previous chapters should not be confused with the initial hesitation of individual advertisers to spend serious money the web. There is an element of contradiction in the fact that the industry worked to ensure that the internet would become an advertising medium, but remained reluctant to commit significant resources once the ad funding model was more or less secured. But the political campaigns led by the likes of the Coalition for Advertising-Supported Information and Entertainment were defensive in many ways. Their primary goal was to eliminate the chance for new media to develop with autonomy from advertising funding. As I argue below and in the next several chapters, it did not take long for the marketing complex to come online, nor to again band together to wield influence over the particulars of internet development. Ironically, the political efforts of the marketing complex’s old guard helped to generate the conditions for a crop of technology-focused upstarts to take a leadership role in the web’s growing advertising economy. But there were challenges along the way.

Ad Networks in Dodge City

The most important impediments to early web advertising were logistical. For marketers and ad agencies accustomed to relatively streamlined processes for buying ad inventory, the day-to-day business of web advertising was a labor-intensive guessing game played on unfamiliar technical terrain. Although the conventions of television and print advertising were well established, the web lacked the standard practices, formats, and metrics that greased the wheels of off-line ad channels. There was “no standard reference for what an on-line ad should cost or look like,” noted the Financial Post.11 Also missing were the “viewer statistics advertisers in other media take for granted.” Invoking the image of the lawless American colonial frontier, one practitioner compared the situation to “Dodge City,” admitting, “The structure’s a mess right now.”12

As Joseph Turow argues, early web advertisers achieved a watershed moment with the implementation of the mouse click-through as a standard measure of value for banner advertisements.13 For advertisers, the click was compelling because it offered a measure of verification that was unique to interactive media and provided a quantifiable representation of consumer intention. Although clicks could not be attributed to individual web users, they could be tallied to get a general sense of a banner’s level of engagement. In short order, the click-through rate—the percentage of ad impressions that generate clicks—became a universal metric for evaluating the efficacy of banner ad campaigns.

Nevertheless, the practice of buying and selling ad space online remained cumbersome for most advertisers and publishers. Every aspect of a campaign, including the specifics of the creative elements, the placement and duration of ads, and expected audience exposure had to be negotiated.14 David Shen, an early employee at Yahoo, recalls inputting banner rotation schedules by hand into a spreadsheet that served as the master template for the portal’s advertising operations.15 Web publishers were flying by the seats of their pants, in many cases trying to drum up business without dedicated ad sales personnel.

Advertising distribution—the placement of ads on publisher sites and the coordination of campaigns—was one of the industry’s most important operational hurdles. Although internet usage was growing quickly, audiences were unevenly scattered across thousands of websites and online services. There were limited means to replicate the scale of mass advertising favored by many national marketers, and it was unclear how campaigns of any meaningful size could be managed across such a decentralized medium. “If the web is a 100,000-plus channel universe in which each site is its own program,” noted MediaWeek, web advertising “is an exasperating reflection of that chaos.”16

Finding opportunity in this disorder, a new breed of advertising company emerged: the ad network. Blending long-established practices of ad sales outsourcing with the interactive properties of the new digital medium, ad networks positioned themselves as intermediaries between web publishers looking to sell ad inventory and marketers looking to reach sizable audiences. Their point of intervention was logistical, utilizing the technical affordances of web communication to facilitate the distribution of advertising on a large scale. As the first organizations to create simple and efficient systems for banner ad distribution, ad networks built an organizational and technical infrastructure to support subsequent innovations in collecting and deploying user data for targeted advertising.

Beginning in 1996, a core group of ad networks developed a business model that enabled online advertising to expand in a more systemic fashion and primed its embrace of consumer surveillance. No company was more important in these efforts than DoubleClick, described by the Wall Street Journal as a “trendy cyber-agency leading Madison Avenue’s uncertain charge” into the realm of interactive media.17 DoubleClick found a foothold in the new web advertising market not through some breakthrough technology but by providing contracted sales labor and streamlining the process of buying and selling banner ads. DoubleClick and peers like WebRep compiled groups of publisher clients and deployed sales teams to entice marketers to buy banner inventory bundled across networks of different sites. The aim was to relieve publishers of the sales burden while giving marketers a single point of access to make buys at relatively large scale.

These companies called themselves ad networks to mimic the conventions of broadcasting, hoping to provide a sense of familiarity to marketers. DoubleClick even gave itself call letters, dubbing its first bundle of sites WCLK.18 This explicit comparison to television reflects the fact that for all of the hype surrounding interactive media, the nuts and bolts of early web advertising focused in large part on building basic economies of scale, simplifying a chaotic business landscape, and coordinating sales labor. To these ends, ad networks worked to standardize banner formats and units of sale, such as cost per thousand ad impressions (CPM), helping to ease transaction costs and enable comparisons across publishers.

Ad networks’ services were immediately attractive to web publishers for whom the cost of hiring sales personnel was prohibitive.19 Even for those with deep pockets, expertise in the evolving online platform proved to be a scarce resource. By early 1996, major media companies, including CBS and NBC, arguably among the best positioned to sell their own ad inventory, hired outside firms to handle sales.20 So did important web publishers like Netscape, who contracted DoubleClick to monetize the influx of traffic to its corporate site, which was set as the default home page on its popular Navigator browser. The web publishing sector experienced a rush for sales personnel with any modicum of interactive experience; they would take “anyone who [could] translate web gobbledygook into language that marketers [could] understand.”21

Although ad networks were novel on the internet, outsourcing ad sales and bundling disparate audiences were not new ideas. On the contrary, ad sales representation was in many ways the original business model of the adverting agency.22 As early as 1889, J. Walter Thompson sold ad space across some thirty magazines such as Harper’s and Cosmopolitan.23 By facilitating bundled ad distribution, writes Richard Ohmann, early agencies like Thompson “opened up an economic space not previously demarcated and put themselves solidly in it.”24 As turn-of-the-century print advertising grew, ad agencies expanded their purview to include copywriting, media planning, and market research. J. Walter Thompson in particular pioneered a full-service agency approach that enlarged the industry’s scope by centralizing a range of marketing functions and iterating to create new ones. Daniel Pope argues that the emergence and broadening of ad sales representation played an integral role in the “making of modern advertising” and the commercialization of print media.25

A century later, ad networks like DoubleClick followed in Thompson’s footsteps, expanding on ad sales representation toward a more full-service approach. DoubleClick developed proprietary technologies to not only improve and enlarge banner advertising distribution but also to centralize control over these processes. DoubleClick’s management came to view its core business as facilitating “the logistics of internet advertising on a large scale.”26 DoubleClick facilitated this mission by merging with Interactive Advertising Network (IAN), another start-up that had developed a system to deliver ads to many disparate websites from a central web server. Describing the deal’s rationale, IAN’s cofounder Kevin O’Connor told Advertising Age, “We realized we had the same goals: [DoubleClick] had the sales and we had the technology. It was a merger made in heaven.”27 Foreshadowing the company’s future emphasis on building centralized ad technologies, it was IAN’s O’Connor, rather than DoubleClick’s management, who led the new company as CEO.

DoubleClick’s new ad-serving technology leveraged the disaggregated nature of the web’s communication protocols to consolidate the delivery and coordination of banner ads. Here’s how it worked. Behind the scenes, a website that appears to load as a unified entity is really an amalgamation of various media elements assembled by the browser. In a basic setup, a web browser might receive all the files that make up a web page from a single server, but this two-party configuration is not a technical requirement. Different website elements can easily be stored across multiple servers and then compiled into a unified web page with no apparent change in the final display or functionality. For example, in the earliest days of web advertising, a publisher like Fast Company would generally host all of the files that constituted its website on its own servers, including any ads, even if ad sales were contracted out to a network. This scenario changed as ad networks began offering centralized ad serving. So when Fast Company joined DoubleClick in 1996, it outsourced not only its ad sales operation but also its ad delivery and campaign management functions. Setup simply required adding a few lines of code to Fast Company’s site to instruct web browsers to retrieve banner ads from DoubleClick rather than its own server. DoubleClick would handle the rest. Advertising online, once a confusing and laborious process, suddenly became the “easiest [business] model for a web startup to implement.”28 Rather than fiddling with ad tech or chasing down user subscriptions, all web publishers needed to do was “contract their revenue growth to an ad network and focus on building an audience.”29

Because it was largely invisible to web users, the ad network system became known as third-party advertising. When an individual web user (the first party) visits a web publisher (the second party), there is an expectation of communication among the two parties. Working behind the scenes to deliver ads from unaffiliated servers, ad networks were third-party intermediaries into what was previously a two-party exchange. By building their own distribution infrastructure via centralized ad servers, DoubleClick and other ad networks were able to offer publishers fully outsourced advertising services, ameliorating the need for sales labor but also handling key operational functions like ad delivery, campaign management, and billing. As the financial services website Edgar Online told the New York Times, working with DoubleClick allowed the company to “focus on what we’re good at, the selling and marketing of S.E.C.-based products, but not on what we’re not good at—dealing with ad agencies, trafficking ads, and sending advertisers reports.”30 Third-party ad servers swiftly became an industry mainstay, giving marketers and their ad agencies more convenient means to reach economies of scale in their media buys—and, as we shall see, opening new possibilities for targeted messaging. As one ad network executive put it, “What we think will drive growth is an automated system that can do all of the ad buys in real time and automate the results.”31

As the web proliferated and more ad networks cropped up, “the challenge facing media planners” shifted from “finding sites on which to advertise to knowing which tools to use for help.”32 Competing with ad networks, other companies emerged to offer iterations of self-service ad tech management. Among the most important of this group were NetGravity and Accipiter, start-ups that provided turnkey ad servers to web publishers that did not wish to outsource ad sales or bundle their inventory in a network.33 Like ad networks, these software-based tools were far more efficient than manual ad placement. Yahoo reported using NetGravity’s system to go from a three-person staff managing five ad campaigns across twenty web pages to a team of two managing seventy campaigns across sixteen thousand pages.34 As the technology improved, the time delay for executing and modifying ad campaigns was effectively reduced to zero. Banners washed over the web at an increasing pace. By late 1997, DoubleClick alone was serving approximately 750 million ads per month, and capacity was growing.35

Beyond the Banner

The logistical improvements brought by ad networks were significant, but the young industry encountered other equally vexing problems. As banners spread, their novelty began to wear thin. It was something of an open secret among web publishers that the majority of users simply did not click on ads. Banners were like poison ivy: Most people steered clear, and those who did click once or twice usually kept their distance after that. This was especially troublesome because so much of the hype surrounding the web’s commercialization hinged on its interactivity, which was supposed to enable marketers to engage consumers directly rather than simply shout in their general direction, only to be ignored. Without robust interactivity, the low-bandwidth web seemed a poor substitute for existing branding platforms like television. As marketers began to grumble about low click-through rates, the online ad industry responded with a flurry of activity centered on ways to move “beyond the banner.”36

Some web advertising companies attempted to enhance banners with “rich media” experiences and pop-up formats that were harder to ignore, but the idea that gained the most traction among advertisers was that ads simply needed to be more “relevant” to consumers. Greater personalization of messaging was positioned as a solution for reining in the chaos of an interactive medium that seemed to give consumers greater control over their media experiences.37 Of course, personalization required increased knowledge about web users, which dovetailed with evolving needs for data collection and user identification in other sectors such as internet retailing and banking. What the web needed was a memory. The problem was that it was designed to forget.

The web’s communication protocols were originally created to facilitate series of discrete data transfers, not persistent connections. Rather than preserve a continuous “state” of communication among parties, web protocols saved precious computing resources by operating in a “stateless” condition.38 Data was sent back and forth as a string of stand-alone communication events. It was possible to catalog information about certain user interactions—servers could count clicks and capture glimpses of immediate browsing history—but it was difficult to attach such actions to individuals. This made web browsing functionally anonymous but limited the scope of commercial applications. For example, in order for online shopping to function, websites had to be able to recognize that a given series of actions (like putting items into a shopping cart) were connected to a single user. As the Wall Street Journal put it, without the means to tell one person from another, websites were destined to remain “poor shopkeepers.”39

Seeking to improve the relevance of ad targeting, internet advertisers found it especially pressing to develop the web’s capacity to recognize and remember individual users. Ad companies and publishers began to work on solutions that required users to register and log in, but it was Netscape, maker of the leading web browser software, that developed a scalable fix to the problem of statelessness. This was the HTTP cookie, an innovation that enabled web servers to uniquely identify web browsers. Cookies effectively allowed websites to say, “Here, hang onto this file and show it to me the next time you ask me something. It will remind me who you are and what we’ve already talked about.”40 Passing a cookie identifier back and forth created a continuous communication state between browsers and servers, which facilitated a range of new data collection practices on the web. Released as an open technical standard, cookie functionality was quickly integrated into all major browsers and put to a variety of uses spanning e-commerce, credentialization, and personalization.41 It was not long before Network World described cookies as “a de facto standard” web protocol.42 This widespread implementation altered the web’s trajectory by introducing a new capacity for surveillance into what had effectively been an anonymous communications environment.

The internet advertising industry seized on cookie technology as a means to improve ad targeting and relevance. Before cookies, ad metrics companies such as I/PRO had attempted to glean information about web traffic patterns from the routine records stored in server logs. Some degree of analysis was possible using metadata such as IP addresses, but as I/PRO’s founder explained, without a system of unique identification, server logs were “ungainly veins.”43 No company was able to reliably divine the activities of individual users without forcing people to register and log into websites. Cookies were a game changer in this regard, offering any business on the web an unobtrusive means to link online actions with individual web browsers. Because they were built into the web’s standard communication protocols, cookies were automatically passed back and forth behind the scenes during the browsing experience. Voluntary site registrations and user questionnaires became outmoded by new methods of data collection that did not need to bother with “the trouble of getting the user’s permission.”44 By 1997, the Internet Advertising Bureau trade group endorsed cookies as “an essential part of member companies’ business strategies.”45

Cookie Monsters

Although the broader advertising sector recognized cookies as an opportunity to improve ad targeting, only ad networks had the capacity to integrate data collection into ad delivery operations on a large scale. This gave ad networks a unique vantage point to track individuals as they moved about the web across seemingly unaffiliated sites. Cookies were flexible and could be hitched to any number of routine data transactions between web browsers and servers. Third parties could deposit cookies through websites for which they were not the primary hosts, provided that they served at least some element of the page. Any component would do, be it an image, a chunk of text, or a far-flung banner advertisement.

Ad networks capitalized on this opportunity, shrewdly embedding cookies into the banners served to sites across their networks. Each time a banner was served, a cookie was sent with it. This created and maintained a data channel between an ad network like DoubleClick and every user who visited any of its client websites.46 Leveraging this reciprocity, DoubleClick transformed its ad delivery system into an apparatus for systematically gathering information about web users and their behaviors. Dispersed across the web via ad networks’ central servers, banner ads became conduits for circulating cookies on a scale and scope that far exceeded anything in practice at the time. Although any web publisher could deploy cookies on its own site to recognize return visitors, ad networks had the ability to connect the dots as users moved from site to site. Significantly, the appropriation of cookies gave ad networks a way to develop improved user tracking and profiling capabilities and to make a renewed business case to publishers, marketers, and investors regarding the viability of targeted banner advertising.

In the hands of ad network developers, cookies added an unprecedented level of granularity to existing advertising techniques, spawning early forays into what later became called behavioral advertising.47 The regularly updated information stored in cookie files provided the crucial mortar used when building databases of user profiles, enabling surveillance of individual web browsers as a proxy for individual consumers. These capacities were enhanced because they were deployed over sprawling ad networks. More than simply adding a larger volume of consumer information, data collection via ad networks qualitatively changed the possibilities for profiling and ad targeting. One of the most popular innovations was termed clickstream analysis, which involved tracking the movements of individual web browsers over time. This enabled ad networks to compile browsing histories that included information about the time, duration, and order of the sites users visited. As Forbes explained, after receiving an initial DoubleClick cookie, “whenever you visit any of the sites [in the network], the DoubleClick server picks up the [cookie] ID number and tucks away information about your visit. Gradually it builds a pretty complete dossier on you—and your spending and computing habits.”48

DoubleClick moved aggressively into profiling, depositing 40 million cookies and compiling a database of 10 million unique user profiles in its first year.49 To get a sense of scale, DoubleClick’s profile database was 60 percent larger than the entire subscriber base of America Online, at the time the nation’s largest internet service provider.50 In 1997, DoubleClick branded its technology under the label DART, which stood for Dynamic Advertising, Reporting, and Targeting. The DART system could serve targeted ads in near real time by cross-referencing its user profile database with information collected on the fly. The company’s tag line during this period highlighted its dynamic targeting capabilities with the promise to deliver the “right message to the right person at the right time.”

The DART software, deployed across DoubleClick’s network of publishers, represented the first major incarnation of the data-driven surveillance advertising that industry boosters had been promising for years. When asked whether DoubleClick could “enable an advertiser to reach women aerospace engineers who like sports in Southern California,” CEO Kevin O’Connor responded, “If you work for Lockheed [Martin] in Orange County and you’re accessing the sports scores for women’s soccer, we’ve got you.”51 As Crain’s New York Business observed, “No one else [was] doing it in nearly as sophisticated a fashion.”52

Although some major publishers such as Microsoft and the New York Times kept ad services largely in house, Forrester Research declared that “the future of web advertising belonged to ad networks rather than single websites.”53 DoubleClick, among the best positioned to capitalize on network effects and economies of scale, emerged as a market leader. The industry was by no means a well-oiled machine, but the services provided by ad networks made it more practical and enticing for marketers to allocate portions of their ad budgets online. Boosted by the start of what would become the dot-com financial boom, online advertising spending in the United States reached nearly $1 billion in 1997, closing in on spending totals for outdoor media ($1.4 billion).54 As HotWired described it, “The days when the net seemed to exist outside the laws of capitalism [were] just about over.”55

Revolt of the Developers

The original technical specification for HTTP cookies was created at Netscape by a team led by Lou Montulli, one of the company’s first software developers. As Montulli told the New York Times, the goal was to create a tool with the flexibility to be put to many different uses, including “things we hadn’t thought about.”56 One such thing was the online ad industry’s increasingly widespread use of cookies to power surreptitious data collection. Montulli was no privacy hawk, but he was troubled by the way cookies had been appropriated by ad networks to engage in third-party tracking. Working with colleagues, Montulli drafted a revised technical specification that disabled third-party cookies by default, requiring users to opt into advertising from the likes of ad networks.57 Although Montulli had no problems with data-driven advertising on a first-party basis, he “didn’t want cookies to be used as a general tracking mechanism” without explicit user consent.58

This is how David Kristol, an engineer at Bell Laboratories and respected member of the internet standards community, came to be accused of corporate subversion. “I got an outraged call from Kevin Ryan [CFO of DoubleClick] expressing his unhappiness and accusing me of trying to sabotage his business,” he recalled.59 Kristol had coauthored the new cookie specification with Montulli and submitted it for review with the Internet Engineering Task Force (IETF), a volunteer standards setting community that developed internet protocols.

The ad network executive’s anger was indicative of a groundswell of industry aggravation related not only to Montulli and Kristol’s proposal but also to a spate of unwanted public attention. The news media had recently discovered the existence of cookies and was beginning to raise questions about online privacy.60 More worrying still, the Federal Trade Commission was gearing up to hold a workshop to discuss web privacy issues. Though there was little evidence to suggest that President Clinton’s administration would derail web advertising’s trajectory, the White House had not yet officially endorsed a self-regulatory approach to internet data collection.

Upon catching wind of Montulli and Kristol’s revised cookie proposal, internet advertising companies went on the defensive, inserting themselves into what might have otherwise been seen as an obscure technocratic issue. As one industry representative put it: “What concerns us is the tone of the proposal, which is that advertising is not good for us, so we want to avoid it. That begs the question, how is the web going to be funded?”61 Tellingly, the new cookie specification was not actually against web advertising. Nor was it against the use of cookies for tracking and personalization. Nor was it even against third-party advertising when conducted on a voluntary, opt-in basis. What the specification opposed was hidden third-party surveillance as a default setting on the web. The proposal took specific aim at the lack of transparency involved in third-party cookies and the absence of user control over their use. As Kristol explained, “Short of turning off automatic image loading, a user could not avoid receiving third party cookies.”62

A debate began within the IETF not only about whether third-party surveillance was a threat to user privacy and autonomy but also about whether the organization should be involved in privacy issues in the first place. Public email archives reveal heated discussions about the overlapping technical and social dimensions of this issue.63 Proponents of the revised standard conceded that it could produce some fallout by limiting otherwise acceptable cookie practices, but they opted for a “conservative method,” preferring to err on the side privacy.64 Others contended that tracking was a necessary component of web advertising and that disrupting third-party cookies would destroy ad networks and the growing number of publishers that depended on them for revenue.65 Drawing a distinction between legitimate advertising and unaccountable surveillance, Kristol argued that restrictions on third-party cookies would undermine “business models that relied on tracking users, not the advertising business itself.”66

Among the most strident opposition to the proposal came from an employee at Microsoft who argued that the “free market will do a better job of protecting user’s rights” than a restrictive standard.67 The commenter warned that if the IETF tried to implement such “social engineering,” the organization would become irrelevant by showing companies that it was a “dangerous place to create standards.”68 “After all,” the commenter wrote, “vendors want to work with an organization where they are partners in the standards process, not its target.” It is worth noting that Kristol and others had explicitly reached out to industry representatives from the start of the discussion, but only a few organizations elected to participate early on. It was only after the revised specification proved a threat to the developing surveillance advertising business model that industry took notice.

Instead of engaging in a real debate about the privacy costs of its emerging business model, the ad sector ran interference while organizing to support a competing standard that left third-party tracking alone.69 Leading this countereffort was the chief technology officer at Engage Technologies, a company on its way to becoming one of DoubleClick’s biggest ad tech rivals.70 Like the broader political mobilization of the marketing complex, this was another example of competitors joining forces to advance a common interest in shaping the web’s features toward surveillance.

With all the back and forth, the deliberation process at the IETF slowed to a crawl, dragging on for months. Reaching an impasse, Kristol withdrew the provision to limit third-party tracking.71 There would be no official pushback from the internet standards community on advertising and privacy issues. Ultimately, the power to implement any standard rested with the browser software makers, of which there were only two of significance. Netscape and Microsoft controlled nearly the entire market, and as such, their decisions became “de facto standards,” regardless of the IETF’s technical specifications.72 Both companies participated in the IETF discussions to some degree (Montulli was still a Netscape employee), but in the end, they were unwilling to support third-party cookie restrictions.

It could have been that Netscape and Microsoft were simply focused on other business objectives and therefore reluctant to dedicate resources toward privacy concerns. As the product manager for Navigator put it, “We expect this to blow over. We just don’t consider it a significant issue.”73 Kristol’s own reflection is that while Netscape and Microsoft were giving web browsers to consumers for free, they were selling servers and related software to businesses. It simply was not worth the trouble to potentially “anger their paying customers by disabling third party cookies.”74 As Rajiv Shah and Jay Kesan argue in their study of the cookie standards-setting process, “the implication is that society cannot expect firms to meet or adequately protect unprofitable societal concerns.”75

In some ways, the software design decisions made by Netscape and Microsoft years earlier had set the stage for the resolution to this standoff. The first iterations of Netscape’s Navigator browser implemented cookies entirely behind the scenes. There were no options for users to manage cookies; nor was there any indication that they were being placed on users’ machines as they surfed the web.76 From the start, cookies were infrastructural, a background technical feature meant to support a range of second-order practices. Only several years later, in the midst of the IETF debates and mounting public scrutiny, did browser makers add user controls over cookies.77 But even then, browsers were configured to accept all cookies by default, including those from third-party ad networks. Had they wanted to support Montulli and Kristol’s revised specification, Netscape and Microsoft could have easily reversed this design choice. Instead, they allowed third-party tracking to recede into the web’s normal state of operation.

It so happened that adding user controls while leaving third-party tracking intact was the exact solution offered by Dwight Merriman, the lead engineer at DoubleClick. Blocking third-party cookies “should be available to the user as an option,” he wrote to the IETF, but all cookies should be enabled “by default.”78 The reason? Disabling third-party cookies automatically was “basically equivalent to not allowing them at all, because 99% of the population will see no reason to change the default” setting. Some within the IETF working group disagreed. “That DoubleClick has formed a business model around a loophole in the original cookie draft is not, as I see it, any reason to compromise the privacy of future web users,” wrote one participant. “I think it is deplorable that you would ask to modify an agreed-to standard for your commercial gain.”79

Deplorable or otherwise, leaving third-party cookies as the default configuration was a significant victory for ad networks and the surveillance advertising business model at large. “Because the vast majority of Web users never bother to change their cookie preferences,” summarized Advertising Age, “the effect on companies that use cookies as targeting tools will be minimal.”80 Research on the power of default settings corroborates this conclusion. As Rajiv Shah and Christian Sandvig have shown, defaults place significant power to regulate human behavior in the hands of technology designers, particularly when adjusting settings is left to the underinformed, as was the case for the majority of web users in this period.81 In his work on digital rights management systems, Tarleton Gillespie argues that objectionable practices like surveillance “seem to be more palatable to users when they arrive as an organic part of a new technology, rather than being imposed on a technology already in existence.”82 The IETF’s unsuccessful challenge to third-party cookies was a moment of what Gillespie calls “technological regulation,” which helped to ingrain surveillance not as a political question of design but simply how the internet works. This is precisely what would be expected in the regulatory void of the government’s neoliberal approach to internet privacy.

Cookies are still used today, but they are by no means the only mechanism of internet surveillance. Overemphasizing their functional role, or that of any particular technology, obscures one of the key lessons of this history. It is not only what cookies did (identify browsers and collect data) but also how they were deployed that helped set the web on the trajectory of surveillance advertising. Cookies were the first widespread implementation of consumer surveillance deployed natively within a media platform. Integrated into the web’s communication protocols, cookies built a capacity for surveillance into the technical guts of the web in the early stages of its development and popularization. Implemented behind the scenes, cookies normalized surveillance as part of the default experience of internet use. In their appropriation of cookies, ad networks set up the technical foundations for the increasingly invasive forms of consumer surveillance that followed. As ad networks grew, so did surveillance advertising, and the fundamental character of the web was transformed from anonymous to identifying.

The following years saw an expansion and refinement of the business models and technologies of surveillance advertising. By the late 1990s, DoubleClick and handful of other ad networks were clear industry leaders. Although still not profitable, they were earning revenue and relentlessly promoting themselves through public relations and their own advertising campaigns. And while falling short of fully satisfying the demands of marketers, the sophistication of their ad-serving systems and the scope of their reach surpassed contemporary alternatives and continued to point to the intensified use of the web as a medium for consumer surveillance. As the chief executive of the Engage ad network told BusinessWeek, coming to appreciate the value of consumer data was his industry’s “single most important revelation.”83 The next chapter examines how surveillance advertisers were given a major boost, materially and ideologically, by the dot-com investment bubble, which propelled the U.S. economy into the twenty-first century.

4. The Dot-com Bubble