eBay: the search ads that looked profitable and did not add sales

Case study · Marketing 465 · Paid Media

Platform reports showed eBay’s search ads paying off many times over, but when eBay switched them off in a controlled test, almost nothing changed, which shows why paid media has to be judged on incremental results, not on clicks that would have happened anyway.

The situation

By 2012, eBay was one of the largest buyers of search ads in the United States. Its researchers wrote that the company had historically managed over 100 million keywords and keyword combinations, updated daily and fed into Google, Microsoft and Yahoo!. In a footnote, they put eBay’s US paid search spend at about $51 million a year, based on a reported monthly figure of $4.25 million.

The ads looked like a strong investment. Every time someone clicked an eBay ad and bought something, the purchase showed up in the reports.

The challenge

The problem was a simple question nobody could answer from the dashboards: would those shoppers have bought anyway? Many people typed “eBay” into Google because they already planned to shop there. eBay’s free, organic listing sat right below the paid ad. A click on the ad got credit for the sale, but the ad may not have caused it.

Three eBay researchers, Thomas Blake, Chris Nosko and Steven Tadelis, decided to test it. Instead of reading correlations, they ran experiments: turn the ads off for some people, keep them on for others, and compare sales.

What they did

The team ran two kinds of tests in 2012, one for brand keywords and one for everything else.

• Brand keywords: In March 2012, eBay stopped buying ads on searches containing “eBay” on Yahoo! and MSN, while it kept buying them on Google as a comparison. A later test on Google followed in July 2012.
• Non-brand keywords (searches like a product name): eBay used Google’s geographic targeting to stop paid search in roughly 30 percent of US designated market areas (DMAs) for about 60 days, while the rest of the country kept seeing ads as a control group.
• Customer data: Because eBay knew each buyer’s purchase history, the team could compare the effect of ads on new customers, lapsed customers and frequent shoppers.

What happened

The results were close to the opposite of what the platform reports suggested. When the brand ads went dark, people simply clicked the free listing instead. In the non-brand test, sales in the markets without ads barely moved.

The researchers first released the findings in 2013 as an eBay Research Labs paper, which drew wide press coverage. The final version appeared in the journal Econometrica in January 2015.

• Brand ads: about 99.5 percent of the clicks eBay stopped paying for were “immediately captured by natural search traffic” (Blake, Nosko and Tadelis, 2014/2015).
• Non-brand ads: the whole paid search program added an estimated 0.66 percent to sales, a result that was not statistically significant.
• Return on spend: a standard, non-experimental analysis put the return at over 1,400 percent. The experiment put it at about negative 63 percent, meaning eBay got back roughly 37 cents in short-term revenue for each dollar spent.
• Who the ads did reach: the ads had their largest effect on new registered users and on people who had not bought in over a year. For people who bought more than about three times a year, the effect was close to zero, and those frequent buyers accounted for most of the ad spending.

Why it worked

The test worked because it asked the right question. A platform report answers “which sales came after an ad click?” An experiment answers “which sales happened because of the ad?” Those are different questions, and for a brand as well known as eBay, the gap between them was huge.

Two forces created the gap. First, people searching for “eBay” already intended to go there, so the ad was taking credit for demand the brand had built some other way. Second, eBay’s loyal shoppers were the people most likely to search and click, so ads and sales rose together even though one was not causing the other. A holdout group, people who did not see ads, was the only way to separate the two.

Where it gets complicated

The findings do not mean search ads fail. Google’s own research, based on more than 400 studies of advertisers who paused search ads, reported in 2011 that on average 89 percent of paid clicks were not replaced by organic clicks when ads stopped. A 2012 Google follow-up found that when an advertiser already held the top organic spot, about 50 percent of ad clicks were still incremental. Google has an interest in these results, and eBay’s experiment was more tightly controlled, but the point stands: results vary by brand.

eBay is an unusual case. It is a household name with strong organic rankings and millions of repeat buyers. Even Slate’s 2013 coverage noted that paid search may still pay off for smaller companies without that name recognition. The study also measured short-term sales only. It did not test long-term brand effects or what happens when a competitor bids on your brand name after you stop.

There is also a practical limit. eBay could run a geographic holdout across hundreds of markets with millions of transactions. A small brand with a $10,000 monthly budget may not have enough data to get a clear answer from the same kind of test.

The marketing principle

Paid media buys reach and speed, but the dashboard is not the truth. Platforms count the sales that followed an ad, not the sales the ad caused. Before you scale spend, ask what would have happened without it, and test that with a holdout when you can. For eBay, incremental return was the number that mattered, and it told a very different story from ROAS.

Sources

• Consumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment (NBER Working Paper 20171), National Bureau of Economic Research (2014)
• Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment, Econometrica (2015)
• Did eBay Just Prove That Paid Search Ads Don’t Work?, Slate (2013)
• Why Companies May Be Overpaying for Web Search Ads, Chicago Booth Review (2015)
• Studies Show Search Ads Drive 89% Incremental Traffic, Google Research (2011)
• Impact of Organic Ranking on Ad Click Incrementality, Google Research (2012)

---

Back to Paid Media · All case studies

Brand names, images and trademarks belong to their owners and appear here for educational purposes only.