Netflix: testing the one image that decides whether you press play

Case study · Marketing 465 · Campaign Planning and Creative Testing

Netflix treated each title’s thumbnail as an ad, wrote down what it wanted to learn, changed only the image, and set its winner rule in advance, and the tests showed that the picture is one of the biggest levers it has.

The situation

Every title on Netflix is sold by a small image. There is no salesperson and usually no trailer playing. A member scrolls a row, sees a thumbnail, and decides in a moment whether to look closer.

In 2016, Netflix’s head of product creative, Nick Nelson, shared what the company’s early 2014 consumer research had found. Artwork was the biggest influence on a member’s decision to watch, and it made up over 82% of their focus while browsing. Members spent an average of 1.8 seconds considering each title. Netflix also said that members who are not engaged within about 90 seconds will likely lose interest and do something else.

The challenge

If the image does most of the selling, the wrong image costs viewing. But taste is a weak guide. A designer can argue for a moody landscape and another for a close-up of the star, and both can sound right.

Netflix also had to be careful about what counted as a win. An image that pulls people toward one show might only steal hours from another show. And an image that gets clicks but misrepresents the title would hurt trust. So the team wrote three goals before testing: help members find a story they want faster, raise engagement for each title and in total, and avoid misrepresenting titles.

What they did

The program, described on the Netflix Technology Blog in May 2016, started small. For the documentary The Short Game, Netflix ran a simple A/B test. Each test group saw a different image for the same title. Nothing else changed. The team tracked click-through rate, total play time, the share of very short plays and how much of the title people finished.

That first test showed that members were sensitive to artwork changes, so Netflix scaled up. The working hypothesis was that picking the best image across many titles would raise total streaming hours.

• One variable: within a test, only the image changed. The title, row and description stayed the same.
• A winner rule set in advance: the main metric was take rate, meaning plays (not counting very short ones) divided by the number of times the image was shown.
• Guardrail metrics: the team checked that short plays and completion did not get worse, so a winner could not be a clickbait image.
• Explore, then exploit: show several images until one wins with statistical significance, typically within a few days, then show the winner to everyone.

What happened

Netflix called the results of its multi-title test “unambiguous.” Both the share of viewing and total streaming hours went up, and the secondary metrics were not harmed. Netflix did not publish the size of the lift, so the exact gain is unknown.

The tests also produced patterns that a creative team could reuse.

• Faces showing complex emotion beat calm or neutral faces, because they signal the tone of the show. The winning Unbreakable Kimmy Schmidt season 2 image was Netflix’s example (Netflix, 2016).
• Villains and polarizing characters did well, especially in kids and action titles. For Dragons: Race to the Edge, two villain images significantly outperformed all others (Netflix, 2016).
• An image’s chance of winning dropped sharply when it showed more than three people, which shaped Orange Is the New Black artwork (Netflix, 2016).
• Winners differed by country. Sense8 had different winning images in different regions (Netflix, 2016).
• Over many months, simply rotating images was not as effective as finding a better one (Netflix Technology Blog, 2016).

Why it worked

Netflix tested the biggest lever first. Its own research said the image drove the decision, so that is where it spent its testing effort, rather than on button text or layout details.

Each test isolated one change, so the team knew what caused the difference. The goal was defined before launch, including what would not count as success. Checking total viewing and quality of viewing kept the team from rewarding images that only shifted attention or tricked people into a short play. And each result fed a growing log of lessons, like emotion, villains and small casts, that designers could use on the next title.

Where it gets complicated

Testing can push past one best image. In December 2017, Netflix described choosing artwork per member based on viewing history. A comedy fan might see Robin Williams on the Good Will Hunting tile. Netflix reported a significant lift in an A/B test but gave no number. It also named the risks: clickbait images, titles becoming harder to recognize, and unclear credit for which image drove a play.

In October 2018, writer Stacia L. Brown pointed out on Twitter that artwork for the comedy Like Father, starring Kristen Bell and Kelsey Grammer, showed two Black actors with small roles. Other users shared similar examples, and some felt misled about what the film was. Netflix said, “Reports that we look at demographics when personalizing artwork are untrue,” and that it uses only a member’s viewing history. Even so, an image chosen because it wins clicks can still misrepresent a title.

There are practical limits too. Niche titles get fewer views, so tests take longer to reach a clear answer, and Netflix had to log every image shown on every device, which it said came to billions of impressions a day. Most brands need longer runs, bigger changes or fewer variants to learn anything.

The marketing principle

Creative testing works when it starts with a clear goal and a hypothesis, changes one thing, and decides the winner rule before launch. Netflix’s lesson is to test the biggest lever first, often the image, and to measure the result you actually want, not just clicks. A winning image still has to tell the truth about what it is selling.

Sources

• The Power of a Picture, Netflix (2016)
• Selecting the best artwork for videos through A/B testing, Netflix Technology Blog (2016)
• Artwork Personalization at Netflix, Netflix Technology Blog (2017)
• Netflix users suggest the service is targeting black viewers with inaccurate movie posters, The FADER (2018)
• Netflix Denies Serving Up Posters Based on Users’ Race, TheWrap (2018)

---

Back to Campaign Planning and Creative Testing · All case studies

Brand names, images and trademarks belong to their owners and appear here for educational purposes only.