Netflix: testing the thumbnail, then learning when a winning image is the wrong one

Case study · Marketing 475 · Analytics and Visual Optimization

Netflix found that its title artwork drove viewing decisions, so it tested images on purpose, judged them on the metric that mattered, and learned that the image with the most clicks is not always the best one.

The situation

Every title on the Netflix home screen is represented by one small image. That image does a lot of work. In 2016 Netflix said its research showed members look at the artwork first, then decide whether to read more about a title.

Nick Nelson, then Netflix’s head of product creative, wrote in May 2016 that consumer research from early 2014 found members spent an average of 1.8 seconds considering each title. The same research, he wrote, found artwork was the biggest influence on a member’s decision to watch and made up more than 82% of their focus while browsing.

The challenge

For years each title had one main image, usually supplied by the studio. Nobody knew if it was the best image for the job. Opinions about which picture was “better” were easy to have and hard to prove.

Netflix also had a measurement problem. A more exciting image might pull clicks away from other titles without increasing total viewing. Or it might get people to press play and then quit after a few minutes. The team needed a test design and a metric that would separate real gains from those traps.

What they did

Netflix ran a series of A/B tests where the only thing that changed was the artwork. The same title, the same description and the same placement were shown with different images to different groups of members. The team described the work on the Netflix Technology Blog in May 2016.

• The main metric was take rate: plays divided by impressions, with very short plays left out so a quick click and exit did not count as a win (Netflix Technology Blog, 2016).
• The team also tracked total streaming hours, the share of short plays and how much of a title people finished, to check that a winning image was not just shifting viewing from one title to another.
• Tests moved from single titles, such as the documentary The Short Game, to many titles at once. An “explore” phase showed several images to a sample of members, then an “exploit” phase gave everyone else the winner.
• In 2017 Netflix went further and personalized artwork. Its tech blog gave the example of Good Will Hunting: a member who watches a lot of romance might see Matt Damon and Minnie Driver, while a comedy fan might see Robin Williams (Netflix Technology Blog, 2017).

What happened

Netflix reported that better artwork for The Short Game significantly increased its audience and engagement, and that testing across many titles significantly raised total streaming hours without hurting quality measures like completion. Netflix did not publish exact percentages. The 2017 post said personalized artwork produced a significant lift in core metrics, with the biggest gains for titles a member had never interacted with.

• Faces with expressive, complex emotion beat neutral or stoic expressions, because they signal the tone of the show (Netflix, 2016).
• Recognizable or polarizing characters did well. For the kids’ series Dragons: Race to the Edge, two images of villains significantly outperformed all others (Netflix, 2016).
• An image’s chance of winning dropped sharply when it showed more than three people (Netflix, 2016).
• Winning images for the same title, such as Sense8, differed by country (Netflix, 2016).

Why it worked

Netflix set the goal before judging the creative. The goal was more total viewing that people actually stuck with, not more clicks. That is why the team defined take rate to exclude very short plays and why it checked completion and total hours alongside it.

The tests were clean. Only the image changed, so any difference could be traced to the image. And the team did not stop at picking winners. It looked across winners and losers for shared traits, such as emotion, villains and crowd size, and turned those patterns into guidance for designers creating the next round of images.

Where it gets complicated

Optimization can reward the wrong thing. Netflix’s own 2017 post warned about “clickbait” images that “entice a member to start playing but ultimately result in low-quality engagement,” and said artwork should be engaging, informative and representative of the title. The authors also noted that changing a title’s image too often can make it harder for members to recognize.

In October 2018 that risk became public. Several Black subscribers posted examples of thumbnails that featured Black supporting actors with little screen time. Writer Stacia L. Brown pointed to Like Father, whose leads are Kristen Bell and Kelsey Grammer, but whose thumbnail showed two Black actors who she said had “maaaaybe a 10 cumulative minutes of screen time” (The FADER, 2018). Netflix told The FADER: “Reports that we look at demographics when personalizing artwork are untrue. We don’t ask members for their race, gender, or ethnicity.” It said the only information it used was viewing history.

Both things can be true. The system may never have used race, and the image could still mislead the viewer about what the film is. A model trained on plays will find whatever image gets plays. Deciding whether that image is honest is a human judgment the numbers cannot make.

The marketing principle

Data shows which visuals work, but only for the goal you set. Netflix chose a metric that rewarded real viewing, tested one change at a time and looked for patterns across winners and losers. The 2018 backlash shows the limit: a visual can win the test and still break the promise to the audience. Data informs, and people still decide what is fair to show.

Sources

• The Power of a Picture, Netflix (About Netflix newsroom) (2016)
• Selecting the best artwork for videos through A/B testing, Netflix Technology Blog (2016)
• Artwork Personalization at Netflix, Netflix Technology Blog (2017)
• Netflix users suggest the service is targeting black viewers with inaccurate movie posters, The FADER (2018)
• Is Netflix Targeting Users By Race?, /Film (2018)

---

Back to Analytics and Visual Optimization · All case studies

Brand names, images and trademarks belong to their owners and appear here for educational purposes only.