How adding a photo to a landing page impacts donations

Hillsdale College

Experiment Summary

Timeframe: 02/27/2019 - 03/03/2019

Hillsdale College recently mailed out thousands of letters to school administrators and teachers across America. In these letters, they offered to send a free pocket-sized copy of the U.S. Constitution for every single child in every classroom. Hillsdale received an astonishing number of replies requesting copies of the Constitution. To fund the shipment to all of these schools, Hillsdale launched a four-day email campaign asking donors to send a donation to help send pocket-sized Constitutions to schoolchildren across the U.S.

After sending the first two emails out to a split landing page test, where they tested the presence of gift handles that explained what each suggested gift amount would be used for, they concluded that adding just one line of copy around the gift handles did not increase likelihood of giving.

The Hillsdale team wondered about including a photo of some of the students who received Constitutions onto the landing page. Rather than telling the prospective donor about what they were doing, they could show them with a photo. Would adding a photo of schoolchildren receiving the Constitution increase donations? They hypothesized that adding the photo would show proof, causing an increase in trust, and would increase emotional appeal. They hypothesized that each of these would increase donations. This experiment was tested on their non-donor file.

Research Question

Will adding a photo to a landing page increase the number of donations?

Results

This experiment has a required sample size of 132,833 in order to be valid. Since the experiment had a total sample size of 502,498, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

    0% increase in traffic
× 12.9% decrease in conversion rate
× 0% increase in average gift

Key Learnings

Because this experiment only ran for one day, the experiment results shouldn’t be rolled out to the rest of the donation pages. However, this result was enough for the Hillsdale team to roll out to this landing page for the duration of the campaign.