Illinois Policy Institute

How implementing a goal thermometer on a high urgency campaign donation page impacted donations- attempt #2

Experiment ID: #145579

Illinois Policy Institute

Experiment Summary

Timeframe: 04/17/2023 - 05/01/2023

As part of a March high urgency campaign for the Illinois Policy Institute, we want to test the impact of adding a goal thermometer on the campaign donation page. The goal is to reach a $50,000 matching gift. The thermometer syncs with GA to track up-to-date campaign revenue, so donation page visitors will see how much money has been raised towards the campaign goal at the time they’re visiting the page.

We’re curious to see what kind of impact the visible progress and end goal seen through the thermometer will impacted donations.

This is our second try at this test. We extended the deadline of the campaign and will re-run this test.

The initial test resulted in a 7.8% increase with a 34.8% level of confidence to the treatment and 9% increase in revenue with a 65.4% level of confidence.

Let’s see how our second test performs.

Research Question

We believe that implementing a goal thermometer for donation page visitors amidst a high urgency campaign will achieve an increase in donations because page visitors will be motivated to help contribute to and reach the end goal of the campaign.

Design

C: Control

T1: Variant 1

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence	Average Gift
C:	Control	11.8%			$0.00
T1:	Variant 1	6.3%	-46.6%	83.5%	$0.00

This experiment has a required sample size of 213 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

The key learning from this experiment is that the implementation of a goal thermometer for donation page visitors did not result in an increase in donations. In fact, the treatment group saw a significant 46.6% decrease in donations compared to the control group with a nearly valid 83.5% level of confidence.

The treatment went in the wrong direction compared to going in the right direction the first time we ran this experiment. That said, each experiment did produce an increase in revenue through the treatment.

1st test: 9% increase in revenue with a 65.4% level of confidence.

2nd test: 16.7% increase in revenue with a 56.4% level of confidence.

Oddly, mobile donations through the treatment increased by 58.7% in the first test but saw a 77.5% decrease the second test. The second test did have a significantly smaller sample size, however.

It would be wise to test this again for a future high urgency campaign. An additional hypothesis would be that a more attainable goal would help produce an increase in donations. The goal was not even halfway reached for this campaign, which may have spoiled donor motivation because they felt their gift would not make a big enough dent towards accomplishing the giving goal. Revenue went up in both experiments, so it’d be interesting to see if it would go up the next time this experiment is run as well.

Experiment Documented by NextAfter

Question about experiment #145579

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.