Alley Cat Allies

How explaining the root causes of an issue affects donor conversion rate

Experiment ID: #23694

Alley Cat Allies

Experiment Summary

Timeframe: 10/10/2019 - 10/21/2019

Alley Cat Allies, as a part of their annual Global Cat Day campaign, asked for supporters and prospects to pledge to be advocates for all cats everywhere, but especially those that do not have people to defend them today. This initial step helped align the audience around a common goal while also collecting contact information for prospective donors.

After supporters signed the pledge, they were taken to an instant donation page that asked them to take the next step towards saving cats lives.

In writing the donation page, we wondered if explaining the root causes for why things need to change was important for us to drive donor conversion rate, or revenue.

Research Question

Does explaining the “root cause” increase donor conversion rates (or revenue) as a part of the Global Cat Day campaign?

Design

C: GCD - Long Form

T1: GCD - Short Form

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence	Average Gift
C:	GCD - Long Form	5.8%			$0.00
T1:	GCD - Short Form	6.7%	14.2%	95.9%	$0.00

This experiment has a required sample size of 6,536 in order to be valid. Since the experiment had a total sample size of 14,185, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

0% increase in traffic
× 14.2% increase in conversion rate
× 0% increase in average gift

Key Learnings

Findings:

With a 95.8% level of confidence, we observed a +14.2% increase in donor conversion rate with the version of the donation page that did not explain the root causes.

Although we did not validate this portion of the experiment, we also observed that the revenue also increased by +17.5% (with a level of confidence of 93.4%).

Key Learnings:

This experiment implies that over-explaining root causes can distract donors from what they really want to do with their gift… which is to solve the problem (or create the impact).

When splitting the experiment data between NEW VISITORS and RETURNING VISITORS, we made a few observations, as well.

For NEW VISITORS, we observed an even greater lift overall (+18.1% in donor conversion rate with a 95.6% level of confidence). This is where most of the data that was used to validate the overall experiment was produced. This is interesting, because NEW VISITORS are most likely to be unaware of what the root causes to this problem were when compared to returning visitors (who have already visited the website), logic might lead you to believe that they needed that information to make a gift.

For them to be more responsive to the shorter version (that did not explain the root causes), is very telling; it proves that they (the uninitiated to the problem’s root causes) didn’t need that information to make a gift.

For RETURNING VISITORS, we observed a directional increase with the shorter copy of just +4.6% in donor conversion rate, but the experiment did not validate, as it produced only a 33% level of confidence.

What this means is that the copy used to describe the root cause issues was also (at least directionally) less effective than focusing these returning visitors on the outcomes (instead of spending time to tell them what caused the problem in the first place). However, since it did not validate, further experimentation may be required moving forward to verify this learning with returning visitors.

Conclusion:

Overall, the key learning from this experiment appears to be that we do not need to spend too much time (or copy) on the donation page during a campaign like this explaining why the problem exists, but rather focus the audience on what they can do to solve the problem(s) we’re talking about.

Experiment Documented by Greg Colunga

Greg Colunga is Chief Operating Office at NextAfter.

Question about experiment #23694

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.