Canadian Taxpayers Federation

How simplifying petition language impacted revenue

Experiment ID: #30182

Canadian Taxpayers Federation

Experiment Summary

Ended On: 04/23/2020

The Canadian Taxpayers Federation (CTF) is a federally incorporated, not-for-profit citizen’s group dedicated to lower taxes, less waste and accountable government.

As a part of their efforts to urge the Canadian federal government in their efforts to provide relief to Canadian citizens in response to COVID-19 (coronavirus), they put together an open letter campaign that allowed Canadian citizens to urge their lawmakers for fiscal responsibility in response to the pandemic.

The open letter was followed by an instant donation page which provided open letter signers to make a gift to help reach more people to sign the open letter.

Research Question

Would simplifying the contents of the open letter (by decreasing the evaluated “reading grade level” from 10th grade down to a 6th grade reading level) impact letter signature rates, donor conversion or revenue?

Design

C: Control - 10th Grade Reading Level

T1: Treatment - 6th Grade Reading Level

Results

	Treatment Name	Revenue per Visitor	Relative Difference	Confidence	Average Gift
C:	Control - 10th Grade Reading Level	$0.23			$41.10
T1:	Treatment - 6th Grade Reading Level	$0.44	90.8%	95.3%	$65.60

This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was met so these experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

0% increase in traffic
× 19.5% increase in conversion rate
× 59.6% increase in average gift
90.8% increase in revenue

Key Learnings

With 95.2% level of confidence, we observed a +90.8% increase in revenue for open letter signers who became donors in the subsequent instant donation page.

This increase was driven by two metrics, specifically:

An increase in donor conversion rate for treatment signers by +22.5% (although the LoC was only 66.5% and did not validate), and
The average gift for donors in the treatment experience was +59.6% higher than the control donors.

Ironically, the email signup rate was rather unaffected by the treatment (improving only 0.7%, with a LoC of 25.8%).

Key Takeaways:

When you read the two treatments, the simplification of the letter content didn’t really “take anything away” from the content—in fact, the simplification made the letter appear more direct to the lawmaker it was being sent to. Our theory is that this increased “directness” of the message was a better representation of the reader/signer’s message, and when presented with an ask, they gave more generously on the instant donation page.

Because the organization was looking to maximize return on investment, the treatment was rolled out to 100% of the audience. Further experimentation may be required to verify if this approach could work again in the future.

Experiment Documented by Greg Colunga

Greg Colunga is Chief Operating Office at NextAfter.

Question about experiment #30182

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.