Americans for Prosperity

How simplifying the reading level of an open letter impacted downstream revenue

Experiment ID: #21128

Americans for Prosperity

Experiment Summary

Ended On: 05/19/2020

As a part of the COVID-19 (Coronavirus) response efforts, American’s For Prosperity made available to concerned Americans an opportunity to add their name to an open letter to Congress as they planned a government response to provide citizens aid, as well as focused stimulus dollars to “restart the economy” again after the pandemic.

The open letter had become a wonderful engagement piece for the housefile, so they decided to open it up to prospective supporters to give them the opportunity to add their name alongside tens of thousands of other signers, as well.

Because the open letter was so popular, they decided to put a donation appeal behind it and test acquiring new donors, and did so very effectively.

But as time went on, the open letter signup (or email capture) rates were beginning to decline, so they wanted to experiment with increasing email signups.

The original letter was carefully crafted by government affair and policy experts and when graded in a readability tool, it was projected to be at a 15th grade reading level. If the 5th grade reading level they tried previously was determined to be “oversimplified,” they then thought to try a 12th grade reading level in an attempt to bring it closer to the original letter contents.

Research Question

Would simplifying the reading level of the open letter increase email signup, donor conversion or revenue?

Design

C: 15th Grade Reading Level

T1: 12th Grade Reading Level

Results

	Treatment Name	Revenue per Visitor	Relative Difference	Confidence	Average Gift
C:	15th Grade Reading Level	$0.17			$21.64
T1:	12th Grade Reading Level	$0.11	-32.5%	92.7%	$18.84

This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was not met so these experiment results are not valid.

Key Learnings

We observed that with a 92.7% level of confidence, that this “simplified readability” open letter decreased revenue by -32.5%.

This decrease in revenue was driven by two factors:

A directionally valid 24% decrease in donor conversion rate (LoC: 80%), and
A 13% lower average gift for the fewer number of donors that gave through the treatment experience.

It’s worth noting that we did observe a fully validated increase in email signup rate on the simplified open letter by +12.7% (LoC: 100%). This is particularly interesting, because although it did facilitate more email signups, that greater volume of people moving through to the donation page were NOT as motivated as those coming through the higher grade reading level in the control experience.

This experiment’s findings seem to validate the original hypothesis, which was that the increased reading speed works against the psychological approach and learnings gathered by adding the mini checkboxes—in that we found when we can slow someone down as they move through the funnel, that they become more likely to read and consume the content within the open letter. And when they do that, they are more likely to convert as a donor on the subsequent instant donation page because they fully understand and align with the contents designed to grow their motivation.

Experiment Documented by Greg Colunga

Greg Colunga is Chief Operating Office at NextAfter.

Question about experiment #21128

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.