How deploying a coaster premium affects donor conversion
Texas Public Policy Foundation
Experiment Summary
Timeframe: 05/08/2019 - 06/28/2019
Texas Public Policy Foundation ran a campaign to fund a legal case to defend John Yearwood against the federal government. They had a coaster premium emblazoned with Texas’ famous motto “Come and Take It” that they had used in direct mail, and wanted to test it on the donation page to see if it would increase conversion. They created a simple treatment that added an image of the coaster to the page and split traffic to determine a winner.
Research Question
Will the presence of a premium incentivize more people to donate?
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 1.4% | ||
| T1: | Coaster | 0.73% | -48.8% | 79.1% |
This experiment has a required sample size of 1,701 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
After more than two months, the coaster premium showed a directional decline in donations, though it wasn’t statistically valid and did not reach a valid sample size. Since this wasn’t able to move the needle, TPPF decided to go back to the drawing board and find a new test to deliver a radical result.
Question about experiment #16759
If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.