The Heritage Foundation

How adding a flag icon to a donation form tab affects recurring donation rate

Experiment ID: #180240

The Heritage Foundation

Founded in 1973, The Heritage Foundation is a research and educational institution—a think tank—whose mission is to formulate and promote conservative public policies based on the principles of free enterprise, limited government, individual freedom, traditional American values, and a strong national defense.

Experiment Summary

Timeframe: 05/30/2024 - 06/11/2024

The Heritage Foundation had previously run a test on their membership renewal page where they added a small flag image to a tabbed donation form’s monthly gift tab in order to appeal to the patriotism of their donors, which led to an increase in donations.

Wondering whether this concept would appeal to donors on another direct appeal donation page, they also tested this flag image in the donation form’s $100 gift array button and in the form submit button, which led to a decrease in donations at a near-valid level of confidence.

As a follow-up experiment on the direct appeal donation page, they wanted to test whether the flag icon in the monthly gift tab and the form submit button would lead to an increase in donations, namely recurring donations.

They launched an A/B test where the control had no flag icon anywhere on the form, and the treatment had the flag icon in the monthly gift tab and the form submit button.

Research Question

We believe that using a small American flag in a donation form’s monthly gift tab and submit button to build affinity for direct appeal prospects will achieve an increase in conversion rate because the flag image will help align their motivation with the intended action.

Design

C: Control

T1: Flag Icon

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence	Average Gift
C:	Control	0.00%
T1:	Flag Icon	0.17%	100.0%	68.3%

This experiment has a required sample size of 2,332 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

The key learning from this experiment is that adding a small American flag image to a donation form’s monthly gift tab and submit button did not lead to a valid increase in donations, specifically recurring donations. In fact, the treatment group saw a 40.3% decrease in transactions for all traffic compared to the control group. However, it is important to note that the statistical confidence level of 78.8% is below the valid threshold of 95%, indicating that these results may not be reliable.

The treatment also resulted in just a 0.17% increase in recurring donations at a 68.3% level of confidence. With such a small sample size, these results cannot be relied upon to inform future action.

In future experimentation, it is crucial to ensure that the sample size is adequate to achieve a statistically valid result. Additionally, it is important to carefully consider the impact of visual elements such as flag icons on donor behavior. Other ideas for follow-up testing include testing a different icon than an American flag or testing this concept on a donation page following a content offer.

Experiment Documented by NextAfter

Question about experiment #180240

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.