Focus on the Family

How adding impact language to a direct-ask pop up banner during a high urgency campaign affects revenue – marriages saved V3 second relaunch

Experiment ID: #118781

Focus on the Family

Experiment Summary

Timeframe: 12/11/2022 - 12/31/2022

Focus on the Family (FOTF) is a global Christian ministry dedicated to helping families thrive. They provide help and resources for couples to build healthy marriages that reflect God’s design and for parents to raise their children according to morals and values grounded in biblical principles. Every year during the calendar year-end high-urgency campaign, FOTF places a top-of-page pop-up banner that contains a direct ask to support their mission. This banner (known internally as the “wiki-banner”) has been tested into over many years, with experiments to improve donation metrics. Since many people visit FOTF.com during this season to make a year-end gift, the “wiki-banner” serves as a low-friction tactic to get people to the donation page quickly. It is important to note that the visiting audience to FOTF’s website is served different content / experiences based on pages they have consumed in the past…specifically a “marriage audience” and an “in favor of pro-life audience – Santicity of Human Life” (also known as SOHL).

For this specific experiment targeting the “marriage audience”, we were focused on changing the language on the gift array buttons to reflect the impact that a gift would make on the amount of marriages saved. The theory was that if we could directly show how many marriages could be saved with a specific gift amount, the visitor would be moved to give a bigger gift to make a greater impact. As screen real-estate is at a premium for this “wiki-banner”, we designed an experience that showed the number of marriages saved on a button on the wiki-banner, and in a “hover-state” it would flip to the equivalent dollar amount of the gift. We even added a free-form field that would allow you to add how many marriages you wanted to save, and it would calculate that amount in real-time. After making a selection, the user is directed to the donation form where the amount they chose is auto-populated on the giving form, to reduce friction. It is also important to note that we were testing for revenue lift on desktop only (not mobile due to screen size limitations).

NOTE: This is the third version of the experiment (V3 relaunch) due to technical challenges of the V2 version. That write-up can be found here.

Research Question

We believe that changing the language on the gift array of a direct ask pop-up to show the impact of the gift will result in greater revenue

Design

C: Control

T1: Marriage Language

Results

	Treatment Name	Revenue per Visitor	Relative Difference	Confidence	Average Gift
C:	Control	$16.75			$578.93
T1:	Marriage Language	$15.23	-9.1%	81.6%	$533.40

This experiment was validated using 3rd party testing tools. Based upon those calculations, a significant level of confidence was not met so these experiment results are not valid.

Key Learnings

If you notice, this experiment is labeled V3 relaunch. We launched this one AFTER the V2 version that we ended on 12/11 due to internal FOTF technical challenges. This version ran from 12/11 to 12/31. This experiment was running in parallel with a similar experiment that was targeted to only the SOHL audience. Unlike the previous 2 versions of this experiment, the results showed a decrease in the revenue by around 9% with an 82% Level of Confidence.

While previous versions of this experiment showed a positive lift (even validating in the V1 version), this one showed a decrease, but not until the final days of the experiment. As we experienced so many technical challenges with launching this experiment, we kept a watchful eye on the results on a daily basis. From 12/11 through 12/29, the treatment was actually leading in revenue….but on the last 2 days of the year, the opposite actually happened.

This led us to even more questions like…

Does the timing of the experiment change the results?
Did donors in “non-peak” times get inspired to give more based on messaging?
Were the donors in the last 2 days motivated to give regardless of messaging?
Was the change in messaging more of a friction point in the last 2 days of the year?
Would it be better to change messaging in the wiki based on the urgency and timing of the campaign? (Ex., show more impactful language earlier in the campaign, then make the most urgent times as simple and frictionless as possible)

There were several key take-aways that came from this series of experiments.

Timing/motivation may play a big role in the effectiveness of impact language
Make sure you are always monitoring your experiments for data anomalies, and don’t be afraid to end an experiment early and re-launch if the data does not “look right”
We cannot say definitively that the impact language led to great revenue during the majority of the high-urgency campaign…but we can say that there is enough data to show a positive trend and learnings that we will take into our next high-urgency campaign experiment

Experiment Documented by NextAfter

Question about experiment #118781

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.