KCBI

How do campaign emails focused on specific outreaches compare to general campaign emails

Experiment ID: #59242

KCBI

Experiment Summary

Timeframe: 04/21/2021 - 04/29/2021

In preparation for KCBI’s Spring Campaign, we were curious if listing all of the outreaches the an organization is fundraising for clouds the impact and value proposition for each individual outreach. In other words, does talking about keeping radio on the air, saving lives with ultrasounds, and bringing Bibles to prisoners all in one email lessen the value prop and so the donor response? Would it be better to talk about each individual outreach in an email bring more depth the the value of the individual outreaches and so improve donor response. So we tested this theory by splitting their housefile into two groups. One group got 4 emails that talked through all of the outreaches in all four emails. The other group got 4 emails that focused on each outreach individually (i.e. one email was focused on ultrasounds, one email was focused on prison ministries).

Research Question

Focusing on specific outreaches individually will increase the donor conversion.

Design

C: Control

T1: Treatment #1

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence
C:	Control	0.49%
T1:	Treatment #1	0.41%	-16.5%	54.4%

This experiment has a required sample size of 52,347 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.

Key Learnings

The outcome of this test was inconclusive, but we have a few hypotheses as to why.

First, it is important to note that there was no significant increase in the click rate on the different emails. There was a 13% increase at a 66% Level of Confidence, so nothing concrete or indicative of higher motivation either way.

Our hypothesis is that there was not enough distinction in the messages. Many of the test emails replace a paragraph with outreach specific message. If we were to retest this, we would make the emails more focused on the outreaches and make them more central to the message.

However, as we looked in Google Analytics, we found the level of traffic to the donation page was the same for the test and the control. Meaning the same number of unique people went to the donation page from both segments of the test. But, the conversion rate for the people who came from the control is higher. The donation page mentioned all of the outreaches, similar to the control emails. So people coming from the control would experience continuity of message. Whereas people coming from the test emails would then be presented with a donation page that lists all of the outreaches. Our hypothesis is that this created cognitive dissonance in the people in the test segment.

So if we test further, we would make donation pages specific to the outreach itself to take the test through the entire funnel.

Experiment Documented by NextAfter

Question about experiment #59242

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.