NextAfter

How an ultra-personal, interruptive ad format affects email acquisition and CPA

NextAfter

Experiment Summary

Ended On: 11/03/2023

For our content offers aimed at name acquisition, we’ve traditionally had a difficult time finding a treatment that outperformed a simple 3D render of the resource, especially in terms of CPA — despite the widely parroted best practice of putting value prop copy on the image.

Our hypothesis has always been that the simple presentation of a 3D render on a plain white background worked for 2 primary reasons:

The simple design was more authentic than a heavily-designed ad
The simple design, in its simplicity, stood out from the crowd of busier, more colorful ads and posts

But still, to be convinced of the value of the offer, the prospect had either to read the ad’s “primary text” or click through to the landing page.

And we continued to suspect that if there was a way to prominently feature the value of an offer on the ad image itself, where a user’s attention is naturally drawn, without sacrificing the authentic, human appeal of the 3D render, it may be able to grab the prospect’s attention, in a unique way, and more effectively.

For this test, we pitted our usual control — a 3D render of the offer as the ad image with value proposition copy in the “primary text” field — against a simple screen grab of a note we created using the Apple Note app on our phone, which communicated the value of the offer, along with common, everyday language used for the primary text.

Because we were testing a new concept, we changed both the ad image and primary text but made sure both fit into our overall hypothesis. We split traffic between the control and the treatment and let the test run until it reached statistical validity and acquired more than 50 conversions.

Research Question

We believe that presenting an ultra-personal format in an unfamiliar context will produce higher acquisition rates and a lower CPA by using incongruence to “jar the prospect to attention” and entice them to engage with the offer.

Design

C: Control

T1: Treatment #1

Results

	Treatment Name	Conv. Rate	Relative Difference	Confidence
C:	Control	0.26%
T1:	Treatment #1	0.66%	151.0%	99.8%

This experiment has a required sample size of 2,299 in order to be valid. Since the experiment had a total sample size of 11,280, and the level of confidence is above 95% the experiment results are valid.

Flux Metrics Affected

The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:

0% increase in traffic
× 151.0% increase in conversion rate
× 0% increase in average gift

Key Learnings

The key learning from this experiment is that presenting everyday elements in uncommon contexts can significantly improve acquisition rates and lower CPA.

The treatment used unexpected juxtaposition to catch the prospect’s attention, resulting in a 106.6% increase in email acquisition for all traffic compared to the control. But that’s not all. Here are the highlights:

107% increase in email acquisition — 39 v. 17
86.49% increase in CTR — 64.9% v. 34.8%
87.3% increase in outbound clicks — 103 v. 55
56.4% decrease in CPA — $9.68 v. $22.19
46.5% decrease in cost-per-click — $3.69 v. $6.90

This suggests that by using a creative approach to funnel design, one that subtly blends the ordinary with the unexpected, we can entice prospects to engage by creating an experience beyond just “showing them an ad.”

The key to the success of this concept, we believe, is presenting everyday design elements and language outside of their expected context, using day-to-day, non-promotional language to stoke curiosity and build momentum through to the landing page.

In future experimentation, we should continue exploring the surprising juxtapositions of personal elements. It would be interesting to test this concept throughout the entire funnel.

And while we are intrigued and inspired to keep testing this concept, we are also aware of the potential for “fatigue” with the specific application we used in this test. Therefore, while we encourage the “note app” treatment for low-investment to test the concept, and will try it with a few different offers, we believe we will eventually need to test this concept using different tactics and applications.

Experiment Documented by Patrick Kitchen

Question about experiment #56986

If you have any questions about this experiment or would like additional details not discussed above, please feel free to contact them directly.