How adding an image of an offer in an email affects click rate
Experiment ID: #16908
Hillsdale College
Experiment Summary
Ended On: 07/11/2019
Hillsdale College recently launched a new online course titled “The Second World Wars.” They are now offering this course as a DVD boxed set and are offering it as a premium offer to their housefile. They wanted to know whether or not adding an image of the DVD boxed set to the email would have any effect on click rate.
Research Question
Does adding an image of an offer in an email affect click rate to the donation page?
Design
C: Control
Array
(
[0] => Array
(
[ID] => 16472
[id] => 16472
[title] => Control-97-2.jpg
[filename] => Control-97-2.jpg
[filesize] => 0
[url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2.jpg
[link] => https://www.nextafter.com/experiments/how-adding-an-image-of-an-offer-in-an-email-affects-click-rate/attachment/control-97-2-jpg/
[alt] =>
[author] => 1
[description] =>
[caption] =>
[name] => control-97-2-jpg
[status] => inherit
[uploaded_to] => 16471
[date] => 2021-02-25 11:35:48
[modified] => 2021-02-25 11:35:48
[menu_order] => 0
[mime_type] => image/jpeg
[type] => image
[subtype] => jpeg
[icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png
[width] => 1434
[height] => 2454
[sizes] => Array
(
[thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-150x150.jpg
[thumbnail-width] => 150
[thumbnail-height] => 150
[medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-175x300.jpg
[medium-width] => 175
[medium-height] => 300
[medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-768x1314.jpg
[medium_large-width] => 640
[medium_large-height] => 1095
[large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-598x1024.jpg
[large-width] => 598
[large-height] => 1024
[1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-898x1536.jpg
[1536x1536-width] => 898
[1536x1536-height] => 1536
[2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-1197x2048.jpg
[2048x2048-width] => 1197
[2048x2048-height] => 2048
[ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-600x400.jpg
[ab-block-post-grid-landscape-width] => 600
[ab-block-post-grid-landscape-height] => 400
[ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-600x600.jpg
[ab-block-post-grid-square-width] => 600
[ab-block-post-grid-square-height] => 600
[tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2.jpg
[tp-image-grid-width] => 409
[tp-image-grid-height] => 700
[post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-150x150.jpg
[post-thumbnail-width] => 150
[post-thumbnail-height] => 150
[admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-29x50.jpg
[admin-width] => 29
[admin-height] => 50
[full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2.jpg
[full-width] => 1434
[full-height] => 2454
[thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-372x637.jpg
[thumb_372-width] => 372
[thumb_372-height] => 637
[single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-365x625.jpg
[single_post_thumbnail-width] => 365
[single_post_thumbnail-height] => 625
[thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-19x32.jpg
[thumb_32_32-width] => 19
[thumb_32_32-height] => 32
[thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-28x48.jpg
[thumb_48_48-width] => 28
[thumb_48_48-height] => 48
[thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-37x64.jpg
[thumb_64_64-width] => 37
[thumb_64_64-height] => 64
[thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-75x128.jpg
[thumb_128_128-width] => 75
[thumb_128_128-height] => 128
[thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-39x67.jpg
[thumb_122_67-width] => 39
[thumb_122_67-height] => 67
[thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-89x153.jpg
[thumb_153_153-width] => 89
[thumb_153_153-height] => 153
[thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-95x163.jpg
[thumb_130_163-width] => 95
[thumb_130_163-height] => 163
[thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-103x177.jpg
[thumb_232_177-width] => 103
[thumb_232_177-height] => 177
[thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-380x650.jpg
[thumb_272_325-width] => 380
[thumb_272_325-height] => 650
[thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-202x346.jpg
[thumb_271_346-width] => 202
[thumb_271_346-height] => 346
[thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-158x271.jpg
[thumb_271_271-width] => 158
[thumb_271_271-height] => 271
[thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-101x172.jpg
[thumb_272_172-width] => 101
[thumb_272_172-height] => 172
[thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-269x461.jpg
[thumb_372_461-width] => 269
[thumb_372_461-height] => 461
[thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-292x500.jpg
[thumb_372_500-width] => 292
[thumb_372_500-height] => 500
[thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-158x270.jpg
[thumb_385_270-width] => 158
[thumb_385_270-height] => 270
[thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-141x241.jpg
[thumb_416_241-width] => 141
[thumb_416_241-height] => 241
[thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-240x410.jpg
[thumb_448_410-width] => 240
[thumb_448_410-height] => 410
[thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-479x820.jpg
[thumb_448_410_retina-width] => 479
[thumb_448_410_retina-height] => 820
[thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-223x382.jpg
[thumb_574_382-width] => 223
[thumb_574_382-height] => 382
[thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Control-97-2-231x395.jpg
[thumb_776_395-width] => 231
[thumb_776_395-height] => 395
) ) )
Results
| Treatment Name | Click Rate | Relative Difference | Confidence | |
|---|---|---|---|---|
| C: | Control | 0.95% | ||
| T1: | Treatment -- Image of DVD | 0.99% | 4.0% | 67.6% |
This experiment has a required sample size of 509,079 in order to be valid. Unfortunately, the required sample size was not met and a level of confidence above 95% was not met so the experiment results are not valid.
Key Learnings
There was a 4% lift in the treatment version with the image of the DVD. The confidence level of this experiment was not at or above the 98th percentile, so this experiment shows no change in results. Even with a larger sample size, this test is not likely to validate. This is because there is such a small difference in click rate that a larger sample size will not determine any valid conclusion. This image test can be run instead on the landing page where people actually convert. We hypothesize that people like to see visuals with text as it helps them better understand what is being asked of them/offered to them.