How a delayed pop-up affects email acquisition rate
Experiment ID: #93149
Scouts
Experiment Summary
Timeframe: 05/02/2022 - 05/22/2022
We built a National Scouting survey pop-up modal to live on the BSA homepage to drive survey completions. Internal teams were concerned with site UX and asked that we set pop-up to fire after the user was on the page for 15 seconds. We believed a shorter duration of 5 seconds would produce a better result and so we created a split test for half of the users to see a pop-up at 15 seconds and the other half to receive it after 5 seconds had elapsed.
Research Question
We believe that triggering a pop-up in a shorter duration for site visitors will achieve a better survey completion/email capture rate.
Design
C: Pop-up (5 secs)
Array
(
[0] => Array
(
[ID] => 57728
[id] => 57728
[title] => Pop-up_(5_secs).png
[filename] => Pop-up_5_secs-2.png
[filesize] => 722940
[url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2.png
[link] => https://www.nextafter.com/experiments/how-a-delayed-pop-up-affects-email-acquisition-rate/attachment/pop-up_5_secs-png-30/
[alt] =>
[author] => 0
[description] =>
[caption] =>
[name] => pop-up_5_secs-png-30
[status] => inherit
[uploaded_to] => 29435
[date] => 2024-02-08 23:28:12
[modified] => 2024-02-08 23:28:12
[menu_order] => 0
[mime_type] => image/png
[type] => image
[subtype] => png
[icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png
[width] => 2048
[height] => 965
[sizes] => Array
(
[thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-150x150.png
[thumbnail-width] => 150
[thumbnail-height] => 150
[medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-300x141.png
[medium-width] => 300
[medium-height] => 141
[medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-768x362.png
[medium_large-width] => 640
[medium_large-height] => 302
[large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-1024x483.png
[large-width] => 640
[large-height] => 302
[1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-1536x724.png
[1536x1536-width] => 1536
[1536x1536-height] => 724
[2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2.png
[2048x2048-width] => 2048
[2048x2048-height] => 965
[ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-600x400.png
[ab-block-post-grid-landscape-width] => 600
[ab-block-post-grid-landscape-height] => 400
[ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-600x600.png
[ab-block-post-grid-square-width] => 600
[ab-block-post-grid-square-height] => 600
[tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2.png
[tp-image-grid-width] => 700
[tp-image-grid-height] => 330
[post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-150x150.png
[post-thumbnail-width] => 150
[post-thumbnail-height] => 150
[admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-50x24.png
[admin-width] => 50
[admin-height] => 24
[full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2.png
[full-width] => 2048
[full-height] => 965
[thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-372x175.png
[thumb_372-width] => 372
[thumb_372-height] => 175
[single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-980x462.png
[single_post_thumbnail-width] => 980
[single_post_thumbnail-height] => 462
[thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-32x15.png
[thumb_32_32-width] => 32
[thumb_32_32-height] => 15
[thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-48x23.png
[thumb_48_48-width] => 48
[thumb_48_48-height] => 23
[thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-64x30.png
[thumb_64_64-width] => 64
[thumb_64_64-height] => 30
[thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-128x60.png
[thumb_128_128-width] => 128
[thumb_128_128-height] => 60
[thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-122x57.png
[thumb_122_67-width] => 122
[thumb_122_67-height] => 57
[thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-153x72.png
[thumb_153_153-width] => 153
[thumb_153_153-height] => 72
[thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-130x61.png
[thumb_130_163-width] => 130
[thumb_130_163-height] => 61
[thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-232x109.png
[thumb_232_177-width] => 232
[thumb_232_177-height] => 109
[thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-544x256.png
[thumb_272_325-width] => 544
[thumb_272_325-height] => 256
[thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-271x128.png
[thumb_271_346-width] => 271
[thumb_271_346-height] => 128
[thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-271x128.png
[thumb_271_271-width] => 271
[thumb_271_271-height] => 128
[thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-272x128.png
[thumb_272_172-width] => 272
[thumb_272_172-height] => 128
[thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-372x175.png
[thumb_372_461-width] => 372
[thumb_372_461-height] => 175
[thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-372x175.png
[thumb_372_500-width] => 372
[thumb_372_500-height] => 175
[thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-385x181.png
[thumb_385_270-width] => 385
[thumb_385_270-height] => 181
[thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-416x196.png
[thumb_416_241-width] => 416
[thumb_416_241-height] => 196
[thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-448x211.png
[thumb_448_410-width] => 448
[thumb_448_410-height] => 211
[thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-896x422.png
[thumb_448_410_retina-width] => 896
[thumb_448_410_retina-height] => 422
[thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-574x270.png
[thumb_574_382-width] => 574
[thumb_574_382-height] => 270
[thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-776x366.png
[thumb_776_395-width] => 776
[thumb_776_395-height] => 366
) ) [1] => Array
(
[ID] => 57729
[id] => 57729
[title] => Pop-up_(15_secs).png
[filename] => Pop-up_15_secs-2.png
[filesize] => 722940
[url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2.png
[link] => https://www.nextafter.com/experiments/how-a-delayed-pop-up-affects-email-acquisition-rate/attachment/pop-up_15_secs-png-30/
[alt] =>
[author] => 0
[description] =>
[caption] =>
[name] => pop-up_15_secs-png-30
[status] => inherit
[uploaded_to] => 29435
[date] => 2024-02-08 23:28:18
[modified] => 2024-02-08 23:28:18
[menu_order] => 0
[mime_type] => image/png
[type] => image
[subtype] => png
[icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png
[width] => 2048
[height] => 965
[sizes] => Array
(
[thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-150x150.png
[thumbnail-width] => 150
[thumbnail-height] => 150
[medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-300x141.png
[medium-width] => 300
[medium-height] => 141
[medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-768x362.png
[medium_large-width] => 640
[medium_large-height] => 302
[large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-1024x483.png
[large-width] => 640
[large-height] => 302
[1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-1536x724.png
[1536x1536-width] => 1536
[1536x1536-height] => 724
[2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2.png
[2048x2048-width] => 2048
[2048x2048-height] => 965
[ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-600x400.png
[ab-block-post-grid-landscape-width] => 600
[ab-block-post-grid-landscape-height] => 400
[ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-600x600.png
[ab-block-post-grid-square-width] => 600
[ab-block-post-grid-square-height] => 600
[tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2.png
[tp-image-grid-width] => 700
[tp-image-grid-height] => 330
[post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-150x150.png
[post-thumbnail-width] => 150
[post-thumbnail-height] => 150
[admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-50x24.png
[admin-width] => 50
[admin-height] => 24
[full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2.png
[full-width] => 2048
[full-height] => 965
[thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-372x175.png
[thumb_372-width] => 372
[thumb_372-height] => 175
[single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-980x462.png
[single_post_thumbnail-width] => 980
[single_post_thumbnail-height] => 462
[thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-32x15.png
[thumb_32_32-width] => 32
[thumb_32_32-height] => 15
[thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-48x23.png
[thumb_48_48-width] => 48
[thumb_48_48-height] => 23
[thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-64x30.png
[thumb_64_64-width] => 64
[thumb_64_64-height] => 30
[thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-128x60.png
[thumb_128_128-width] => 128
[thumb_128_128-height] => 60
[thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-122x57.png
[thumb_122_67-width] => 122
[thumb_122_67-height] => 57
[thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-153x72.png
[thumb_153_153-width] => 153
[thumb_153_153-height] => 72
[thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-130x61.png
[thumb_130_163-width] => 130
[thumb_130_163-height] => 61
[thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-232x109.png
[thumb_232_177-width] => 232
[thumb_232_177-height] => 109
[thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-544x256.png
[thumb_272_325-width] => 544
[thumb_272_325-height] => 256
[thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-271x128.png
[thumb_271_346-width] => 271
[thumb_271_346-height] => 128
[thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-271x128.png
[thumb_271_271-width] => 271
[thumb_271_271-height] => 128
[thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-272x128.png
[thumb_272_172-width] => 272
[thumb_272_172-height] => 128
[thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-372x175.png
[thumb_372_461-width] => 372
[thumb_372_461-height] => 175
[thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-372x175.png
[thumb_372_500-width] => 372
[thumb_372_500-height] => 175
[thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-385x181.png
[thumb_385_270-width] => 385
[thumb_385_270-height] => 181
[thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-416x196.png
[thumb_416_241-width] => 416
[thumb_416_241-height] => 196
[thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-448x211.png
[thumb_448_410-width] => 448
[thumb_448_410-height] => 211
[thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-896x422.png
[thumb_448_410_retina-width] => 896
[thumb_448_410_retina-height] => 422
[thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-574x270.png
[thumb_574_382-width] => 574
[thumb_574_382-height] => 270
[thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-776x366.png
[thumb_776_395-width] => 776
[thumb_776_395-height] => 366
) ) )
T1: Pop-up (15 secs)
Array
(
[0] => Array
(
[ID] => 57728
[id] => 57728
[title] => Pop-up_(5_secs).png
[filename] => Pop-up_5_secs-2.png
[filesize] => 722940
[url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2.png
[link] => https://www.nextafter.com/experiments/how-a-delayed-pop-up-affects-email-acquisition-rate/attachment/pop-up_5_secs-png-30/
[alt] =>
[author] => 0
[description] =>
[caption] =>
[name] => pop-up_5_secs-png-30
[status] => inherit
[uploaded_to] => 29435
[date] => 2024-02-08 23:28:12
[modified] => 2024-02-08 23:28:12
[menu_order] => 0
[mime_type] => image/png
[type] => image
[subtype] => png
[icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png
[width] => 2048
[height] => 965
[sizes] => Array
(
[thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-150x150.png
[thumbnail-width] => 150
[thumbnail-height] => 150
[medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-300x141.png
[medium-width] => 300
[medium-height] => 141
[medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-768x362.png
[medium_large-width] => 640
[medium_large-height] => 302
[large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-1024x483.png
[large-width] => 640
[large-height] => 302
[1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-1536x724.png
[1536x1536-width] => 1536
[1536x1536-height] => 724
[2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2.png
[2048x2048-width] => 2048
[2048x2048-height] => 965
[ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-600x400.png
[ab-block-post-grid-landscape-width] => 600
[ab-block-post-grid-landscape-height] => 400
[ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-600x600.png
[ab-block-post-grid-square-width] => 600
[ab-block-post-grid-square-height] => 600
[tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2.png
[tp-image-grid-width] => 700
[tp-image-grid-height] => 330
[post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-150x150.png
[post-thumbnail-width] => 150
[post-thumbnail-height] => 150
[admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-50x24.png
[admin-width] => 50
[admin-height] => 24
[full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2.png
[full-width] => 2048
[full-height] => 965
[thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-372x175.png
[thumb_372-width] => 372
[thumb_372-height] => 175
[single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-980x462.png
[single_post_thumbnail-width] => 980
[single_post_thumbnail-height] => 462
[thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-32x15.png
[thumb_32_32-width] => 32
[thumb_32_32-height] => 15
[thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-48x23.png
[thumb_48_48-width] => 48
[thumb_48_48-height] => 23
[thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-64x30.png
[thumb_64_64-width] => 64
[thumb_64_64-height] => 30
[thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-128x60.png
[thumb_128_128-width] => 128
[thumb_128_128-height] => 60
[thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-122x57.png
[thumb_122_67-width] => 122
[thumb_122_67-height] => 57
[thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-153x72.png
[thumb_153_153-width] => 153
[thumb_153_153-height] => 72
[thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-130x61.png
[thumb_130_163-width] => 130
[thumb_130_163-height] => 61
[thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-232x109.png
[thumb_232_177-width] => 232
[thumb_232_177-height] => 109
[thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-544x256.png
[thumb_272_325-width] => 544
[thumb_272_325-height] => 256
[thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-271x128.png
[thumb_271_346-width] => 271
[thumb_271_346-height] => 128
[thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-271x128.png
[thumb_271_271-width] => 271
[thumb_271_271-height] => 128
[thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-272x128.png
[thumb_272_172-width] => 272
[thumb_272_172-height] => 128
[thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-372x175.png
[thumb_372_461-width] => 372
[thumb_372_461-height] => 175
[thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-372x175.png
[thumb_372_500-width] => 372
[thumb_372_500-height] => 175
[thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-385x181.png
[thumb_385_270-width] => 385
[thumb_385_270-height] => 181
[thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-416x196.png
[thumb_416_241-width] => 416
[thumb_416_241-height] => 196
[thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-448x211.png
[thumb_448_410-width] => 448
[thumb_448_410-height] => 211
[thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-896x422.png
[thumb_448_410_retina-width] => 896
[thumb_448_410_retina-height] => 422
[thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-574x270.png
[thumb_574_382-width] => 574
[thumb_574_382-height] => 270
[thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_5_secs-2-776x366.png
[thumb_776_395-width] => 776
[thumb_776_395-height] => 366
) ) [1] => Array
(
[ID] => 57729
[id] => 57729
[title] => Pop-up_(15_secs).png
[filename] => Pop-up_15_secs-2.png
[filesize] => 722940
[url] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2.png
[link] => https://www.nextafter.com/experiments/how-a-delayed-pop-up-affects-email-acquisition-rate/attachment/pop-up_15_secs-png-30/
[alt] =>
[author] => 0
[description] =>
[caption] =>
[name] => pop-up_15_secs-png-30
[status] => inherit
[uploaded_to] => 29435
[date] => 2024-02-08 23:28:18
[modified] => 2024-02-08 23:28:18
[menu_order] => 0
[mime_type] => image/png
[type] => image
[subtype] => png
[icon] => https://nextafter-1a91a.kxcdn.com/wp-includes/images/media/default.png
[width] => 2048
[height] => 965
[sizes] => Array
(
[thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-150x150.png
[thumbnail-width] => 150
[thumbnail-height] => 150
[medium] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-300x141.png
[medium-width] => 300
[medium-height] => 141
[medium_large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-768x362.png
[medium_large-width] => 640
[medium_large-height] => 302
[large] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-1024x483.png
[large-width] => 640
[large-height] => 302
[1536x1536] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-1536x724.png
[1536x1536-width] => 1536
[1536x1536-height] => 724
[2048x2048] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2.png
[2048x2048-width] => 2048
[2048x2048-height] => 965
[ab-block-post-grid-landscape] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-600x400.png
[ab-block-post-grid-landscape-width] => 600
[ab-block-post-grid-landscape-height] => 400
[ab-block-post-grid-square] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-600x600.png
[ab-block-post-grid-square-width] => 600
[ab-block-post-grid-square-height] => 600
[tp-image-grid] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2.png
[tp-image-grid-width] => 700
[tp-image-grid-height] => 330
[post-thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-150x150.png
[post-thumbnail-width] => 150
[post-thumbnail-height] => 150
[admin] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-50x24.png
[admin-width] => 50
[admin-height] => 24
[full] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2.png
[full-width] => 2048
[full-height] => 965
[thumb_372] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-372x175.png
[thumb_372-width] => 372
[thumb_372-height] => 175
[single_post_thumbnail] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-980x462.png
[single_post_thumbnail-width] => 980
[single_post_thumbnail-height] => 462
[thumb_32_32] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-32x15.png
[thumb_32_32-width] => 32
[thumb_32_32-height] => 15
[thumb_48_48] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-48x23.png
[thumb_48_48-width] => 48
[thumb_48_48-height] => 23
[thumb_64_64] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-64x30.png
[thumb_64_64-width] => 64
[thumb_64_64-height] => 30
[thumb_128_128] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-128x60.png
[thumb_128_128-width] => 128
[thumb_128_128-height] => 60
[thumb_122_67] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-122x57.png
[thumb_122_67-width] => 122
[thumb_122_67-height] => 57
[thumb_153_153] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-153x72.png
[thumb_153_153-width] => 153
[thumb_153_153-height] => 72
[thumb_130_163] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-130x61.png
[thumb_130_163-width] => 130
[thumb_130_163-height] => 61
[thumb_232_177] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-232x109.png
[thumb_232_177-width] => 232
[thumb_232_177-height] => 109
[thumb_272_325] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-544x256.png
[thumb_272_325-width] => 544
[thumb_272_325-height] => 256
[thumb_271_346] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-271x128.png
[thumb_271_346-width] => 271
[thumb_271_346-height] => 128
[thumb_271_271] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-271x128.png
[thumb_271_271-width] => 271
[thumb_271_271-height] => 128
[thumb_272_172] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-272x128.png
[thumb_272_172-width] => 272
[thumb_272_172-height] => 128
[thumb_372_461] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-372x175.png
[thumb_372_461-width] => 372
[thumb_372_461-height] => 175
[thumb_372_500] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-372x175.png
[thumb_372_500-width] => 372
[thumb_372_500-height] => 175
[thumb_385_270] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-385x181.png
[thumb_385_270-width] => 385
[thumb_385_270-height] => 181
[thumb_416_241] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-416x196.png
[thumb_416_241-width] => 416
[thumb_416_241-height] => 196
[thumb_448_410] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-448x211.png
[thumb_448_410-width] => 448
[thumb_448_410-height] => 211
[thumb_448_410_retina] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-896x422.png
[thumb_448_410_retina-width] => 896
[thumb_448_410_retina-height] => 422
[thumb_574_382] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-574x270.png
[thumb_574_382-width] => 574
[thumb_574_382-height] => 270
[thumb_776_395] => https://nextafter-1a91a.kxcdn.com/wp-content/uploads/Pop-up_15_secs-2-776x366.png
[thumb_776_395-width] => 776
[thumb_776_395-height] => 366
) ) )
Results
| Treatment Name | Conv. Rate | Relative Difference | Confidence | Average Gift | |
|---|---|---|---|---|---|
| C: | Pop-up (5 secs) | 0.26% | $0.00 | ||
| T1: | Pop-up (15 secs) | 0.11% | -56.4% | 99.9% | $0.00 |
This experiment has a required sample size of 6,768 in order to be valid. Since the experiment had a total sample size of 41,336, and the level of confidence is above 95% the experiment results are valid.
Flux Metrics Affected
The Flux Metrics analyze the three primary metrics that affect revenue (traffic, conversion rate, and average gift). This experiment produced the following results:
0% increase in traffic
× 56.4% decrease in conversion rate
× 0% increase in average gift
Key Learnings
The 5 second pop-up clearly outperformed waiting 15 seconds to trigger the modal. The 5 second variant acquired 53 emails versus the 23 acquired by the 15 second version. The 15 second version underperformed the 5 second version by 56% with a 99.9% confidence rate, proving that the shorted trigger duration was clearly the stronger performing variant.
Hypothesis for this performance has more to do with time on page and were users even on the page for 15 seconds to see the longer triggered variant on the page or had they already navigated elsewhere within the BSA site.
Follow up test will be to shorten the pop-up trigger from 5 seconds to 2 seconds. Test to follow this one will test trigger time versus triggering pop-up upon page scroll.