Resources Home

A/B testing is something that not a lot of nonprofits are doing well – but those that are running a/b tests are seeing major lifts in donations and revenue. So how exactly do you start setting up and running a/b tests at your nonprofit that lead to major lifts? I’m going to show you how in the A/B testing guide for nonprofits.

In the 8 steps below, you’ll learn exactly how to find where to test, what to test, and how to test. But before we get there, let’s look at why you need to be testing.

Why is a/b testing so important?

The answer here is actually really simple. Relying on your own intuition is no better than flipping a coin to determine which version is better.

But why should you take my word it? Let’s look at an example or two…

Which of these donation pages is going to bring in the most donations? The all text page or the page with a video?

If you’ve read more of our blogs than just this one, you probably know the answer already. But this is an area we get questioned on more often than almost anything else.

The correct answer?

The all text page saw a 560% increase in donations!

Without testing, we would have no idea. And even if you’re one of the few fundraisers that would have picked an all text page over a page with a video, would you have been willing to risk a 560% change in donations without testing it first? 

Let’s look at one more test that’s a little more nuanced.

Which email below brought in more donations? If you can’t read the text, just click on the image to pull it up full screen.

Honestly, I could make an argument for why either of these should win based all the fundraising “best practices” that are circulating out there.

Here’s the answer…

Email B increased donations by 360%!

Did you get that one right? Even if you did, were you 100% confident? Confident enough to risk a 360% change in donations?

Where should you start a/b testing?

The simplest answer is to ask yourself the question, “Where can I get the most return for my effort?”

But I bet if you asked that question even to your closest colleagues, you would get wildly different responses. So here’s what I would suggest…

Start testing areas of your fundraising that influence these 3 key metrics: web traffic, donations, and average gift.

These 3 metrics each have a direct impact on revenue. And if you’re a/b tests start improving revenue, it will be much easier to get others to care about what you’re doing.

If you want some specific test ideas to start with, check out these 5 common fundraising “best practices” that you should stop assuming work, and start testing new, proven strategies.

Ok. I could go on and on about where to start testing, but let’s get into the A/B testing guide. Here are the 8 key steps to launching an effective and valid nonprofit A/B test.

1. Identify Your Conversion Goal

First, you need to define the goal that you’re trying to accomplish. Without a clearly stated goal up front, you will never have a clear understanding of whether or not your test was successful.

Your conversion goal will give you the framework to design your a/b test and craft your hypothesis.

If you want to improve your donation page, your conversion goal might be the “total number of donations.”

If you want to traffic from a banner ad to a landing page, your conversion goal might be “clicks” or “landing page visits.”

If you want to improve your email newsletter form, your conversion goal might be “form submissions.”

Once you’ve identified the specific metric you’re hoping to improve, you can move on to step #2.

2. Make Sure You Can Measure Your Conversion Goal.

If you can’t track it, you can’t A/B test it. And if you can’t A/B test it, you can’t optimize it. And if you can’t optimize it, well…then you’re potentially leaving huge amounts of donations on the table as you saw in the examples above.

Google analytics is your essential tool for measuring your goals.

To measure your conversion goals, there is one thing you need to set up, and one thing you really, really should set up to get the most out of you’re a/b testing.

You need to set up conversion goals in Google Analytics.

Need some help setting up your Google Analytics goals? There’s a great post from Neil Patel on 4 types of Google Analytics goals. In short, you can set up 4 different types of goals based on:

• URLs
• Visit Duration
• Pages Per Visit
• Google Analytics Event

For almost all of the a/b testing you’ll start out with, you’re going to either use a URL goal (triggered when someone visits a specific URL like your donation form’s confirmation page) or a Google Analytics Event goal (triggered by an event that fires when a form is submitted).

You really, really should set up eCommerce tracking.

Now, I understand that eCommerce tracking is much harder to set up than a basic conversion goal. So if you can’t get it set up right away, that shouldn’t stop you from testing. But the more you run a/b tests, the more you’re going to want to track the actual revenue that’s resulting from your testing.

Why is eCommerce tracking so important?

eCommerce tracking will let you see exactly what you’re a/b tests are doing to your revenue. In some cases, you might be getting more clicks or visits to a landing page, but actually hurting your revenue. This may sound counter-intuitive, but it happens more often than you might think.

Here’s an a/b test where we an increase in email clicks, but a decrease in donations.

If you don’t measure revenue, your test could appear to be positive, but actually hurt you where it matters most.

3. Craft Your A/B Testing Hypothesis

Once you know exactly what you want to improve, and you know that you can measure your goal, you need to define your hypothesis.

A good hypothesis will address the specific idea that you can think can make an impact on your conversion goal.

Example Hypothesis: “Removing friction from the giving process by eliminating unnecessary form fields will increase donations.”

This hypothesis tells you the specific variables that you’re a/b test will look at. It makes it clear that your treatment or challenger page will have fewer form field than your control (original page).

In this example, your treatment might remove fields like “gift designation,” or “Make this gift in honor of…,” or other fields that are not absolutely essential to processing the donation.

Changing multiple variables at once

Some would argue that your hypothesis has to isolate one specific variable. If you change too much, you don’t really know what part of your test actually made an impact.

While this is true, your hypothesis can be crafted in a way that allows you test multiple elements at once – so long as they support the foundation of the hypothesis.

Example: A more personal email will lead to more donations.

This hypothesis addresses a specific idea, but it opens the door to change multiple elements in your email. Your control could be a heavily designed template, and your treatment can be a plain-text email with more personal copy. While multiple elements are changed, it all supports the underlying hypothesis.

Here’s an a/b test where multiple variables changed, yet the experiment remained sound.

Once you have your hypothesis created, write it down! You don’t want to forget why you ran the test, or what the over-arching idea was. You’ll want to keep your hypothesis so you can document what you’ve learned after the a/b test is complete.

4. Calculate Your Estimated Sample Size

Before you run your a/b test, you need to make sure that it’s possible to get a valid result.

To do so, you have to calculate your estimated sample size. All this means is that you need to figure out how many people need to see your a/b test in order to get a reliable result.

For instance, if your test increases donations by 50%, but only 20 people actually visited your donation page, it’s possible that this increase in donations was just the result of random chance.

There are some great tools out there to calculate exactly how many people need to see your a/b test in order to get a valid result. Here are a couple to choose from:

• Optimizely’s Sample Size Estimator (basic and self-explanatory)
• AB Test Calculator from Online Dialogue (a little more advanced, but super cool)

A Quick Walkthrough of Sample Size Calculation

If this is all new to you, here’s a quick explanation of how to use the Optimizely tool I listed above.

First, enter you Baseline Conversion Rate

This is the conversion rate that you would normally expect to see. If it’s a donation page, your conversion rate would be the number of donation divided by the number of visitors.

If your conversion goal is email clicks, your conversion rate would be the number of clicks divided by the number of emails sent.

Second, enter the Minimum Detectable Effect. This is the minimum amount of change you’d like to be able to measure. So if you hope to see a minimum of a 20% increase in donations as a result of your test, enter 20%.

Third, enter your desired level of Statistical Significance. We always recommend using 95% for this number. Statistical Significance is the likelihood that you’ll see this same result in the future.

For example, a 95% statistical significance essential means you’ll see the same result 95 out of 100 times. A 50% statistical significance is basically the equivalent of a coin toss – the result could go either way with equal odds.

After entering these 3 numbers, you’ll receive your Sample Size per Variation. This is the amount of traffic (or people) you need to see each version of your experiment. If you need 1,000 samples per variation, that means 1000 people need to see your control and 1000 people need to see your treatment.

Once you’ve calculated your needed sample size, you need to make sure you’re a/b test is actually capable of getting enough traffic.

If your donation page doesn’t get enough traffic, test something earlier in the donation process. Try testing a fundraising email first.

5. Design Your Treatment

Half way there. The planning stage of you’re a/b test is done. Now it’s time for the fun part: designing your treatment.

Your test design is made up of at least 2 variants – your control and your treatment. The control is your original page, email, form, etc. The treatment is your challenger or the new design you want to test.

To design the treatment for your a/b test, you’ll want to keep your hypothesis in mind. If your hypothesis is as simple as “Removing the image in the email will increase clicks,” then your design will be really easy.

All you have to do is get rid of the image.

Designing for a more complex hypothesis gets tricky. If your hypothesis is something like “A more empathetic messaging tone will increase donations from an email fundraising appeal,” you have a little more work to do.

Every element you change has to support your hypothesis. With the example above, you shouldn’t change the color of your call-to-action links, or use a completely different email design. But you would likely have major changes to your email copy throughout.

This can get pretty complicated, so it’s best to have a colleague double check your a/b test design to make sure aligns with your hypothesis.

Our friends at ConversionXL have a great post on how to craft one of these more “radical redesigns.” You can read more about radical treatment designs here.

Once your treatment is designed, you’re ready to set up your experiment.

6. Set Up Your Experiment

You’re getting to the home stretch! Time to set up your well-planned experiment.

Setting up an experiment on your website

It used to be that you had to shell out some cash for a tool like Optimizely in order to run a good a/b test. But with Google Optimize, the vast majority of your testing can be done for free.

So that’s what’s next. If you don’t have a Google Optimize account, you can create one just using your normal Google Analytics login. If you need help getting it set-up, this post from Google has got you covered.

Once your account is all set up, you’ll create a new experiment. In many cases, you can edit the actual page elements right from Google Optimize without having to touch any code.

You can set your URL targeting, change what percentage of your web traffic sees your control and treatment, and set your conversion goal. Remember how we set that up in step 2? This is where all that hard work pays off.

Google will even help you preview your experiment to make sure everything’s being tracked properly. Here’s what your dashboard looks like once your experiment is running.

Once you’ve got everything configured, do one last test to make sure everything’s firing properly. Open the page you’re testing in an “Incognito Window” and see if it gives you the control or treatment. Then close the window and open a new one until you’ve seen and tested both your control and treatment.

Setting up an email experiment

Most email marketing tools will let you run an a/b test without any additional tools, fancy coding, or jerry-rigging of the platform. If you don’t know how to do it with your email tool, contact customer support.

If your email tool can’t run a/b tests, there’s a way to hack it. You can manually divide your email list into 2 parts and send two separate emails. Just make sure your lists are divided randomly, and not between key segments like “donors vs non-donors.”

*If you have to hack it like this, you’re using the wrong email tool. Time to start looking for a new platform.

If you don’t have an email tool, start with Mailchimp. It’s free up to 2,000 contacts and it will let you run a/b tests. It’s by far the best tool to use if you have a small list or are just starting out. Plus, there are tons of integrations to get your data into other common online fundraising tools.

7. Validate Your Results and Document Your Learnings

You’re a/b tests don’t matter if no one learns from them. 

If you don’t document your results, you’ll never remember what you learned. And one day you’ll be sitting in a meeting where someone asks, “Why don’t we have a video on our donation page anymore?”

If you document your experiment, you can easily show that the video on your page was killing donations. If you don’t document your experiment, it’s just their word against yours.

Logging your experiments is super easy and totally free on WinstonKnows.com 

It has never been easier to document your experiments. We built this slick tool called WinstonKnows.com that will give you your very own research library, allow you to log every experiment you run, and give you an infinite archive to keep track of everything you’re learning.

Plus, there’s a cool dashboard to show all your big wins. You can use that to help get yourself a little promotion when the time is right.

But it gets even better…

You can also connect your Google Optimize account, Mailchimp, Hubspot, Unbounce, and other common testing tools so your a/b test results get pulled in automatically. It’s literally like magic.

When you’re all done logging you’re a/b test, WinstonKnows.com will give you a short little URL that you can use for step 8…

8. Share Your Results and Change the World

If what you learned from you’re a/b test never gets seen by anyone else besides yourself and your closest colleagues, how can you ever hope to see major growth?

If you work at a nonprofit, sharing these learnings can lead to entire organizational culture transformation. And organizational culture change can be one of the biggest factors leading to online fundraising growth.

If you’re a consultant that works with nonprofits, the best way to get more buy-in from those organizations is to share your learnings with them. And by doing so, you empower those nonprofits to apply those learnings in other areas in order to increase their impact.

No problem or challenge is ever solved by withholding data. So if you want to see generosity increase and the causes you care most about have a bigger impact, sharing your learnings is essential.

Ready to put this process into practice?

This step-by-step workbook will walk you through the 8 key steps to setting up and running an a/b test, and in the end you’ll be equipped with everything you need to get started with your testing.

Learn more and get your free workbook at https://www.nextafter.com/ab-testing-guide/