A/B Testing

Open rates and conversions are most important to us!

We think the best metric for evaluating A/B tests is conversions, whether that’s donations or advocacy actions or other form actions.

Depending on the audience size, though, sometimes there aren’t statistically significant differences in the numbers of donations, so we sometimes fall back to using open rate or click rate to determine whether A or B was the winner.

Our email lists are so small that a single A/B test is not statistically significant. Instead, I try the same alternatives over a series of email to learn my audience. The metric is open rate (for engagement) or conversions (for fundraising).

Some questions I’ve considered are:  

Does including a photo improve the conversion rate?

Does a raw link work better than a button?

What tone gets more engagement?

Then, I use A/B testing over 2 or three emails and then evaluate the differences between A and B. I’m still looking for trends rather than absolutes. 

I think that’s a great point - you may not always get a hugely significant winner every time. Watching for those trends across multiple emails allows you to really understand what speaks to your audience!