This is the first in a new on going series on conversion optimization and testing. If you would like to make more money with your website I suggest you subscribe to my blog so that you don’t miss out on these future updates.
Testing and optimizing websites is crucial and something that most website owners overlook; however, this is something I focus on with every single website in my portfolio. Ever since this blog relaunched in early January 2010 I’ve shared numerous stories where I’ve tested different ways to make money online, the impact on search engine traffic after moving a blog, how to drastically reduce the load time on a blog and lesser known traffic generation techniques just to name a few. Today I’m going to show you how color choice effects a person’s decision to sign up for a newsletter or not.
What did I test?
I specifically tested the newsletter sign up box in the upper right hand corner of the blog’s homepage. Or to be more specific, I only tested out the color of the arrow that pointed at my newsletter sign up box in green, red and black. Here are the three different boxes I tried:
(Images have been sized down to fit)
Why did I test just the color of the arrows?
Some people may be quick to point out things like “Why didn’t you change the color of the ‘$5,000’ from green to red or black?” or “Why didn’t you change the color of the “Get Your Guide” button to match the color of the arrow?” etc. Well, the reason why I kept this test simple is because the more variables that are added into a test scenario the longer it takes to get enough data to determine with reasonable confidence whether or not one option is truly better or not. I specifically choose to test only the color of the arrow in this test because quite frankly I noticed that EVERYONE uses a red arrow to point at their newsletter sign up box and I wanted to find out if there is any proven reason for that or if people just think that red is best.
Interpreting the results:
As the title of this post has already indicated yes green performed the best out of the three colors; however, the difference is very small and could almost be discounted due to insufficient data. See below:
I use Aweber to easily create these forms and highly recommend them as a newsletter service.
There are a few different reasons why green out performed the other colors that must be considered in future tests:
1. The $5,000 text was green. So with the green option both the arrow and $5,000 were green. By not matching $5,000 with the other colors it could have clashed and resulted in less people clicking through than had the colors matched.
2. Green could have been the best color to pair with the yellow submit button.
3. There could always be more data. Even with nearly 2,500 impressions for each option more data always helps.
4. Not enough data to draw conclusive results between the three colors (green got lucky)
There are several other reasons I could list off, but every time I run a test I analyze factors to address how the data could have been influenced. But what is better than coming up with a list of ways the data could have been influenced? More testing! I can run tests to address the two concerns I have listed above without even addressing word choice yet.
Closing Thoughts:
If you aren’t testing elements on your websites than you need to start today! Once you think you’re done – you’re not. There is always something more to test.
The fact that I even got to the point of testing just the color of the arrow is because I have already tested more of the obvious elements in a newsletter sign up box. For example, when I first set up newsletter code on one of my other blogs I did so without any incentive for visitors to sign up to my newsletter. The results? No one signed up. I added an incentive, more people signed up. I added an image of the incentive (even though it’s a digital good) and even more people sign up. Now I know that if I just remove the black and red arrow versions from the site than I’d get about a 15% increase in newsletter sign ups from that section of my blog; however, I am going to keep testing other variables and eventually get even better results. If you look now, you’ll see the box is already different.
Upcoming: I will be sharing other examples of testing I’ve done on this blog to improve my newsletter sign ups as well as the effect guest posting has had on my readership. It will be very exciting and I hope stick around to see the results.
Update: Some of my more mathematically minded readers have pointed out via standard deviation formula’s I need more data to draw the conclusion that one color has outperformed one or the other more. So in future blog posts I’ll be sure to highlight examples with more data. Stay tuned…
I’m not convinced that you have enough data to draw this conclusion.
The null hypothesis is that red, green, and black are equivalent. Since you had 158 total signups, the null hypothesis says that on average, you would expect to have seen 52.67 signups for each of red, green, and black, with a standard deviation of 7.26.
Then black had 41 signups, so it was 1.61 standard deviations below the mean.
Red had 54 signups, so it was .18 standard deviations above the mean.
Green had 63 signups, so it was 1.42 standard deviations above the mean.
None of these are really remarkably different from our expectations, so we can’t reliably conclude that the null hypothesis is wrong. (It MIGHT be wrong. It MIGHT be true that green is better than red or black. But the statistics aren’t really strong enough to conclude that yet.)
In my view, your null hypothesis should be considered “not disproved yet” and you should collect more data to be sure.
I’m a computational physicist and the above analysis is very primitive–what I could scratch out in two minutes on a pad of paper. A real analysis would take a bit more work.
Hey Randy,
I think with almost any test you’ll never have enough data to draw a definitive conclusion, but again that’s why I’m still testing the newsletter sign up box and in the future test I’ll run it for even longer now that I’ve found there is just a slight difference between all 3 colors (or as you suggest – not a large enough difference to prove one is better than the other). Perhaps I’m just not using the accurate terminology from a statistical point of view and you’re right I could make some changes to my wording.
What would you like to see in a follow up analysis that wouldn’t be so primitive? I’m obviously not a statistician and you clearly have some expertise in the matter so what might you suggest in a follow up post besides more data?
As an FYI I made a few adjustments to the post to more clearly state that the data doesn’t show conclusive results and that more testing is necessary. In either case, I’m running some other tests via Google’s website optimizer which I believe runs standard deviation formula’s automatically and once there has been enough data it indicates that within the test results.
When that test is done it will provide more value than the one I’ve posted here above – what do you think though?
Hi Chris: I’m out of town and just now had a chance to log in and saw your question. Actually, I meant that my own analysis was primitive, not yours.
It’s not a simple problem to decide when you’ve got enough data to make a decision. One has to define things like the probability that you’re willing to accept for a false positive. In science, the usual choice for this is that you’re willing to accept false positives 5% of the time, but this isn’t engraved in stone, and there’s no particular reason for choosing this, other than convention.
I was on the road most of today and spent some time thinking about the problem and it’s actually fairly complicated, especially when you’re doing a three-way split test. I think a monte-carlo simulation would probably be quick and easy to get a decent answer.
Hey Chris,
What are you using for spit testing? Doesnt look like google website optimizer…
Thanks!
Dan Brock
P.S. We still on for the webinar sometime this week?
.-= Dan Brock´s last blog ..Selling the Benefits vs. Selling the Features =-.
I just used Aweber’s code. Google Website Optimizer doesn’t run well within WordPress sites so I wasn’t able to run that unfortunately.
Yes, testing is VERY important in the online world, and testing an opt-in form or “squeeze page” is one of the easiest since it’s just about monitoring your conversions. Google provides a free service that makes split-testing easy: google.com/websiteoptimizer
The best way to do it is to change just ONE thing, assess your results after 100 conversions (opt-ins, or even sales if you’re testing a sales page) and then move onto the next element.
Thanks again Chris for a great post and reminding us the importance of testing.
.-= Jonathan Beebe´s last blog ..ListZEN Officially Launched � Join For Free =-.
So are you taking this data to your Amazon sites and going to put more green? Or are you just taking this data and saying, “Colors need to match not clash”.
I should try this on a large PPV campaign.
.-= browie´s last blog ..Thanks Volk and Chow =-.
I think I need more data – just look at the responses from Randy 😀
Chris,
Have you tried split testing the submit button color? I think that would be more important than the arrow color.
A lot of people say yellow submit buttons suck, and green work better. Worth a test for sure.
.-= Dan Brock´s last blog ..Selling the Benefits vs. Selling the Features =-.
I didn’t find this particular post that helpful, but I am really looking forward to your series of posts on blog optimization, and the results!
.-= Estate Yard´s last blog ..How much does a Realtor� make per deal? =-.
Thanks for the feedback, it helps me to figure out better what types of articles people want to see from me.