I don't want to be data-driven.   


I run a company that sells software on the web. Sales depend a lot on how well our splash page 'converts'. For the past 3 months, I've been religiously AB testing every part of the splash page, hoping to increase its conversion rate.

I learned very quickly that the single most important part of the splash page (when it comes to conversions) is the headline. I've also learned that conversions go up when the headline describes the value proposition of the product, and that they go down when the headline describes features.

Every time an AB test finishes, I've diligently applied the winner as 'the new permanent', even if I don't personally like it. In this sense I've been totally data-driven.

A week ago I began a new headline AB test. One of the variations has a cruel streak. It says, "Fire your [insert employee that we make redundant] for $[insert fairly small amount] a month". The value proposition is clear: cut your costs.

A friend emailed me a few days later and told me she thought the headline was too harsh. I'd actually forgotten about it, because I wasn't getting served that variation. I felt bad, and decided to check out the test results. My guess was that its performance would be mediocre, and that I'd remove it.

The cruel headline was a clear winner, beating the next best headline by 6%. I felt conflicted. On one hand it's the best conversion rate of any headline I've tested in the past 3 months. On the other hand, I don't want people to associate me and my product with that message. I don't want to be data-driven.


The cruel result is titled 'Fire'. This is a screenshot from Optimizely

I replied to the email rudely.

"It is the best performing AB test variant (of about 6 variants). I aspire to be data driven, rather than harshness or lack thereof driven."