Sure, you can show users mockups, prototypes, or even a paper napkin. But that falls flat when it comes to actual behavior, because the difference between what people say they will do and what they will actually do in a given situation can be quite large.
You need to find creative ways to test how people will actually, truly react to a feature. There is one fool-proof way to do this, which I call the “just kidding” test.
I started my product management career at Citrix as an APM. I was tasked with increasing new bookings for GoToMyPC – the decade-old “cash cow” product which, despite it’s decidedly aged name & feel, had over 200,000 users and brought in a cool $80 million per year.
Working on this product had its pros & cons. The main con was that nobody paid much attention to what we did. But on the flipside, since nobody expected much of us anyways, we had more opportunity to experiment with new ideas. If they worked, everyone loved you for bringing in more revenue! If they didn’t work, well, nobody really expected much to work anyways.
I’d created some effective retention & upsell programs, which were working quite well. But what I really wanted to start was a referral program. At the time, GoToMyPC had the highest retention & NPS score of all Citrix products. So I figured, let’s try to capitalize on that love & loyalty by incentivizing our customers to share with their friends. Everyone liked the idea, but we had zero development budget for it. Zero.
So I did some research and found a new company called Extole, a company that activated rewards programs through social media. We partnered with them, and as part of the deal,they agreed to build not only the back-end logic, but also the front-end landing pages.
I had the ability to email users directly, so I created a quick email test program to gauge the viability of the idea.
After some very brief validation chats with a few existing customers, I decided to test three different approaches. Each group received an email asking them to refer their friends to GoToMyPC for one of three rewards.
In exchange for referring a friend…
- The first group would get a free month of GoToMyPC
- The second group would get free 500 minutes of connection time
- The third group would get free day passes
The interesting thing here is that we didn’t actually have the ability to fulfill these options on the backend!
I needed to be able test these ideas without actually building them. The best we could do was manually give people a free month of service. That’s it. This was the only lever I had.
This would be fine for the Free Month group. But what about the other groups?
Here’s the rub: I decided that if someone referred a friend from the Free Minutes or Day Pass group, they’d be brought to a landing page that said the following:
Hello. Thanks for your interest in our new referral program!
We’re still working through some of the program details – but in the meantime we’re going to give you something even better.
Instead of a single day pass, we’re actually going to give you GoToMyPC free for the next 3 months.
“Thanks for referring! Oh, you wanted a day pass? Sorry, here’s a free month instead.”
Was this bait-and-switch? Technically, yes.
But given the “switch” was much better than their original offer, the COGS in software is close to zero, and we’d learn a ton from this test, we as a team were collectively okay with it. Learning quickly without building is the name of the game here; and that’s exactly what this would allow us to do.
So how did my just kidding test work?
After getting executive buy-in (a bit tricky at a large conservative company), I set the test live. I sent out an email to a few thousand customers, and went to bed that night feeling good about things, eagerly anticipating the morning’s results….
…But when I woke up the next morning, I was surprised to see there were no takers on the offer.
Huh, that’s strange, I thought. But it wasn’t unheard of. Maybe the program needed some time to ramp? Who knows. I double-checked everything was working properly, tweaked the subject line a bit, and then sent to another few thousand customers.
To my dismay, the next morning there were still no conversions. None. Okay, I thought, this isn’t good. Either something’s broken, or this idea is just plain not working.
I then upped the reach of the email for a few more days. But by the end of the week, after thousands of emails sent & hundreds of clicks, we had just one single conversion.
Ugh. The program was bombing, hard.
I was pretty devastated. How could this be? We all liked the idea, and email tracking showed that “click interest” was there. People were opening the email and clicking the links. But why weren’t they referring their friends?
Not knowing what else to do, I picked up the phone. I started calling those who clicked on the email, but didn’t convert. I knew I needed to get into their heads and find out what was going on. I planned on calling about 20 customers – just quick 20-minute chats to understand their psychology. But I only ended up making it through about a dozen calls, because by that time a clear pattern had already emerged…
The reason that GoToMyPC customers weren’t referring their friends was simple. These weren’t 17-year olds sharing cat videos on YouTube, they were 40-55 year-old accountants who don’t know the first thing about Twitter! They barely had Facebook accounts, and had never before heard the word ‘Reddit.’
In a nutshell, the reason they weren’t using social media with their friends was because they didn’t have a big network of friends using social media! This was a classic mistake of not fully empathizing with your audience; or rather, building for the audience you want vs the audience you have. In hindsight it was so obvious, it’s incredible none of us saw it coming.
But here’s the kicker. Through the discussions, it was clear something else was happening as well… They may not have wanted to refer friends, but they really seemed to like the idea of day-passes…
In fact it turned out that a good number of our customers only needed GoToMyPC a few days a month anyways, and were really interested in that type of packaging. I believe one customer’s exact words were:
I’d buy those day passes you were advertising. Can you just sell me those instead?”
Upon realizing this, I immediately shifted the nature of the interviews & tests to validate the day pass idea. I wanted to make sure the day pass option was a better idea than free minutes, or a weeklong pass, etc.
After some final validation, as you may have guessed, we determined that the referral program was not going to work, but that the GoToMyPC Day Pass was indeed a viable idea.
So there you have it. This is how in a matter of weeks, I started out with one hypothesis, and ended up with an epiphany of an entirely new pricing model; forming and validating it all without spending any money, or even writing a single line of code.
No costly developers, no engineering time, just an idea validated in the leanest, cleanest way possible.
It’s messy and it’s awkward, but it’s sort of what product management is all about.