I at present lead a small information group at a small tech firm. With every thing small, we’ve loads of autonomy over what, when, and the way we run experiments. On this collection, I’m opening the vault from our years of experimenting, every story highlighting a key idea associated to experimentation.
And right here we’ll share a shocking end result from an early check on our referral-bonus program and use it to debate the way you may slender your choice set for experiments (a minimum of after they contain people).
Background: It’s COVID and we have to rent a zillion nurses
IntelyCare helps healthcare amenities match with nursing expertise. We’re a glorified nurse-recruiting machine and so we’re at all times seeking to recruit extra successfully. Nurses come to us from many sources, however those that come through referrals earn increased critiques and stick with us longer.
The 12 months was 2020. IntelyCare was a child firm (nonetheless is by most requirements). Our app was new and most options had been nonetheless primitive. Some examples…
- We had a manner for IntelyPros to share a referral hyperlink with associates however had no monetary incentives to take action.
- Our utility course of was a serious slog. We required a small mountain of paperwork for evaluate along with a cellphone interview and references. Solely a small subset of candidates made it throughout to working.
Throughout a recruiting brainstorm, we latched onto the concept of referrals and agreed that including monetary incentives could be straightforward to check. One thing like, “Get $100 when your buddy begins working.” Zero creativity there, however an concept doesn’t should be novel to be good.
Understanding that many individuals may refer repeatedly in the event that they earned a bonus, and figuring out that our utility course of was nothing in need of a gauntlet, we additionally puzzled if it is likely to be higher as an alternative to offer clinicians a small prize when their associates begin an utility.
A small prize for one thing straightforward vs a giant prize for one thing troublesome? I imply, it is dependent upon many issues. There’s just one technique to know which is greatest, and that’s to attempt them out.
The referral check
We randomly assigned clinicians to one in all two experiences:
- The clinician earns an additional $1/hour on their subsequent shift when their referral begins a job utility. (Tremendous straightforward. Beginning an utility takes 1–2 minutes.)
- The clinician earns $100 when their referral completes their first shift. (Tremendous onerous. Some nurses race by means of it, however most candidates take a number of weeks and even months in the event that they end in any respect).
We held out an equal third of clinicians as a management and let clinicians know the principles through a collection of emails. There’s at all times a threat of spillovers in a check like this, however the considered one group stealing all of the referrals from the opposite group appeared like a stretch, so we felt good about randomizing throughout people.
Decidedly non-social: Many individuals hear these two choices and ask, “Did you consider making an attempt prosocial incentives?” (Instance: I refer you, you do one thing, we each get a prize). Studies show they’re often better than individual incentives they usually’re fairly frequent (instacart, airbnb, robinhood,…). We thought of these, however our finance group grew to become very unhappy on the concept of us sending $1 every to a whole lot of people that could not ever turn into workers.
I suppose Quickbooks doesn’t like that? Sooner or later, you simply settle for that it’s greatest to not mess with the finance group.
For the reason that $1/hr reward couldn’t be prosocial with out changing into a serious headache, we restricted payouts in each packages to the referring particular person solely. This offers us two referral packages the place the important thing variations are timing and the payout quantity.
Seems timing and presentation of incentives matter. Loads. Social incentives additionally matter. Loads. In the event you’re making an attempt to growth-hack your referral program, you’ll be sensible to think about each of those dimensions earlier than growing the payout.
Nerdy apart: Fascinated with issues to check
Product information science, with minimal exception, is enthusiastic about how people work together with issues. Typically it’s an internet site or an app, but it surely may very well be a bodily object like a pair of headphones, a thermostat, or an indication on the freeway.
The science comes from altering the product and watching how people change their conduct in consequence. And you’ll’t do higher than watching your prospects work together together with your product within the wild to study whether or not a change was useful or not.
However you’ll be able to’t check every thing. There are infinite issues to check and any group tasked with experimenting must lower issues all the way down to a finite set of concepts. The place do you begin?
- Begin with the product itself. Ask people who find themselves accustomed to it how they prefer it, what they want was totally different, Sean Ellis, NPS, the Mom Test, and so forth. That is the frequent place to begin for product groups and nearly everybody else.
- Begin with human nature. For a lot of a long time Behavioral Scientists have documented explicit patterns in human conduct. These scientists go by totally different names (behavioral economists, behavioral psychologists, and so forth.).
In my humble opinion, the 2nd of those beginning factors is severely underrated. Behavioral science has documented dozens of behavior patterns that may inform how your product may change most successfully.
A couple of honorable mentions…
- Loss aversion: individuals hate shedding greater than they like successful
- Peak-End: individuals bear in mind issues extra favorably after they finish on a optimistic be aware
- Social vs Market Norms: every thing modifications when individuals pay for items and companies as an alternative of asking for favors
- Framing: individuals make selections primarily based on how data is introduced
- Left-digit Bias: perceptions of a value are disproportionately influenced by the main digit ($0.99 = 🔥, $1.01 = 🥱)
- Present Bias: individuals hate waiting
These psychological shortcuts aren’t a silver bullet for product and Marketing. We’ve examined many of those in numerous settings to no impact, however some have labored. Maybe you’ll find $160M under your couch cushions.
Zooming in on current bias
Many people display a conduct sample the place they like small, instant rewards over bigger rewards sooner or later. Social scientists name this present bias and measure it with powerful questions like…
Individuals who select instant, smaller rewards have some measure of current bias.
Current bias can differ throughout individuals and circumstances. You may be a affected person individual in most conditions. However while you’re hungry or drained… not a lot. That is one cause why A/B checks are so helpful in comparison with, say, surveys or interviews.
For our experiment, the query is just not so totally different from the lab questions:
- Would you fairly have ~$8 someday this week or $100 someday within the coming weeks and perhaps by no means?
Experiment outcomes
Many people (current firm included) thought the $100 supply would do greatest. It sounds greater. It seems higher in an electronic mail. It’s straightforward to elucidate.
I additionally thought we’d get extra functions from referrals with the $1/hr program, however didn’t assume that might carry by means of to extra referrals. I figured individuals would take the cash and run.
To my shock the $1/hr program delivered extra functions through referrals and extra working clinicians, all at a a lot decrease price.
The $100 supply led to a 65% improve in referrals in comparison with our management group. That is large!. Many individuals like the sort of program as a result of we’re solely giving rewards when a referral is profitable. It seems like the cash is effectively spent.
The $1/hr supply, nevertheless, is even higher. Referrals elevated by 81% in comparison with our management group. And regardless that the rewards had been smaller and paid out earlier, referrals from these IntelyPros nonetheless went on to begin working for us at about the identical fee because the $100 group.
Sure, many IntelyPros earned rewards for referrals that utilized after which did not cross the end line, however the math nonetheless labored out. Even with an imperfect conversion fee from utility to working, the overall price per new working IntelyPro was lower than half of the $100 group’s price.
ABT: At all times be testing
The position that referrals play for a enterprise can change for a lot of causes. As an organization turns into greater and extra acquainted, referrals turn into much less essential. You don’t want individuals to unfold the phrase as a result of the phrase is already out.
We ran this check in 2020 throughout a world pandemic. We ran a smaller-scale check a 12 months later and noticed related outcomes. Would we see the identical outcomes now, in 2025? Onerous to say. Issues are a lot totally different within the healthcare staffing house these days.
That is but another excuse why testing is so essential. One thing like referral bonuses might work wonders for firm Some time being a dud for firm B. The upside of product experiments is that the world is your laboratory. The draw back is that it’s onerous to know what generalizes to different merchandise.
Key takeaways for individuals who made it this far
- Once you’re working experiments to enhance your product and accumulate information from the people utilizing your product, behavioral science is a fruitful place to search for check concepts.
- Typically the concepts that appear much less nice prove to have the very best influence!
- Timing and presentation matter for incentives. Earlier than you improve the scale of some payout, maybe put some thought into the effort and time required to earn that payout.
(This submit was refurbished from my own 2022 post on behavioraleconomics.com)
