Observe your digital product ‘in the wild’ with the scientific method
Remember when you used to conduct experiments in school? Well, if it were up to the team at Product Ventures, product experimentation would be nearly identical to the scientific method.
In fact, we believe it is vital for creating lovable features in product development and management.
This article explains how it works using Amazon Prime as an example.
The scientific method for product development and management
“If you’re not taking this approach to experimentation, you’re just doing pseudoscience.”
Step 1: Identify the specific question you want to study (added step for PM)
Example “How can I increase sales for Prime users on the mobile app?”
Step 2: Observe the world
In Prime’s research, your team found that users had trouble locating the filter menu on the Prime app. Users were overwhelmed with the number of results. Past experiments reducing search results resulted in lower revenue, so you need another strategy.
Step 3: Formulate a hypothesis
“If you don’t have a hypothesis, you don’t have an experiment.”
An educated guess could be that revenue would increase if users could narrow search results.
Is this hypothesis testable? Yes, he could measure revenue.
Does it have an explanation? Yes, users can decide faster with more relevant and narrowed search results.
Step 4: Design an experiment
Before this step, do usability testing to remove variables. You come up with a specific design for a toggle at the top of the navigation bar and target only US prime customers with the iOS app.
Step 5: Run the experiment
You can now run the experiment over a two-week period to account for the novelty effect, learning curve, and unusual circumstances such as holidays.
Step 6: Analyze the results
Revenue increased by 2.5 per cent with a p-value of .05.
When analyzing this data, take four concepts into account.
1. Statistical significance: The likelihood that the numeric difference between the control and treatment was not due to random sampling or error.
2. Rejecting the null hypothesis: The idea that there is no statistical difference between control and the treatment.
3. P-value: The likelihood that the hypothesis is valid.
4. Confidence interval: Results on the positive side of the confidence interval lead us to assume revenue will continue to increase.
Step 7: Prove or reject your hypothesis
It’s time to craft a story. Does this result make sense? What did you learn about the product and its users? What other behaviors changed?
Step 8: Communicate your results throughout the organization (added step for PM)
The team followed processes and filled out a template to share these new findings, ensuring everyone in the organization could benefit from the work.
Be a good wannabe scientist: tips for conducting quality experiments
• Choose the right short and long term metrics for your product development process.
• Align success metrics across your whole company.
• Be suspicious. If you see a result that you were not expecting, run the experiment again.
• If you torture data, it will tell you whatever you want. Slice your data as little as possible to avoid false positives.
• Lean against rolling out flat features just because you invested time and money.
• Create a template to document ideas and findings so everyone in the company can benefit.
As Carl Sagan once said, somewhere, something incredible is waiting to be known. With the right product development processes, you are sure to find it.
At Product Ventures, we help partnering teams build useful, high quality digital products.
We bring expert level product skills within your organization. We play a hybrid role of; Product Owner / Manager, and Product Coach.
We collaborate with your customers, as well as your product and executive teams to deliver value in a deliberate way.
Connect with us to discuss your digital product needs.
