Tag Archives: Product Development

Know what you’ve learned, not what you’ve done

Posted on by

At Homebrew, Hunter and I are very focused on “Why” a founder has chosen to start a company and what motivates him or her to attack the specific problem or opportunity he or she sees.  But also important is the “how”, the approach the entrepreneur has taken to address the opportunity.  Often, when we starting talking about the “how”, we hear about a lot of different ideas being considered and experiments being run in an effort to find product/market fit.  But at the seed stage, entrepreneurs often focus only on what they’re doing without being equally attentive to what they’re learning.  “Being busy” by itself does not equate to building a company. You should be learning with every step so that you can find a scalable model of success.

Focusing on the key questions and how best to answer them

To create an organization that learns and doesn’t just do, I’m a big advocate of the scientific method approach to building product (and companies more generally).  The scientific method is a simple framework that can help startups focus, experiment, learn and iterate quickly and effectively. Below is a description of that method along with a simplified example of an experiment we ran at Twitter (not the actual data).

Purpose/Question – As obvious as it sounds, you need to start with the question you want to answer.  Surprisingly, lots of startups take the “see if the spaghetti sticks” approach, just putting something out in the world and then somehow gauging the response.  Without clarity around what question you want answered, it’s difficult to design the right experiment and to draw the right conclusions from the data you collect.  In particular, it’s critical to know what metrics are relevant to the question you’re trying to answer.

Example: How can we increase the sign up rate of users visiting the Twitter homepage?

Twitter homepage in April 2011:

Research – If you have a question, it makes sense to consider all of the potential answers, even if many of them are dismissed quickly.  Research, whether that’s talking to potential users, evaluating similar products, conducting simple surveys or brainstorming as a team, does a few things.  Research helps uncover unspoken assumptions about the answer, identify unexpected potential answers and inform the design of the right experiment.

Example: Ideas that emerged from user research, looking at site analytics and from team ideation included better descriptions of what Twitter is, a video that explains how to use Twitter, showing popular tweets, simplifying the homepage by removing trending topics, simplifying the homepage by removing the search box, etc.

Hypothesis – What do you believe to be true?  That is the essence of your hypothesis.  And proving or disproving that statement is the goal around which your experiment should be designed.  Any test run without a hypothesis is unlikely to lead to learnings that impact product direction in the correct way because the experiment likely doesn’t have a control for what you believe to be true.

Example: Simplifying the homepage to focus on sign up by removing the search box will increase the sign up rate.

Experiment – This is what most startups focus on but only in the sense that experimentation means putting something out in the world.  More important than the idea of experimenting is the design of the experiment.  Simply put, you need to know what question you’re trying to answer, which answer you believe to be correct and which variables you believe impact that answer.

Example: Show the homepage without the search box to a randomly selected, statistically significant sample of users and compare the sign up rate to users that see the existing homepage during the same period of time.

Analysis – The reality in most product experiments is that you can’t isolate or control all of the variables, so it’s important to not be a slave to the data.  Data from the experiment usually needs to be considered one input into product thinking and not the answer in and of itself.  Accordingly, it’s critical to be honest about what the data does and does not say in relation to the hypothesis and question at hand.

Example: The sign up rate increased by 12% without the search box. But by removing the search box did we lose sign ups from people who searched and then signed up from the search results page?

Conclusion – In organizations both small and large, nothing is more important than providing the proper context for product decisions.  So when you arrive at conclusions from your experiment, be sure to share them quickly, clearly and repeatedly.  Was the hypothesis proven or disproven?  Did the outcome result in more questions and experiments or answers that you feel comfortable moving forward with? Communicate what you intend to do in reaction to the conclusions and start the scientific method process all over again.

Example: At Twitter, results from experiments and planned next steps were summarized and emailed to an internal mailing list for anyone in the company to review.  When changes went into production, another email was sent outlining the changes.  

The homepage that resulted from this experiment and several others was launched in December 2011.

Here is the current homepage:

Next time you’re working on experimenting and iterating to get to product market fit, remember the scientific method.  If you remember to have a hypothesis and to design an experiment that tests that hypothesis cleanly, you’ll be learning and not just experimenting.

Better isn’t good enough

Posted on by

I previously wrote about what I think is required for a successful mobile product.  With all of the activity in the social messaging/communications/networking category (Whisper, Confide, Secret and Wut being the latest buzzed about apps), I thought I’d dig into one of the key points I made, which I believe is even more true in this category.  It’s not good enough to be better, you have to be different ─ in a way that helps address an entirely different use or possibly a similar use case dramatically differently.

In the case of messaging, it’s not enough to innovate along an existing dimension.  You need to create an entirely new product dimension.  It’s clear when you look at the breakout messaging apps that they clearly did something different relative to the apps that came before them, and in most cases that helped those apps address different user needs.  Facebook popularized status updates within a private network.  Twitter made status updates public (changing who sees the update).  Instagram made status updates visual (changing the format of the update).  WhatsApp made status updates (via SMS) free.  Snapchat made status updates ephemeral (changing the permanence of updates).  And what of the apps that were hyped and have seemingly gone away?  What new dimensions did Path, Frontback and MessageMe introduce?

Of the “hot” new apps, it remains to be seen which will pass the test of time.  Whisper makes public status updates anonymous (changing who sent the update).  Confide has made text status updates private and ephemeral.  Secret has made the anonymous status update visible only to a semi-private group.  Wut has anonymous, private and ephemeral status updates(?).  Are any of these apps introducing new dimensions that address different use cases or needs?  Fortunately, or possibly unfortunately, social messaging companies doesn’t typically fit our Bottom Up Economy thesis.  So we don’t have to bet which of these products will become the next Facebook, Twitter or Instagram.  But if I had to bet, I’d pick the one that does the most different thing best.  Because better just isn’t good enough when launching a product.

Four truths about mobile products, user mindset and achieving scale

Posted on by

Like many others, I’m spending more and more time on my phone, and I’m not making more calls.  I’m getting information, buying stuff, collaborating with others and much more.  There are a handful of mobile apps that I use religiously to do these things.  Why do I use these specific apps as regularly as I do?  I suspect that the answer for me and for many others who have the same behavior (a few select apps used frequently) boils down to a few simple things.

Scale is the outgrowth of doing just one thing really well.  WhatsApp, Instagram and Shazam are great examples of products and companies that expertly address a single, well-defined need in a simple way that satisfies a large number of people.  They haven’t added tons of new features to address additional use cases or at least they didn’t begin to do that until they had already significant user scale.  A single-purpose app makes it easier for a user to remember why she should use that app at the moment that she has a specific need.

Users inherently have a tendency towards mental inertia.  Once a user begins thinking of an app as addressing a particular need, it’s really hard to get him to think about it or use it differently.  An app developer who adds lots of features to an app risks confusing users and detracting from the core use case that the app is meant to address.  Unlike the desktop web, where tabs, menus, filters, etc. can be used to add features, it’s likely that in the mobile world the only successful way to add features will be to build entirely separate apps (see Twitter’s Vine and Facebook’s Messenger), often under separate brands.  Can you think of a single app that you use that does many different things well?

Better isn’t good enough.  Once a user begins to think of an app as addressing a need well, it’s really hard to get him to switch to another app to address the same need, even if it does it “better” (see Facebook’s Poke vs. SnapChat).  It’s not enough to be better in mobile, you need to be first to get to scale or you need to be different to fight inertia (not to mention switching costs, network effects, etc.).  WhatsApp continues to thrive in the face of increasing competition because it was first to address a specific need well.  SnapChat and Instagram succeeded not only because they did only one thing extremely well, but also because they did something different from Facebook and Twitter.  They addressed entirely new use cases and didn’t settle for competing via marginal feature innovation (which seems often to be the case in the messaging category, as one example).

Great products unlock user acquisition.  How did Uber grow virally?  The challenges of mobile app discovery and distribution have been well documented. App store distribution, pay-per-install ads, incentivized referral programs, etc. all face obstacles in mobile.  If you don’t have a product that requires users to invite others to benefit from the app (i.e., Facebook), there is only one true answer to the distribution problem.  The best distribution strategy is to build a killer product that generates tremendous word-of-mouth.  Uber, HotelTonight and Mailbox are examples of mobile apps that delivered amazing user experiences that in turn led to viral growth via word-of-mouth.  More than ever before, being the first to deliver an elegant solution to a user problem can be the key to dominating distribution and hence an entire market.

Surprisingly, when I thought about the points above, it seemed that what is true in mobile has largely been true on the web as well.  While technology changes, human behavior is pretty ingrained.  The mind craves simplicity and consistency and resists complexity and change.  Mobile app developers who want to achieve scale will be well served by satisfying the mind.

Shared Context in Product Development and the How of “Why”

Posted on by

Product development doesn’t come without its fair share of strong points of view, impassioned debates and occasional yelling.  These disagreements don’t result from team members wanting to do the wrong thing.  Instead, arguments about priorities, features and timeframes are typically caused by the lack of shared context about “why”.  Whether you’re a product manager, engineer or designer, if you’re the “decider”, your job is to make sure that everyone on the team has a clear understanding about why the goals, priorities and decisions are what they are from the very beginning of your work together.  Without shared context there can’t be shared purpose, shared risk or shared outcomes.  Here are some tips for making sure you and your team are operating from the same base knowledge and set of assumptions.

Don’t assume.  Even if you think that everyone knows the priorities, has been looking at the same data or been privy to the same conversations, don’t assume that the team has shared context.  Leaving it to each person to interpret conversations, make sense of data and draw their own conclusions inherently leads to misalignment and lack of buy-in regarding decisions.  Not surprisingly, as teams begin to work, the absence of shared context has severe costs – lost time and productivity, finger pointing and discouraged team members.

Write it down.  If you force yourself to write down the “Why” behind decisions that are being made, it will both clarify and improve your thinking.  You’ll be able to understand how others might react to the explanation of why and what issues they may have.  Sharing that written explanation of why with your team will allow all of you to have a common language with which to talk about problems, solutions and goals.  It will also provide a reference document that can be used by the team after the why has been discussed and you’re all no longer in the same room.

Repeat early and often.  The why is not something to be shared once and then be forgotten.  Talking about why at the very earliest points of product development lays the foundation for close collaboration and tight alignment.  Repeating the why during each step of the development process ensures that the team continues to have a common understanding and shared goals and allows the group to reaffirm or change the why as new data, requests and issues emerge.

Seek understanding, not agreement.  Assuming it’s clear that you’re the “decider” and that you’ve already gathered input from the team, the purpose of sharing the why is to help everyone understand the reasons for the decisions that are being made, not to have everyone agree that those are the right decisions.  That said, team members need to feel that their points of view were incorporated into the thinking that led to the why (writing it down helps provide evidence of being heard!).  The right outcome is to have a collaboration amongst team members based on shared context, not to be consensus-driven.

While establishing the why is critical in product development, it’s an equally important behavior in nearly all professional and personal contexts.  Next time you’re working on a project with your team, friends or spouse think about whether there is shared context and how setting that context might help you work together more effectively.