The survey that asked the wrong people

A product team surveyed its users about a proposed change and received an overwhelmingly positive response. The change was made and usage fell.

The survey had been sent inside the product, to people who had opened it that week. This is the single most common sampling error in product research, and it is almost invisible because the sample looks like exactly the right one: these are the users, answering about the thing they use. What it excludes is everyone for whom the product is already marginal — people who open it monthly, people who stopped last quarter, and people who tried it once and left. Those groups contain most of the information about what is wrong, and none of them was asked.

The technical description is that the sampling frame was defined by the behaviour being studied. A survey about engagement delivered to the engaged will report that engagement is fine. The same error appears when a school surveys attenders about attendance, or a service surveys claimants about access.

Fixing it requires reaching outside the product, which is harder and slower: email to lapsed accounts, interviews with people who abandoned registration, and panels recruited from the target population rather than the user base. Response rates are poor and the findings are consistently less comfortable, which is the sign that the frame has changed.

The cheap first step is simply to label the frame on every chart. A satisfaction figure reported as 'among users active in the last seven days' invites the right question from whoever reads it, while the same figure reported as 'user satisfaction' does not. Most of the damage from this error is done by the caption rather than by the survey.