Creating respondent segments to identify what drives the concept KPI

Two core parts of a survey used to evaluate a concept are the set of key performance indicators (KPIs):

1. Typically, a form of purchase likelihood 5-point rating scale, with top or top-two box percentages, or averages, used to provide a total sample summary of those ratings
2. A set of performance-based evaluations, provided by respondent ratings. These are usually of two types:
   1. Lower-Order KPIs (rated on 5-point scales) to measure general criteria of concept acceptance that serve as gate-keepers necessary for the respondent to actually want to purchase the product depicted in the concept. Examples are Value for the money, Believability, Uniqueness, and Meets Needs.
   2. Actionable Attributes (also rated on 5-point scales or presented as a checklist for selection) of the product depicted, typically dealing with perceptions of the product’s physical features (e.g., ease of handling, delivery of stated benefits).

Ratings of the performance-based evaluations provide an integral part necessary for concept optimization. The ratings are a basis for profiling the concept and providing summaries in the form of respondent segments. 

The ratings are also the basis for what is referred to as “driver analyses”, identifying characteristics and features of the concept that can, with improvement or modification, increase the level of the KPI. 

In the Zappi portfolio, Activate It (Zappi’s late-stage concept test) brings these two integral parts together, maximizing the insight extracted from concept test data and providing direction, prescription, for optimization*.

Overview of concept evaluation

The total sample KPI performance is used as an internal benchmark, to be improved upon. Top box purchase likelihood ratings are preferred because of their use in market sizing.

A segment can be created through the linking or intersection of two or more** performance-based evaluations. The points of intersections for those evaluations are usually the top two box ratings provided to each. A segment is then defined as those respondents who provided a top-two box rating to each of the evaluations being linked.

Segments provide two summaries related to concept optimization:

1. Percentage of the total sample captured in the segment. The percentage of respondents in the segment is a good description (e.g., a larger segment size is an indication of better concept performance) but tells nothing about the segment’s statistical relationship with, or effect on, the KPI.

2. A simple way to assess that statistical relationship is to calculate the KPI (e.g., top box purchase likelihood percentage) within the segment and compare it to the total sample value of the KPI. The difference is a measure of the relationship between the KPI and the segment.

The strength of the statistical relationship between the segment and KPI is measured by the increase in the KPI among respondents classified into the segment. This is then compared to the total sample KPI result. Both (the size of the segment and the strength of the statistical relationship) are meant to be maximized.

The difference in KPI, total sample vs within segment, is standardized to create a measure of effect size. Extensive use of this measure has led to interpreting the strength using the following triage:

• Small Effects: between .2 and .49
• Medium Effects: between .5 and .79
• Large Effects: greater than .79

Example of concept evaluation

Ratings of Advantage and Distinctiveness provide an evaluation of the concept as depicting a product that is both better and different than other products on the market. These are considered lower-order KPIs that serve as gatekeepers. The positive (top two box ratings) performance of both measures is considered necessary for a respondent to “breakthrough” to provide a top box purchase likelihood rating.

Gathering respondents who provided top two box ratings to both measures is one way to quantify the size of this segment. Using data from a salty snack concept, 32% of 400 respondents were classified in this segment, by virtue of having provided top two box ratings to both the Advantage and Distinctiveness measures. 

The KPI, percentage top box purchase likelihood, within this segment, was 55. This 55% is referred to as Breakthrough Opportunity Score, an upper bound on what the total sample top box purchase likelihood percentage could be if all respondents in the sample provided top two box ratings to both Advantage and Distinctiveness (if everyone in the sample rated the concept as better and different).

For comparison, the total sample percentage top box purchase likelihood, as a benchmark, was 23. At face value, the segment KPI provides of lift of 2.5 (55% divided by 23%), indicating a great potential to improve concept reception. 

Effectively communicating the concept as both different and better can lead to a sizable increase in purchase likelihood. To provide a summary of statistical relationship, and for normative purposes, the difference between segment and total sample percentage top box purchase likelihood, (55 – 23), is re-expressed as an effect size*** of .64, a medium effect.

Notes:

* Many ratings used by Zappi have accumulated normative databases. Survey results from a new concept test can then be compared to their norms (specifically, the average normative value from the database). The comparison of survey results to norms can have at least an implied “driver analysis” interpretation, e.g., for a specific performance-based evaluative measure, the survey result exceeding its norm is taken as a possible reason (if not a cause) for also exceeding the KPI norm. And conversely, the reason for a poorer KPI value could be assigned to those evaluations whose results were below their norms. Reliance on survey vs norms performances tells only a part of the interpretative story. What has been missing is an assessment of the strength of the statistical relationship between each evaluative measure and the KPI. This can be rectified by consideration of each evaluative measure’s effect size. A potential evaluative “driver” of the KPI can be defined by both its performance vs the norm and its effect size.

** Segments can be created from a single performance-based measure, e.g., classifying only those respondents who offered a rating of “4” or “5” to the Meets Needs rating.

*** The effect size is an assessment of the magnitude of the difference between two statistical summary measures (e.g., means, percentages). The difference is typically standardized (e.g., division of the difference by a pooled or average estimate of the standard deviation). The standardization allows for the comparison of effects across data from different studies which may have used different scales and summary statistics and come from different countries at different points in time. Effect sizes form the basis for a meta-analysis, an analysis of the statistical results accumulated over many studies or separate pieces of research. The accumulated evidence, in the form of effect sizes taken from the different studies or pieces of research, is used to increase the statistical power, sensitivity and generalizability of conclusions that could not be obtained from one study alone.

Within the context of this article, the effect size is calculated as the difference between top box purchase likelihood percentages, (total sample – segment), which is then divided by the standard deviation of the total sample percentage. To the example, (55 – 23) / 50, where 50 is a close approximation to the standard deviation.