Norms

Making the right comparisons is critical to decision making.  In the platform, you can benchmark in a flexible, dynamic way, enabling you to draw on the best available benchmarks for the decision and the stimulus at that moment in time.  
Learn more about possible relevant benchmarks in this blog.

This article compiles guidance on what benchmarking is possible and where to find and how to understand norms when using Zappi, including:

What is a norm?

A norm is a statistical concept. It refers to the aggregate response of a certain population, against which a similar subject or answer set is compared. Simply put, comparing your results to a norm allows you to determine "what good looks like" for key metrics.

Back to top

What norm or benchmark should I use?

We encourage you to build up your data asset and tag the characteristics of your ideas, ads and products so you can learn and compare them over time.
One way to learn from this data asset is to create your own norms and benchmarks to inform better and better decisions from over time. You can curate specific benchmarks of stimuli that you know have been successful in-market.  Or stimuli that share similar characteristics to each other. You can even benchmark when looking at sub groups you have predefined (gender or age). These targeted comparison points can often be more appropriate than very large norms made up of very different stimuli over many years among different audiences.  
Whilst offering this flexibility to benchmark in useful ways for your business and decision, you’ll also find recommended norms within our solutions that go beyond comparing only to the research you have done in the platform. These recommended benchmarks come from Zappi research experts who have created the solutions based on an understanding of the business decisions that need to be made and how best to analyze different datasets. You can read more about these below.

Back to top

Which measures in my results have norms?

In general, norms are generated for measures that are asked the same way in every survey. Norms availability differs by Zappi product, so the quickest way to see what measures have norms is to open your results, select a chart, and view which norm is automatically applied and what others may be available. 
To see which norm is automatically applied, look for a blue box above the chart.

To select a different norm, select the Norms tab next to the chart and scroll to compare your results to other populations. Often these will be either less specific (i.e., a global or country norm with no further filtering) or more specific (i.e., a country norm filtered by your category, or stage of development of the asset (Animatic, Final cut, etc.)).

Back to top

What assets are in norms?

In the image above, you can easily see the number of assets each norm is based on next to the label. Those that are greyed out have not yet reached the minimum number of assets (see next section) to make comparison statistically relevant. For transparency, the number of tested assets in the norm you select is also included in the footnote of any slide you share or export from the platform.

We do not allow anyone to view the actual assets included in a norm, as our norms include proprietary data from our customers.

What is the minimum number of assets for a norm to populate?

From the 26th June 2023, the minimum is 20 stimuli for a Zappi country/category Norm where it used to be 30.  This will help you benefit from a relevant comparison sooner, particularly for new categories or countries.  As databases grow and diversify, they will become more stable. While the norm includes less than 50 cases, we encourage using your own internal or competitive benchmarks alongside these smaller norms.

  • Many solutions, categories, countries already had 30 or more cases in which case you will experience no difference.

Back to top

How does Zappi calculate norms?

Norms are created by aggregating the results of a measure across all of the assets tested in Zappi that meet the criteria you've selected. Most often you will see norms based on the average (mean) answer to a rating scale question asked in your survey. 

In most norms there is no relationship between category and sampling. This means that a norm for "food and drink" in the UK can be composed of assets tested with a "nationally representative" sample, but also "older females" and "men under 35" sample, as long as the asset was tested in the UK with food and drink consumers. 

Some notable exceptions to the above: 

Zappi Creative Suite:
  • The “Lovers” norm is based on the top three box scores (i.e., scoring 8-10 on a 0-10 scale), rather than the mean.
  • Key message norms are created by averaging the proportion of respondents who chose a given message.
Prioritize It:
  • Prioritize It norms are based on category audiences. 
  • You can also create norms by tag, combined categories, and more.
MARC Rapid Results:
  • MARC norms are based on category audiences. 
  • Most are aggregates of top boxes, and only the top 25% of best-performing concepts in a category are included in the norm.

Back to top

I can't find a relevant norm, what should I do?

While our overall database includes hundreds of thousands of stimuli, there are instances in which there isn't a norm available for a particular market, category, or industry.  

If you are in this situation here are some suggestions:
  • Use your own benchmark made up of fewer, but highly relevant, cases including for example your previous ideas/products/ads and those of your competitors
    If you have no cases, include a benchmark cell (asset) so you can compare the results. Ideally this could be an ad/idea/product that you know has been successful in market.  Or maybe a competitor you want to beat. Or a previous ad from within  a campaign. 
  • Create a norm using a broader geographic region, or adjacent category using our advanced norm functionality.

Back to top

Early stage ad testing and norms

We've crafted a standalone FAQ for those using the Amplify suite to test early stage creative ideas, including guidance on how to interpret and use norms at early stages in the creative process. To read that, click here.

More norm options

Selecting a norm date:

To provide more flexibility to your analysis, you have the ability to select the date on which the norm will be calculated for the majority of our products. The dates available from which to calculate your norm values will include:

  • The 1st of the current month
  • The 1st of the month your selected study was created in
  • Today

Advanced norms:

This option enables you to define a custom selection of countries and industries to be included in a norm, to suit your needs.  

For example, you may not have conducted sufficient studies in Colombia, for instance, to be able to build a country norm, so you might roll up those that you have done with any others you have done in neighboring Central American countries to create a custom norm that is more relevant to your products or advertisements than a global norm would be.

User- or customer-defined norms:

These are norms made up of all the assets you've tested within your platform domain. This enables you to compare your results to your own historical performance. For example, you might compare your ads or concepts with the best you ever tested, or you might compare them with any of your other tests that have the same tags, so you can analyze your performance on particular themes. In most Zappi tools with norms, the minimum number of stimuli required to view a user-defined norm is 2. The exceptions to this are:

  • Amplify suite and Prioritize It = 30 minimum for percentiles, 2 minimum for mean scores
  • MARC = 20 minimum

Back to top