Norms

What is a norm?

Making the right comparisons is critical to decision making. With Zappi, you can benchmark in a flexible, dynamic way, enabling you to draw on the best available benchmarks for the decision and the stimulus at that moment in time.  

A norm is a set of aggregate survey responses, against which a similar subject or answer set is compared. Simply put, comparing your results to a norm allows you to determine "what good looks like" for key metrics.

How does Zappi calculate norms? 

Norms are created by aggregating the results of a measure across all of the assets tested in Zappi that meet the criteria you've selected.

A norm can be:

• Shown alongside survey results and aggregated in the same way as in your report. They are most often a mean score or a top box, and sometimes tested against for statistical significance.
• Used in the calculation of a percentile score, to show where a tested idea scores in comparison to the rest of the database

What norm or benchmark should I use?

You can use both your own norm and the recommended country or country  > category norm to inform your business decisions. You’ll also find recommended norms within our solutions that go beyond comparing only to the research you have done in the platform. These recommended benchmarks come from Zappi research experts who have created the solutions based on an understanding of the business decisions that need to be made and how best to analyze different datasets. You can read more about these below.

Country and Category Norms:

Zappi stores the results for every survey that we have ever run. When another survey is run with the same measures, we can provide benchmarks based on the data that we have collected previously. Most often these take the form of country norms or country > category norms. You can compare your results to other relevant surveys, to contextualize the performance of your idea.

Use Case: This type of norm is best for concepts that you want to compare to similar concepts in the same country or country and category combo.

Customer Norms:

With Zappi, you can build up your data asset and tag the characteristics of your ideas, ads and products so you can learn and compare them over time.

One way to learn from this data asset is to create your own norms and benchmarks to inform better decisions. You can curate specific benchmarks of stimuli that you know have been successful in-market. Or stimuli that share similar characteristics to each other. You can even benchmark when looking at sub groups you have predefined, like gender or age.

Use Case: This type of norm works best for comparing new ideas to your previous ideas that you know performed well in-market. You can also run tests on competitor’s ideas and compare your concepts to theirs.

Zappi Curated Norms:

Our curated norms are built from a set of up to 100 ads in select countries tested using Zappi Amplify TV. The norms are fully funded and carefully curated by Zappi to reflect the market. Each norm is updated annually to remain fresh year-to-year. Read more about Zappi Curated Norms.

Use Case: This type of norm gives you a benchmark that reflects the real market. It gives you an idea of how your ad would perform in the real world, where it needs to compete for customer attention alongside ads from all other categories at the same time.

Back to top

Which measures in my results have norms?

In general, norms are generated for measures that are asked the same way in every survey. Wherever norms are available, you will see them at the top of your Quick Report, and on your charts. 

Quick Reports norms

You can see what norm is applied to your quick report by hovering over the heading.

We keep the original norm for the quick report because it is a snapshot in time of how your concept performed at the time of testing. You should be able to refer back to this data when you're reviewing how your ad actually performed in the real world vs. the prediction and be confident that the data shows the same result as it did when you made your decisions. We want you to be able to review the data behind your decisions in the future and be confident that the quick report still shows the data that you based those decisions on.

Charts

To analyze using a different norm, select the Norms tab next to the chart and scroll to compare your results to other populations. Often these will be either less specific, such as a global or country norm with no further filtering, or more specific, like a country norm filtered by your category, or stage of development of the asset like an animatic, or final cut.

You can also specify the Norm date in the norms tab.

You can see what norm is applied by checking the blue bar at the top. If the bar is grey it means the norm has not yet reached the minimum number of assets to make comparison statistically relevant.

Back to top

What assets are in norms?

In the image above, you can easily see the number of assets each norm is based on next to the label. For transparency, the number of tested assets in the norm you select is also included in the footnote of any slide you share or export from the platform.

We do not allow anyone to view the actual assets in norms that include proprietary data from our customers. 

What is the minimum number of assets for a norm to populate?

The minimum is 20 stimuli to build a norm. This will help you benefit from a relevant comparison as soon as possible. As databases grow and diversify, they will become more stable. While the norm includes less than 50 cases, we encourage using your own internal or competitive benchmarks alongside these smaller norms or a curated norm.

I can't find a relevant norm, what should I do?

While our overall database includes hundreds of thousands of stimuli, there are instances in which there isn't a norm available for a particular market, category, or industry.  

If you are in this situation here are some suggestions:

• Use your own benchmark made up of fewer, but highly relevant cases including your previous ideas/products/ads and those of your competitors.

  If you have no cases, include a benchmark cell (asset) so you can compare the results. This could be an ad/idea/product that you know has been successful in-market, a competitor you want to beat, or a previous ad from within a campaign.  

• Create a norm using a broader geographic region, or adjacent category using our advanced norm functionality.
  • Not available in Amplify and Activate It suites.

Early stage ad testing and norms

Since early stage testing is a little nuanced, check out our article on Amplify Early stage ad testing.

Back to top

More norm options

Selecting a norm date

To provide more flexibility to your analysis, you can select the date the norm will be calculated for the majority of our products. Your options are:

• The 1st of the current month
• The 1st of the month your selected study was created in
• Today

Advanced norms:

This option enables you to define a custom selection of countries and industries to be included in a norm, to suit your needs.  

For example, you may not have conducted sufficient studies in Colombia, for instance, to be able to build a country norm, so you might roll up those that you have done with any others you have done in neighboring Central American countries to create a custom norm that is more relevant to your products or advertisements than a global norm would be.

User- or customer-defined norms:

These are norms made up of all the assets you've tested within your platform domain. This enables you to compare your results to your own historical performance. For example, you might compare your ads or concepts with the best you ever tested, or you might compare them with any of your other tests that have the same tags, so you can analyze your performance on particular themes. In most Zappi tools with norms, the minimum number of stimuli required to view a user-defined norm is 2. The exceptions to this are:

• Amplify suite, Prioritize It = 20 minimum for percentiles, 2 minimum for mean scores
• MARC = 20 minimum

Back to top