Norms
Overview
Making the right comparisons is critical to decision making. With Zappi, you can benchmark in a flexible, dynamic way, enabling you to draw on the best available comparisons for the decision you need to make.
You can use an aggregate of prior survey responses, a norm, as your benchmark. Comparing your results to a norm allows you to determine "what good looks like" for key metrics.
A benchmark can be:
- Shown alongside survey results and aggregated in the same way as in your report. They are most often a mean score or a top box, and sometimes tested against for statistical significance.
- Used in the calculation of a percentile score, to show where a tested idea scores in comparison to the rest of the database
How does Zappi calculate norms?
Zappi offers multiple different types of norms to suit your particular needs.
| Norm Type | Description |
|---|---|
| Curated |
Zappi Curated Norms are country-specific, high-quality benchmarks designed to mirror the real-world marketplace. They are fully funded and carefully curated by Zappi to accurately reflect the market. Each norm is updated annually to remain fresh year-to-year. Use Case: This type of norm gives you a benchmark that reflects the real market. It gives you an idea of how your ad or product would perform in the real world, where it needs to compete for customer attention alongside ads or products from all other categories at the same time. |
| Fixed |
A fixed norm is built by Zappi, but is not continuously updated like a Curated Norm. A fixed norm can be requested by a customer to fit a particular need. These norms may be representative of different groups of adverts or ideas. They are not continuously updated, though customers can request an update to their customized norm later on if needed. Use Case: This type of norm is often purpose-built for specific scenarios. For example, our Super Bowl Norms are great for testing against ads that were created specifically for the event. |
| Organic |
Organic norms are created by aggregating the results of a measure across all of the assets tested in Zappi. Most often these take the form of country norms or country/category norms, made up of all the data across Zappi that meets the chosen criteria. Each customer can also have their own organic norms, created by their own past testing. These norms include all historical data and evolve over time as more assets are tested. Use Case: Organic norms allow you to compare your survey results to a large pool of similar results from anything tested in Zappi. Organic customer norms allow you to compare your new assets to your previous assets that you know performed well in-market. You can also run tests on competitor’s ideas and compare your concepts to theirs. |
Which measures in my results have norms?
In general, norms are generated for measures that are asked the same way in every survey. Wherever norms are available, you will see them at the top of your Quick Report, and on your charts.
Norms in Quick Reports
You can see what norm is applied to your quick report by hovering over the heading.
Important Note!
The norm used in your quick report will reflect a snapshot in time when the project was run. If you come back to your project later and re-analyze or customize your charts, you may end up with slightly different numbers, especially if you use a different norm during that analysis.
We keep the original norm for the quick report because it is a snapshot in time of how your concept performed at the time of testing. You should be able to refer back to this data when you're reviewing how your ad actually performed in the real world vs. the prediction and be confident that the data shows the same result as it did when you made your decisions. We want you to be able to review the data behind your decisions in the future and be confident that the quick report still shows the data that you based those decisions on.
Norms in Charts
Some charts use norms as a benchmark comparison. Usually, they have a default norm applied.
To analyze using a different norm, select the Norms tab next to the chart and scroll to compare your results to other populations. Often these are either less specific, such as a global or country norm with no further filtering, or more specific, like a country norm filtered by your category, or stage of development of the asset like an animatic, or final cut.
You can also specify the Norm date in the norms tab.
Selecting a norm date
To provide more flexibility to your analysis, you can select the date the norm will be calculated for the majority of our products. Your options are:
- The 1st of the current month
- The 1st of the month your selected study was created in
- Today
For automatically generated Quick Reports, the norm is set for the date the study was completed.
You can see what norm is applied by checking the blue bar at the top. If the bar is grey it means the norm has not yet reached the minimum number of assets to make comparison statistically relevant.
Norms FAQ
What assets make up the norms?
For Organic and Fixed norms, you can easily see the number of assets included in each norm next to the name. For transparency, the number of tested assets in the norm you select is also included in the footnote of any slide you share or export from the platform.
We do not allow anyone to view the actual assets in norms that include proprietary data from our customers.
Zappi Curated Norms for Advertising are the only exception. We keep a list of every ad that makes up the Curated Norm for each country for full transparency.
How many assets are needed to create a norm?
20 stimuli is the minimum to build a norm. This is the smallest number that can give you a statistically relevant comparison. As databases grow and diversify, the norm will become more stable. While the norm includes less than 50 cases, it’s best to use a Curated Norm or your own internal or competitive benchmarks alongside these smaller norms.
I can't find a relevant norm, what should I do?
While our overall database includes hundreds of thousands of stimuli, there are instances in which there isn't a norm available for a particular market, category, or industry.
If you are in this situation here are some suggestions:
- Use your own benchmark made up of fewer, but highly relevant cases including your previous ideas/products/ads and those of your competitors.
- If you have no previous testing, include a benchmark cell (asset) so you can compare the results. This could be an ad/idea/product that you know has been successful in-market, a competitor you want to beat, or a previous ad from within a campaign.
- Use a broader geographic region, or adjacent category using our advanced norm functionality to filter.
Norms for early stage ad testing
Since early stage testing is a little nuanced, check out our article on Amplify Early stage ad testing.