Quality Score

In this article you will find details on the Quality Score, including:

What the Quality Score is

The Zappi quality score looks at 14 signals of quality at an individual respondent level and creates a composite score.  It looks at things such as gibberish/nonsense, straight lining across questions, richness of response, repetition, irrelevant or offensive responses.  
It is used after the survey has finished for that respondent to look holistically at all of the signals of quality rather than each in isolation and is therefore a very powerful measure of quality.  
The quality score is looked at across respondents and research projects to continuously ensure and improve the quality of our data:
  • Group markets according to confidence in quality
  • Compare sample providers
  • Design better surveys 
  • Detect bots and fraudulent respondents

How we have used the Quality Score to group markets in to red/amber/green on quality

We know that online sample quality is different by market (all suppliers and agencies will acknowledge this). We therefore use our quality score to put different checks in place to assure the quality in all the countries.
We use the quality score to look across respondents and research projects to group markets according to confidence in quality.
The confidence in quality dictates which tier a market falls into, which customers can access that market and how we align our research and extra quality checks in line with the known risk.  
  • Green markets - hit benchmark on quality score
  • Amber - quality score is greater than 1.5x lower than the benchmark
  • Red - quality score is greater than 2x lower than the benchmark

The quality checks we do for each of the market groups and implications

In all markets, we use Zappi’s market leading automated checks to remove poor quality responses (bots or from people).  In markets with the greatest risk (red/amber), we introduce an additional layer of human checks on top of the automated checks through the respondent quality service.
Through this service, a consultant is manually checking every project according to a set of specified criteria.
Poor quality responses are removed from the dataset and the study is refielded to ‘top up’ data. In red markets these ‘top up’ responses are then checked and this is followed by a second round of refields.
Whilst these human checks slow down your access to data in the higher risk markets, they ensure that the data you act upon has been checked and validated before your decision making, so you can act in confidence.  The responses removed during these manual checks are providing a training set of data we can use to train our automated checks to detect more bad quality responses in the future, which will help us get back to super speed even in the difficult markets.   

What we do for new or less frequently used markets

Markets that are new or less frequently used are all initially in the ‘yet to be classified’ group of markets.  
In order to identify which group (red, amber, green) they fall in to based on our quality score, we need to have run a minimum of 5 projects within a single solution (to ensure enough data from a consistent survey - same length, number stimuli, number of open questions etc)
Until we can classify them, we implement the same rigorous checks as we do for amber markets: Zappi’s market leading automated quality checks AND Human respondent quality service. This adds some time to fieldwork but ensures you can act confidently on results.

Which markets fall into which tier

Share the table with others using this link

Region RED Markets AMBER Markets GREEN Markets YET TO BE CLASSIFIED markets
(Follows Amber checks)
Europe, Middle East and Africa Egypt, India (English), India (Hindi)  Russia, Saudi Arabia, Saudi Arabia - (English) Austria, Belgium (Dutch), Belgium (French), France, Germany, Ireland, Italy, Netherlands, Nigeria, Norway, Pakistan, Poland, Slovakia, Slovenia, South Africa, South Africa (Afrikaans), Spain, Sweden, Switzerland (French), Switzerland (German), Switzerland (Italian), Turkey, United Arab Emirates - (English), United Arab Emirates (Arabic), United Kingdom Algeria, Bahrain, Bulgaria, Congo (Dem. Rep.), Croatia, Czech Republic, Denmark, Estonia, Ethiopia, Finland, Ghana, Greece, Hungary, Israel, Jordan, Kazakhstan, Kenya, Kosovo, Kuwait, Latvia, Lithuania, Morocco, Mozambique, Oman, Portugal, Qatar, Romania, Rwanda, Serbia (Latin), Tanzania, Ukraine, Uzbekistan
Asia Pacific Japan, Singapore, Taiwan, Vietnam
Australia, China, Indonesia, Korea (South), New Zealand, Philippines, Philippines (English), Thailand Cambodia, Hong Kong, Malaysia, Malaysia (English)
North America Canada (English), Canada French, United States, United States - (Spanish)
Latin America Argentina, Brazil, Chile, Colombia, Mexico, Peru Costa Rica, Ecuador, El Salvador, Guatemala, Haiti, Jamaica, Panama, Puerto Rico