Quota sampling for surveys and how to get a balanced sample

Most survey samples are not wrong because of the questions. They are wrong because of who answered.

Open a survey to a panel and the people who respond first are not a random slice of anyone. They are the people who check their phone most, have the most free time, and are most interested in your topic. Leave it running and you get a sample that looks big and tells you about the wrong people.

Quotas fix this. You decide the shape of the sample before fieldwork starts, and recruitment stops for each group once it is full.

What a quota is

A quota is a target for how many completed responses you want from each group. For example:

• 50% female, 50% male
• 40% aged 25 to 34, 35% aged 35 to 44, 25% aged 45 and over
• 60% living in the UAE, 40% in Saudi Arabia

When a group reaches its target, new people from that group can no longer take part, and the remaining places go to the groups that still need people.

Why it matters more in MENA

Online panels across the region skew in predictable ways. Depending on the audience, you will often see more women than men, more young people than older ones, and more people in big cities than elsewhere. Nationality mix inside the GCC adds another layer: a "UAE sample" that is 90% one nationality is not a UAE sample.

None of this is a problem if you know about it and set quotas. It is a serious problem if you report the raw numbers as though they represent the market.

Quotas or weighting?

There are two ways to correct an unbalanced sample.

Quotas shape the sample during fieldwork. You get the right number of people from each group in the first place.

Weighting corrects the sample afterwards, by counting under-represented people more heavily in the analysis.

Weighting is useful, but it has a cost. If you only got 20 fathers and need them to count as 150, every one of those 20 answers is amplified, and a couple of unusual responses can move your headline numbers. Quotas avoid that by collecting enough people from every group to begin with. Use quotas for the groups you care about, and keep weighting for small final adjustments.

How to set good quotas

1. Start from the decision, not the demographics. If the business question is whether females and males want different things, you need enough of each to compare. That matters more than matching the population exactly.
2. Keep the number of dimensions small. Two or three dimensions is plenty for most studies. Every extra one makes groups smaller and fieldwork slower.
3. Make every group big enough to analyse. If you plan to compare groups, aim for at least 30 completes in each, and ideally 50 or more. A group of 12 can be reported, but not compared with confidence.
4. Use real proportions when you want market estimates. If the goal is "what share of parents would buy this", match the population you are estimating for. If the goal is comparison, equal groups are often better.
5. Check the panel can deliver. A quota for a group that barely exists on the panel will not fill, however long you wait. Look at panel counts before you launch.

Independent or interlocking

There are two ways to combine quotas across dimensions.

Independent quotas fill each dimension on its own. You get 50% women overall and 40% aged 25 to 34 overall, but not necessarily the right number of women aged 25 to 34.

Interlocking quotas set targets for every combination, such as women aged 25 to 34 and men aged 35 to 44. They are more precise, but the number of cells grows quickly and the small ones become very hard to fill.

For most product and marketing research, independent quotas on two or three dimensions give a balanced sample without slowing fieldwork to a crawl.

The mistakes that skew results

Counting places when people start, not when they finish. If places are counted at the start, people who drop out still hold them, and your groups close before they are full. Count completes.

Letting abandoned places sit forever. The opposite problem. Someone who starts and walks away should only hold their place for a limited time.

Quota on one thing, screen on another. If your quota is on gender but your screener asks "mother or father", make sure the two agree. Otherwise a mismatch quietly lands people in the wrong group.

Forgetting the screener. Quotas balance the people who qualify. They do nothing about people who should not be in the study at all. Screen first, then apply quotas.

Adding sample without keeping the balance. If you extend a study from 100 to 300 responses, the extra 200 should keep the same proportions, not go to whoever arrives fastest.

How quotas work on UsersArabia

Quotas are available for surveys and unmoderated tests, including surveys hosted on your own tool such as Tally or Typeform.

• Set target percentages by gender, age, country of residence, city, nationality, industry or income.
• Applicants who pass your screener and fit an open group are accepted automatically and can start straight away.
• Places count when someone finishes. Accepted participants hold a place for 24 hours, and if they do not finish, it opens up again.
• Each dimension is filled separately, and you can watch every group fill on the study page.
• Add more participants to a live study and the groups keep their percentages, so 50/50 of 100 becomes 50/50 of 300.

Set up a survey with quotas, or read about screening questions that actually filter.