Understanding small samples & margin of error
When reviewing subgroup/filtered Idea scores, it’s important to understand the impact reduced data has on our confidence in results.
When n=80 for a filtered group, we are only confident that the true interest score for that group is within around +/- 11% of the interest score in the dashboard (the margin of error). Note that the smaller the sample size is, the higher the margin of error is going to be. And at n=50, the margin of error is close to +/- 14%.
See the below table for each sample size and its corresponding margin of error:
Margin of Error (+/- %)
This uncertainty will be true for any given question, for example, top 2-box in a monadic grid.
For this reason, we warn you when sample sizes drop below n=80, and ideally filtered subgroup analysis is done with a base of at least n=200.