In this article:

A stat test, or more formally a hypothesis test seeks to find out if an observed effect is real or if it is random. To do this we compare an observation from a sample to some other expectation. For example, we may observe in a study that women like chocolate more than men, and we could stat test this against the expectation that men and women like chocolate equally.

Stat testing your Idea Split is like a lemonade stand competition between two kids, Abigail and Bradley. In this competition, we want to find out if one kid sells significantly more lemonade than the other.

Imagine you're watching a lemonade stand competition between Abigail and Bradley. Both kids are selling lemonade to people passing by, but we don't know who sells more. To find out, we count how many cups of lemonade each kid sells.

This is similar to what the Idea Split Stat Test does. It looks at two Ideas (like Idea A and Idea B) and checks if there's a big difference in the responses.

Before the competition starts, we guess that both kids will sell the same number of lemonade cups (this is our starting guess, or "null hypothesis"). But we're open to being proven wrong (this is our "alternative hypothesis").

After the competition, we compare the number of lemonade cups Abigail and Bradley sold using the stat testing. If the test shows a big enough difference in their sales, we can say our starting guess was wrong and that one kid sells significantly more lemonade.

So, the Stat Testing is like a tally counter we use in a lemonade stand competition to find out if there's a big difference between the number of lemonade cups two kids sell. It's very useful in many areas, like figuring out which kid is a better lemonade seller.

Confidence level is how confident we are that the answers are representative. Testing at a 95% level means we'll only find significance when there is less than 5% chance the difference is random. In other words we are 95% confident the difference in results is non-random.

We've made using Stat Testing in Idea Splits super simple! All you need to do is toggle the 'Stat Testing' button:

For the Stat Testing to work, you will need to pin the idea you want to test all other ideas against by selecting the pin button.

Interpreting your results is very simple! Simply put: If the result is highlighted, it is statistically significant.
​
If it is green, it is higher than the pinned Idea in a statistically significant way.
​
If it is red, it is lower than the pinned Idea in a statistically significant way.
​