Free analysis tool

Significance test for two percentages

You've compared two groups — two departments, two page versions, two consecutive months — and one is a few points higher. Before you act on it, this tool tells you whether that gap is significant at this sample size or could have come out the other way with a different sample.

Start for free Sample size calculation

Reports guide

Compare two groups

Group 1

Number of “yes” (x)

Sample size (n)

Group 2

Number of “yes” (x)

Sample size (n)

Alpha level (α)

0.05 is the common convention

72.5% vs 67.5%

Not significant

The difference is not significant. With this sample size, a gap this big could be nothing but sampling noise; don't base decisions on it.

z: 0.69
p-value: 0.49
Difference: 5%
Confidence interval: -9.2% to 19.2%

Why “72% vs. 68%” says nothing on its own

Suppose unit A has 73% satisfaction and unit B 68%, with 80 responses each. A 4-point difference looks real. But the p-value of this comparison is 0.49 — meaning that if there were really no difference between the two units, you'd see a difference this large or larger in about 49 out of every 100 samples. That's not “significant”.

The same two percentages with 800 responses per group give a p-value of about 0.08, and it is still not significant. The difference didn't change; the sample size did.

Rule of thumb: with groups under 100 people, don't treat differences of less than 10 percentage points as decisive unless this tool says otherwise.

What exactly this tool calculates

Two-proportion z-test (pooled variance)

p̂ = (x₁ + x₂) ÷ (n₁ + n₂)Pooled proportion under the “no difference” hypothesis
z = (p₁ − p₂) ÷ √( p̂(1 − p̂)(1/n₁ + 1/n₂) )Test statistic
p-value = 2 × (1 − Φ(|z|))Two-tailed; Φ is the standard normal distribution function
CI = (p₁ − p₂) ± z* √( p₁(1−p₁)/n₁ + p₂(1−p₂)/n₂ )Confidence interval of the difference

A p-value below the alpha level (usually 0.05) means the difference is “significant”: it is unlikely to be due to chance.
If the confidence interval includes zero, you can't say which group is higher.
“Significant” is not the same as “important”: with a very large sample, even a half-percent difference becomes significant, yet it may not be worth acting on.

How many responses it takes to see a difference

To detect a 5-point difference around 70% with 95% confidence and 80% power, you need about 1,320 responses per group. For a 10-point difference, about 330 per group is enough.

Required sample size per group (around 70%, 0.05 alpha, 80% power)
The difference you want to detect	Responses per group
3 percentage points	3,664
5 percentage points	1,320
10 percentage points	330
15 percentage points	147

These numbers are approximate and get somewhat larger for proportions near 50%.

If any of the four cells (yes/no for each group) is below 5, the normal approximation is unreliable. The tool warns you; in that case use Fisher's exact test instead of z, or collect a larger sample.

Where it comes in handy in Porsino

In Porsino reports, when you view results by unit, branch or any other field, the response count is written next to each percentage. Whenever you want to compare two groups, put those two numbers here.Reports guideExplains how to build segments.

Frequently asked questions

How is this test different from chi-square?

For comparing two proportions, the two-proportion z-test and the 2×2 chi-square (without correction) give exactly the same p-value; z² equals the chi-square statistic. The advantage of z is that it also gives the direction of the difference and a confidence interval.

Where does the 0.05 alpha level come from?

It's a common convention in the social sciences, not a law of nature. 0.01 is used for costly decisions and 0.10 for exploratory work. What matters is choosing the level before you see the result.

My two groups aren't the same size; is that a problem?

No. The formula works with different n₁ and n₂. It's just that the smaller group determines the overall precision of the comparison.

Can I compare the means of two groups too?

This tool is for proportions (percentages). For a mean score on a 1–5 scale you need a two-sample t-test, which requires each group's standard deviation; get that from the report's Excel export.