Statistical Methodology

This page documents exactly how PaperSurvey computes every statistic it reports. For each dashboard widget it gives the formula, the conventions behind it and a link to a worked help article. The same functions power the dashboards, the exports and the public calculators, so a number you see here matches the number you get after scanning a real survey.

Descriptive statistics Correlation and regression Significance tests Cross tabulation Surveys and scales Quiz and assessment Customer experience How we verify these formulas References

Descriptive statistics

Summarise a set of numeric responses: central tendency, spread and the shape of the distribution.

Mean

Meanx̄ = Σx / n
The arithmetic average of all numeric responses.

Conventions

For choice questions, each option counts as its numeric value.
Blank and non-numeric answers are excluded from n.

Read the worked example

Standard deviation

Standard deviations = √[ Σ(x − x̄)² / (n − 1) ]
How spread out responses are around the mean. A low value means responses are similar, a high value means they vary widely.

Conventions

Sample standard deviation (divides by n − 1), treating responses as a sample of your audience. Matches Excel STDEV.S and R sd().

Read the worked example

Descriptive statistics

Meanx̄ = Σx / n
Arithmetic average.
Medianmiddle value of the sorted responses
Half of responses are below this value. Less affected by outliers than the mean.
Modemost frequent value(s)
Shows all tied values; empty when no value repeats.
Standard deviations = √[ Σ(x − x̄)² / (n − 1) ]
Spread around the mean.
Variances² = Σ(x − x̄)² / (n − 1)
Squared spread around the mean.
Standard errorSE = s / √n
How precisely the sample mean estimates the true mean.
Coefficient of variationCV = (s / x̄) × 100
Spread relative to the mean, in percent.
Quartileslinear interpolation between closest ranks
Q1 and Q3 bound the middle half of responses.
Interquartile rangeIQR = Q3 − Q1
Width of the middle half of responses.
Skewness[n / ((n−1)(n−2))] Σ((x − x̄)/s)³
Asymmetry of the distribution. Positive means a longer right tail.
Kurtosisexcess kurtosis, bias corrected
How heavy the tails are compared with a normal distribution.
Geometric meanexp(Σ ln x / n)
Average for multiplicative data. Defined for positive values only.
Harmonic meann / Σ(1/x)
Average for rates. Defined for positive values only.
Outliersvalues outside Q1 − 1.5 IQR and Q3 + 1.5 IQR
Tukey fence rule.

Conventions

Sample formulas (n − 1) throughout, matching Excel STDEV.S, VAR.S, SKEW and KURT.
Percentiles use linear interpolation, the same as Excel PERCENTILE.INC and R type 7.
Blank and non-numeric answers are excluded from n.
Sample standard deviation (n − 1), matching Excel STDEV.S.

Read the worked example

Correlation and regression

Measure how two questions move together, with both linear and rank-based coefficients.

Correlation

Pearson rr = Σ(x − x̄)(y − ȳ) / √[ Σ(x − x̄)² Σ(y − ȳ)² ]
Strength and direction of the linear relationship between the two questions, from −1 to +1.
R²R² = r²
Share of variation in one question explained by the other.
Spearman ρPearson r computed on ranks
Rank-based correlation, robust to outliers and curved relationships. Ties receive average ranks.

Conventions

Only respondents who answered both questions are included.
p-values are two-tailed, from t = r √[(n − 2) / (1 − r²)] with n − 2 degrees of freedom.

Read the worked example

Significance tests

Test whether the differences and associations in your data are more than sampling noise.

t-tests

One-sample t-testt = (x̄ − μ) / (s / √n)
Tests whether the average response differs from the comparison value μ.
Average changed̄ = mean(after − before)
Average within-respondent change between the two time points.
Paired t-testt = d̄ / (s_d / √n)
Tests whether the average change is zero. Two-tailed p-value with n − 1 degrees of freedom.

Conventions

Two-tailed p-value with n − 1 degrees of freedom.
Sample standard deviation (n − 1).
By default μ is the scale midpoint; you can change it in the block settings.
Only respondents who answered both questions are included.
Two-tailed p-value with n − 1 degrees of freedom, where n is the number of pairs.
Only respondents with both a before and an after answer are included in paired statistics.

Read the worked example

Odds ratio and relative risk

Odds ratioOR = (a × d) / (b × c)
How much the odds of the outcome differ between the two groups in the 2 by 2 table.
Relative riskRR = [a / (a + b)] / [c / (c + d)]
Ratio of outcome rates between the two groups.
95% confidence intervalexp( ln OR ± 1.96 × √(1/a + 1/b + 1/c + 1/d) )
If the interval includes 1, the association is not statistically clear.

Conventions

When any cell is zero, 0.5 is added to every cell (Haldane-Anscombe correction) so the ratio stays defined.

Read the worked example

Cross tabulation

Compare two categorical questions in a contingency table and test them for independence.

Chi-square and Cramér’s V

Chi-square testχ² = Σ (observed − expected)² / expected
Tests whether the two questions are related. Expected counts come from the row and column totals.
Cramér's VV = √[ χ² / (n × min(rows − 1, cols − 1)) ]
Strength of the association from 0 to 1. Above 0.1 is small, 0.3 medium, 0.5 large.
Fisher's exact testexact hypergeometric probability, two-tailed
Used alongside chi-square for 2 by 2 tables, reliable even with small counts.

Conventions

Degrees of freedom = (rows − 1) × (columns − 1); no continuity correction is applied.
A warning appears when any expected cell count is below 1, or when more than 20% of cells have an expected count below 5, where chi-square is less reliable.
On 2 by 2 tables the Fisher exact p-value is shown alongside chi-square, and is treated as the primary result.

Read the worked example

Surveys and scales

Score multi-item scales and instruments, and measure how consistently their items hang together.

Reliability

Cronbach's alphaα = [k / (k − 1)] × (1 − Σs²ᵢ / s²ₜ)
Internal consistency of the k scored questions. Values above 0.7 are usually considered acceptable.
KR-20[k / (k − 1)] × (1 − Σpq / s²ₜ)
Cronbach's alpha specialised for right-or-wrong items.
Standard error of measurementSEM = s × √(1 − reliability)
Typical distance between an observed score and the underlying true score.

Conventions

Every processed respondent is included; a scored question they did not answer counts as 0 points.
Item and total variances use the sample formula (n − 1).

Read the worked example

Scale scores

Instrument totalsum of the respondent’s item points
Each respondent’s answers are converted to points and summed into one instrument total.
Standard deviations = √[ Σ(x − x̄)² / (n − 1) ]
Spread of instrument totals.
Severity bandsfirst band whose range contains the total
Each respondent is assigned to the first matching band.

Conventions

Only processed respondents with at least one scored answer are included; unlike the test-score widgets, a respondent with no scored answers is left out rather than counted as 0.

Read the worked example

MaxDiff

Best-worst scorescore = times chosen best − times chosen worst
Simple count-based MaxDiff score. Higher means more preferred.

Conventions

Items are ranked by net score; ties break by the number of best votes.

Read the worked example

Quiz and assessment

Grade tests, summarise scores and analyse how well each question performs.

Test scores

Mean scorex̄ = Σ score / n
Average total score.
Median scoremiddle value of the sorted scores
Half of respondents scored below this.
Standard deviations = √[ Σ(x − x̄)² / (n − 1) ]
Spread of total scores.
Pass rate(scores ≥ threshold / n) × 100
The threshold is the pass percentage applied to the maximum possible score.
Score distributioncount of respondents at each total score
Each bar counts respondents with that total score.

Conventions

Respondents with no scored answers are included with a total score of 0.

Read the worked example

Item analysis

Difficultyp = (correct answers / respondents) × 100
Percentage answering the question correctly. Higher means easier.
DiscriminationD = p(top 27%) − p(bottom 27%)
How well the question separates high scorers from low scorers. Values above 0.3 discriminate well.

Conventions

The top and bottom groups are the highest and lowest 27% of respondents by total score, the standard split from classical test theory.

Read the worked example

Customer experience

Turn feedback into the experience metrics teams track: NPS, CSAT and Customer Effort Score.

Net Promoter Score

Net Promoter ScoreNPS = % promoters (9-10) − % detractors (0-6)
Ranges from −100 to +100. Passives (7-8) count toward the total but not the score.

Conventions

Percentages are computed on all responses, then the final score is rounded to a whole number.
Periods with no responses are shown as gaps rather than zero.

Read the worked example

CSAT

CSATCSAT = (satisfied responses / all responses) × 100
Percentage of respondents at or above the satisfaction threshold.

Conventions

A response counts as satisfied when its numeric value meets the threshold set in the block settings (default 4).

Read the worked example

Customer Effort Score

Customer Effort ScoreCES = Σx / n
The average of all effort ratings.

Conventions

Blank and non-numeric answers are excluded from n.

Read the worked example

How we verify these formulas

Every formula on this page is verified against SciPy, the reference scientific computing library, by an automated test suite that runs on every code change. The same verified functions power the dashboards, the exports and the public calculators.

In practice this means both of our statistical engines, the one behind the dashboards, reports and exports and the one behind the public calculators and live editing, are checked against one golden fixture of SciPy results, so a paper survey, a web survey, an export and a calculator all agree to the last decimal. The How we verify our statistics article explains the process and shows how you can reproduce any number yourself from the raw exports.

References

The methods above follow the standard statistical literature. These are the primary sources for the conventions and corrections used.

Welch, B. L. (1947). The generalization of "Student’s" problem when several different population variances are involved. Biometrika, 34, 28–35. The unequal-variance t-test used when comparing two independent groups.
Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2, 110–114. The approximate degrees of freedom that pair with the Welch t-test.
Haldane, J. B. S. (1956) and Anscombe, F. J. (1956). The estimation and significance of the logarithm of a ratio of frequencies. Annals of Human Genetics, 20, 309–311; On estimating binomial response relations. Biometrika, 43, 461–464. The add-one-half correction applied to a 2 by 2 table so the odds ratio stays defined when a cell is zero.
Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley. The 1.5 times interquartile range fences used to flag outliers.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334. Cronbach's alpha, the reliability coefficient for multi-item scales.
Kuder, G. F. and Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2, 151–160. KR-20, the reliability coefficient specialised for right-or-wrong items.
Cohen, J. (1960) and (1988). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46; Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum. Cohen's kappa for rater agreement and Cohen's d for effect size.
Hyndman, R. J. and Fan, Y. (1996). Sample quantiles in statistical packages. The American Statistician, 50, 361–365. The type 7 percentile definition, matching Excel PERCENTILE.INC and R’s default.
Abramowitz, M. and Stegun, I. A. (1964). Handbook of Mathematical Functions. National Bureau of Standards. The rational approximations used to evaluate the normal, t, chi-square and F distribution functions.
Haberman, S. J. (1973). The analysis of residuals in cross-classified tables. Biometrics, 29, 205–220. The standardised and adjusted residuals reported alongside the chi-square test.

Get Started with PaperSurvey.io Software

Get Started

Start your 14-day free trial now, no credit card required.