Significance Check Lab¶

Run the Significance Check Lab MicroSim Fullscreen

About This MicroSim¶

The learner will judge, for each of eight comparisons of two rates, whether the difference is statistically significant at the 5% level using the rule that the absolute value of z must be at least 1.96. Evaluate-level work is judging against a criterion. Committing a verdict before z appears makes the learner weigh the size of the gap against the sample size, and the boundary item at z = 1.92 forces attention to the 1.96 cutoff.

This MicroSim belongs to the chapter Measurement Rigor Kpi Reporting. It uses JavaScript with the shared quiz kit in sims/shared-libs/. All data in it is illustrative.

How to Use¶

Read the case and commit an answer before any result is shown.
Press Check (or Commit). The sim shows whether the answer is correct and explains why.
A wrong answer gets one more attempt; after the second miss the answer is shown.
After the last item an exploration panel unlocks. It is practice only and is not scored.

Iframe Embed Code¶

You can add this MicroSim to any web page by adding this to your HTML:

<iframe src="https://Kschuman-aon.github.io/genai-roi/sims/significance-check-lab/main.html"
        height="322px"
        width="100%"
        scrolling="no"></iframe>

Lesson Plan¶

Audience¶

Professionals and graduate students learning to estimate and defend the cost and return of generative AI.

Duration¶

10-15 minutes

Bloom's Taxonomy¶

Evaluate (judge)

Prerequisites¶

statistical significance check, standard error, z statistic, pooled rate

Activities¶

Predict (5-10 min): work through the committed challenges before any answer is revealed.
Explore (5 min): use the controls unlocked after the last challenge to see how each input changes the result.

Assessment¶

For each of eight comparisons the learner commits a verdict, "Significant" or "Not significant", before z is shown. A verdict is correct when it matches the Content table. Mastery is 7 of 8 correct on the first attempt. Exploration is not evidence.

Common Misconceptions¶

(1) A difference that looks large in percentage terms is significant. (2) A non-significant result proves there is no difference. (3) Sample size does not matter if the rates are the same.