Reading: "Gamified Flashcards Improve Exam Scores in Introductory Biology" (fictional)

This reading covers only the abstract, a methods excerpt and a results excerpt. I have no introduction, full methods, tables, figures or discussion.

1. Pass 1: What is this paper?

• Question: Does a gamified flashcard app improve learning in introductory biology?
• Type: A non-randomized comparison of two course sections (observational or quasi-experimental).
• Main claim: "We conclude that gamified flashcards cause large improvements in learning and should be adopted widely."
• Headline result: "Section A scored 7.4 points higher on the final exam (p<0.01)."
• Why it matters (per the authors): They say schools should adopt the app widely.

Plain-language summary: Two sections of one biology course were compared. One section was offered the app and the other wasn't. The offered section scored higher on the final. The authors conclude the app caused the gain. The design can't support that. The two groups differ in many ways besides the app, so the gap could come from those differences instead.

2. Pass 2: How does it work?

• Sample: Section A (n=112) was offered the app and section B (n=98) was not. Both sections were at one university. Students chose their section at registration.
• Intervention: The app was optional, and 61% of section A installed it. That is about 68 students, though the excerpt doesn't state the count. Use of the app beyond installing it is not stated.
• Outcome: The final exam score. Each instructor wrote their own exam questions.
• Comparison: Section A vs. section B means. The excerpt also gives a split of section A by installation:

• Analysis: Only a p-value is reported (p<0.01). The test used, any adjustment for covariates, and confidence intervals are not stated.
• Figures and tables: None were provided. The numbers above come from the text.

Terms:

• Confound: a third factor that differs between groups and could explain the result.
• Self-selection: participants choose their own group or treatment.
• Intention-to-treat: comparing everyone offered the treatment with everyone not offered, regardless of use.

3. Pass 3: Does the evidence support the claim?

• Does the comparison fit the claim? No. Sections differ in instructor, meeting time (Monday morning vs. Friday afternoon) and student self-selection into sections. Any of these could produce a 7.4-point gap.
• Other explanations:
  • Instructor: teaching quality and style are fully tangled with the app. With one instructor per condition, they can't be separated.
  • Timing: students who choose Monday mornings may differ in motivation or schedule from those who choose Friday afternoons.
  • Exam: "Final exam questions were written by each instructor." The two sections may have sat different exams, so score differences may reflect difficulty or style. This is a measurement problem.
  • Self-selection into the app: Among section A, installers scored 81.2 and non-installers scored 70.9. Motivated students are likely both to install and to score higher. Non-installers (70.9) look about the same as section B (69.8). This is consistent with the app doing nothing and the gap being driven by who installs. It is also consistent with a real effect, so it doesn't settle the question.
  • Chance: with only two sections, there is little replication, and a p-value can't fix confounding.
• Uncertainty and effect size: No confidence intervals are given. The standard deviation and the scale of the exam (out of 100?) are not stated, so I can't say whether 7.4 points is "large." The 7.4 figure also looks diluted by the 39% who didn't install. The abstract's own number can't tell us how big the effect is for users.
• Choices after seeing results: Not stated. The installer vs. non-installer split looks like a post hoc analysis, but I can't confirm it.
• Conclusions beyond the data: Yes, in two ways. "Cause" is not justified by non-random assignment with these confounds. "Adopted widely" generalizes from one university, one course and two sections.
• Limitations stated: Not stated in the excerpts. The missing ones are instructor and time confounding, self-selection, and different exams.
• Checkability: "No data or code is shared." Nobody else can re-analyze this.

4. Takeaways and open questions

What you can reasonably conclude: In this study, students in the section offered the app scored higher on average. Students who used the app scored higher than those who didn't. Nothing here shows the app caused either difference.

What remains uncertain: Whether the app has any effect, how large it is, and whether it would work elsewhere.

What would strengthen the claim:

• Random assignment of students to app access, within the same section and instructor.
• A common exam, graded blind to condition.
• Baseline measures such as prior GPA or a pre-test, adjusted for in the analysis.
• Several instructors or sections, so that instructor effects can be separated from the app's.
• Reporting effect sizes with confidence intervals, and sharing the data and code.

Questions to check in the full paper:

• Were the exams the same or overlapping?
• Are there baseline comparisons of the two sections?
• How was app usage measured?
• Was the analysis pre-registered?

What to read next: Look for randomized trials of spaced-repetition or gamified study tools in higher education, and for a review of how such studies handle self-selection.

5. What could not be assessed

• The statistical test, standard deviations, and effect size in standard units.
• Whether the exams were the same.
• Baseline equivalence of the sections.
• The authors' own limitations discussion.
• How "gamified" was defined and how much students used the app.
• Anything in the data or code, which are not shared.

This is a critique of the evidence in the excerpts. It isn't a final judgment on the paper or the authors.