Page 716 - ISC PROCEEDINGS 21.4

The four conversational tasks are summarized according to their learning objectives
in Table 1.
Table 1. Structured conversational tasks by learning objective
Core Learning Practice
No. Conversational Task Content
Objective Article
Distinguish IRR, Students query the AI to understand why the
aOR, RR, HR by SCCS design uses IRR instead of aOR/RR; Maret-Ouda
et
J
al. J
1 statistical nature interpret the IRR value and 95% CI in clinical Gastroenter
and conditions of language; analyze the significance of a CI
application that does not include 1 ol. 2023 [16]
Students query the AI to perform step-by- Bhatt DL et
Calculate and step calculation of ARR and NNT from two- al.
interpret ARR
2 group event rates; interpret the clinical (COGENT).
and NNT from
clinical trial data significance of NNT; critique the limitations NEngl J Med.
of HR alone for prescribing decisions 2010 [18]
Read and
interpret meta-
analysis results; Students query the AI to interpret pooled RR
understand the and confidence interval; analyze why a Wu Z et al.
Front
3 relationship large meta-analysis may not reach p < 0.05; Pharmacol.
between sample evaluate I² and the remaining uncertainty
size, effect size, regarding the PPI-pneumonia association 2025 [17]
and statistical
significance
Identify and Students query the AI to enumerate and
control for classify potential confounders in the SCCS Maret-Ouda
confounding design and meta-analysis; evaluate the et al. [16] &
4 Wu et. al.
variable in authors’ confounding control strategies; [17]
observational compare confounding control strength
studies between SCCS and RCT (combined)

Each conversational task comprises 3-4 cognitively oriented questions following
Bloom’s taxonomy levels 2-5. The instructor did not intervene in the content of student-AI
interactions. Guidance was provided before each session, and debriefing was conducted
afterward.
2.6. Outcome variables and indicators
Primary outcome: proportion of students achieving correct answers for each
pharmacoepidemiological indicator domain (binary variable; unit: percentage points).
Secondary outcomes: (1) task completion time (minutes); (2) subjective self-confidence
(Likert 1-5); (3) comparison of results between ChatGPT and Gemini groups at T1.
2.7. Statistical analysis
All eligible students enrolled in the Pharmacoepidemiology course were included
using census sampling. Because the study aimed to evaluate an educational intervention
within a single class, no a priori sample size calculation was performed.
Binary variables (pre-post correctness rates in the same individuals): McNemar test.
Normally distributed continuous variables (task completion time): paired t-test. Ordinal
variables (Likert): Wilcoxon signed-rank test. Independent comparison of the two AI

715