Page 715 - ISC PROCEEDINGS 21.4

2.4. Intervention protocol
Phase 1 - Allocation and Orientation (Week 0)
Students were allocated by simple randomization (lottery) to Group A (n = 19,
ChatGPT GPT-4o/4o-mini) and Group B (n = 18, Gemini 1.5 Pro/Flash). All students then
attended a 30-minute orientation session covering: (1) principles of effective prompt
engineering in academic contexts; (2) the mandatory rule of verifying AI responses against
original articles; and (3) how to identify AI-generated misinformation (hallucination).
Phase 2 - Baseline Assessment T0 (End of Week 0)
Students independently completed a baseline competency assessment consisting of
15 items (10 MCQs + 5 short-answer questions) covering all seven
pharmacoepidemiological indicator domains, without AI assistance. Actual task
completion time was recorded. The assessment was applied to two example articles
distinct from the T1 set.
Phase 3 - AI Intervention (Weeks 1 and 2)
Each 90 minute session followed the sequence: (1) 15 minutes of independent
article reading and analysis before AI use; (2) 60 minutes of AI-assisted learning using the
pre- designed conversational task protocol, with progressively increasing complexity; (3)
15 minutes of verification, cross-checking against the source article, and group discussion.
Task allocation:
Session 1 (Tasks 1-2): Interpretation of relative indicators (IRR, aOR, RR, HR) from
the SCCS article - Conversational Tasks 1 and 4.
Session 2 (Task 3): Calculation and interpretation of ARR and NNT from the COGENT
RCT - Conversational Task 2.
Session 2 (Task 4): Reading a meta-analysis; interpreting pooled RR, I², and the
implications of a result that does not reach p < 0.05 - Conversational Task 3.
Phase 4 - Post-intervention Assessment T1 (End of Week 2)
Students completed a parallel-form assessment on two new articles without AI
assistance, using the same format as T0.
2.5. Structured instructor - guided AI chatbot protocol
Rather than allowing students to use AI chatbots freely, the study implemented a
structured instructor-guided chatbot protocol in which the instructor pre-designed four
standardized conversational tasks with specified context, inline data, and cognitively
oriented questions. The task set was content-validated by two pharmacoepidemiology
experts and pilot-tested with five students prior to full implementation.
Task design principles: (1) full article context and specific numerical data were
provided inline to minimize the risk of AI hallucination; (2) question sequences followed
an escalating cognitive order from conceptual explanation (Bloom level 2) through applied
calculation (level 3) to clinical judgment (level 5); (3) students were instructed to ask the
AI for step-by-step explanations rather than final answers alone.
Before each session, the instructor: (i) distributed a task sheet describing the
conversational task and its corresponding learning objective; (ii) demonstrated for 10- 15
minutes how to query the AI and how to verify responses against the original article;
(iii) reminded students not to accept AI-generated answers without cross-
referencing the data in the article’s results tables. After each session, the instructor
conducted a 10- minute debrief on points where AI responses were inaccurate or
required additional clinical context.

714