The influence of ChatGPT on medical student learning: a systematic review of educational outcomes and cognitive impact

Журнал "Сеченовский вестник".  Лист редактора к этой статье можно посмотреть здесь.

Sechenov Medical Journal. Editor's checklist for this article you can find here.

РЕЦЕНЗЕНТ А / REVIEWER A

ПЕРВЫЙ РАУНД РЕЦЕНЗИРОВАНИЯ / FIRST ROUND OF PEER-REVIEW

Scientific quality: Grade B: Very Good

Language quality: Grade A: Priority publishing

This review aims to synthesize controlled evidence on the impact of ChatGPT as a supplemental tool in medical education, focusing on measurable outcomes in knowledge, clinical reasoning, and performance. Its primary contribution lies in identifying and critically appraising the first three randomized trials on the topic, demonstrating that structured ChatGPT feedback can improve or maintain short-term learning outcomes particularly in history-taking, routine clinical reasoning, and problem-based learning while highlighting the continued superiority of expert guidance for complex cases. Key strengths include rigorous PRISMA-guided methodology, clear and balanced interpretation of limited but high-quality evidence, and practical, cautious recommendations for integrating AI as an adjunct under faculty oversight.

Comments on Scientific Content:

1. Overstated Generalizability: Conclusions drawn from only three highly heterogeneous RCTs (different interventions, outcomes, learner levels) are inherently preliminary. The claim of effects being "consistent across contexts" is too strong. The text should more clearly state that these are separate, early signals, not an established, generalizable pattern.
2. Limited Outcome Validity: The reviewed evidence is confined to short-term, in-education assessments. The critical link to long-term clinical competence or patient outcomes remains entirely unstudied. The discussion must more explicitly caution against equating improved test scores with enhanced clinical ability.
3. Key Methodological Flaw in PBL Trial: The positive Mini-CEX findings from the unblinded, single-assessor study (Hui et al., 2025) carry a relatively high risk of detection bias. These results should be labeled as preliminary and requiring an additional validation.
4. Poor Reproducibility of Interventions: A substantial weakness across included studies is the lack of detail for replicating the AI interaction (exact ChatGPT version, prompt engineering, output validation). The review should recommend more trials as a core requirement for scientific rigor and reproducibility.

Specific Comments on Scientific Content, Accuracy, and Clarity:

1. Abstract, Line 6-7: Preferably to use Simple Past (Not Perfect) tense: "Searches... identified three trials..."
2. Eligibility Criteria: The policy for mixed-population studies seems to be a potential source of selection bias, and this limitation should be acknowledged.
3. Data Extraction/Results: try to report how "disclosure of AI use" was handled across studies and if it correlated with outcomes.
4. Results (Chinese Trial): The link between Mini-CEX gains and the intervention's "emphasis on question framing" is speculative; so, this link should be rephrased as a hypothesis.
5. Table 1, Hui et al.: For D2, explicitly state that the core concern is the unblinded single-rater Mini-CEX assessment, not just participant unblinding.
6. Discussion, "Strengths...": The claim of "generalizability" is too strong. Revise to state that positive signals in different settings merit further study, not that generalizability is proven.
7. PRISMA Flow Diagram: Ensure the provided diagram matches the narrative counts exactly (1,481 screened, 297 assessed, 3 included).

CONCLUSION: minor revision.

SECOND ROUND OF PEER REVIEW

Thank you for your consideration of the proposed changes to the text of the article. At this point, I believe the article can be published.

CONCLUSION: accept for publication

РЕЦЕНЗЕНТ B / REVIEWER B

ПЕРВЫЙ РАУНД РЕЦЕНЗИРОВАНИЯ / FIRST ROUND OF PEER-REVIEW

Scientific quality: Grade B: Very Good

Language quality: Grade A: Priority publishing

Strengths

1. The paper is a genuine systematic review with a clear PICO question, restricted to controlled studies in medical students.
2. The search strategy across five databases, eligibility criteria, screening and data extraction are described transparently; RoB 2 is applied appropriately, and important concerns (e.g. unblinded Mini‑CEX ratings, lack of registration of primary trials) are openly acknowledged.
3. The synthesis is structured by pedagogic role, outcome type, and task complexity. The authors draw appropriately cautious conclusions: ChatGPT can improve or maintain short‑term learning outcomes under supervision, whereas expert feedback remains superior for complex reasoning tasks.

Issues requiring revision

1. The evidence base is small. Please state explicitly that this reflects the very early stage of controlled research on ChatGPT, rather than selective inclusion, for example: “The small number of eligible trials reflects the early stage of controlled evidence on ChatGPT in medical education rather than restrictive inclusion criteria.” To improve transparency and reproducibility, could you please add the full search strings in the Supplementary Materials, including the exact date(s) the searches were run?
2. The review was not prospectively registered. This is not fatal for publication, but it does require clearer justification and mitigation: Add a sentence such as: “The protocol was not prospectively registered; however, eligibility criteria, search strategy, and synthesis methods were pre‑specified and consistently applied.” Please also address directly: Why was the review not registered in PROSPERO or a similar registry at inception, and would you consider registering a protocol for future updates or related reviews?
3. All included studies report immediate or ≤10‑day outcomes. The manuscript should more sharply distinguish short‑term learning outcomes (test scores, CRI‑HTI, KFQ, Mini‑CEX) from broader educational effectiveness and clinical/patient outcomes Please consider qualifying formulations such as “improves educational outcomes” to “improves short‑term learning outcomes” and add a brief paragraph in the Discussion/Limitations noting the absence of data on long‑term retention, clerkship performance or any downstream impact on clinical care.
4. In its current form, the manuscript appears somewhat unbalanced: the Discussion section is markedly shorter than the Strengths, limitations, and implications section and contains very few citations. A total of seven references for a systematic review is clearly insufficient. Please expand the Discussion, increase the number of references, and relate your findings more explicitly to the broader literature on AI and large language models in medical education.
5. Add 1–2 paragraphs summarizing potential risks of ChatGPT use (hallucinations, over‑reliance, bias, academic integrity, assessment integrity) and how the reviewed interventions attempted to manage these risks (disclosure, supervision, verification of outputs).

CONCLUSION: major revision.

SECOND ROUND OF PEER REVIEW

The authors have made significant improvements to the manuscript and addressed all of the reviewers' comments.

CONCLUSION: accept for publication

РЕКОМЕНДАЦИИ НАУЧНЫХ РЕДАКТОРОВ ЖУРНАЛА / RECOMMENDATIONS

OF THE SCIENTIFIC EDITORS OF THE JOURNAL

The editorial team of the journal follows international publication ethics standards and aims to ensure that all manuscripts meet the requirements of COPE and major scientific databases. As part of the editorial process, each manuscript is reviewed by a scientific editor.

Based on the editorial assessment, we kindly ask you to address the following revisions:

• Abstract

The abstract must be structured in the following format: Objective, Methods, Results, Conclusions.

It is recommended to present your own results and conclusions, rather than only summarizing the findings of the included studies.

• Keywords

Recommendation: please replace some keywords with terms that differ from those used in the title and abstract in order to improve indexability.

• Contact Information and Information about the authors

Please add academic degrees (e.g., MD, PhD, etc.) for all authors.

In addition, ORCID identifiers should be provided where applicable.

• Author Contribution

A more detailed author contribution statement is required (for each author individually), using the CRediT taxonomy:

https://www.elsevier.com/researcher/author/policies-and-guidelines/credit-author-statement

• Methods

Please specify the date of study selection in the main text. We saw this date in Supplementary file.

Also, add the initials of the reviewers who conducted the screening process.

• Figure (PRISMA diagram)

Please correct “Records identified from databases (n = 4)” to reflect the total number of records retrieved (i.e., the sum of all records in this block).

• Discussion

Please add references where they are currently missing.

Please revise the placement of references where they are incorrectly combined.

Additionally, please verify the relevance of several references. Some of them do not correspond to fragments of the manuscript. Alternatively, indicate which part of the cited source corresponds to the text of the manuscript in comments (preferably with a quotation).

• References

Please complete the references by adding DOIs and other missing bibliographic details