The influence of ChatGPT on medical student learning: a systematic review of educational outcomes and cognitive impact

Abbreviations:

• AI – artificial intelligence
• CRI-HTI – Clinical Reasoning Indicator-History Taking Inventory
• KFQ – key-features questions
• LLM – large language model
• Mini-CEX – Mini-Clinical Evaluation Exercise
• PBL – problem-based learning
• RCT – randomized controlled trial

The advent of artificial intelligence (AI) applications, such as ChatGPT, has had a major impact on many industries, including medical education. As medical schools advance to include innovative technologies, the literature on how AI can be employed to enhance the learning outcomes of medical trainees continues to grow. OpenAI's sophisticated large language model (LLM), ChatGPT, has demonstrated competence in various domains of medical education. It has been reported to be useful in creating experiences for individuals who learn, developing critical thinking skills, and supporting problem-based learning (PBL) interventions among medical students [1][2]. In addition to providing tailored learning experiences, ChatGPT enables the consistent delivery of quality learning across contexts.

Despite these benefits, empirical evidence of the effectiveness of ChatGPT in medical education remains limited. Surapaneni [3] reported a performance gap between medical students and responses produced by ChatGPT for certain tests, which raises questions regarding accuracy and applicability. Moreover, the use of AI-based clinical reasoning and skills assessment raises important ethical and pedagogical issues, a theme that is further illustrated by the duality in perceptions of AI technology in clinical learning settings [4].

This review aims to synthesize controlled evidence on whether ChatGPT, used as a supplement to usual instruction, improves medical students’ knowledge, clinical reasoning, and near-term performance, with the goal of identifying educational effects, boundary conditions, and implications for safe curricular use.

MATERIALS AND METHODS

Protocol and reporting framework

This systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020) guidelines. The review question was framed using the PICO structure: What is the impact of ChatGPT on the educational outcomes and cognitive development of medical students? The protocol specified the inclusion/exclusion criteria, information sources, screening methods, data extraction fields, risk-of-bias assessment by study design, and a data synthesis plan. This review was not prospectively registered with PROSPERO. Registration was not completed at inception due to resource and time constraints during the initial protocol development phase. We will prospectively register any future update or extension of this review to strengthen transparency. Nevertheless, the eligibility criteria, outcomes of interest, and synthesis approach were finalised before screening commenced and were applied consistently throughout the review process.

Eligibility criteria

We included original empirical studies published in English from 2022 onwards that enrolled medical students and evaluated ChatGPT as part of an educational intervention with measurable results. Eligible designs included randomized controlled trials (RCT) or nonrandomized controlled trials, controlled cohort studies, and mixed-methods evaluations with quantitative endpoints. We excluded editorials, letters, opinion pieces, reviews, purely technical descriptions without analysable educational data, studies not involving medical students (unless a distinct medical-student subgroup could be extracted), non-learning uses of ChatGPT (e.g., administrative tasks), and non-English publications.

Information sources and search strategy

A comprehensive search was performed in five bibliographic databases with pre-specified Boolean logic adapted to each platform: PubMed, Scopus, ScienceDirect, SpringerLink, and Web of Science. Search strings combined terms for the technology (e.g., “ChatGPT,” “large language model,” “artificial intelligence”) with medical education terms (“medical students,” “medical education”) and outcome terms (“learning outcomes,” “performance,” “clinical reasoning,” “critical thinking”, Table 1). The literature search was conducted on 25 June 2025. Searches were limited to English-language records and publication years from 2022 onwards, where database filters permitted. No study design filter was applied during the search stage. The results were exported on the same search wave, merged in a reference manager, and de-duplicated prior to screening.

Study selection

The screening was conducted in two stages by two independent reviewers (L.A.C. and R.I.). Titles/abstracts were screened against the eligibility criteria, followed by full-text assessment of potentially relevant reports. Disagreements were resolved through discussion or, if required, by a third reviewer (M.A.M.). The screening workflow was supported by Rayyan¹, a web-based platform used to facilitate blinded title/abstract screening [5]. The citation-checking workflow was supported by scite.ai², a web-based citation analysis platform used as a supplementary tool for verification, including checking citation contexts. The PRISMA flow diagram documents the process and counts: 1481 unique records were screened after the removal of 478 duplicates; 297 full texts were retrieved and assessed; 294 were excluded based on eligibility (review articles, n = 74; no analyzable data, n = 132; wrong study design, n = 87; non-English, n = 1). Three studies met all the criteria and were included in the narrative synthesis (Fig. 1).

Data extraction and management

A structured data-extraction form, piloted on a subset of studies, captured the following: study setting and design; participant characteristics (stage of training, sample size); description of the ChatGPT/LLM intervention (role in the learning activity, prompt structure, timing relative to instruction, presence/absence of human oversight); comparator condition(s); outcome measures and instruments (e.g., written knowledge tests, Key-Features Questions, Clinical Reasoning Indicator–History Taking Inventory (CRI-HTI), Mini-Clinical Evaluation Exercise (Mini-CEX)); follow-up interval(s); statistical results (effect estimates, measures of variability, p-values); and author-reported implementation details (e.g., disclosure of AI use, debriefing, verification steps). Two reviewers independently extracted the data; inconsistencies were reconciled by consensus with reference to the source report.

Outcomes

The primary outcomes were objective educational measures aligned with the intervention’s learning aims, including near-term written knowledge performance and validated indices of clinical reasoning and skills (e.g., CRI-HTI subdomains and Mini-CEX ratings). Secondary outcomes included learner-reported indicators relevant to cognitive impact (e.g., perceived learning effectiveness, breadth of clinical exposure, and critical stance toward AI following disclosure and debriefing). Where available, immediate and short-delay (≤10 days) post-intervention results were collected to gauge the early retention.

Risk of bias and study quality assessment

Risk-of-bias judgements were performed at the study level by design category using validated tools. RCTs were appraised using Version 2 of the Cochrane tool for assessing the risk of bias in randomized trials (RoB 2) (domains: randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result). ROBINS-I (Risk Of Bias In Non-randomized Studies-of Interventions) and MMAT (Mixed Methods Appraisal Tool) were pre-specified for non-randomized and mixed-methods studies but were not applied because all included studies were randomized. Disagreements were resolved by discussion; where reporting limited firm judgements, domains were rated “some concerns” with justification.

Data synthesis and analysis

Given the small number of eligible studies and heterogeneity in interventions (LLM role, presence of structured feedback, task complexity), comparators, and outcome instruments, a narrative synthesis was specified as the primary approach. Planned quantitative synthesis required ≥ 5 trials with commensurate outcome measures; this threshold was not met because of non-overlapping instruments (CRI-HTI, Key-Features Questions (KFQ), Mini-CEX) across studies. If conditions permitted, continuous outcomes would have been summarized as mean differences or standardized mean differences with 95% confidence intervals using random-effects models, with heterogeneity quantified by I² and explored via subgroup analyses (e.g., pre-clinical vs. clinical stage, routine vs. complex case tasks, human oversight vs. AI-only feedback). Instead, we synthesized findings by mapping interventions to learning mechanisms (e.g., immediate task-specific feedback, rehearsal of history-taking, PBL scaffolding), outcome family (knowledge, clinical reasoning, clinical performance), and task complexity, noting consistencies and divergences across settings. Where immediate and short-delay outcomes were both reported, we qualitatively described patterns to indicate early retention.

RESULTS

Study selection and characteristics

Of the 1481 unique records screened, three RCTs met the inclusion criteria and were included in the review of educational outcomes in medical students. The studies were conducted in Germany, Turkey, and China, and each evaluated ChatGPT as a supplement to established teaching formats rather than as a replacement. The validated indices included CRI-HTI, KFQ, and Mini-CEX. The interventions fell into three pedagogic roles: (i) simulated patient encounters with LLM-delivered formative feedback, (ii) AI-generated written feedback on text-based clinical reasoning problems, compared directly with expert feedback, and (iii) ChatGPT-assisted PBL integrated into a brief clinical rotation. Outcomes comprised validated indices of clinical reasoning (CRI-HTI; KFQ), theory examinations, and Mini-CEX domains, with short-delay assessments reported. The sample sizes ranged from 21 to 129 randomized participants, with follow-up completion reported in all trials. Reporting of model version, prompting and output verification was limited across trials.

Interventions and comparators

In the German trial, pre-clinical students completed four six-minute LLM-simulated history-taking encounters; only the intervention arm received structured ChatGPT feedback after each case, generated against CRI-HTI rubric [6]. Two blinded human raters scored the transcripts (intraclass correlation coefficient was 0.924). The comparator was an identical simulation without feedback.

In Turkey, first-year students undertook five days of spaced ContExtended Question practice for urinary tract infections; arms differed only in the source of explanations provided after each step (expert-written vs. ChatGPT-generated) [7]. Clinical reasoning was measured using the KFQs immediately and 10 days later.

In China, fifth-year interns on a two-week urology rotation were randomized to PBL+ChatGPT (pre-class exploration and in-class discussion with AI support, with instructor oversight) versus traditional teaching [8]. The primary outcomes were a 100-point theory exam (pre-and three days post-lecture) and Mini-CEX ratings across seven domains; student satisfaction was secondary (Table 2).

Primary outcomes

Across the two reasoning-focused trials, ChatGPT improved or matched outcomes when deployed as a feedback engine, with caveats for complex cases.

In the German simulation RCT (n = 21 analyzed), after four sessions, LLM feedback improved final CRI-HTI scores versus simulation without feedback (3.60 ± 0.13 vs. 3.02 ± 0.12; F(1,18) = 4.44, p = 0.049; partial η² = 0.198), with gains concentrated in ‘creating context’ (p = 0.046) and ‘securing information’ (p = 0.018), but not focusing questions (p = 0.265). The rater agreement for transcript scoring was excellent (intraclass correlation coefficient was 0.924). These findings indicate that immediate task-specific LLM feedback can shift observable reasoning behaviors over a short training period.

In the Turkish RCT (n = 129 randomized; ≥115 tested), ChatGPT feedback vs. expert feedback produced no difference in overall KFQ scores at immediate testing (ChatGPT 74.7 ± 15.1 vs. Expert 78.5 ± 20.6; p = 0.26) or at 10 days (ChatGPT 76.0 ± 14.5 vs. Expert 78.0 ± 21.2; p = 0.57). For complicated urinary tract infection items in delayed testing, expert feedback outperformed ChatGPT (p < 0.001). Notably, the disclosure of AI use increased students’ critical approach to AI, with medium to significant effects. Together, these data suggest that ChatGPT can provide non-inferior formative explanations for routine problems, while expert oversight remains advisable for nuanced scenarios.

In the Chinese rotation RCT (n = 42), both groups improved from baseline, but PBL+ChatGPT achieved a higher post-course theory score at three days (93.90 ± 3.65) than traditional teaching (90.33 ± 4.08; p < 0.01). This short-term knowledge advantage was accompanied by higher Mini-CEX ratings in medical interviewing, clinical judgement, and overall competence, with no differences in other domains. Mini-CEX gains favouring PBL+ChatGPT were concentrated in interviewing, judgement, and overall competence. These domain-specific improvements may reflect the intervention’s emphasis on structured question framing and synthesis, however, this interpretation remains hypothetical. All assessments were completed within standardized time windows and by a single assessor, which supports procedural consistency but may limit blinding. Because a single, likely unblinded assessor completed the Mini-CEX, these skill domain gains warrant cautious interpretation.

Secondary outcomes (acceptability and perceived learning)

Medical students rated the LLM-supported activities favorably. In the German study, participants described the simulated encounters as realistic and feasible, mirroring the observed behavioral gains when feedback was present [6]. In Turkey, students in the ChatGPT feedback group developed a notably more critical outlook [7]. As a result, they became more skeptical about the accuracy and reliability of AI-generated information and more cautious in accepting AI content without question. In the Chinese trial, satisfaction was uniformly high after PBL+ChatGPT, with no reports of dissatisfaction [8].

Synthesis of effects and certainty

Because interventions and instruments differed across trials, meta-analysis was not appropriate; instead, narrative synthesis showed consistent benefits of immediate, task-specific feedback, parity with expert feedback for routine problems, and short-term knowledge and Mini-CEX gains with ChatGPT-assisted PBL under supervision.

Risk of bias

We appraised the risk of bias using RoB 2 across five domains (Fig. 2). Overall, the included trials showed a generally acceptable methodological profile for short-term educational outcomes. The randomization process was judged to be at low risk of bias in all three studies: Brügge et al. used a two-group randomized design with balanced baseline characteristics, Çiçek et al. applied computer-generated randomization with comparable groups, and Hui et al. used randomized allocation with equal group sizes.

Deviations from intended interventions were unlikely to materially affect the results in the German and Turkish trials, where training procedures, exposure, and assessment conditions were standardized across arms. In the Turkish trial, participants were blinded to the feedback source, and the only intended difference between groups was whether the explanations were expert-written or ChatGPT-generated. By contrast, the Chinese trial raised some concerns because intervention delivery was unblinded, which may have influenced performance-related outcomes.

Missing outcome data were judged to be at low risk of bias across the three studies, with minimal exclusions or complete post-course assessment reported and no evidence of differential attrition. Measurement of outcomes was also robust in two trials: the German study used two independent blinded raters and the validated CRI-HTI instrument, with excellent inter-rater agreement, whereas the Turkish study relied on objective standardized key-features scoring. In the Chinese trial, the theory test was objective; however, Mini-CEX was assessed by a single likely unblinded evaluator, raising concerns about detection bias. Therefore, Mini-CEX findings, although directionally favorable, should be interpreted with caution.

The domain “selection of the reported result” was conservatively rated as “some concerns” in all three studies because none reported prospective registration or a pre-specified analysis plan. Taken together, the overall risk of bias was judged as low to some concerns for the German and Turkish trials and as some concerns for the Chinese trial. The evidence base is therefore methodologically acceptable for evaluating short-term educational outcomes, but conclusions regarding clinical skills should remain cautious when based on unblinded performance ratings.

Summary of main findings

Taken together, the best available randomized evidence supports an optimistic but circumspect conclusion: when used to augment existing teaching, it provides immediate, structured feedback; by scaffolding PBL or streamlining explanations, ChatGPT improves or maintains near-term educational outcomes for medical students, with clear added value in efficiency and reach. The signal is strongest for short-cycle learning targets (history-taking behaviors, script refinement, and theory recall), while complex clinical problems still benefit from human calibration. Across trials, disclosure and supervision varied. In the Turkish trial, disclosure of AI use was reported to increase students’ critical stance toward AI. Contrastingly, other trials provided supervision and structured feedback but did not quantify disclosure-related effects.

DISCUSSION

Principal findings and interpretation

Across three RCTs, ChatGPT used alongside usual teaching equaled or improved the learning outcomes. The small number of eligible trials reflects the early stage of controlled research on ChatGPT in medical education rather than selective inclusion. Effects were observed in written knowledge, clinical reasoning during history taking, and selected Mini-CEX domains. Two themes recur. These findings are consistent with established learning theory. Feedback is one of the strongest influences on learning, but its effects depend on timing, specificity, and whether it directs attention to the task and process rather than to the learner alone [9]. These findings are educationally plausible because the included interventions emphasized repeated practice, timely feedback, and structured support. In the reviewed trials, ChatGPT appeared most useful when it was embedded within clearly defined learning activities rather than used as an unrestricted answer-generating tool.

First, feedback is the engine of the learning process. For example, ChatGPT provided prompt, task-based feedback after simulated patient encounters, resulting in improved clinical reasoning (measured on a validated instrument) compared to a no-feedback control group of students. When used to replace experts as the source of feedback on clinical key-features problems, ChatGPT was found to be as effective as expert-written feedback overall at immediate and 10-day testing, demonstrating scalable parity for routine problems. This pattern suggests that ChatGPT may be better suited to routine, structured tasks than to complex, ambiguity-rich clinical reasoning. In such settings, expert feedback still appears to provide a meaningful advantage. This pattern is also compatible with retrieval practice theory, which predicts that repeated testing and corrective explanation can strengthen learning more effectively than passive review alone [10].

Second, there is the question of context and complexity. In the same RCT, expert feedback outperformed ChatGPT for more complex items at late testing, indicating that cognitive scaffolding for complex multi-step reasoning is still better provided by experienced clinicians or by blended human-AI feedback calibrated to the task [7]. Beyond test scores, the Chinese controlled cohort suggests the spillover into the performance domain: ChatGPT-assisted PBL not only improved near-term theory scores but also enhanced Mini-CEX ratings for interviewing and clinical judgment, outcomes that may be relevant to supervised clinical performance [8]. In addition, ChatGPT increases the pace of formative feedback and expands case exposure. When integrated into formalized activities, it enhances knowledge and chosen performance indicators. Where issues require subtlety, the insight of an expert provides an edge [7]. The discovered limits are informative and represent design choices rather than intrinsic constraints.

Strengths, limitations, and implications

This synthesis is based on the use of validated tools (e.g., CRI-HTI, Mini-CEX) and authentic assessment tasks (key features, observed encounters). Positive results were observed across three settings (Germany, Turkey, and China), supporting further study across diverse curricula, however, they do not establish generalizability.

However, there are conditions under which highlighting is necessary for the reader. First, most outcomes were short-term, and we have limited information on longer-term retention, transfer to authentic clerkship performance, or downstream patient-centered outcomes. One plausible risk of AI-supported learning is over-reliance on the tool, whereby improved immediate task performance may not translate into durable competence if students defer reasoning rather than internalize it. This concern is particularly relevant when outcomes are measured over short intervals. Second, the learning tasks studied on history taking, key features reasoning in one topic, and short-horizon PBL represent important but narrow sections of the early curriculum [6]. Third, the primacy of expert feedback on challenging cases at late testing is a salutary reminder that blind use of LLM results to "replace" faculty is unwise; complexity calibration and human oversight seem to be essential for safety and quality [7]. LLM outputs can be fluent yet incorrect, and learners may inadvertently internalise inaccuracies or biased reasoning if responses are accepted uncritically [11]. Accordingly, educational use should be paired with explicit verification practices, particularly for safety-critical or high-complexity content. Lastly, our approach to mixed-population studies may introduce selection bias if medical-student subgroup data were selectively reported or not extractable.

Methodologically, future studies could pre-register protocols, determine power for non-inferiority or superiority prespecified outcomes, report adherence to intervention protocols (e.g., prompt templates, versioning), and disclose guardrails (instructional hints, verification steps), which may modulate both efficacy and safety. In parallel, curricular integration should acknowledge the “black box” features of LLMs and the risk of plausible-sounding inaccuracies or biased outputs.

Implementation also occurs within a pre-existing landscape of academic integrity challenges in medical training. Generative AI may amplify familiar risks (e.g., unacknowledged assistance, plagiarism, and inappropriate collaboration), particularly where institutional policies are unclear or inconsistently applied [12][13]. In controlled, supervised interventions, the educational signal is clearer. However, in unsupervised settings, it becomes harder to distinguish AI-supported learning from AI-facilitated misconduct. Therefore, for this reason, clear permissible-use guidance, assessment design that samples unaided reasoning, and teaching students how to document and justify AI use are therefore essential complements to any educational deployment of ChatGPT [12][13].

Three design choices appear to be important. First, feedback should be structured, timely, and task-focused; in the simulation trial, ChatGPT’s outputs were mapped to CRI-HTI behaviors and delivered immediately after the task was completed [6]. Second, align support to task complexity: routine problems tolerate automated feedback, whereas complex or safety-critical scenarios merit clinician oversight or blended experts, which is the AI commentary [7]. Third, LLMs should be used to widen exposure and rehearsal opportunities, but they should be treated as adjuncts to live bedside teaching, not substitutes for patient contact and mentorship [7][8].

Generative AI also raises academic and assessment integrity concerns, as unsupervised use may blur the boundary between supported learning and unpermitted assistance. Assessment policy should evolve with time. Where ChatGPT is available, assessment for learning can plausibly include AI-mediated practice (e.g., formative key features with ChatGPT feedback), while assessment of learning must ensure that summative tasks adequately sample unaided reasoning and performance. Explicit instruction on when to use AI (e.g., when setting up for PBL but not while responding to objective structured clinical examination) will allow students to benefit from AI while maintaining academic integrity and skill building.

Key questions now are the durability of gains at 3–6 months, transfer to objective structured clinical examinations and workplace-based assessments, the complexity thresholds at which AI-generated feedback underperforms expert guidance, and whether blended expert, the AI feedback narrows that gap. Adequately powered multicenter randomized trials should compare ChatGPT-only, expert-only, and blended models stratified by case complexity and accompanied by cost-effectiveness analyses. Reproducibility requires clear reporting of the model version, prompt templates, guardrails, and protocol adherence.

CONCLUSION

Across three randomized studies on medical students, ChatGPT used as a supplement to usual instruction matched or improved short-term outcomes in knowledge, clinical reasoning, and selected workplace-based domains. Benefits were clearest when the model delivered immediate, structured feedback or scaffolded PBL; by contrast, complex, nuanced problems continued to favor expert guidance. On balance, the evidence favors an adjunctive role for ChatGPT, which is expanding practice and feedback at a low marginal cost while preserving human oversight for complex reasoning and high-stakes competencies.

AUTHOR CONTRIBUTIONS

Felix Y.S. Tan conceived and designed the study, developed the methodology, performed the formal analysis, and prepared the original draft of the manuscript. Lydia C. Ang and Rahima Ibrahim participated in the investigation and screening process and contributed to the review and editing of the manuscript. Munawwarah Abdul Majeed participated in the investigation, curated the data, performed adjudication, and contributed to the review and editing of the manuscript. Mohd Aliff H. Mohd Aris performed validation procedures and contributed to the review and editing of the manuscript. Sherly D. George conceived and designed the study, developed the methodology, supervised the project, performed validation procedures, and contributed to the review and editing of the manuscript. All authors approved the final version of the article.

Conflict of interest. The authors declare that there is no conflict of interests.

Financing. The study was unfunded (own resources).

Use of artificial intelligence. No artificial intelligence tools were used in the preparation of this manuscript.