Step by step guide for the implementation and assessment of Intelligent Tutoring Systems

Intelligent Tutoring Systems (ITS) are advanced computer-based learning environments that aim to provide personalized, adaptive, and interactive instruction by emulating many of the key functions of a human tutor. They leverage techniques from Artificial Intelligence (AI), cognitive science, and educational psychology to tailor the learning experience to the individual needs, pace, and understanding of each learner. ITS are designed to support self-regulated learning, provide immediate and individualized feedback, adapt to the learner’s evolving competence, and are often employed for teaching complex problem-solving skills or decision-making in intricate domains like AMR/AMS. A typical ITS architecture comprises several core interconnected modules:

  • Domain Model (Expert Knowledge): Contains the expert knowledge of the subject matter to be taught (e.g., principles of antimicrobial selection, mechanisms of resistance, AMS guidelines).
  • Student Model (Learner Model): Dynamically assesses and stores information about the learner’s current knowledge state, understanding, misconceptions, problem-solving strategies, learning progress, and affective state.
  • Pedagogical Model (Tutor Model): Contains instructional strategies and rules for how to teach, what to teach next, when to intervene, and how to provide appropriate feedback, hints, or explanations based on the student model and learning objectives.
  • User Interface Model: Facilitates interaction between the learner and the ITS, presenting information, problems, simulations, and feedback.

Table of contents

  1. Planning a Intelligent Tutoring Systems
  2. Defining roles in a Intelligent Tutoring Systems
  3. Assessing a Intelligent Tutoring Systems
  4. Suggested Intelligent Tutoring Systems prototype
  5. Reference
Priority topics in NAPs Icon

Planning a Intelligent Tutoring Systems

The following steps should be taken into consideration when planning the use of an Intelligent Tutoring System on AMR/AMS:

  1. Needs Assessment and Domain Definition for AMR/AMS: Clearly identify the specific, often complex, AMR/AMS knowledge domain or cognitive skills the ITS will target (e.g., appropriate empirical antibiotic selection for multi-drug resistant organisms (MDROs), implementing diagnostic stewardship for complex infections, understanding PK/PD principles for antibiotic optimization, managing AMR outbreaks from a One Health perspective). Define clear learning objectives. (Niculescu, 2016, outlines iterative design stages including needs assessment and cognitive task analysis).
  2. Knowledge Engineering (Domain Model Construction): Elicit, structure, and formalize expert knowledge from AMR/AMS specialists (e.g., Infectious Diseases physicians, AMS pharmacists, clinical microbiologists, epidemiologists, veterinarians). This forms the basis of the ITS’s “expertise.”
  3. Student Modeling Design: Determine how the system will diagnose and track the learner’s evolving knowledge, skills, common misconceptions, and problem-solving approaches related to AMR/AMS. This may involve analyzing learner inputs, response times, error patterns, and help-seeking behavior. (Vrushabh Hete et al., 2024, emphasize user profiling for tailoring content).
  4. Pedagogical Module Design (Tutoring Strategies): Define the ITS’s teaching strategies, including how it will present problems, provide hints, deliver explanations, correct errors, and guide the learner towards mastery. This could involve case-based reasoning, Socratic dialogue, guided discovery, or problem-solving exercises.
  5. User Interface and Interaction Design: Create an engaging and intuitive user interface that effectively presents AMR/AMS problems (e.g., simulated patient cases, farm scenarios, environmental data), allows for learner input, and delivers system feedback. This may incorporate multimedia, simulations, or even integrate with VR/AR for immersive experiences. (Vannaprathip et al., 2025, Source: 220800, describe a VR-based ITS for surgical decision-making in dentistry; Lampropoulos, 2025, reviews the integration of ITS with AR/VR).
  6. System Integration, Development, and Testing: Develop the software integrating all modules. Conduct rigorous iterative testing with target users to evaluate functionality, adaptivity, usability, engagement, and pedagogical effectiveness. (Dada et al., 2024, conducted a feasibility study for an ITS on AAC curriculum).
  7. Deployment and Evaluation in Context: Deploy the ITS and plan for its evaluation in the intended learning environment, collecting data on learner performance and experience.
AMR One Health policy group Icon

Defining roles in a Intelligent Tutoring Systems

Facilitator’s role (Primarily System Designers, Domain Experts, and Educational Researchers): The primary human involvement is in the complex and iterative design, development, knowledge engineering, and rigorous evaluation phases of the ITS. Once deployed, the “facilitator” role shifts to monitoring overall system performance, analyzing aggregated learner data to identify areas for system refinement or curriculum improvement, and potentially integrating the ITS into a broader educational program. Direct, real-time interaction with individual learners during the tutoring process is typically handled by the ITS itself.

Participant’s role (Learner): Actively interact with the ITS by working through presented AMR/AMS problems, tasks, or simulated scenarios. Respond to system prompts, questions, and challenges. Critically engage with the personalized feedback, hints, and explanations provided by the system. Learn through an adaptive, guided, and often self-paced pathway, taking responsibility for their learning process.

AMR One Health policy group Icon

Assessing a Intelligent Tutoring Systems

Methods

  • ITS Internal Tracking and Analytics: Detailed logging and analysis of learner interactions within the system, including decisions made, errors committed, time taken on tasks, concepts mastered, problem-solving pathways followed, and help-seeking patterns. (Niculescu, 2016, mentions evaluation as a key stage in ITS design).
  • Performance on In-System Tasks: Assessment of performance on specific problem sets, simulated cases, or skill-based exercises embedded within the ITS.
  • Pre- and Post-ITS Assessments: Objective measures of changes in AMR/AMS knowledge, clinical reasoning skills, diagnostic accuracy, prescribing appropriateness, or problem-solving efficiency using validated instruments or custom-developed tests. (Fodouop Kouam, 2024, discusses evaluating the effectiveness of ITS on improving programming skills).
  • User Satisfaction and Usability Studies: Surveys and qualitative feedback (e.g., think-aloud protocols, interviews) to assess learners’ perceptions of the ITS’s usability, engagement, clarity of feedback, adaptivity, and overall learning value.
  • Comparative Studies: In research settings, comparing learning outcomes of students using an ITS versus those receiving traditional instruction or other educational interventions. (Kulik and Fletcher’s meta-analysis, frequently cited e.g., in Neagu et al., 2020, found ITS to be generally effective).

Tools

ITS log file analysis software, built-in performance dashboards and learner analytics within the ITS, embedded assessment modules with automated scoring, standardized knowledge tests relevant to AMR/AMS, clinical vignette evaluation rubrics, validated usability and user experience questionnaires, user feedback forms.

AMR One Health policy group Icon

Suggested Intelligent Tutoring Systems prototype

Target Audience: Prescribers (Doctors, Veterinarians, Dentists), Clinical Pharmacists/Dispensers, advanced practice nurses, and potentially Researchers or Public Health Officials needing to develop complex decision-making skills in AMR/AMS.

Learning Objectives:

  • Improve diagnostic reasoning for common and complex infectious diseases, considering potential AMR.
  • Enhance proficiency in selecting appropriate empirical and definitive antimicrobial therapy, optimizing dose, duration, and route, based on patient factors, local epidemiology, diagnostic results, and current AMS guidelines.
  • Develop advanced skills in interpreting complex microbiology data, including susceptibility testing and mechanisms of resistance.
  • Strengthen ability to apply PK/PD principles to individualize antimicrobial therapy for challenging infections or special patient populations.

Curriculum/Activities:

  • The ITS presents learners with a series of interactive, branching clinical scenarios or virtual patients with complex infections, potentially involving MDROs or diagnostic uncertainty. (Inspired by systems like “AMR Case-Solver” previously described, and the VR-ITS for dental surgery by Vannaprathip et al., 2025).
  • Guided Problem-Solving: The ITS guides users through steps such as: initial patient assessment, ordering and interpreting diagnostic tests (virtual lab results, imaging), formulating differential diagnoses, selecting empirical antimicrobial regimens with justification, and then adapting therapy based on evolving clinical data and microbiology results.
  • Adaptive Feedback and Scaffolding: The system provides immediate, tailored feedback on learner decisions (e.g., “This antibiotic has too broad a spectrum for this likely pathogen based on local data,” or “Consider the patient’s renal function for this dosage”). It offers hints, prompts for further information gathering, or provides expert explanations if the learner struggles or makes suboptimal choices. The difficulty of scenarios adapts based on learner performance.
  • Experiential Learning Elements: Decisions made by the learner could impact the virtual patient’s outcome within the simulation (e.g., clinical improvement, adverse drug event, development of further resistance), providing a safe way to experience consequences. (Goldberg and Boyce, 2018, discuss experiential ITS that incorporate the environment to contextualize concepts).
  • Student Model Tracking: The ITS builds a model of the learner’s understanding of specific AMR/AMS concepts (e.g., knowledge of antibiotic spectra, interpretation of antibiograms, application of de-escalation principles).

Evaluation of the Prototype’s Effectiveness:

  • Detailed analysis of learner performance trajectories within the ITS, including accuracy of diagnostic and therapeutic choices, time to effective treatment, and common error patterns.
  • Comparison of pre- and post-ITS performance on standardized clinical reasoning tests or complex A
  • MR case vignette evaluations.
  • Measurement of changes in confidence in managing complex AMR cases.
  • User feedback on the ITS’s usability, the realism of scenarios, the quality and timeliness of feedback, and overall perceived impact on their clinical decision-making skills.
  • In a research setting, comparison of skills acquired through the ITS versus other educational methods (e.g., traditional lectures, non-adaptive case studies).

Reference

  • Dada, S., Flores, C., Bastable, K., Tönsing, K., Samuels, A., Mukhopadhyay, S., Isanda, B., Bampoe, J. O., Stemela‐Zali, U., Karim, S. B., Moodley, L., May, A., Morwane, R., Smith, K., Mothapo, R., Mohuba, M., Casey, M., Laher, Z., Mtungwa, N., & Moore, R. (2024). Use of an intelligent tutoring system for a curriculum on augmentative and alternative communication: Feasibility for implementation. International Journal of Language & Communication Disorders, 59(6), 2279–2293. https://doi.org/10.1111/1460-6984.13084
  • Fodouop Kouam, A. W. (2024). The effectiveness of intelligent tutoring systems in supporting students with varying levels of programming experience. Discover Education, 3(1), 278. https://doi.org/10.1007/s44217-024-00385-3
  • Goldberg, B., & Boyce, M. (2018). Experiential Intelligent Tutoring: Using the Environment to Contextual ize the Didactic. In Lecture Notes in Computer Science (pp. 192–204). Springer International Publishing. https://doi.org/10.1007/978-3-319-91467-1_16
  • Kurniawan, B., Meyliana, Warnars, H. L. H. S., & Suharjo, B. (2025). Intelligent tutoring system in army: A systematic literature review. AIP Conf. Proc, 3200(1), 040027. https://doi.org/10.1063/5.0255652
  • Lampropoulos, G. (2025). Augmented Reality, Virtual Reality, and Intelligent Tutoring Systems in Education and Training: A Systematic Literature Review. Applied Sciences, 15(6), 3223. https://doi.org/10.3390/app15063223
  • Menor, J. V. (2023). Design, development and effectiveness of an intelligent tutoring system using neural network. 030011. https://doi.org/10.1063/5.0125246
  • Molina Hurtatiz, Y. E., Pascuas Rengifo, Y. S., & Millan Rojas, E. E. (2015). SISTEMAS TUTORES INTELIGENTES COMO APOYO EN EL PROCESO DE APRENDIZAJE. Redes de Ingeniería, 6(1), 25. https://doi.org/10.14483/udistrital.jour.redes.2015.1.a02
  • Neagu, L.-M., Rigaud, E., Travadel, S., Dascalu, M., & Rughinis, R.-V. (2020). Intelligent Tutoring Systems for Psychomotor Training – A Systematic Literature Review. In Lecture Notes in Computer Science (pp. 335–341). Springer International Publishing. https://doi.org/10.1007/978-3-030-49663-0_40
  • Niculescu, C. (2016). Intelligent tutoring systems-trends on design, development and deployment. In Conference proceedings of “eLearning and Software for Education (eLSE)”, (pp. 280-285). Carol I National Defence University Publishing House. https://doi.org/10.12753/2066-026x-16-218
  • Purwaningtyas, D. A. (2024). Technology and characteristics of intelligent tutoring system for air traffic controller surveillance training: A systematic review. AIP Conf. Proc, 3077(1), 040012. https://doi.org/10.1063/5.0201749
  • Tuyboyov, O., Turdikulova, B., Davlatova, R., & Norov, S. (2025). The role of AI-driven intelligent tutoring systems in enhancing mechanical engineering education. AIP Conf. Proc, 3268(1), 070038. https://doi.org/10.1063/5.0257379
  • Vannaprathip, N., Haddawy, P., Schultheis, H., & Suebnukarn, S. (2025). SDMentor: A virtual reality-based intelligent tutoring system for surgical decision making in dentistry. Artificial Intelligence in Medicine, 162, 103092. https://doi.org/10.1016/j.artmed.2025.103092
  • Vera, G., Daniel, V., & Award, G. (2015). Implementation of an Intelligent Tutorial System for Socioenvironmental Management Projects. International Association for Development of the Information Society (ED562096). https://eric.ed.gov/?id=ED562439
  • Vrushabh, H., Chetan, C., Vinita, K., Suankit, H., & Milind, U. (2024). INTELLIGENT TUTORING SYSTEM: UTILIZING USER PROFILING IN REGULATORY AF FAIRS COURSES. International Education and Research Journal, 10(3). https://doi.org/10.21276/ierj24018735591493