Internal Reviews

The Internal Review Team

The organization of the Internal Review Team will be completed by the PGME Office. The PGME Office will also facilitate the selection of an internal review date that aligns with the program’s availability and the availability of Internal Review Team members.

The Internal Review Team will include the following:

• A faculty member experienced in postgraduate medical education from another discipline who will act as Chair
• A Program Director from another discipline
• A Resident Representative from another discipline

Program Invested Parties

Programs will need to ensure the following invested parties are available during the day of the internal review:

• Program Director
• Teaching faculty
• Residency Program Committee (RPC) members
• Competence Committee (CC) members
• Program Administrator
• Department/Division Chair
• Current residents/trainees of the program

Participation can be in-person or via videoconference or teleconference.

Planning the Internal Review Day

The PGME Office is responsible for facilitating date selection for the internal review. Once a date is confirmed, the program and the Internal Review Team will be notified. The PGME Office will then send the program a sample agenda and any other additional documents (i.e. curriculum mapping templates, updated standards, etc.) either directly via email or via a OneDrive shared resource folder. This will be completed no later then six months in advance of the review.

The completed agenda must be returned to the PGME Office ten weeks before the internal review date. The program is responsible for the logistical arrangements for its own review date. This includes room booking, virtual meeting equipment, morning refreshments and lunch (if the internal review is being held in person). Costs are the responsibility of the program.

Preparing the Required Documentation, Evidence and Previous AFI Responses

Providing Narrative Responses and Supporting Evidence (CanAMS Instrument)

Programs will use their online CanAMS instrument profile to provide documentation, evidence and narratives related to the required accreditation standards. Training for programs on how to use the AMS system is available here. Additional training can be scheduled with the PGME Office. 

For many programs, the documentation and narratives from the last accreditation site visit will still be populated, meaning programs will only need to update new documentation and narratives as required (i.e., changes that have been made based on quality improvement, standards updates, or in response to the last accreditation visit, etc.).

Please remember to save your instrument reponses by selecting the "Save" button. 

All documentation must be saved in the program’s instrument profile at least eight weeks in advance of the review date.

Responding to Previous Areas for Improvement (AFIs)

Please use the "AFIs" tab within CanAMS to populate narrative responses to your program's previous AFIs. While this tab is currently not accessible to internal reviewers, the PGME Office will ensure your responses are shared with them prior to the review. 

Please provide responses to both AFI-2Y and AFI-RR. Only narrative repsonses are require; evidence documents need not be attached as they will be provided within the instrument. However, if your program would like reviewers to examine a particular instrument document as evidence, please ensure that the file name is mentioned in the AFI narrative response. 

Additional information on AFI responses will be provided by PGME, along with other internal review preparatory resources, when your review date is confirmed. 

Document Review Files

At least three weeks in advance of the review date, programs will submit to the PGME Office the RPC minutes (previous two years), Competence Committee minutes (previous two years) and resident files via OneDrive, OWL, or Teams.

Residents’ Report

The PGME Office will facilitate the process of collecting a report from program trainees. The Residents’ Report template will be sent by the PGME Office to the Program Administrator to disseminate to residents. The residents will then send a completed report directly to the PGME Office (without cc'ing anyone from the program). The Residents’ Report will remain confidential and will not be shared with the program.

The Residents’ Report must be returned at least eight weeks prior to the internal review date.

Training

For those who are new to internal reviews, online asynchronous training is available, developed by CanERA. The training covers the following topics:

• Navigating CanAMS
• Accessing and reviewing evidence
• Accessing and navigating the accreditation report instrument
• Providing narrative feedback

This training has been developed for surveyors taking part in accreditation visits at various universities; therefore, not all information in this training is directly applicable. However, it will introduce new Internal Review Team members to the process of providing feedback.

Access to Program Documentation and Reports

The PGME Office will provide the Internal Review Team with the following items a minimum of two weeks in advance of the review:

• Access to the program’s instrument profile in the AMS
• Agenda (including the participants for each meeting)
• General Standards for Accreditation of Residency Programs
• Specific Standards, Objectives of Training, and Specialty Training Requirements
• Previous Accreditation Survey Report
• Sample Interview Questions
• Sample Internal Review Report Template

Each Internal Review Team member must review the above documentation at least one week in advance of the review and submit any questions or points of clarification to the PGME Office to relay to the program. A minimum of one week in advance of the review, the PGME Office will share the RPC minutes (previous two years), Competence Committee minutes (previous two years) and resident files via OneDrive.

On the review day, the program may provide an updated copy of the agenda, if required. If participants are calling in via teleconference or videoconference, this should be clearly indicated on the schedule so that the Internal Review Team members know who to expect in person and who will be joining virtually.

Conducting an Internal Review

Internal Review Team members will interview all key program invested parties (i.e. Program Director, residents, faculty, etc.) and evaluate the program based on relevant accreditation standards.

To determine whether a program has met a requirement, Internal Review Team members will:

• Review all available program information within the AMS;
• Ask additional questions and check additional documentation (i.e. resident files, meeting minutes) as needed during the review.
• Select “meets”, “area for improvement” for each requirement using the internal review report template provided by the PGME Office based on the program information in CanAMS and stakeholder interviews.

Internal Review Team members should include detailed feedback for requirements rated “does not meet”, including reference to specific indicators (i.e., 1.1.1.1, 3.2.1.2, etc.), as well as feedback for positive aspects of the program. 

Important considerations for internal reviewers:

• Ensure all participants feel comfortable sharing their feedback.
• Ensure all participants can provide input.
• Stay on schedule to ensure there is enough time to complete the review.

The internal review mirrors the agenda for a regular onsite survey as best as possible. Each program will be provided with an agenda template from the PGME Office at least six months in advance of the selected review date. The template will provide guidance on the various meetings required to take place, as well as their order.

The PGME Office will also provide commonly asked questions to the programs in advance of the review to assist the program in their preparation.

Surveyor Preparation (30 min)

• This meeting must be scheduled first.

Program Director (60 minutes)

• This meeting must be scheduled immediately following the surveryor preparation.

The following meetings must be scheduled in the middle of the day, in any order:

• Program Administrator meeting (15 minutes).
• Division/Department Chair (15 minutes) – include the name and specific title so it is clear whether it is the Department or Division Chair.
• Surgical Foundations Director (30 minutes) – applicable to surgical programs only (including all Department of Surgery programs, Neurosurgery, Otolaryngology - Head and Neck Surgery, and Obstetrics & Gynaecology).
• Resident(s) (60 minutes) – groups of a maximum of 20 residents. For larger programs, this can be arranged by PG year or by grouping junior and senior residents. All resident meetings should last 60 minutes; thus, if your program requires three meetings to accommodate 50 residents, your program will have three meetings at 60 minutes each. 
• Faculty/Teaching Staff (45 minutes) – Program Directors and Department Chairs should NOT attend this meeting.

Competence Committee (CC) (30 minutes)

• A meeting should be held with the Competence Committe members. Program Directors and Department Chairs are able to attend this meeting. 

Residency Program Committee (45 minutes)

• This must be scheduled as the last review meeting.
• The Program Director, Department/Division Chair (if applicable), and Program Administrator (if applicable) will be invited to attend the LAST 15 minutes of this meeting (there may be an additional 15-minute meeting scheduled with the Program Director directly following this meeting, if required).
• Resident representative(s) on the RPC attend this meeting and must be identified on the list of attendees.
• When faculty members are members of the RPC, it is not necessary for them to attend the meetings with faculty/staff.
• No meetings should be scheduled after this meeting.

Private Discussion Time for Surveyors (60 minutes)

• It allows the Internal Review Team to come to a consensus on the apparent strengths and weaknesses of the program and to prepare for the exit meeting with the Program Director.
• This should be scheduled after the last meeting of the day (RPC meeting).
• Program Directors should be available during this time in case the survey team ends their discussion early.

Exit Meeting (15 minutes)

• This provides the surveyors with the opportunity to present their findings to the Program Director and obtain clarification on certain issues.
• The internal review team should identify any program strengths and areas for improvement and allow the program to provide feedback on any comments made to determine whether an issue is a real area for improvement or simply a perception.

The template provided by the PGME Office will be used by the Chair of the Internal Review Team to prepare a written report. The report must include the strengths and areas for improvement of the program and specific recommendations for continued development and improvements.

The report should also include detailed feedback for requirements with identified areas for improvement, including specific identification of the indicators that were not met (i.e. 1.1.1.1).  Additional feedback is encouraged to recognize positive aspects of the program as well. A summary of areas for improvement should be included in the summary section of the report, as well as any leading practices and/or innovations. 

Once the report has been written by the Chair, it must be circulated to the other Internal Review Team members (Resident and Faculty representatives) for their review and input. The report must be submitted to the PGME Office no later than two weeks following the internal review.

The PGME Office will disable the Internal Review Team members’ access to the program’s AMS instrument profile upon completion of the review.

The Internal Review Team Chair’s report will be reviewed by the Internal Review Subcommittee (IRC) approximately 1-4 months after the Internal Review date. The IRC typically meets quarterly. The IRC will discuss the information provided in the Internal Review Team Chair’s report and make final decisions and recommendations for the program. The PGME Office will then send a finalized Internal Review Report to the Program Director, Program Administrator, and Department/Division Chair. PGME recommends the report and/or report findings be disseminated to all program stakeholders (i.e., residents, faculty, etc.) and discussed at the RPC.

The Chair of the IRC or the Associate Dean, PGME, will periodically present a summary of internal review findings at the PGME Committee.

The program will be provided with a follow up chart including areas for improvement and space for a timetable of planned work to document progress toward improvements.  The PGME Office and Internal Review Subcommittee will monitor the progress of these areas for improvement at the 6-month and 12-month marks following the program's receipt of their Decision Letter and Internal Review Report.

In general, Internal Review Reports are deemed to be internal documents of the University. The report, nor the results, will be shared with the Royal College/CFPC for the purposes of a residency program review, and will not impact a program's accreditation status. However, reports of the internal reviews of all programs are to be available to the institutional accreditation survey team just prior to the regular Royal College/CFPC survey to enable the survey team to assess the efficacy of the internal review process.