Surgery Newsletter, UofT, Chair's Column, September 1997

Table Of Contents Next Page

[Guest Chair's Column]
[Evaluation of Faculty Postgraduate Teaching by Surgical Residents]
[PGY-1 Orientation for New Surgical Residents]

GUEST CHAIR'S COLUMN

Technical Skills Laboratory

The Department of Surgery is committed to enhancing the technical skills of our residents. In order to do so, a decision has been made to develop a centre which will focus on technical skills acquisition that will serve as an adjunct to the operating room. The development of a skills centre will allow residents to acquire fundamental skills in an appropriate learning environment, one that will allow repeated practice with feedback. It is our firm conviction that with this approach to teaching combined with the excellent operative experiences our residents are receiving in the operating room will have the potential to greatly enhance the learning of surgical skills. In addition, the skills centre will serve as an important base for research in skill development in general and surgical skills acquisition specifically.

The University of Toronto Surgical Skills Centre to be located at Mount Sinai Hospital plans to transform the way surgical residents learn basic operative skills. The initial focus will be to provide junior residents with a core curriculum in these skills. It is anticipated that basic skills will be developed to a desirable level outside the operating room. The resident will then be better equipped to participate in the surgical procedure and this will make greater educational gains while in the operating room. It is expected that the core curriculum will decrease the variability in surgical skills and change the shape of the learning curve. Intermittent, planned evaluation will help us to define ways in which we can provide residents with feedback and address their individual needs to further develop their technical skills. The Surgical Skills Centre will not replace but rather will enhance the operating room experience. The Surgical Skills Centre Curriculum Committee is currently developing a two-year core curriculum which will begin its first session July 1st, 1998.

Each Division will be responsible for teaching aspects of the core curriculum that pertain most closely to their specialty (eg. neurosurgery - lumbar puncture). A Surgical Teaching Team will be selected carefully by each of the Division Chairs. A Faculty Development Workshop for the Surgical Teaching Teams will be held in November. The workshop will provide faculty with tips on effective coaching of residents during procedural skills to maximize their learning. Visiting Professor, Dr. Janet Starkes, will enlighten us on the cognitive and perceptual-motor issues in skills acquisition. Dr. Starkes has done extensive research in the area of motor skills expertise and has recently focused much of her attention on microsurgery.

Educational research in skills acquisition and evaluation will be an essential component of the Surgical Skills Centre. The Surgical Education Research Group has had significant interest in this area. They have received eleven research grants ($398,450) and have fifteen publications relating to surgical skills. Surgical education research residents and fellows in their MEd. program will have the opportunity to conduct research in the Surgical Skills Centre. Fourteen residents and fellows have completed higher degrees in education with the University of Toronto Surgical Education Research Fellowship Program. Two of the surgical education research residents have been registered in the Surgeon Scientist Program while completing their MEd. The University of Toronto has become part of a consortium of schools aimed at mounting a joint curriculum of laboratory-based teaching of technical skills. The consortium will develop a program of research in technical skills acquisition and plan a coordinated effort to analyze the efficacy of ex-vivo teaching of technical skills.

The Surgical Skills Centre Mission Statement

1) To change the way fundamental surgical skills are taught and evaluated.
2) To establish a facility wherein surgical skills can be taught and evaluated.
3) Provide a platform for continuing education in surgical skills.
4) Create a laboratory for research in surgical skills acquisition.
5) Create a laboratory for research and development of surgical skills innovation.

Ms. Wendy McCulloch has been hired as the Coordinator of the Surgical Skills Centre. Wendy has experience as an operating room nurse, and excelled in her more recent position as the Clerkship Coordinator for the Department of Surgery at The Toronto Hospital. She has done an outstanding job while administering the Objective Structured Assessment of Technical Skills (OSATS) and is expert in making OSATs a success. Wendy is a member of the consortium of Surgical Skills Centres currently involving five centres across North America.

The Mount Sinai Hospital Foundation is managing the fundraising. On June 19th, 1997, surgical industry partners attended an informative evening at Mount Sinai Hospital. The vision and mission of the Surgical Skills Centre was presented to them. Those attending have demonstrated significant interest in the endeavour and negotiations for partnerships with industry are currently underway. Zimmer has committed all orthopaedic equipment and $180,000 toward capital costs.

This is an exciting step forward in surgical education. The Department of Surgery is on the leading edge of a fundamental change in the way surgical skills are acquired and evaluated. The strength of our faculty will undoubtedly bring international recognition to The Surgical Skills Centre.

Carol R. Hutchison
Director, U of T Surgical Skills Centre at MSH

Evaluation of Faculty Postgraduate Teaching By Surgical Residents

The majority of surgical faculty participate in training of residents at the University of Toronto. The effectiveness of this teaching needs to be assessed for several reasons:
  1. It is a requirement of the Royal College of Physicians and Surgeons of Canada that postgraduate teaching be objectively and anonymously evaluated by the residents. All Royal College Accreditation surveyors therefore routinely examine and comment on the process of postgraduate teaching evaluations.
  2. Performance in postgraduate education by the faculty is closely scrutinized in considering application for promotions within our department.
  3. Mid-term and 3 year reviews of new recruits to our department require assessment of performance in postgraduate education.
  4. Reviews of institutions, headships, surgeons-in-chief positions and chairs of divisions also require evaluation of postgraduate teaching of the individuals in a particular unit.
  5. Evaluation of postgraduate teaching performance is one of the factors considered in the determination of academic productivity and assignment of merit points.

Clearly this process of postgraduate teaching evaluation has very significant implications not only for faculty in general but also program directors, external reviewers and residents.

The method of evaluation of postgraduate teaching has undergone significant changes over the last 10 to 15 years and attempts to improve the reliability and effectiveness of this tool continue. Most recently, attempts have been made to use numerical scales in order to more objectively evaluate teaching performance in Postgraduate Surgery. One of the key elements of this entire process is the maintenance of anonymity so that the residents may provide this evaluation without fear of reprisals. In smaller programs this maintenance of anonymity may prove difficult. In the present format each resident is required to evaluate each consultant on the service by completing an Evaluation of Consultant form. This form outlines the length of the rotation, the degree of contact with the consultant, the frequency on contact in the operating room, the ward and ambulatory setting. In addition the consultant is evaluated on the quality of formal teaching, the quality of informal day to day teaching, the degree to which the consultant questioned and challenged the housestaff, the degree to which the consultant stimulates enthusiasm for learning. Other areas evaluated by the residents include the approachability of the consultant for discussion, the provision of appropriate independence in looking after patients, the provision of adequate opportunity to see ambulatory patients, contribution to improvement of operative skills, the establishment of good rapport with the housestaff, provision of direction and feedback, establishment of a good role model as a surgeon, provision of a good role model as a teacher and adequacy of the consultant's practice for teaching purposes. The residents are also required to rate the overall teaching performance of the consultant. All of these categories are rated on a 5 point scale with A being excellent, B being very good, C being good, D being fair and E being poor. In addition, the A,B,C,D and E categories are given scores of 5 to 1 respectively. A teaching effectiveness score is computed by summing all of the scores from the individual items (each of which has a maximum score of 5) and the overall score is standardized to a maximum of 20 which is then termed the Teaching Effectiveness Score (TES). In addition, residents are required to provide written comments on the teaching performance of the faculty for review boards, promotion committees, etc.

After the teaching effectiveness scores have been generated, a histogram is constructed and the percentile ranking for all members of the department is determined. The teaching effectiveness scores, a copy of the histogram for the entire department as well as an individual breakdown of the scores in the different categories and the percentile ranking of each faculty are provided to the surgeons-in-chief and university division heads for distribution to the individual faculty member. In the material provided to the individual faculty member the names of other members of the faculty and their rankings are not included in order to maintain confidentiality. The TES, percentile ranking and names of individuals from one specialty are not provided to any division head or other faculty member outside that particular specialty. The period of 2 years was chosen so that an adequate number of evaluations could be available for analysis. This time factor would also tend to preserve resident anonymity.

In addition to these bi-annual determinations of teaching effectiveness scores, separate scores are required for presentation and assessment of faculty during 3-year reviews, annual consideration for promotion and consideration of postgraduate teaching awards. Because of the obvious importance of this entire evaluation process it is essential that reliable data can be generated. This not only requires anonymous evaluations but also a sufficiently large number of responses from the residents. Clearly responses of less than 50 percent from the residents can lead to misleading evaluations. Unfortunately, this has been a problem in the past in that too many of the residents fail to complete the evaluation forms. In the past we have excluded performance evaluation that only involved 3 or less form completions. However, staff members requested that even this type of evaluation would be meaningful to them. In review of this entire process during our last 2 year review from 1994 to 1996 it was pointed out very strongly by many members of our department that less than 50 percent compliance with a small number of evaluation completed by residents would obviously generate unreliable data and should not be distributed or included in the overall ranking of faculty members. Because the calibre of postgraduate teaching is generally very high among our surgical faculty the percentile ranking of TES represents a very skewed distribution with 95% of the TES ranging between 13 and 19. A TES of 13-14 which is equivalent to 65-70% of the maximum score of 20 has a very low percentile ranking and this has caused some very good surgical teachers to regard the percentile ranking as being misleading. This factor has to be considered in the interpretation of the TES and percentile rankings. In other words, a low percentile ranking does not necessarily represent poor surgical teaching. In an attempt to improve compliance among the residents and thus increase the reliability of this important process we have instituted a new method for collecting the completed evaluation forms. At present, the residents are required to complete the forms and place them in a sealed envelope to be delivered to the institutional division head prior to the In-Training Evaluation Report being discussed with the trainees. Preliminary review of this process has indicated a significant improvement in compliance although it is still not 100 percent. With increased compliance it should be possible to provide more frequent feedback to the faculty as the number of completed evaluations increase.

Several faculty members have requested more frequent and timely evaluations than the two-year reports. To fulfil this legitimate need additional abbreviated teaching evaluation forms are being completed by the residents and forwarded to their respective program directors. These evaluations are then discussed with the faculty members by the program director and respective division heads.

Apart from the provision of feedback and assessment of teaching for consideration of candidates for promotion, awards and reviews, the results of the teaching evaluation process are also applied to identify areas of weakness in our Postgraduate Training Program. With identified deficiencies attempts can then be made to correct such deficiencies and improve the quality of the program. Likewise, failure to improve these deficiencies would identify areas within our program where trainees will not be assigned routinely.

In addition to the evaluation of individual surgical consultant performance in postgraduate teaching the residents play an important role in assessing the different clinical rotations within the respective institutions. The information provided through this form covers areas of patient care, level of responsibility allowed, the type of clinical exposure, the organization of the teaching program, the quantity and quality of teaching as well as provision of feedback. In addition, an assessment is made of the hospital services including the interaction with nursing staff, the hospital laboratories, the degree to which objectives are outlined and met, the degree to which an appropriate on-call schedule is maintained as well as the provision of adequate services within the hospital for the trainees such as on-call accommodation. Finally, in the resident evaluation of service forms the trainees are required to rate the overall quality of the learning experience, the education to service ratio and to indicate whether they would recommend a rotation in that specialty within the particular institution. This type of information is crucial in identifying strengths and weaknesses of the program and could be used in deciding the most appropriate assignments of residents to the individual institutions and services within our program.

The above description of surgical postgraduate teaching evaluation at the University of Toronto highlights the importance of this process but a key element in its success is the provision of reliable information which requires compliance by the residents, maintenance of anonymity and confidentiality so that the residents would have no fear of reprisals. This process is obviously very labour intensive. We are very grateful for and look forward to the continued cooperation of the members of the surgical faculty in ensuring that this process is conducted as flawlessly as possible so that its integrity may be maintained.

J. Ali, Director,
Postgraduate Education

PGY-1 Orientation for New Surgical Residents

On July 7, several departmental faculty met with 35 new PGY-1 residents who have been assigned to the various divisional programs. After welcoming remarks from Chair John Wedge who reviewed the size and stature of the department, a number of speakers addressed various operational items. Complete and timely registration would guarantee the residents a pay cheque according to Pam Hawes who also indicated that the Office of the Associate Dean acts as an advocate for the resident. Or, as Professor Murray Urowitz put it last year, it is to "make sure there is enough 'student' in the 'resident'".

Professor Jameel Ali acting as the MC, stressed that the new residents pay attention to the information contained within the booklet of departmental guidelines which had been dispersed.

Professor Martin McKneally is responsible for the POS course which begins again on September 9; he indicated how the 2-year structured lecture program will assist the residents with their eventual passage of the Royal College POS examination.

Professor Ori Rotstein's infectious enthusiasm came through again as he spoke of the Surgeon Scientist Program by reiterating information from his earlier editorial (Surgery Newsletter, April/May, 1997). As a stimulus to the provision and evaluation of a resident's technical skills Dr. Dimitri Anastakis informed the residents about the new Surgical Skills Teaching Centre (at the Mount Sinai Hospital), which was announced by Professor Wedge in the last issue of Surgery Newsletter. In late August, the PGY-1 residents took part in a 3 day technical skills course which will measure skill aptitudes learned in one of 3 ways - by reading alone, or by working on a bench model or, on a cadaver. The aim is to define the best instructional technique for one to transfer cognitive information to mechanical performance. And finally, the importance of the surgical resident as a teacher was encouraged by Dr. John Murnaghan. He urged the residents to assume a role as a teacher indicating that teaching is an attitude as well as an activity. The payback is that junior instructors "are more likely to learn when they are in the teaching process".

The new PGY-1 residents are:

General Surgery

Anna Fu
Cameron Gelder
Asher Khitab
Deepa Kumar
Crystal Pallister
Kinga Powers
Bao Quy Tang
Shannon Trainor
Leonard Tse
Ali Jawas
Monther Kabbani
Yasser Botros

Orthopaedic Surgery

Plastic Surgery

Neurosurgery

Urology

Cardiac Surgery

Thoracic Surgery

Table Of ContentsNext Page

This Web page was produced using a Beta version of HTML Transit 3.0