Introduction

The Faculty of Science introduced an alternative peer review system in Semester 1, Academic Year (AY) 2006/2007 to improve current methods of evaluating the pedagogical work of its academic staff. Improvements to the peer review system have been ongoing in NUS, with the last major change in 2001.

In the past, the faculty required academic staff (except new appointees) to conduct the peer review for their colleagues. This proved problematic for the following reasons:

• With a long list of teaching staff to review every semester, the faculty would end up engaging a large number of reviewers, which often led to inconsistent evaluations and even disagreements between reviewers and the teachers under review.
  
  
• Many peer review reports lack detail or contain only general comments.
  
  
• Reviewers tend to provide sympathetic evaluations, perhaps mindful that they would come under peer review in future.
  
  
• The exercise is time-consuming and it can be a logistic challenge to arrange the peer review sessions.

To address these problems, trial changes were introduced to the system. Instead of sourcing for reviewers from the entire faculty, two standing teams of peer reviewers comprising experienced teachers were formed. Each team had two reviewers, with one team covering the physical and mathematical science departments, and the other the chemical and biological science departments. New teams are selected every semester, and reviewers are relieved of teaching duties for that semester so they can focus on conducting fair and useful peer reviews. They will give staff constructive feedback and follow up with discussions, if necessary.

Data from the peer review exercises conducted before and after the new system was implemented have been compiled and the key findings are presented here.

Comparing Data

We compared scores collected in Semester 2, AY 2005/2006 (Figure 1) using the previous peer review system, with scores from Semesters 1 and 2 of AY 2006/2007 (Figures 2 & 3) when the new system was implemented. While Figure 1 shows that most staff received gradings of either 4 or 5, Figures 2 and 3 indicate a bell-shaped distribution of scores collected under the new system, even though different review teams were employed. As the new system allows reviewers to make comparisons across the entire faculty, their scores are expected to be more reliable.

Figure 1. Peer review scores for Semester 2, AY 2005/2006 under the previous review system

Figure 2. Peer review scores for Semester 1, AY 2006/2007 under the new peer review system

Figure 3. Peer review scores for Semester 2, AY 2006/2007 under the new peer review system.

We also calculated the scores differently by taking the average scores for the evaluation questions. In Figures 4 to 6, the scores given refer to the overall scores displayed on the first page of the peer review report. When we compared the scores given and those calculated during the two periods, we observed that under the previous system, there was a larger scatter between the overall scores given and the scores calculated (Figure 4). In contrast, Figures 5 and 6 show very close correlations between the overall and calculated scores, indicating that using smaller dedicated teams of reviewers provided more accurate and reliable scoring.

Figure 4. Comparison between scores given and scores calculated in Semester 2, AY 2005/2006 under the previous review system

Figure 5. Comparison between scores given and scores calculated in Semester 1, AY 2006/2007 under the new peer review system

Figure 6. Comparison between scores given and scores calculated in Semester 2, AY 2006/2007 under the new peer review system

Qualitative Comments

The reviewers' comments from the two systems provided further useful information. Under the previous system, typical comments included:

• "This is a new course for Dr. A. PowerPoint slides were prepared and used in teaching. They are also available online."
  
  
• "Dr. B should encourage students' participation by asking some simple questions and checking whether the students are following the lecture. Lecture notes are well prepared. Delivery is clear and organised."
  
  
• "Dr. C is able to show his knowledge, enthusiasm and ability to motivate students to learn and think. The lecture materials are well designed and prepared. The presentation is generally vivid and interactive."

The comments in these examples lacked detailed explanations and did not address issues such as the staff's teaching abilities, level of interaction with students, presentation standards and clarity of speech. Although the last reviewer gave feedback on such issues, he failed to cite specific examples to support his points.

In contrast, review teams under the new system gave detailed and informative comments:

• "Dr. D very actively engaged his student s throughout the lecture by asking probing questions related to the lecture content and student s responded well. He began by reviewing what was taught in the previous session. Throughout his lecture he would pause to ask if students followed him and if they had questions. He ended the session with a summary of key points discussed. He also used examples closer to home to reinforce these points. One can see that he used effective methods to achieve the module's learning objectives."
  
  
• "Dr. E began with a recap of the previous lecture. He was well prepared, confident and most of all very knowledgeable of the subject. His previous industrial experience came in handy a s he constantly mentioned current industrial practices. Throughout his lecture, he constantly paused and asked students questions to ensure they were given time to absorb the teaching materials. Due to the smaller class size (35 students), he actively engaged students using this method. Dr. E's unaccented English was not very good but he seems to be able to explain very difficult concepts to the class. He also communicated effectively by maintaining eye contact. He should be particularly commended for his effort to link his teaching materials to materials students had previously learned in other science modules. This should reinforce students' learning. Dr. E's lecture was well-organised and his notes were adequate. The PowerPoint slides however were not spectacular. They were mostly simple and not very flashy. He could have made better use of audio-visual technology to give a first-rate lecture. Dr. E used many illustrations during the lecture to reinforce his teaching, but it was unclear why they were not included in the students' handouts. However, the slides used in other lectures (as found in his module folder) included beautiful pictures and diagrams."
  
  
• "The lecture dealt with polymers. Dr. F started by showing two advertisements featuring polyethylene terephthalate (PET) and fabric that blocks ultraviolet radiation. This is a good start to the lecture's main theme. During the lecture, Dr. F asked students several relevant and probing questions, including comparing the usage and application of low-density against high-density polyethylene, why polypropylene is less dense than low-densit y polyethylene, whether nonbiodegradable polystyrene is environmentally less friendly than paper, and how does non-stick Tef lon stick to cookware. Another good use of current issues was the discussion whether PET bottles should be recycled. At the end of the lecture, Dr. F posed three questions (relevant to the lecture) for students to think about. Dr. F spoke clearly with good diction. His slides were very well prepared, with many eye-catching graphics."
  
  
• "Dr. G first gave a summary of topics covered in the last lecture. The lecture dealt mainly with defects in solids and alloys. He used diagrams to give clear explanations of the differences and origins of various types of defects (e.g. stoichiometric defects versus non-stoichiometric defects). He often used mnemonics to remind students about concepts like FISH (i.e. Frenkel defects are associated with interstitial and Schottky defects are associated with holes) which helped students appreciate the origins of the defects. In his discussion of alloys, he pointed out some interesting examples of its usefulness: sterling silver, 22-carat gold and pewter. His slides were nicely prepared. Some questions were incorporated but with answers provided. It may be better to omit the answers and explain them verbally instead, although students might complain that they had to take notes for the answer. During the break, Dr. G walked around the lecture theatre, pausing to interact with students. His delivery was good and he always maintained eye contact. At the end of the lecture, he gave a summary of the key points covered." In these examples, the teams gave detailed feedback about teaching methodology and style which were substantiated with relevant examples. Such comments were clearly more informative and useful.

Logistics & Acceptance

Under the new system, departments compile a list of academic staff to be reviewed for the teams at the start of each semester. The list includes pertinent information about the staff under review and the modules they teach. This gives the teams more time to discuss how they would conduct the reviews to ensure consistency and fairness.

It is also noted that staff were generally more receptive of the new peer review system. There were complaints that staff were unfairly evaluated under the previous system due to perceived bias or leniency by some reviewers. However, there have been no such complaints about the new system so far, and staff appear receptive to this objective method of conducting peer reviews.

Conclusion

Staff's feedback on the new peer review system has been encouraging. Both reviewers and staff under review have welcomed the new system. We anticipate that promotion and tenure committees will also f ind it more reliable and useful. We will continue to use this new system and finetune the peer review process as we receive more feedback.

The quotations by peer reviewers in this article have been edited to ensure clarity and readability.