Last semester, CDTL organised a seminar on conducting open book examinations (OBEs) with presentations by four panelists from the faculties of Arts and Social Sciences, Engineering, Law and Science. The following article was contributed by panel member Professor K. P. Mohanan, from the Department of English Language and Literature.

A report on CDTL's OBE seminar

There was consensus among the panel members that open book examinations are generally more suitable than closed book examinations for testing the application of knowledge to novel situations, as well as for testing higher order thinking abilities.

Associate Professor Walter Woon [Faculty of Law] emphasized that what students learn in a course would ultimately be determined by the goals implicit in the system of assessment. We are at liberty to set these target "hoops" where we want them: rote memorization, exposition of knowledge content, mechanical application, critical thinking or creative thinking, at easy, medium or difficult levels. Our students are intelligent and efficient: they will jump through whatever hoops we set.

I outlined a set of possible educational goals for university courses in the sciences and humanities, and illustrated different types of examination questions appropriate for each of these goals. I also demonstrated that open book examination questions, which demand the processing of novel information, are particularly suitable for questions that simultaneously test mastery of knowledge and thinking abilities.

Associate Professor Winston Seah [Faculty of Engineering] showed how one can test both knowledge and thinking abilities through a type of question that requires students to attempt a real life task of the kind that professional engineers are typically faced with. By changing the variables of this question each year, the examiner can generate novel questions without the danger of recycling, and hence this pattern is not too taxing for the examiner.

Professor S. M. Tang [Faculty of Science] brought in a cautionary note by pointing out that while open book examinations are particularly suitable for testing the student's ability to apply knowledge to novel situations, one must not over-emphasize this aspect at the cost of knowledge content and depth of understanding.

Another warning bell was that many lecturers and students that Associate Professor Seah talked to were not in favour of open book examinations, on the grounds that they are likely to be difficult for the examiners to set, and for the students to answer.

A number of recurrent issues emerged in both the speakers' presentations and the discussion session that followed. What follows is my response to some issues that appear to reflect the major concerns of the teaching community.

A response to some recurrent concerns

What kinds of subjects are suitable for OBEs?

The suitability of open book exams depends not on the subject matter of a course, but on the kinds of questions we want students to answer, which in turn depends upon what is being tested in an exam. Ultimately, it boils down to what we want them to attain at the end of the course. If a course expects students to memorize the information given to them, and reproduce it in the exam, an open book examination is unsuitable. On the other hand, if a course expects students to be able to process new information, an open book examination is more appropriate. If a course aims to test understanding through exposition, open book examinations are not suitable; if it aims to test understanding through application to novel situations, then open book examinations are suitable.

Suppose we want to find out if students have understood the so-called nature­nurture debate on the roles of genetic predisposition and environmental factors on human behavior. We could design an exam question like: "Critically evaluate the debate between nature and nurture in human behavior."

This question is definitely not suitable for an open book exam, because, to answer this question, all students have to do is copy material from a textbook or a prepared model answer they have brought to the examination hall.

In an open book exam, we would ask a different kind of question. For instance, we could construct a question that tests the student's understanding of the debate on the basis of information they have not come across in their textbooks or lectures. We might, for instance, begin the question by pointing out that children who are born blind still smile when they are happy. We would then ask students to discuss the nature­nurture controversy on the basis of this information. Does it provide evidence in support of one of the competing hypotheses? How can we formulate the argument?

To answer this question successfully, students should be familiar with the nature­nurture controversy, and understand it in a deep sense. They should also be able to think through the new evidence. They cannot answer the question by lifting material from a textbook or lecture, or reproducing ready-made answers. The crucial concept here is that of processing new information in the context of what is already learnt.

If the ultimate aim of a course is understanding and thinking, then open book exams are very useful, but if the goal is rote learning and regurgitation, then closed book exams are clearly better.

How are open book exams different from closed book exams?

Intelligently designed closed book exams can be used to test thinking, but they are typically used as tests of memory. Open book exams cannot be used as plain tests of memory. If we want to ask questions like: "Describe the Rutherford model of the atom." or "Give the structural formula for methane." or "Critically evaluate the biological account of aging." then we cannot use open book exams because students could copy the answers from prepared answers. Such questions prompt students to regurgitate the information given to them in their lectures and readings. Flags like "critically evaluate" are only cosmetic, since what students are going to do is repeat the lecturer's critical views.

Students prepare for traditional examinations by: looking at previous question papers, spotting probable questions, constructing/finding model answers to these questions and memorizing these answers.

Constructing a model answer on one's own deserves credit. However, given that model answers are easily available, usually from senior students, all that students need to do to prepare for an exam is to memorize such answers.

If students can spot the likely questions and come ready with prepared answers for an open book exam, the exam does not test even memory. Hence, open book examinations cannot allow questions whose answers students can prepare in anticipation. One consequence of this is that open book examinations cannot afford to recycle questions.

What kinds of abilities are tested in OBEs?

A closed book exam typically tests (in practice) the ability to memorize and reproduce. An open book exam lends itself to a wide range of higher order abilities. Suppose we design a question that contains a two page passage on evidence of life in Mars, with a set of questions to test the students' ability to understand a piece of academic prose, evaluate the arguments and claims, consider alternative interpretations of the same facts and so on. Such a question would be ideal for testing the students' critical reading ability. If the processing of the passage crucially requires the knowledge content of the course, the tasks can also test how well the students have mastered this content, and how well they can acquire new knowledge on the basis of what they already know.

Suppose we design a question that contains a brief case study of a company that is currently running at a loss, with a set of questions to test the student's ability to analyze the probable causes of the problem, come up with a proposal to solve the problem, consider alternative proposals, and decide on the best solution. Such a question simultaneously tests the students' understanding of the concepts taught in a course, their creative faculty and decision-making abilities.

Undergraduate students in chemistry are generally told that the combination of atoms in a molecule may involve different kinds of bonds: single bonds, double bonds, triple bonds and so on. An ethene molecule, for instance, is believed to have a double bond between the two carbon atoms it contains, while an acetylene molecule is believed to contain a triple bond. If an examination question requires students to write down the structural formulae of these two compounds, they can regurgitate the correct answer without a shred of understanding.

The first step towards building a generation of thinking students is to sensitize them to issues of evidence and alternatives. One of the reasons for assuming double bonds and triple bonds in ethene and acetylene is the valency theory. Hydrogen is assumed to have a valency of one, while carbon is assumed to have a valency of four. Ethene and acetylene have two carbon atoms each, so we would expect these molecules to have eight hydrogen atoms. Contrary to the expectation, there is evidence to believe that ethene has four hydrogen atoms, and acetylene has two. This puzzle is explained by the double bond and triple bond hypotheses. Suppose an examination question proposes that we explain the number of hydrogen atoms in ethene and acetylene by assuming that carbon has a variable valency of four, two and one. This variable valency hypothesis would allow each carbon atom to combine with two oxygen atoms (as in carbon dioxide), two hydrogen atoms (as in ethene) or a single hydrogen atom (as in acetylene). We can now ask students to choose between the two explanations in terms of available evidence. The answer to this question cannot be found in any of the standard textbooks. To answer this question, students will need not only a considerable knowledge of the facts of chemistry, but also the ability to think critically by pulling these facts together.

Is knowledge content ignored in OBEs?

Thinking does not take place in a vacuum. Solving problems, inventing explanations, evaluating alternatives, testing theories and so on require close familiarity with and understanding of a large body of available facts, theories, analyses, and explanations. An open book examination can test the students' mastery of content indirectly by testing how well the student is able to apply this knowledge to new information. In fact, such application-oriented questions are a superior means of testing the mastery of content.

Even memorization is tested through such applications, provided that what is memorized has a crucial role in the processing of new information. We do expect chemistry students to know that carbon has a valency of four, and physics students to know the equation that expresses Newton's law of gravitation. We do not treat such knowledge as meaningless memorization because without this knowledge further thinking in the subject will be impossible. What open book examinations will eliminate is rote memorization with the goal of simply reproducing what is memorized, not memorization that enhances the efficiency and speed of thinking in a given domain.

How do OBEs affect the learning process?

The traditional learning strategy of spotting, preparing and memorizing answers will not work for open book exams. Instead, students will have to practice activities that develop abilities tested by open book exams. If we have to take a test in swimming or piano playing, we don't try to memorize a set of ready-made answers. Rather, we engage ourselves in activities that enhance our ability to swim or play the piano. The same holds for open book exams.

Students who are used to traditional examinations take a long time to figure out how to study for an open book exam. It is therefore necessary to help the students change their study habits by giving them quizzes and mock examinations. One of the speakers remarked that in his courses, more than fifty percent of the students simply flunk in the first open book quiz, because they come in with memorized answers without any practice in thinking that calls for processing novel information in terms of what they have learnt. The post-mortem of the quizzes helps students see why they did badly, how they can study to do better and so on. By the time they come to the third quiz, most students have a reasonable idea of what open book examinations call for. They stop looking for ready-made answers to questions. After the third quiz, they are ready for the final examination.

How will OBEs affect the teaching process?

If we are going to use open book examinations, it may be necessary to replace the lecture mode of teaching with an interactive mode. In a traditional lecture in a university, the lecturer delivers a fifty-minute lecture, and the students listen passively and take down notes. In the student's mind, each lecture is converted into a possible answer for a potential examination question. They write down the points made by the lecturer on each topic, including the lecturer's criticisms, and reproduce the points in the final examination. The lecture method is ideally suited for regurgitation in closed book exams.

In an open book examination, the focus shifts from the reproduction of information to the processing of information. What this means is that the focus shifts to the testing of certain abilities closely tied up with the knowledge content presented in the course: the ability to apply a theory, test a theory, propose an explanation, interpret the meaning, infer predictions, design an experiment, find logical inconsistencies and so on. We cannot help students develop these abilities by lecturing to them. The crucial concept here is active interaction: students actively interact with the teacher, instead of listening passively to the teacher.

One may think of a teacher as a master craftsman, and students are apprentices. One of the speakers said that he thought of himself as an experienced potter. His classroom is not a lecture hall, but a pottery workshop. Students make clay objects in the workshop, individually and collectively, and receive feedback from both the teacher and other students. The basic idea is that students construct and critically evaluate a large part of the knowledge that we expect them to acquire in a course. By doing this, they develop their creative and critical faculties while acquiring a body of knowledge, and are able to extend the knowledge beyond the class room and examinations. The teacher, like a master craftsman, helps students construct and evaluate the knowledge.

In the lecture method, the teacher is at the center of the picture. In the interactive workshop method, the focus shifts from what the teacher does to what students do in the classroom. If we use open book exams, the lecture method must be replaced by the workshop method, or some other student-centered interactive method to trigger learning in students.

Needless to say, not all lectures are necessarily geared towards regurgitation. In a graduate course in MIT, Caltech or Stanford, we find some of the top people in the field communicating their recent ideas to students in the form of lectures. But these are contexts in which students are already well trained in their thinking abilities, and are not afraid of challenging the lecturer and converting the lecture into a discussion. In a context where students are not trained in the active mode of learning, however, the lecture mode is perhaps not the best.

To conclude, open book exams trigger a transformation of the modes of teaching and learning which enhance the students' creative, critical and decision-making faculties. The overall result can be summed up in two words: enhanced intelligence.