Barnes, Jeffrey --- "Book Review: A Lateral Approach to Assessment" [1998] LegEdRev 10; (1998) 9(2) Legal Education Review 213

BOOK REVIEW:



A LATERAL APPROACH TO ASSESSMENT

JEFFREY BARNES*

Peggy Nightingale et al, Assessing Learning in Universities, Sydney, NSW, University of New South Wales Press, 1996, pages 1–315.

Price $39.95 (softcover) $35.95 (softcover at academic price direct from publisher)

ISBN 0 86840 408 X

INTRODUCTION

Drama lessons for advocates ... designing more effective “Wanted” posters ...taking the pulse of High Court judges — this work certainly excites thought about assessment (!) The authors were “looking for a way to encourage people to break free of the traditional methods of assessment in their disciplines”. (4) Instead of structuring a book of assessment materials around the various disciplines, they have produced a book which purports to be structured around abilities which are “common to all disciplines”. (4) This is a difficult brief which, unsurprisingly, only partly succeeds.

The book is compiled by a team drawn from the Professional Development Centre at the University of New South Wales. The production was funded by a grant from the Committee for the Advancement of University Teaching. The materials themselves are largely the work of various academics who responded to the project team’s call for innovative and carefully thought out assessment proposals in their various disciplines. The project team then had the challenging task of turning the various contributions into case studies which were (hopefully)-to be of more general use; in other words, of use to other disciplines. They settled on eight clusters of abilities, which provide the structure for the package. The clusters are:

(i) thinking critically and making judgements

(ii) solving problems and developing plans

(iii) performing procedures and demonstrating techniques

(iv) managing and developing oneself

(v) accessing and managing information

(vi) demonstrating knowledge and understanding

(vii) designing, creating, performing

(viii) communicating. (3)

Each cluster is presented as a module, and the eight modules are described as a “portfolio of current practice”. (11) The book is more, however, than simply a package of assorted assessment tasks loosely arranged under the above headings (and subheadings). It also contains a synthesis of educational theory and research on each of the module topics and thus constitutes a handy, if brief, guide for the university teacher. Each of the modules contains a concise introduction to the ability cluster and in some cases to the particular abilities with which the cluster is concerned. The case studies commonly contain information on:

• context of the exercise
• abilities being assessed
• description of the assessment task
• sample assessment questions
• assessment procedure
• criteria for assessment
• marks, grades and feedback
• strengths and limitations of the exercise

While the case studies were generated by academics, the precise source of the materials varies. Most are previously unpublished, though some are taken from published work. But, using the teacher’s words as much as possible, virtually all of the studies have been edited to address the matters listed above in a consistent manner. The originators are identified, so there is a possibility, presumably, of correspondence if further information is required. Following the case studies, there is a commentary by a member or members of the project team on the case studies, and in some cases further discussion of the assessment method by the module author or authors. A glossary of assessment terms, an annotated bibliography of works referred to in the case studies, and an index (including an index to disciplines) completes the work.

Four of the 62 case studies are drawn from law schools. They involve simulated client interviews in the second module (solving problems and developing plans); a research assignment in the fifth module (accessing and managing information); assessing class participation and self assessment of class participation, both in the eighth module (communicating — orally). All of these studies are well thought out, but it is obvious that if the work is to appeal to law teachers, it must do so because of the tasks developed in other disciplines. The assumption the authors make is that, in whatever discipline students are placed, as graduates they form a group who ought to have core “abilities and characteristics (knowledge, skills and attitudes)”. (2) The work is based on other assumptions about assessment which presumably are meant to apply across the board. We are told that:

Traditional forms of assessment have usually focussed on ranking students according to the knowledge that they gained in a subject or course. ...

At worst — and far too frequently — the kinds of assessment methods chosen force students into surface learning — rote learning and regurgitation of isolated facts and formulae — quickly acquired to meet exam pressures and just as quickly forgotten. (6, 7)

To assess the work, it is convenient to turn to the intended outcomes of the project set out at the beginning of the work: (1)

1. to synthesise the literature on assessment in higher education so as to make it accessible and relevant to the teachers of a wide variety of disciplines;
2. to develop materials which not only provide models of assessment processes but also relate assessment to curriculum design and pedagogy and to the enhancement of student learning;
3. to develop materials which address principles of good practice at the level of school or department as well as at the level of individual subject, thus encouraging the exercise of academic leadership;
4. to develop materials which encourage consideration of alternatives to traditional methods of assessment in the various disciplines;
5. to develop materials which will suit the needs of individual academics who wish to consider modest changes to their practice as well as the needs of individuals or groups who wish to expend considerable energy in a major reappraisal of their practice;
6. to develop materials which will assist colleagues working in academic staff development or educational development to offer workshops, seminars and programs of study on assessment issues.

In the space available, I shall discuss what seem to be the more important of these goals.

CONSIDERING ALTERNATIVES

The key goal for the project, I would argue, is the fourth: the encouragement of the consideration of alternatives. Let me first state some positives about the way the work seeks to achieve this goal. A number of the assessment exercises originally designed for other disciplines could be usefully employed in law schools, and could even be innovative in this respect. Of special interest to this reviewer were “portfolios” and an “autobiography”.

A “portfolio” is a package of items of written work chosen as well as assembled by a student (rather than the teacher) to encourage the student to evaluate his or her own learning and to demonstrate what learning has taken place. The example given (case study 21) was suggested by an academic who teaches Psychology to pre-service and in-service teachers.1 To assist students to choose their own portfolio, the teacher compiled a list of items from which students could choose if they wished. Items included: a self-set essay, in note form; a few objective test items, explaining the correct answer and why each item was selected; a one or two page letter to a friend who is thinking about taking this unit next year, describing the unit and offering advice; a list of questions that could be used to assess the unit; and anything the student thinks is appropriate. This exercise gives students the opportunity to think about the learning process in a very practical, creative and individual way. It gives the students responsibility, but, since they are assessed in terms of how much the student has understood the topics and reflects on their profession, it achieves a congruence between course objectives, learning processes and assessment. Although a few students “reacted negatively, wishing to the end to be lectured to and assessed by exam or set essay assignment” (94), the results speak for themselves: one third As; 40 per cent Bs. One example of an “autobiography” in the work was developed in a town planning context, because, according to the teacher, “students of town planning often do not appreciate the extent to which their own backgrounds (predominantly middle class, suburban and white) are likely to influence the way they approach their role and the decisions they make”. (99) In this exercise (case study 23) the students were “asked to write a paper which describes specific incidents from their own life which were significant in forming and confirming their values and sense of identity”. The students “are asked to develop this into a broader discussion of what they consider to be their place in the community and finally to consider in what ways their particular collection of characteristics, values and opinions are likely to be an advantage or a disadvantage in a planning career.” It can be strongly argued that similar questions can be put to new law students. The social background of law students is even more restricted.2 But what benefit is likely to be gained by such reflection? In the introduction to the section, “The ability to understand and manage one’s own feelings and be sensitive to the feelings of others”, there is a balanced discussion of this issue by the module author. (98)

Other exercises, not often used in law but worth considering, are journals (or reflective diaries), and alternative modes of marking: peer and self evaluation. Each of these matters is thoughtfully and critically illustrated and commented upon.

Many teachers, therefore, may find the publication to be a useful resource if contemplating new ways of assessing students. Does this mean the book is likely to succeed? First of all, it should be noted that portions of the book appear to have little or no relevance for law. For example, it is hard to see how any of the exercises in the third module, “Performing procedures and demonstrating techniques”, dealing with abilities such as computation, taking readings, using equipment, following laboratory procedures, following protocols and carrying out instructions, is relevant. Having said that, seven out of eight modules are relevant to law to a varying extent.

Secondly, the book’s description of assessment practice, underscoring the need for alternatives, is flawed in the case of law. While I have argued elsewhere3 that there is a need for alternatives to the dominant mode of assessment in law school, the problem examination, the book is written from the perspective that assessment in universities has at least in the past concerned itself too much with the narrow task of assessing knowledge, and with rote learning and regurgitation. This assumption seems much more applicable to the science-based disciplines than to law. Not for a long time has law assessment concerned itself with purely knowledge-based assessment. When overemphasised, the limitations of the problem examination are sorely exposed, but it is not an exercise which rightly attracts the assumption of the work’s authors. Regurgitation of a body of knowledge (principles of law) is not an adequate response in a problem examination, since that mode of assessment calls for the application of a body of knowledge, using skills of legal reasoning, to certain hypothetical facts.

Thirdly, how many law teachers will avail themselves of the work? Using this book as a resource requires lateral thinking, since there are insufficient law case studies to permit instant adoption. The book’s intended purpose is clearly to facilitate borrowings from other disciplines. Of course, those who regularly experiment in their teaching would be attracted to a book such as this. But I fear that the book will fall on many deaf ears. Despite the best intentions of the authors, it is difficult to see how case studies, many admirable, from other disciplines will motivate hard-pressed law teachers to change long entrenched habits. It is doubtful whether merely arranging the exercises under common headings such as “Thinking critically” and “Communicating” is enough. There are practical and legal obstacles too. Teachers who teach in multiple streams or who co-teach with colleagues do not have the luxury of simply deciding what is the best assessment arrangement in theory. Legal requirements can also restrict assessment avenues in subjects required for admission as a solicitor or barrister (or both).

Fourthly, since the case studies are mostly drawn from disciplines other than law, the nuances in each discipline are likewise mostly absent. I have already mentioned that the premise of the book — to get away from fact-based assessment — is unwarranted in the case of law. In law we need to know how to broaden the assessment from simply the law examination and the essay, which are both overdone at the expense of other modes of assessment.

MAKING THE LITERATURE ACCESSIBLE

The work makes an ambitious attempt to break down traditional barriers in thinking about assessment. I have explained how, in regards to law, the achievement is likely to be very modest. But in relation to the first goal of synthesising the literature on assessment, few problems are evident. The writing is concise, accessible to the non specialist and frequently insightful; only rarely does it offer the trite remark (such as advising us to use topics of current concern: 29). The module authors have also avoided overloading the text with references to the literature (which with the Harvard style of referencing makes for tedious reading). The writing on thinking critically and making judgments (module l), for instance, makes useful distinctions between “reasonable thinking”, “reflective thinking” and “focused thinking”. (15) Other well written, pithy accounts relate to such topics as student learning (91–92), multiple choice tests (151), student writing (207–209), and setting essay questions (224–226).

GOOD PRACTICE AT THE LEVEL OF THE SCHOOL

Other goals receive less attention in the work. The third goal of addressing principles of good practice at the level of the school or department fits into this category. One of the few places where recognition is given to the role of the institution or delegated decision making body is in the Introduction:

the first principle to be kept in mind is that the choice of assessment method should allow reasonable judgments to be made about the extent to which the student has achieved the aims, objectives or intended outcomes of the educational program. If, for example, the educational program claims to produce graduates who are innovative problem solvers then there should be many occasions throughout the program where students are asked to solve problems and develop innovative solutions and are assessed accordingly. (10)

Is lip service paid to this goal? Most of the attention is on the individual teacher, as Peggy Nightingale, the author of one of the modules, observes:

This brings us back to the theme running through these modules on assessment, that teachers should be examining their own assessing and examining in order to learn about what works and what doesn’t, and in order to make adjustments, if necessary, to the results to achieve fair outcomes. (226)

When virtually all the attention is on the individual teacher, a distorted picture of the context in which assessment is formed and ought to be formed is presented. I have argued elsewhere4 that the Faculty (or other delegated decision making body) ought to have a much greater role in the setting of modes of assessment, so I will content myself with a brief summary.5 The first point is that academics are not alone in their teaching and assessment (the two are inextricably entwined). Teaching in the one degree program makes for an interconnectedness, and other teachers may have a legitimate interest in a subject which they currently do not teach. For example, the task setting and assessment in introductory subjects is relied on by teachers of later year subjects.

Secondly, there is a need for co-ordination. Assessment is too important a matter to be left entirely up to the idiosyncrasies of individual teachers or the vagaries of group membership or group dynamics. Without faculty involvement and co-ordination in assessment design the following problems may arise or be perpetuated:

Some assessment tasks may be required without clear justification to our peers. Some essential tasks may not be undertaken at all, or insufficiently.

Some tasks may be unnecessarily duplicated within a particular degree program.

Some tasks may overload students with work at a particular time.

Thirdly, faculties are appropriately constituted to make large scale decisions about assessment policy. They can take account of a wide field of interests. Importantly, they can give students a right to be heard. Faculties with outside membership can permit the users of assessments — such as the legal profession in our case — to inform academics about the ultimate function assessment should perform and does perform in their experience. Since employers and others will rely on the degree, they would seem to have a legitimate claim to be heard on the kinds of tasks upon which academics will pass judgement.

Finally, it may be noted that faculties and other comparable decision making bodies and offices of the Universities currently do “teach, in the sense that they make decisions which do affect assessment and teaching generally. They approve the subjects which may be taught by faculty members; they affect the teaching methodologies which may be employed by structuring classes into lectures, tutorials, etc; and they influence the teaching climate which prevails by determining class maximum sizes. Therefore, it cannot be said that decisions on assessment are restricted to the actual teachers.

MEETING THE NEEDS OF INDIVIDUAL ACADEMICS

The fifth goal of the project was to develop materials which suited the “needs of individual academics”. Any work on assessment which did not take account of the limited (and frequently diminishing) resources available to academics would be seriously deficient. This work does not ignore resource issues, though unexpectedly it is not the central concern6 Some of the case studies seem expensive in terms of resources, for example the staff/student ratio recommended in case study 11. Some cases studies acknowledge the cost factor as a limitation (case study 9). Module authors (86) as well as the project team acknowledge it too. (10) More interesting are the exercises which claim to be efficient. In a case study illustrating a critique of a current concern drawn from Geography, the teacher concerned advises that: In an attempt to provide quality feedback with maximum efficiency (for teaching staff), individual comments on the reports were kept to a minimum but more complete explanation and advice was given to students in a comments sheet ... outlining common strengths and weaknesses in relation to the assessment criteria. Individual comments on the reports focused on the strongest and weakest features. (27)

Aside from the time spent in producing such a comprehensive document, there is the (explicit) assumption that students will make the effort to reconcile the marking guide with their own report and grade. Some students may have difficulty doing this.

STRUCTURE AND PRESENTATION

The ambitious structure — to synthesise the abilities of graduates under 8 major headings (frequently subdivided into sub-abilities) is generally clear. The exception is the eighth module on Communicating which concerns the assessment of student oral and written communication. It is confusing for marking and grading — quite distinct issues from assessment design — to be dealt with in this module. Whereas the book is presented as being organised around eight abilities of students (3), in the last module the book slips into advice about the abilities of academics. So while I find it helpful for marking and grading to be discussed, the authors have unduly complicated the eighth module. Marking and grading are not assessment methods per se. Rather, they are the sequel to assessment methods (which may not be particularly concerned with communication). I would have preferred therefore to see marking and grading dealt with separately from the design of assessment exercises.

On the whole, the book is well presented and attractive to use. The case studies frequently include extracts from the actual assessment material distributed by the teacher concerned, so the book acts as a handy set of precedents. Two brief criticisms, however, can be made of the presentation. The word “judg(e)ment” is not spelt consistently, and the index is inconsistent and unhelpful under the item “Law”.

CONCLUSION

Though the book sets out a large number of case studies of assessment in Australian universities, thoughtfully annotated, it is not a ready-made resource for law teachers. Conceptually though, it is an innovative work which tests the boundaries of what it means to be a university graduate. It is certainly interesting and even refreshing to see what the disciplines do share. There are clearly, at a general level, goals the various disciplines share, but in my opinion a discipline-based book of precedents would be more valuable to many law academics with limited time on their hands. This is not to say the book does not fill a gap in the literature, which is either theoretical, or practical but discipline based.

More successful, though not ground breaking, is the synthesis of the literature on assessment. Although there are many other works on assessment in universities: this one manages to be economical and illuminating in its analysis.

* Senior Lecturer, School of Law and Legal Studies, La Trobe University.

© 1999. (1998) 9 Legal Educ Rev 213.

1 The teacher concerned is John Biggs, of the University of Hong Kong, a prominent writer on educational theory.

2 D Weisbrot, Recent Statistical Trends in Australian Legal Education (1990–91) [1991] LegEdRev 11; 2 Legal Educ Rev 219, at 227–237. Weisbrot makes the point that law students have more elite social backgrounds than all other groups of students (except medicine students in some respects), measured by the educational backgrounds of the students’ parents (234–235) and their attendance at private schools (235–236). For more recent developments (which have not greatly affected that picture), see J Goldring and S Vignaendra, A Social Profile of New Law Students (Sydney: Centre for Legal Education, 1997), reviewed by J Nelson[1997] LegEdDig 48; , (1997) 6(1) Legal Education Digest 9 (referring to impact of newer and regional law schools); and E Clark, Australian Legal Education a Decade After the Pearce Report (1997) 8 Legal Educ Rev 213, at 217–218. Cf A Ziegert, Social Structure, Educational Attainment and Admission to Law School (1992) 3 Legal Educ Rev 155.

3 J Barnes, The Functions of Assessment: A Re-examination (1990–91) [1991] LegEdRev 10; 2 Legal Educ Rev 177.

4 J Barnes, Planning Assessment — Whose Responsibility?, in A Window Between Worlds, Papers from a Conference for the Monash University Community, 24–26 November 1992 (Clayton, Vic: Higher Education Advisory and Research Unit, Monash University, 1993) 11.

5 I have also taken the opportunity to clarify my argument.

6 An excellent source of ideas on assessing efficiently is also produced by the Professional Development Centre at the University of NSW; it is L Andresen et al, Strategies for Assessing Students: A Guide for Setting, Marking, Grading and Giving Feedback on Assignments, Tests and Examinations (Kensington, NSW: Professional Development Centre, University of NSW, 1992). It is No 1 in a series entitled “Teaching with Reduced Resources”.

7 Barnes, supra note 3 and M LeBrun & R Johnstone, The Quiet (R)evolution Improving Student Learning in Law (Sydney: Law Book Company Limited, 1994) 177–225 refer to and discuss the general literature.