Johnstone, Richard; Patterson, Jenny; Rubenstein, Kim --- "Improving Criteria and Feedback in Student Assessment in Law" [1996] LegEdRev 11; (1996) 7(2) Legal Education Review 267

IMPROVING CRITERIA AND FEEDBACK IN STUDENT ASSESSMENT IN LAW

RICHARD JOHNSTONE,* JENNY PATTERSON** & KIM RUBENSTEIN***

INTRODUCTION

Spurred on by both the 1987 Pearce Report1 and the general changes to higher education spawned by the “Dawkins revolution” from 1988, there has been much critical self-evaluation leading to profound improvements to the quality of teaching in Australian law schools.2 Despite the changes there are still areas of general law teaching practice which have lagged behind recent developments in our understanding of what constitutes high quality teaching. One such area is assessment criteria and feedback.

The project Improving Feedback in Student Assessment in Law is an attempt to remedy this. It aims to produce a manual containing key principles for the design of assessment and the provision of feedback, with practical yet flexible ideas and illustrations which law teachers may adopt or modify. Most of the examples have been developed by teachers at the University of Melbourne Law School. The project was supported in 1996 by a Committee for the Advancement of University Teaching grant and the manual will be published late in 1997.3 This note summarises the core principles which are elaborated further in the manual.

ASSESSMENT DEFINED

Assessment has been very broadly defined as “occurring whenever one person ... is conscious of obtaining and interpreting information about the knowledge and understanding, or abilities and attitudes of ... [another] person.”4 For the purposes of this note, assessment has a narrower meaning, and takes place when there is a formal appraisal of, or judgment made about, a student’s work, which is communicated to the student. Assessment is sometimes described as having formative and summative aspects. The formative aspect generally refers to the on-going, diagnostic assessment of students, where judgments made about student work are used to form or improve a student’s competence by reducing “trial and error” learning by the student.5 Feedback plays an important part in formative assessment. Summative assessment is often described as the “credentialling” aspect of assessment—that which makes a public statement summing up the degree to which students have or have not demonstrated certain skills and knowledge.6 Summative assessment is not intended to have any immediate impact on student learning.7 Although the distinction between formative and summative aspects of assessment can be clearly stated, in practice assessment is rarely purely formative or summative. Most law school assessment might traditionally be termed summative in that it ultimately results in the provision of a grade to students. But in emphasising the provision of feedback we highlight the formative, diagnostic aspect of assessment.

THE IMPORTANCE OF CRITERIA AND FEEDBACK IN ASSESSMENT

Recent theories about student learning emphasise that students learn by actively engaging with subject matter, and by constructing meaning for themselves, thereby transforming their understanding of the subject matter.8 Little knowledge and few skills of any kind can be acquired satisfactorily through mere description to students of the knowledge or skill by the teacher. Feedback on students’ attempts to engage with the subject matter, and student reflection and action in response to the feedback, is essential to learning. “Action without feedback is completely unproductive for the learner”.9 Royce Sadler suggests10 that, to be acquired, most skills

require practice in a supportive environment which incorporates feedback loops. This usually includes a teacher who knows which skills are to be learned, and who can recognize and describe a fine performance, demonstrate a fine performance, and indicate how a poor performance can be improved.

Feedback provides information about the gap between the desired level of performance (expressed in goals and criteria) and the student’s actual level of performance. The teacher establishes the goals and criteria for performance and communicates these to students, with the intention that students share an understanding of the goals and criteria. The teacher then makes comparisons between actual and desired performance, and conveys information (feedback) about the comparison to the student, so that the student can reduce the difference between the established criteria and the student’s actual performance.11 Feedback might include information which indicates: the basis upon which the teacher awarded a particular grade; where relevant, how the student fell short of the required standard of performance; the misconceptions in the student’s understanding of the subject matter revealed by her or his performance in the assessment task; and what the student could do to gain a better understanding of the subject matter, or to perform better in future activities or assessment tasks. Effective feedback helps the student to reflect on her or his performance in and understanding of the subject matter, to identify strengths and weaknesses, and suggests how the student might improve where weak, or build upon what the student does well.12

In the context of student assessment, to be useful the feedback must relate to the specific criteria by which particular assessment tasks have been assessed. The particular assessment tasks, in turn, should assess the extent to which students have achieved the subject objectives. In short, the subject objectives, assessment tasks, specific assessment criteria and feedback must be integrated. Improving Feedback in Student Assessment in Law identifies four stages which facilitate the provision of effective and meaningful feedback to students on their assessment tasks. The four stages are:

1. Precise formulation of subject objectives;
2. Choice of assessment tasks best suited to the chosen objectives;
3. Articulation of clear criteria for marking the assessment tasks; and
4. Development of methods for providing feedback to students.

FORMULATING SUBJECT OBJECTIVES

We consider an aim to be a teacher’s broad statement of educational intent, and an objective as a specific statement about what the learner will be able to do better as a result of completing the subject.13 Among the different types of objectives are:

• Intellectual objectives—that is, the acquisition and understanding of knowledge, and the ability to analyse, apply, synthesise or evaluate such knowledge;
• Affective objectives—which focus on students’ own development in areas such as self-confidence, reflection, independence and initiative, and the development of values, attitudes and interests;
• Practical skills objectives—which involve the development of skills in areas such as written and oral communication, legal research, negotiation, statutory interpretation, drafting and interviewing.

Teachers establish objectives as part of their design of their subject. Objectives help teachers to focus on how students are to make progress in the subject, what content and activities they should include in the subject and in what order, and how they can structure their teaching to support the kinds of learning that students will need to be engaged in to learn the subject matter of the course.14 They provide criteria against which teachers guide, assess and monitor student learning.15 They also focus students’ attention on what teachers regard as the most important aspects of the subject, so that students have a clear idea of what they need to concentrate on to do well in the subject. Finally objectives provide students with a clear understanding of what they should be able to do at various stages of the subject, so that they can reflect on their progress through the subject.

These reasons for formulating objectives illustrate the links between objectives and assessment. Assessment tasks should determine the extent to which students have achieved the subject objectives. The criteria for assessing student performance are determined in the light of the objectives and assessment tasks. Feedback provides students with information about the extent to which their performance in assessment tasks measures up to the objectives of the subject, as particularised in the assessment criteria.

CHOICE OF ASSESSMENT TASKS BEST SUITED TO THE CHOSEN OBJECTIVES

It is widely accepted that assessment can drive the curriculum, in that students will tend to focus on those areas of the subject which are directly assessed, and tend to skim over the rest.16 Therefore the choice of assessment tasks is pivotal in focusing student attention on the most important learning objectives. A few key principles can assist teachers in the selection of assessment methods.17

First, given the functions of assessment in motivating student learning and defining the “hidden curriculum”, when choosing assessment methods the teacher should “try to ensure that assessment procedures promote and reward desired learning activities and outcomes.”18 The assessment tasks must, therefore, be tied to the important learning objectives articulated for the subject (the essential knowledge and skills), so that students focus their efforts on these objectives. For example, if an objective of the subject is to produce students who are competent at providing accurate legal advice, then students should be given repeated opportunities in the subject to apply legal principles to fact situations, and should be assessed accordingly.

Second, the assessment methods should be chosen to support learning, and not to undermine it.19 This means that the assessment methods must give rise to accurate judgments about students’ understanding and competence in carrying out the required skills in the subject. Assessment tasks should be aimed at revealing misconceptions students might have about the subject matter, so that only students with a robust understanding of the subject do well in it. Too much assessment might undermine efforts to promote good student learning, by reducing the time teachers have to provide students with feedback on their work, and reducing; student motivation and forcing them to take short cuts in their learning.20

Third, not only should there be consistency between the aims and objectives of the subject and the assessment tasks, but these two elements of subject design will have a major impact in shaping the teaching methods and media to be used in the subject.

Instead of automatically choosing an open-book examination or a research essay as assessment methods, teachers might choose other forms of assessment. Some examples include:

• a take-home examination, which could include tasks such as drafting a brief to counsel, a case note, preparation of a contract or a written judgment;
• a reflective essay on set written material, or on a student’s experience of observing a court hearing or tribunal sitting;
• mooting; and
• class participation, or presentation of a class paper.

Depending on the subject objectives, more creative and adventurous responses may be possible. For example, in a Criminal Law or Evidence subject teachers could assess their students’ ability to conduct accurate legal research as well as explore theoretical and policy issues (notions of “justice”, the construction of law, or the interaction of law and society) through the preparation of a script suitable for a television drama.

In a Constitutional Law subject, the teacher might require students to participate in a simulated “mini-Constitutional Convention”. Groups of six to eight students could be required to (i) research, (ii) analyse and (iii) reach conclusions about particular constitutional issues—for example, dual citizenship and qualifications for election to the Commonwealth government; the role, powers and future of the Senate; or the treaty making power of the Commonwealth government. Students would examine the current law, including, if appropriate an historical analysis, and then consider the arguments for and against reforming the constitution. They would present their conclusions to the entire class and answer questions from the class. They might also be required to submit a written version of their work to the teacher, which might include an outline of how the group conducted its research. This latter requirement not only encourages students to consider the processes involved in legal research, but also provides an incentive for the whole group to contribute to the oral presentation and written report.

This type of assessment task encourages students to develop and demonstrate skills in research, and oral and written presentation. It requires students to consider a legal problem from theoretical, political and substantive perspectives. It also requires students to work together and co-operate in the assessment task.

The key question to ask when choosing assessment tasks is: to what extent does the task enable students to demonstrate that they have achieved the specified subject objectives? While we acknowledge that most students and teachers are restricted by the limited time available, we argue that teachers should consider more than one assessment task in their subjects, because it is unlikely that a single assessment task could appropriately gauge student achievement over the whole range of chosen subject objectives.

ARTICULATION OF CRITERIA FOR MARKING THE ASSESSMENT TASKS

Once teachers have determined the subject objectives and the assessment tasks which will reinforce those objectives and test whether students have achieved the outcomes envisaged by the objectives, the next stage is to specify and communicate to students the assessment criteria. In assessing students’ work, and in providing feedback, teachers must be able to articulate the standard of work required from students, and to be able to judge the quality of the students’ work in relation to the required standard. For students to be able to use the feedback to improve, they need themselves to have a clear idea of the teacher’s conception of the required standard, so that they can make sense of the feedback, and use it to work out how to improve their performance. Apart from a simple argument that it is only fair and reasonable that students should know what the teacher has in mind when setting an assessment task,21 we suggest that establishing and communicating to students criteria for assessment is important to both teachers and students because criteria:

• focus teachers on ensuring congruency between the objectives and assessment tasks of the subject;
• focus students on what it is that teachers want them to achieve in the assessment, so that they put time and effort into the important aspects of the subject, and do not waste time on things that they are not required to do, or resort to superficial learning strategies because they are unclear about what to study;22
• provide a basis for qualitative judgments in awarding grades, and a foundation for consistency and predictability in marking between students;
• enable students to have a yardstick through which they involve themselves to a greater extent in evaluating and improving their own work in the assessment task;23 and
• provide a focus for feedback from teachers.

Assessment criteria outline what it is that teachers expect students to do in the assessment task, and indicate the factors that teachers will take into account in judging student performance in the assessment. Determining assessment criteria is part of subject design, to be undertaken when the assessment task is designed. Teachers should ensure that the criteria are based on the subject objectives, and that the criteria specify how the teacher will make judgments about the quality of student work in the assessment task. The clearer and more complete the assessment criteria, the more focused student learning is likely to be.24

Teachers have two principal means of specifying criteria for student assessment tasks: descriptive statements and exemplars (outstanding examples).25 Teachers can describe the criteria verbally, setting out the characteristics of student performance that students must demonstrate when performing the assessment task. The statement of criteria may specify things that must be present or absent in the piece of work (for example, an introduction and conclusion, clear themes, evidence supporting the argument), or which will be correct or incorrect (grammar, spelling, punctuation), and other features which might be discernible to a greater or lesser extent (structure, coherence). Criteria may also be specified using key examples, or exemplars, chosen to illustrate different standards of work. Exemplars are helpful because they are concrete, although where there are many criteria the teacher will have to provide a number of examples to show that good levels of performance can be constituted in a variety of ways.26

For assessment to be fully integrated into the teaching of a subject, the nature, purpose and criteria of the assessment must be articulated by the teacher and communicated to students (in the teaching materials or through handouts) at the beginning of the subject, and, at the latest, when students are required to begin work on the assessment. Teachers should be careful to ensure that students understand the criteria. For example, teachers should not assume that students know what is meant by expressions such as “appropriate use of content”, or a “well-structured essay”. Teachers must show students what these things mean. Different teachers might emphasise different aspects of the criteria, and teachers should be wary of assuming that there is a common understanding of key criteria or terminology.

Teachers may also want to devote class time to a discussion of the requirements of the assessment tasks. Students will then have opportunities to ask questions about criteria which are unclear to them. While teachers may be reluctant to devote much class time to a discussion of the assessment criteria, the pivotal role of assessment in teaching and learning, and the difficulties students have in understanding exactly what is required in concrete assessment tasks, suggest that this use of class time is as important, if not more important, than time spent covering new material.

DEVELOPMENT OF METHODS FOR PROVIDING FEEDBACK TO STUDENTS

Research indicates that students rank feedback on assessment as an important activity for effective study.27 Without feedback students:

• may not know why they have completed the assessment successfully, and accordingly may not be able to replicate their successful learning strategies in other activities;
• may not know in what aspects of their performance they are failing to produce the kind of work expected by teachers, or may not know how to improve identified weaknesses;
• will not be aware of their misconceptions of the subject; and
• might become increasingly demoralised with their studies, particularly if they do not know why they are not achieving the results they would like or expect.

If the only feedback students receive is a grade they may have a vague idea of what the teacher thought of the quality of their work, but virtually no conception of which parts of the assessment were good, which were bad, which parts of the subject the student understood, and which were misconceived. A grade provides no suggestion of what the student might do to improve her or his learning in the subject, in other subjects, or in future assessment.

As we noted earlier, the criteria for student assessment provide a focus and reference point for feedback.28 In providing feedback to students, teachers provide information about the extent to which students have met those criteria in their assessment tasks, identify misconceptions students may have about the subject matter, and suggest ways in which students might improve their knowledge of the subject, or their execution of the skills being tested by the assessment.

In order for feedback to be most useful to students, the feedback must:

• be specific, timely, constructive and should enhance student motivation to learn;
• be in language that students can understand;
• identify misconceptions in the student’s understanding of the subject;
• enable the student to take steps to remedy these misconceptions;
• assist the student in learning other subjects which the student will take later in the degree; and
• assist the student to complete assessment tasks in subjects which the student undertakes later in the degree program.

Students must also be motivated, and able, to reflect on the feedback, and to use it to improve their performance in the subject and in later subjects.

We are not suggesting that feedback is not worthwhile if any of these features is missing. Indeed, we are well aware that some of these elements are beyond the control of individual teachers. But to the extent that any of these elements is missing, there will be a reduction in the benefits of feedback for student learning.

The means of providing feedback will depend on factors such as: the nature of the assessment task; the number of students undertaking the assessment; the number of assessment tasks in the subject; whether or not the assessment anticipates, or forms the basis for, other assessment tasks in the subject; whether or not the assessment takes place at the end of the subject; the criteria for assessment; and the extent to which later year subjects are explicitly designed to build upon what students have learned in this subject.

Some possible vehicles for providing feedback to students include:

• interviews with individual students in which their performance in the assessment task is discussed in detail.
• a pro forma sheet with space under each criterion for individual written comments.
• a pro forma sheet with a scale ranging from “excellent” down to “not shown” for each criterion, to be circled or ticked as appropriate. There may also be space on the sheet for additional written comments.
• written comments throughout, or at the end of, the body of a written piece of work which reflect the criteria.
• generalised feedback in a discussion with the entire class, in which the teacher identifies general strengths and faults, provides a more detailed discussion of the particular comments provided to individual students, and suggests remedies to common problems.29
• model answers given to students when their marked assessment is returned to them, which provide examples of good responses to the actual assessment task set by the teacher. If students are provided with a variety of model answers, they can see the different ways in which the criteria for the assignment might have been met.

EVALUATION

Teaching is a dynamic activity, requiring constant reflection on matters such as whether the subject matter reflects current legal developments in the subject, whether the subject is structured in a way which best captures student interest and facilitates student understanding, and whether there is coherence between the objectives, assessment and teaching methods in the subject.30 Evaluation is an ongoing process in which a teacher seeks information about the impact of the design of the subject on student learning, using a wide range of sources, including information from students (from student evaluation questionnaires, individual and group student interviews, student diaries and journals), peers (where colleagues sit in on classes or examine teaching materials) and other sources. This information enables the teacher to identify strengths and weaknesses in the design of the subject, and to take steps to remedy identified weaknesses. Evaluation should include all aspects of subject design, including assessment. For example, when evaluating the subject teachers should check:

• with students early on in the subject that students understand the subject objectives, and know what is expected of them;
• that students understand from the outset of the subject how they should be preparing for the assessment;
• that students understand the assessment criteria; and
• that the procedures providing students with feedback were helpful, and that students benefited from the process.

FUTURE DIRECTIONS

In this note we have outlined a simple model for selecting assessment tasks, clarifying assessment criteria, and providing feedback to students on their performance in assessment tasks. We believe that skilled subject design which ensures that teachers choose assessment tasks which reflect the subject objectives, communicate clear and intelligible assessment criteria and provide constructive feedback on students’ performance in the assessment has the potential to improve student learning in law.

In the course of this project we have become aware of key issues which might be the focus of future research. The first is that feedback provided by teachers (or by fellow students) is of little use if students do not understand or apply the feedback to review their progress in the subject or to prepare for later learning. The second is that, in the context of students’ life-long learning, teachers and students should, over time, aim to reduce students’ dependence on information provided by the teacher (feedback), and to ensure that students learn how to evaluate the quality of their own work (self-monitoring). These issues require students to understand the learning objectives and standards of performance they are expected to achieve, and to learn how to monitor their own learning. In short, one of the purposes of providing students with assessment criteria and feedback on their assessment tasks is to provide them with opportunities to learn how to monitor and take greater responsibility for their own learning.31

* Faculty of Law, the University of Melbourne. This note was prepared in the course of the project outlined in the note, and draws on material which will be published in R Johnstone, J Patterson, & K Rubenstein, Improving Criteria and Feedback in Student Assessment in Law (London: Cavendish Publishing Limited, 1997).

** Dunhill Madden Butler, Solicitors.

***Faculty of Law, the University of Melbourne.

© 1997. (1996) 7 Legal Educ Rev 267.

1 D Pearce, E Campbell, & D Harding, Australian Law Schools: A Discipline Assessment for the Commonwealth Tertiary Education Commission: A Summary and Volumes 1–4 (Canberra: Australian Government Publishing Service, 1987).

2 See C McInnis, & S Marginson, Australian Law Schools after the 1987 Pearce Report (Canberra: Australian Government Printing Service, 1994) vii–viii.

3 R Johnstone, J Patterson, & K Rubenstein, Improving Criteria and Feedback in Student Assessment in Law (London: Cavendish Publishing Limited, 1997).

4 D Rowntree, Assessing Students: How shall we know them? revised ed (London: Kogan Page, 1987) 4.

5 R Sadler, Formative Assessment and Design of Instruction Systems (1989) 18 Instructional Sci 119, at 120.

6 For a discussion, see M Le Brun, & R Johnstone, The Quiet (R)evolution: Improving Student Learning in Law (Sydney: Law Book Company, 1994) 181–182, Rowntree, supra note 4, at 121–122; S Brown, & P Knight, Assessing Learners in Higher Education (London: Kogan Page, 1994) 37–41.

7 Sadler, supra note 5, at 120.

8 See generally: N Entwhistle, Styles of Learning and Teaching: An Integrated Outline of Educational Psychology (London: David Fulton Publishers, 1988); Le Brun, & Johnstone, supra note 6, ch 2.

9 D Laurillard, Rethinking University Teaching: A Framework for the Effective Use of Educational Technology (London: Routledge, 1993) 13–14.

10 Sadler, supra note 5, at 120.

11 Id at 120, 142.

12 Rowntree, supra note 4, at 24.

13 D Rowntree, Preparing Materials for Open, Distance and Flexible Learning (London: Kogan Page, 1994) 50; P Ramsden, Learning to Teach in Higher Education (London: Routledge, 1992) 130. There is a growing body of literature on the use/abuse of aims and objectives in tertiary courses, for example, Le Brun, & Johnstone, supra note 6, at 157.

14 Ramsden, supra note 13, at 130.

15 Le Brun, & Johnstone, supra note 6, at 155.

16 Id at 178–181.

17 See generally T Crooks, Assessing Student Performance (Sydney: HERDSA Green Guide, 1988) 10–17; P Nightingale, I Te Wiata, S Toohey, G Ryan, C Hughes & G Magin, Assessing Learning in Universities (Sydney: University of New South Wales Press, 1996) 10; Rowntree, supra note 4, ch 5; Brown, & Knight, supra note 7, chs 5–7; Ramsden, supra note 13, ch 10.

18 Crooks, supra note 17, at 10.

19 Nightingale et al, supra note 17, at 10.

20 Crooks, supra note 17, at 13.

21 See Brown, & Knight, supra note 7, at 102.

22 Id. See also Crooks, supra note 17, at 14.

23 See D Boud, Assessment and the Promotion of Academic Values (1990) 15 Stud Higher Educ 101; Brown, & Knight, supra note 7, ch 5.

24 See Crooks, supra note 17, at 24.

25 See generally Sadler, supra note 5, at 127–128.

26 ld at 128–129.

27 See Brown, & Knight, supra note 7, at 108; Ramsden, supra note 13, at 193–196.

28 See Brown, & Knight, supra note 7, at 113–114.

29 See Crooks, supra note 17, at 26.

30 See generally J Armstrong, & L Conrad, Subject Evaluation: A Resource Book for Improving Learning and Teaching (Brisbane: Griffith Institute for Higher Education, 1994); P Ramsden, & A Dodds, Improving Teaching and Courses: A Guide to Evaluation (Melbourne: Centre for the Study of Higher Education, the University of Melbourne, 1989); Le Brun, & Johnstone, supra note 6, ch 8.

31 For further discussion of self-monitoring, see Sadler, supra note 5.