Rawson, Shirley; Tyree, Alan L --- "Note: Self and Peer Assessment in Legal Education" [1989] LegEdRev 11; (1989) 1(1) Legal Education Review 135

SELF AND PEER ASSESSMENT IN LEGAL EDUCATION

SHIRLEY RAWSON^* AND ALAN L TYREE^**

A PROBLEM

The vast majority of law schools in the western world have the privilege of admitting hard working, creative and intelligent students, no matter how the slippery concept of “intelligence” is measured. Yet every year in almost every course, there are students whose work is assigned low grades because it lacks substance, clarity, creativity and sophistication. Why is such poor quality work produced?

No doubt the answer to this question is complex and is sometimes due to student isolation, alienation, disinterest, overcommitment outside law school, poor teaching methods, vague teaching goals or the student’s emotional, financial, or physical misadventure. This comment aims to identify and address another cause.

One explanation for the phenomenon of student failure is that the poor achievers do not define criteria for good work or, having defined such criteria, fail to apply them to their own work. By contrast, the student who achieves higher marks is able to do so because of some better capacity to determine the appropriate criteria for the work at hand and, equally as importantly, to apply the criteria to the task.

This comment describes some recent work which attempts to improve student performance on the basis of this explanation.

ASSESSMENT SKILLS IN CURRENT LEGAL EDUCATION

The process of defining and applying criteria to evaluate one’s own work is known as “self assessment.” Although self assessment has never been an explicit part of university education, there are certain activities in traditional legal education which indirectly improve students’ assessment skills.

For example, the first time that students are asked to read a case, their reading is likely to be very superficial. They will express some dismay, and possibly anger, when asked questions about the case which they find that they cannot answer. After training in the case law method, they read the case from a different critical viewpoint. They know the criteria to be applied in order to gain official approval and have learned how to evaluate their reading “performance.”

Teaching students to read cases is only one of the skills taught in law schools. Typically, we attempt to teach a certain craftsmanship in legal writing as well as skills in verbal communication. In each case, the skills to be taught are at a much lower level than most of us would call “creativity.” Improvement in each of the skills requires an ability to be self-critical during the process of exercising the skill.

The important thing from our current point of view is that most students do improve in this process of self-analysis during the repetitive experience of completing exams and assignments. They are being taught, or at least are learning, these skills within certain contexts — though the process is often clumsy and concealed.

If it is possible to teach self assessment skills directly to law students, the benefits should be obvious. Graduate lawyers skilled in self assessment should be more effective and efficient in whatever role or career they move into. It is this general, and we believe significant, improvement in the skills of graduate lawyers which makes self and peer assessment suitable for inclusion as part of legal education.

There have been few studies on the benefits of adding a peer assessment component to any self assessment project even though many studies are reported as self and peer assessment projects. Our experience is that student response to suggested peer assessment is usually adverse. However, the authors consider that introducing a peer assessment is a very useful way of assisting in the teaching of self assessment skills. As most formal assessments of the work of individuals in the workforce are carried out by management, it may seem that there is no real need to be able to assess a peer’s work. However, since formal peer review is being used in an increasing number of professions we believe that it is appropriate to include peer review techniques in any program. Moreover, most professionals are constantly called upon informally to evaluate the performance of their peers.

SOME RECENT RESEARCH

Although research in self and peer assessment dates back more than fifty years ¹ much of the early material is ad hoc and not very helpful to the law teacher. Of the more recent material, a great deal is concerned with student marking in subjects such as engineering and mathematics where it is possible for a teacher to prepare a detailed marking guide. The predominant aim of much of this work is simply and candidly to save teacher time. We have no objection to any scheme which will save teacher time. However, this incentive is noticeably absent for most traditional forms of law school assessment, for which marking guides require considerable detail and investment of time in their preparation.

There is, however, a body of research which aims to explore the educational benefits of teaching assessment skills directly. The indications are that there is a direct relationship between assessment skills and the quality of work which a student produces. More excitingly, there is evidence that the skills can be exploited directly with a consequential improvement in student performance. ²

The remainder of this comment outlines some research done with law students in recent years by the authors which supports the statements of the preceding paragraph.

Marks and Assessment Skills

If there is no relationship between marks and assessment skills, then there may be no need to continue with the direct teaching of self and peer assessment. The authors conducted a study in 1988 to test the relationship. The subject of the study was the first year graduate entry at the University of Sydney Law School. Past experience has shown that although these students have already been successful students in other disciplines there is a wide variation in their performance at law school.

The assessment exercise was undertaken on the students’ first significant piece of legal writing in Legal Institutions which dealt with the topic of parliamentary sovereignty. Although the students had been given a small amount of tutorial instruction on methods of legal writing, there had been no attempt to test their understanding of these methods or, more importantly, to provide feedback to them concerning their understanding of the methods.

Criteria were established before the writing of the essay by a process known as the “nominal group” technique. This is an intense interactive session where the students negotiate and agree upon a set of objective criteria for the evaluation of task at hand. The teacher in the subject agreed to mark the essay according to the defined criteria. This method of establishing the criteria ensured that all students were working to the same criteria and that, at the assessment stage, both teacher and students would be using the same criteria.

The completed essays were assessed by the teacher, two students and the author of the essay in that order. The point of leaving the self assessment until the end is that it provides some “distance” from the work. The essays were photocopied and all identifying marks removed for the purposes of the peer assessment.

If the average self and peer mark deviated from the teacher’s mark by more than 10 points the essay was submitted to an independent teacher marker. The independent marker was required to mark 15 essays. This procedure was adopted because we were including the self and peer marks as a formal part of the assessment. As expected, the difference between teacher and student marks was dramatic. The overall correlation of self marks with teacher marks was only .313. The correlation of peer marks with teacher marks was .376.

This divergence cannot simply be attributed to teacher and student using different marking scales. Using rank-order statistics, the Kendall rank correlation coefficient of self and teacher marks was only .201. ³ One consequence of this is that no scale changes would improve the agreement unless the scaling changed the order of marking.

Our interest in the order statistics was the main reason for asking each student to assess two of his or her peers. In almost half the cases (26/59), the student marker did not even get the marks of the two peers in the same order as the teacher!

The most interesting point of the project was the finding that “high achieving” students were substantially better assessors than “low achieving” students. If a “high achieving” student is defined as a student who received a teacher mark above 70, then “high achieving” students and the teacher agreed on the ordering in 57% of all cases whereas “low achieving” students agreed with the teacher on the ordering in only 36%.

Self and Peer Assessment of Legal Essays

In 1984, one of the authors (Tyree) conducted a self and peer assessment exercise with a group of final year law students at the University of New South Wales. Students in two optional final year classes — International Trade Law and the Law of Banking — were required to write a major research paper as the main component in the course assessment. The teacher’s experience in previous years with such papers had been somewhat disappointing as it seemed that students failed to begin their papers early enough and, perhaps as a consequence, the quality of the work was often quite poor.

In cooperation with David Boud of the Tertiary Education and Research Centre, the problem was approached by using a self assessment program. It was thought that the students probably were able, as final year law students, to define the criteria for a good piece of research work. We hypothesized that the poor quality work could be seen as a failure to apply criteria which the students implicitly knew.

Early in the term, the students were required to define objective criteria which would be used in the marking of the final research essay. The method used was the “nominal group” technique. As expected, the students had no difficulty in identifying criteria which were acceptable to the teacher. It was agreed by the teacher that the student generated criteria would be those used in marking.

The students were given a summary sheet of the relevant criteria and, in theory at least, referred to the list as they researched and wrote their essays. They were required to hand in a self assessment sheet ⁴ (in a sealed envelope) along with their essays at the end of the course. This sheet required them to give a mark to each of several categories and to give a brief (one line) reason to support the mark given. The aim of this exercise was to use self assessment to improve the performance of students who were believed to be able to define and apply criteria but who, for reasons of time, often did not do so. In the opinion of the researchers, the aim was achieved. ⁵ Although it can only be a subjective opinion, the teacher thought that the essays were of a significantly higher quality than those obtained from students in preceding years.

Student self marks did not count as part of the formal assessment in this exercise. Because of the highly individual nature of the essays (students chose their own topics which were then approved by the teacher) it was thought inappropriate to ask students to mark their peers.

Although the exercise was not designed to test student marking ability, the results showed that these final year students were far more skilled markers than the first year graduate group previously discussed. The correlation with the teacher mark was .7.

SOME ISSUES

Research in the self and peer assessment area is fragmented among a number of different disciplines. It has been used for a variety of different reasons with the result that reported research results are often difficult to compare and to reconcile. Boud and Falchikov have collected the research which generated quantitative results, and have produced one of the first studies to attempt to make sense of the fragmented literature. ⁶ Boud and Lublin have collected and summarised the work which has been done in Australia. ⁷

There are several issues which are interesting but about which little is known.

Should Self/Peer Marks Count Formally?

We have found great student hostility to the inclusion of self and peer assessment marks as part of the formal assessment process. It is difficult to know if this is a general phenomenon or if it is more visible in the traditionally competitive law school environment.

In the 1988 study by the authors, mentioned above, we used the formula of final mark = .75T + .25 (S + P1 + P2)/3. ⁸ This formula was subject to the “safety net” that the paper would be referred to an independent referee if the second average mark differed from the teacher mark by more than 10 points. It would probably be an understatement to say that the procedure was unpopular!

More recently, we proposed to a graduate class that the formula final mark = .SOT + .20S be used provided that S was larger than T but within 10 points. Students voted 60% in favour of continuing the project, 40% against. When we agreed to drop the formal inclusion of the self mark, the vote was 100% in favour of proceeding with a self and peer assessment project. Note that the individual student could only benefit from our proposed formula!

This last result is so surprising that it is worth noting the comments which students offered in objection. Typical comments included “the teachers know best”, “we are here to be taught not to teach”, “I am worried about my peers giving me a bad mark”, “how anonymous can all this be”? and “we could go to the Dean and complain”! We note that these comments are not all addressed to the formula proposed and so must reflect some deeper disquiet with formal inclusion of marks.

Can Assessment Skills be Taught Directly?

Although we believe that the answer to this question is “yes’’, it must be admitted that there is no research which directly supports our view. What is required is a “vertical” study which follows a group of students over a period of years. These students could be compared with a “control” group or the results could be analysed more subjectively. We are hoping to conduct such a study at the University of Sydney Law School during the next few years.

APPENDIX: EXAMPLES OF SELF-ASSESSMENT FORMS

Self-Assessment of the Research Project

Subject_____________________

The attached sheet includes the edited list of criteria for self-assessment which were generated in the class. These will be the criteria which form the basis for your own assessment of your assignment. Not all of them are necessarily of equal weight and you may wish to place more emphasis on some rather than others.

The most important element of any self-assessment is making judgments about our own performance, not assigning marks. A column has been provided for you to give yourself a mark for each item, but this is of much less importance than the comments you write on the extent to which you met did not meet each criterion in the work you produce.

You are asked to hand in your assessment and an overall mark for your assignment together with our completed assignment. Your self-assessment schedule should be completed, placed in a sealed with your evaluation sheet and handed in with your assignment on the last day of the session. It will not be viewed until after your work has been marked.

SELF-ASSESSMENT SCHEDULE

Subject ___________________

Below are listed the main groups of criteria agreed earlier, plus an additional category for you to use as you think appropriate. Please write detailed notes against each item indicating the extent to which you have met each and a brief note of the ways in which your work has not met them. On the right hand side give yourself a mark (out of ten) for each one of the factors.

Detailed                 Mark/1O

GOALS, RELEVANCE OF MATERIAL

1. aims and objectives defined
2. target audience identified
3. stays within boundaries of the question examined
4. all points covered
5. each point clearly related to main thrust of essay

RESEARCH

1. overall knowledge of subject
2. quantity, quality, depth of research
3. appropriate number of sources and authorities used
4. use of primary as well as secondary sources
5. alternative views noted
6. comparative research where appropriate

SYNTHESIS INNOVATION

1. imaginative topic chosen
2. originality of ideas
3. innovation in application of theory
4. own ideas formulated
5. cross-fertilisation of disciplines where appropriate
6. suggestion for change where appropriate

STYLE, STRUCTURE

1. clarity, conciseness of expression
2. direct and assertive
3. consistent theme
4. interesting
5. use of correct legal language
6. grammar, syntax, spelling correct
7. good use of section headings, paragraphs, etc.

ARGUMENTS AND ANALYSIS

1. arguments logical and well organized
2. clear exposition of the problem
3. issues and sub issues clearly identified
4. sources analysed, not just quoted
5. full discussion and critique of findings
6. well reasoned conclusions
7. no loose ends in conclusions
8. sources well organized
9. no plagiarism

PRESENTATION

1. good typing and general presentation
2. thorough citation
3. interesting to read

PROCESS

1. criticism has been sought of arguments and draft
2. wasted time in research and presentation minimised

OTHER IMPORTANT CONSIDERATIONS (please list)

__________________

Title of Project___________________                

Overall Assessment____________ /10

^[*] Faculty of Law, University of Sydney.

^[**] Faculty of Law, University of Sydney.

© 1989. [1989] LegEdRev 11; (1989) 1 Legal Educ Rev 135.

^[1] D Boud & J Lublin, Self Assessment in Professional Education: A Report to the Commonwealth Education Research and Development Committee (Kensington: University of New South Wales Tertiary Education Research Centre, 1983).

^[2] Ibid.

^[3] MG Kendall, Rank Correlation Methods 3rd ed (New York: Hofner, 1962). The Kendall coefficient is a measure of the degree to which the rankings of the two markers agree. As such, it is independent of the scale of marks used.

^[4] An example of the self assessment sheet is set out at the end of this comment.

^[5] Clearly there are other variables to be considered, including the thesis that any enthusiastic experiment breeds success, before the success of this self assessment strategy per se can be assured.

^[6] D Boud & N Falchikov, Student Self Assessment in Higher Education: A meta-Analysis (in preparation).

^[7] Boud & Lublin, supra note 1.

^[8] Where T = teacher’s mark; S = student’s “self mark”; P1 = mark by first peer; M = mark by second peer.