|
|
This booklet has two parts. Part I: How Are Our Students Doing? reviews the results of provincial and national student evaluations in specific subject areas. Part II: Developing a Policy on Student Evaluation offers suggestions to boards of education who may be considering developing a policy on student evaluation.
There are two provincial and one national evaluation programs.
The two provincial evaluation programs are:
The national evaluation program is:
The information that follows is drawn from the provincial and national
evaluation programs and is presented in two categories:
Student achievement in specific subject areas is summarized in the sections
that follow.
Back to Table of Contents
The science performance of Saskatchewan 13- and 16-year-olds in 1996 was comparable to that of all Canadian students. However, for Canada as a whole, and for Saskatchewan, the percentage of students who achieved mid-range performance met standards, but fewer than expected Canadian and Saskatchewan students achieved top levels of excellence.
An area of strength among Saskatchewan elementary students in 1993 was
their understanding of the nature of science. Areas of weakness included
observing, describing, hypothesizing, questioning, inferring, written communication,
and the effect of science on one personally.
Back to Table of Contents
The 1997 provincial assessment of Grade 5, 8 and 11 students found that Grade 5 student outcomes for good math proficiency met provincial expectations; otherwise students’ overall achievement was below expectations for good and top-level proficiency.
The 1995 provincial assessment found that fewer Grade 5 students than expected achieved excellent and acceptable performance in most areas. At Grade 8 and 11, more students than expected achieved excellent performance and fewer than expected achieved adequate performance in most areas.
In general, in the 1995 assessment, Saskatchewan students were stronger on math concepts, procedures and relationships and weaker on practical applications and problem solving.
In the 1997 national assessment, both Canadian students as a whole and
Saskatchewan students achieved below expectations in math content and problem
solving. However, the percentage of Saskatchewan 13- and 16-year-olds
who achieved adequately was lower than for Canada as a whole.
Back to Table of Contents
The 1998 national assessment of reading and writing showed that:
The 1998 provincial assessment of listening and speaking skills showed that Grade 5 students met provincial expectations, but both Grade 8 and 11 students fell short of expectations in some dimensions of listening and speaking.
The 1996 provincial assessment of Grade 5, 8 and 11 students showed that most Saskatchewan students had adequate or better reading skills and that students met or exceeded expectations in most reading areas. In writing, more students performed at the middle achievement levels than expected, and fewer performed at both very high and very low achievement levels.
There was some evidence that Grade 8 and 11 students’ writing skills had improved between 1994 and 1996.
The 1994 provincial assessment showed that, in general, Grade 5, 8 and 11 students exceeded expectations in reading, and were below expectations for writing for all three grades.
The teaching-learning process is central to the Grade 7, 8 and 9 Health
Education curriculum. This process can be summarized as follows:
|
As part of the assessment of the 1995 Grade 7, 8 and 9 Social Studies
curriculum, students, teachers and administrators were asked to report
on achievement. In addition, students wrote tests to measure their
actual achievement.
The 1998 evaluation of the Grade 1-9 Arts Education program included
a review of student outcomes and experiences. The review found that:
A policy on student evaluation is important for five reasons:
Stakeholder involvement has several advantages. It:
Involving students, parents, teachers and community members in policy development increases the time needed to develop the policy and may also increase the amount of work involved. However, involvement is worth the extra time and effort, because it results in a policy that has a much higher level of acceptance and is more likely to be observed.
The process of policy development is described in detail in the SSTA publication Policy Leadership. This publication on student evaluation policies focuses on the components that could appropriately be included in a student evaluation policy.
Each of these components of a student evaluation policy is discussed
in more detail in the sections that follow.
Back to Table of Contents
The statement of philosophy or rationale that begins a student evaluation
policy gives the reasons for the policy. For example, it might state
that the board of education:
The policy statement is a short, one- to three-sentence statement of what the board intends to do. For example:
Most school boards contain reference to the legislation under which the policy was established. Section 17.8(1) of The Education Act, 1995 says:
Most school division policies on student evaluation list a few fundamental principles of fair student evaluation. The policies make it clear that these principles underlie all student evaluation activities that take place in the division.
The best source for fundamental principles of student evaluation is The Principles for Fair Student Assessment Practices for Education in Canada which appear in Appendix A. These principles were developed by a group comprised of several major Canadian educational organizations and are widely accepted across Canada. However, the full set of principles is lengthy and complex. Most school divisions use The Principles for Fair Student Assessment Practices for Education in Canada as a guide in the development of a few key principles.
Key principles identified in Saskatoon (West) School Division’s policy
on student evaluation appear below.
| Goals | Objectives |
| 1. A wide range of evaluation methods will be used. | 1. Formal methods will be used to assess student progress. Examples
of formal methods are teacher-developed tests and examinations, commercially-developed
evaluation instruments and standardized tests.
2. Informal methods will be used to assess student progress. Examples of informal methods are teacher observation, anecdotal records, performance appraisals, peer appraisal, and self-appraisal. 3. Professional development opportunities and support will be provided to teachers to assist in expanding and enhancing evaluation methods. 4. Standardized tests will be used in accordance with divisional policy. (Policy to be developed.) |
| 2. The purpose of evaluation is to promote student growth and development in the cognitive, affective, and psychomotor domains. | 1. Teachers will incorporate evaluative activities, which will assess
affective and psychomotor objectives as well as cognitive objectives.
2. Student growth in the cognitive, affective, and psychomotor domains should be reported to students and parents. 3. Evaluation is primarily formative in elementary and middle years. 4. A list of the major competencies will be developed at each grade level in each area of learning, as resources are available. 5. Retention at the K-8 level will occur only as part of a carefully designed development plan for an individual student. (Promotion and retention policy to be developed.) |
| 3. Evaluation practices will be fair and equitable. | 1. Teachers will provide evaluation plans to the students/parents as
part of the course outline.
2. Students must know on what basis and when they will be evaluated. 3. Inservice will be provided. Teachers will be familiarized with the principles and practices of fair student assessment. 4. Parents and students will be made aware of the divisional appeals policy (to be developed). 5. Each school will develop a program to annually monitor their evaluation plan and principles. |
| 4. The results of evaluation will be regularly communicated to students and parents. | 1. Mechanisms for reporting student programs will be developed which
are appropriate for the needs of the various school populations.
2. Student progress will be formally reported to parents at least twice per semester at the high school level and at least four times per year at other levels. The report can be a documented conference, a report card, or both. 3. School staffs will work towards increasing the informal reporting process. 4. Students should be active participants in the reporting process. 5. A committee will be formed to study alternatives to the existing grading structure and current report cards. 6. Any changes in the grading structure or report cards will be clearly communicated to parents and students. 7. School staffs will develop a plan to communicate the results of all major tests, assignments, etc. to parents. |
| 5. Evaluation will be continuous and carefully planned to reflect instructional techniques and curricular objectives. | 1. Teachers and administrators will seek to match evaluation devices
to instructional techniques.
2. Final comprehensive examinations based on at least 50 percent of the course content will be utilized in Grades 9 to 12, unless precluded by subject contents. The mark for the comprehensive examination should not represent more than 40 percent of the final mark. 3. Evaluation devices should be weighted to reflect the relative importance of the curricular objectives. |
| 6. Evaluation will be a major focus of professional development activities at the divisional level and school levels for 3-5 years, beginning in the 1993-94 school year. | 1. Inservice sessions will be provided to ensure all teachers know
and understand the divisional evaluation policy.
2. Additional inservices will be provided to build on and enhance the strengths of teachers in the area of evaluation. 3. Opportunities will be provided for teachers to collaboratively plan and implement new evaluation devices and/or strategies. |
The specific issues addressed in the student evaluation policy will
vary from one school division to another depending on local circumstances.
Many student evaluation policies include guidelines for:
Each of these topics is discussed in more detail in the sections that
follow.
Back to Table of Contents
Some Saskatchewan school divisions are beginning to establish benchmarks
and standards to supplement those set through Saskatchewan’s Provincial
Learning Assessment Program (PLAP) and the national School Achievement
Indicators Program (SAIP).
Board policy usually describes the processes that will be used to establish benchmarks and standards, rather than specifying the benchmarks and standards themselves, because these can change from year to year.
The process that is frequently used to establish benchmarks involves:
A complete description of this process appears in Appendix B.
In addition to being an evaluation tool, benchmarking can play an important role in improving student learning.
Benchmarking involves developing written descriptions of student work at various levels (rubrics) and collecting samples of student work at each of the levels (exemplars). The rubrics and exemplars provide very clear examples of the criteria that students must meet in order to get a particular mark. When these rubrics and exemplars are distributed to students and teachers, students can compare their work and identify changes and improvements they need to make in order to reach a certain level.
The benchmarking process involves training teachers to score student work based on rubrics and exemplars rather than on criteria developed by individual teachers. The objective is for all teachers to score the same piece of work in the same way. This process promotes greater fairness and consistency, because it is based on accepted external standards rather than on teachers’ individualistic criteria.
In recent years, use of standardized tests has become a somewhat controversial
topic. Some groups are strong advocates of their use and others are
strongly opposed. Establishing a policy on the use of standardized
tests will help ensure they are not misused and will provide useful information
to the public. Such a policy often includes:
It is sometimes believed that requiring an elementary student who is not doing well to repeat the grade will produce achievement in later grades. In fact, there is a huge body of research that shows that the opposite is usually true.(17) Children who are retained in grade, usually don’t do as well later on as children with comparable achievement who are promoted. The sense of being a loser and of being incapable of learning that comes with school failure often stays with a child for the rest of his or her school career and leads to continuing low performance - an example of the self-fulfilling prophecy in action.
School board policies on retention in grade often contain the following
guidelines:
Student retention in grade is less of an issue at the high school level than the elementary and middle years level. In high school, promotion is on a subject by subject basis and is often determined strictly by marks.
The two sides to this issue are:
Generally, it is preferable to return students’ tests because this promotes greater learning.
Many student evaluation policies contain a statement about the return of completed tests and specify the circumstances under which tests must be returned to the students and the circumstances under which they may be kept by the teacher.
Sometimes students are “recommended”, which means they pass a course
without having to take the final exam. It may be appropriate to include
a statement about recommendations in a student evaluation policy.
This statement might respond to questions such as:
Sometimes teachers give bonus marks on exams and routine assignments.
Bonus marks are typically given for factors such as student attitude, student
effort, completing work on time and/or neatness of finished work.
Bonus marks mean that some students can get a mark in excess of 100 percent.
It may be appropriate to include a statement about bonus marks in a student
evaluation policy. This statement might respond to questions such
as:
Most school division student evaluation policies give students the right
to appeal the grade received on a final exam. The policy may contain
provisions such as the following:
School division student evaluation policies often address both informal and formal reporting to parents.
Informal Reporting
Informal reporting can include telephone calls to parents and notes sent via e-mail or regular mail. It is extremely important that informal reporting focus on students’ strengths and achievements as well as on their problems. Otherwise, parents will come to dread every phone call from school.
Formal Reporting
The policy on formal reporting can include items such as:
The way that school divisions and individual schools handle parent/teacher/student
conferences varies greatly. It may be appropriate to include a policy
statement that responds to questions such as:
Most school division student evaluation policies contain guidelines
concerning missed tests and exams at the high school level. These
guidelines typically address:
Most school divisions will need to address the issue of access to and ownership of student records.
At one time students’ records were held confidential by the school,
and students and parents were not permitted to see them. This is
no longer the case. With increasing openness in the education system,
students and parents now usually have access to all school records relating
to the specific student. In their policy on student evaluation, school
divisions may wish to respond to questions such as the following:
This section of the student evaluation policy can address topics such
as:
Using a Variety of Sources for Evaluation Data
A single data source gives a limited view of student achievement.
Evaluating a student and/or assigning a mark on the basis of a single test,
exam or other data source is bad evaluation practice and very unfair to
the student. A comprehensive picture of student achievement in relation
to course objectives is best obtained by using several different data sources.
These can include:
The Role of Achievement and Effort
When expectations and objectives are the same for all students, an average
student may need to work very hard to achieve top-level performance, while
a highly gifted student may put in little effort and still achieve the
same level of performance. Some student evaluation policies contain
statements in response to questions such as:
Marks for Attitude and Behaviour
Sometimes, teachers also consider attitude and behaviour when assigning marks. A student who demonstrates good achievement, but is troublesome in the classroom will get a lower mark. A student whose achievement is low will still get a good mark because s/he is obedient and well behaved. This is not a good idea. It is best to consider achievement and behaviour separately and to report on them separately. The student evaluation policy might clearly state that achievement and attitude/behaviour are evaluated separately.
Attendance
Another issue to consider in the policy on assigning marks is attendance. Some teachers lower a student’s mark if the student has many absences regardless of the student’s level of achievement. Generally, it is best to consider attendance and achievement separately and to report upon them separately.
The Role of Achievement and Growth
Some teachers give marks for achievement only, others focus primarily
on student growth. For example, a student who performs very well
but shows little growth over the year might get a lower final mark than
a student whose performance is low but shows significant growth over the
year. A student evaluation policy might contain a statement about
the relative importance of achievement and growth when assigning marks.
Some school divisions prefer to evaluate these two elements separately.
Back to Table of Contents
A policy of student evaluation will only fulfill its objectives (promoting student learning, ensuring fairness and consistency for all students, and preventing problems and issues from arising) if the guidelines it contains are well known to all students, teachers and parents.
Thus, many student evaluation policies contain guidelines for making
the existence of the entire policy known and for familiarizing students,
teachers and parents with guidelines for specific issues such as missed
exams and appeals of exams. Some ideas for doing this include:
Assessment practice is broadly defined in the Principles as the
process of collecting and interpreting information that can be used:
(i) to provide feedback to students, and to their parents/guardians where
applicable, about the progress they are making toward attaining the knowledge,
skills, attitudes, and behaviours to be learned or acquired, and (ii) to
inform the various educational decisions (instructional, diagnostic, placement,
promotion, graduation, curriculum planning, program development, policy)
that are made with reference to students. Principles and related
guidelines are set out for both developers and users of assessments.
Developers include people who construct assessment methods and people who
set policies for particular assessment programs. Users include people
who select and administer assessment methods, commission assessment development
services, or make decision on the basis of assessment results and findings.
The roles may overlap, as when a teacher or instructor develops and administers
an assessment instrument and then scores and interprets the students’ responses,
or when a ministry or department of education or local school system commissions
the development and implementation of an assessment program and scoring
services and makes decisions on the basis of the assessment results.
Financial support for the development and dissemination of the Principles was provided principally by the Walter and Duncan Gordon Charitable Foundation, with additional support provided by various Faculties, Institutes, and Colleges of Education and Provincial and Territorial Ministries and Departments of Education in Canada. This support is gratefully acknowledged.
The Joint Advisory Committee invites users to share their experiences in working with the Principles and to submit any suggestions that could be used to revise and improve the Principles. Comments and suggestions should be sent to the Joint Advisory Committee at the address shown below.
The Principles for Fair Student Assessment Practices for Education in Canada is not copyrighted. Reproduction and dissemination are encouraged. Principles reproduced in this document with permission. Please cite the Principles as follows:
Principles for Fair Student Assessment Practices for Education
in Canada. (1993). Edmonton, Alberta: Joint Advisory
Committee (Mailing address: Joint Advisory Committee, Centre for
Research in Applied Measurement and Evaluation, 3-104 Education Building
North, University of Alberta, Edmonton, Alberta, T6G 2G5).
Organization and Use of the Principles
The Principles and their related guidelines are organized in
two parts. Part A is directed at assessments carried out by teachers
at the elementary and secondary school levels. Part A is also applicable
at the post-secondary level with some modifications, particularly with
respect to whom assessment results are reported. Part B is directed
at standardized assessments developed external to the classroom by commercial
test publishers, provincial and territorial ministries and departments
of education, and local school jurisdictions.1
The Joint Advisory Committee recognizes that in the field of assessment
some terms are defined or used differently by different groups of people.
To maintain as much consistency in terminology as possible, an attempt
has been made to employ generic terms in the Principles.
Part A: Classroom Assessments
Part A is directed toward the development and selection of assessment methods and their use in the classroom by teachers. It is organized around five interrelated themes:
I. Developing and Choosing Methods for Assessment
II. Collecting Assessment Information
III. Judging and Scoring Student Performance
IV. Summarizing and Interpreting Results
V. Reporting Assessment Findings
| Principle | Commentary |
| 1. Assessment methods should be developed or chosen so that inferences drawn about the knowledge, skills, attitudes, and behaviors possessed by each student are valid and not open to misinterpretation. | Validity refers to the degree to which inferences drawn from assessments results are meaningful. Therefore, development or selection of assessment methods for collecting information should be clearly linked to the purposes for which inferences and decisions are to be made. For example, to monitor the progress of students as proofreaders and editors of their own work, it is better to assign an actual writing task, to allow time and resources for editing (dictionaries, handbooks, etc.) and to observe students for evidence of proofreading and editing skill as they work than to use a test containing discrete items on usage and grammar that are relatively devoid of context. |
| 2. Assessment methods should be clearly related to the goals and objectives of instruction, and be compatible with the instructional approaches used. | To enhance validity, assessment methods should be in harmony with the instructional objectives to which they are referenced. Planning an assessment design at the same time as planning instruction will help integrate the two in meaningful ways. Such joint planning provides an overall perspective on the knowledge, skills, attitudes, and behaviors to be learned and assessed, and the contexts in which they will be learned and assessed. |
| 3. When developing or choosing assessment methods, consideration should be given to the consequences of the decisions to be made in light of the obtained information. | The outcomes of some assessments may be more critical than others. For example, misinterpretation of the level of performance on an end-of-unit test may result in incorrectly holding a student form proceeding to the next to the next instructional unit in a continuous progress situation. In such "high-stake" situations, every effort should be made to ensure the assessment method will yield consistent and valid results. "Low stake" situations, such as determining if a student has correctly completed an in-class assignment, can be less stringent. Low stake assessments are often repeated during the course of a reporting period using a variety of methods. If the results are aggregated to form a summary comment or grade, the summary will have greater consistency and validity than its component elements. |
| 4. More than one assessment method should be used to ensure comprehensive and consistent indications of student performance. | To obtain a more complete picture or profile of a student's knowledge, skills, attitudes, or behaviors, and to discern consistent patterns and trends, more than one assessment method should be used. Student knowledge might be assessed using completion items; process or reasoning skills might be assessed by observing performance on a relevant task; evaluation skills might be assessed by reflecting upon the discussion with a student about what materials to include in a portfolio. Self-assessment may help to clarify and add meaning to the assessment of a written communication, science project, piece of artwork, or an attitude. Use of more than one method will also help minimize inconsistency brought about by different sources of measurement error (for example, poor performance because of an "off-day"; lack of agreement among items included in a test, rating scale, or questionnaire; lack of agreement among observers; instability across time). |
| 5. Assessment methods should be suited to the backgrounds and prior experiences of students. | Assessment methods should be free from bias brought about by student factors extraneous to the purpose of the assessment. Possible factors to consider include culture, developmental stage, ethnicity, gender, socio-economic background, language, special interests, and special needs. Students' success in answering questions on a test or in an oral quiz, for example, should not be dependent upon prior cultural knowledge, such as understanding an allusion to a cultural tradition or value, unless such knowledge falls within the content domain being assessed. All students should be given the same opportunity to display their strengths. |
| 6. Content and language that would generally be viewed as sensitive, sexist, or offensive should be avoided. | The vocabulary and problem situation in each test item or performance task should not favour or discriminate against any group of students. Steps should be taken to ensure that stereotyping is not condoned. Language that might be offensive to particular groups of students should be avoided. A judicious use of different roles for males and females and for minorities and the careful use of language should contribute to more effective and, therefore, fairer assessments. |
| 7. Assessment instruments translated into a second language or transferred from another context or location should be accompanied by evidence that inferences based on these instruments are valid for the intended purpose. | Translation of an assessment instrument from one language to another is a complex and demanding task. Similarly, the adoption or modification of an instrument developed in another country is often not simple and straightforward. Care must be taken to ensure that the results from translated and imported instruments are not misinterpreted or misleading. |
| Principle | Commentary |
| 1. Students should be told why assessment information is being collected and how this information will be used. | Students who know the purpose of an assessment are in a position to respond in a manner that will provide information relevant to that purpose. For example, if students know that their participation in a group activity is to be used to assess cooperative skills, they can be encouraged to contribute to the activity. If students know that the purpose of an assessment is to diagnose strengths and weaknesses rather than to assign a grade, they can be encouraged to reveal weaknesses as well as strengths. If the students know that the purpose is to assign a grade, they are well advised to respond in a way that will maximize strengths. This is especially true for assessment methods that allow students to make choices, such as with optional writing assignments or research projects. |
| 2. An assessment procedure should be used under conditions suitable to its purpose and form. | Optimum conditions should be provided for obtaining data from and information about students so as to maximize the validity and consistency of the data and information collected. Common conditions include such things as proper light and ventilation, comfortable room temperature, and freedom from distraction (e.g. movement in and out of the room, noise). Adequate workspace, sufficient materials, and adequate time limits appropriate to the purpose and form of the assessment are also necessary. For example, if the intent is to assess student participation in a small group, adequate work space should be provided for each student group, with sufficient space between subgroups so that the groups do not interfere with or otherwise influence one another and so that the teacher has the same opportunity to observe and assess each student within each group. |
| 3. In assessments involving observations, checklists, or rating scales, the number of characteristics to be assessed at one time should be small enough and concretely described so that the observations can be made accurately. | Student behaviors often change so rapidly that it may not be possible simultaneously to observe and record all the behavior components. In such instances, the number of components to be observed should be reduced and the components should be described as concretely as possible. One way to manage an observation is to divide the behavior into a series of components and assess each component in sequence. By limiting the number of components assessed at one time, the data and information become more focused, and time is not spent observing later behaviour until prerequisite behaviours are achieved. |
| 4. The directions provided to students should be clear, complete, and appropriate for the ability, age and grade level of the students. | Lack of understanding of the assessment task may prevent maximum performance or display of the behavior called for. In the case of timed assessments, for example, teachers should describe the time limits, explain how students might distribute their time among parts of those assessment instruments with parts, and describe how students should record their responses. For a portfolio assessment, teachers should describe the criteria to be used to select the materials to be included in a portfolio, who will select these materials, and it more than one person will be involved in the selection process, how the judgments from the different people will be combined. Where appropriate, sample material and practice should be provided to further increase the likelihood that instructions will be understood. |
| 5. In assessments involving selection items (i.e., true/false, multiple-choice), the directions should encourage students to answer all items without threat of penalty. | A correction formula is sometimes used to discourage "guessing" on
selection items. The formula is intended to encourage students to
omit items for which they do not know the answer rather than to "guess"
the
Answer. Because research evidence indicates that the benefits expected from the correction are not realized, the use of the formula is discouraged. Students should be encouraged to use whatever partial knowledge they have when choosing their answers, and to answer all items. |
| 6. When collecting assessment information, interactions with students should be appropriate and consistent. | Care must be taken when collecting assessment information to treat all students fairly. For example, when oral presentations by students are assessed, questioning and probes should be distributed among the students so that all students have the same opportunity to demonstrate their knowledge. While writing a paper-and-pencil test, a student may ask to have an ambiguous item clarified, and, if warranted, the item should be explained to the entire class. |
| 7. Unanticipated circumstances that interfere with the collection of assessment information should be noted and recorded. | Events such as a fire drill, an unscheduled assembly, or insufficient materials may interfere in the way in which assessment information is collected. Such events should be recorded and subsequently considered when interpreting the information obtained. |
| 8. A written policy should guide decisions about the use of alternate procedures for collecting assessment information from students with special needs and students whose proficiency in the language of instruction is inadequate for them to respond in the anticipated manner. | It may be necessary to develop alternative assessment procedures to ensure a consistent and valid assessment of those students who, because of special needs or inadequate language, are not able to respond to an assessment method (for example, oral instead of written format, individual instead of group administered, translation into first language, providing additional time). The use of alternate procedures should be guided by a written policy developed by teachers, administrators, and other jurisdictional personnel. |
| Principle | Commentary |
| 1. Before an assessment method is used a procedure for scoring should be prepared to guide the process of judging the quality of a performance or product, the appropriateness of an attitude or behaviour, or the correctness of an answer. | To increase consistency and validity, properly developed scoring procedures should be used. Different assessment methods require different forms of scoring. Scoring selections items (true/false, multiple-choice, matching) requires the identification of the correct or, in some instances, best answer. Guides for scoring essays might include factors such as the major points to be included in the "best answer" or models or exemplars corresponding to different levels of performance at different age levels and against which comparisons can be made. Procedures for judging other performances or products might include specification of the characteristics to be rated in performance terms and, to the extent possible, clear descriptions of the different levels of performance or quality of a product. |
| 2. Before an assessment method is used, students should be told how their responses or the information they provide will be judged or scored. | Informing students prior to the use of an assessment method about the scoring procedures to be followed should help ensure that similar expectations are held by both students and their teachers. |
| 3. Care should be taken to ensure that results are not influenced by factors that are not relevant to the purpose of the assessment. | Various types of errors occur in scoring, particularly when a degree of subjectivity is involved (e.g., marking essays, rating a performance, judging a debate). For example, if the intent of a written communication is to assess content alone, the scoring should not be influenced by stylistic factors such as vocabulary and sentence structure. Personal bias errors are indicated by a general tendency to rate all students in approximately the same way (e.g., too generously or too severely). Halo effects can occur when a rater's general impression of a student influences the rating of individual characteristics or when a previous rating influences a subsequent rating. Pooled results from two or more independent raters (teachers, other students) will generally produce a more consistent description of student performance than a result obtained from a single rater. In combining results, the personal biases of individual raters tend to cancel one another. |
| 4. Comments formed as part of scoring should be based on the responses made by the students and presented in a way that students can understand and use them. | Comments, in oral and written form, are provided to encourage learning and to point out correctable errors or inconsistencies in performance. In addition, comments can be used to clarify a result. Such feedback should be based on evidence pertinent to the learning outcomes being assessed. |
| 5. Any changes made during scoring should be based upon a demonstrated problem with the initial scoring procedure. The modified procedure should then be used to restore all previously scored responses. | Anticipating the full range of student responses is a difficult task for several forms of assessment. There is always the danger that unanticipated responses or incidents that are relevant to the purposes of the assessment may be overlooked. Consequently, scoring should be continuously monitored for unanticipated responses and these responses should be taken into proper account. |
| 6. An appeal process should be described to students at the beginning of each school year or course of instruction that they may use to appeal a result. | Situations may arise where a student believes a result incorrectly reflects his/her level of performance. A procedure by which students can appeal such a situation should be developed and made known to them. This procedure might include, for example, checking for addition or other recording errors or, perhaps, judging or scoring by a second qualified person. |
| Principle | Commentary |
| 1. Procedures for summarizing and interpreting results for a reporting period should be guided by a written policy. | Summary comments and grades, when interpreted, serve a variety of functions. They inform students of their progress. Parents, teachers, counsellors, and administrators use them to guide learning, determine promotion, identify students for special attention (e.g., honours, remediation), and to help students develop future plans. Comments and grades also provide a basis for reporting to other schools in the case of school transfer and, in the case of senior high school students, post-secondary institutions and prospective employers. They are more likely to serve their many functions and those functions are less likely to be confused if they are guided by a written rationale or policy sensitive to these different needs. This policy should be developed by teachers, school administrators, and other jurisdictional personnel in consultation with representatives of the audiences entitled to receive a report of summary comments and grades. |
| 2. The way in which summary comments and grades are formulated and interpreted should be explained to students and their parents/guardians. | Students and their parents/guardians have the "right-to-know" how student performance is summarized and interpreted. With this information, they can make constructive use of the findings and fully review the assessment procedures followed. It should be noted that some aspects of summarizing and interpreting are based upon a teacher's best judgment of what is good or appropriate. This judgment is derived from training and experience and may be difficult to describe specifically in advance. In such circumstances, examples might be used to show how summary comments and grades were formulated and interpreted. |
| 3. The individual results used and the process followed in deriving summary comments and grades should be described in sufficient detail so that the meaning of a summary comment or grade is clear. | Summary comments and grades are best interrupted in the light of an adequate description of the results upon which they are based, the relative emphasis given to each result, and the process followed to combine the results. Many assessments conducted during a reporting period are of a formative nature. The intent of these assessments (e.g., informal observations, quizzes, text-and-curriculum embedded questions, oral questioning) is to inform decisions regarding daily learning, and to inform or otherwise refine the instructional sequence. Other assessments are of a summative nature. It is the summative assessments that should be considered when formulating and interpreting summary comments and grades for the reporting period. |
| 4. Combine disparate kinds of results into a single summary should be done cautiously. To the extent possible, achievement, effort, participation, and other behaviors should be graded separately. | A single comment or grade cannot adequately serve all functions. For example, letter grades used to summarize achievement are most meaningful when they represent only achievement. When they include other aspects of student performance such as effort, amount (as opposed to quality) of work completed, neatness, class participation, personal conduct, or punctuality, not only do they lose their meaningfulness as a measure of achievement, but they also suppress information concerning other important aspects of learning and invite inequities. Thus, to more adequately and fairly summarize the different aspects of student performance, letter grades for achievement might be complemented with alternate summary forms (e.g. checklists, written comments) suitable for summarizing results related to these other behaviours. |
| 5. Summary comments and grades should be based on more than one assessment result so as to ensure adequate sampling of broadly defined learning outcomes. | More than one or two assessments are needed to adequately assess performance in multi-facet areas such as Reading. Under-representation of such broadly defined constructs can be avoided by ensuring that the comments and grades used to summarize performance are based on multiple assessments, each referenced to a particular facet of the construct. |
| 6. The results used to produce summary comments and grades should be combined in a way that ensures that each result receives its intended emphasis or weight. | When the results of a series of assessments are combined into a summary
comment, care should be taken to ensure that the actual emphasis placed
on the various results matches the intended emphasis for each student.
When numerical results are combined, attention should be paid to differences in the variability, or spread, of the different sets of results and appropriate account taken where such differences exist. If, for example, a grade is to be formed from a series of paper-and-pencil tests, and if each test is to count equally in the grade, then the variability of each set of scores must be the same. |
| 7. The basis for interpretation should be carefully described and justified. | Interpretation of the information gathered for a reporting period for a student is a complex and, at times, controversial issue. Such information, whether written or numerical, will be of little interest or use if it is not interpreted against some pertinent and defensible idea of what is good and what is poor. The frame of reference used for interpretation should be in accord with the type of decision to be made. Typical frames of reference are performance in relation to pre-specified standards, performance in relation to peers, performance in relation to aptitude or expected growth, and performance in terms of the amount of improvement or amount learned. If, for example, decisions are to be made as to whether or not a student is ready to move to the next unit in an instructional sequence, interpretations based on pre-specified standards would be most relevant. |
| 8. Interpretations of assessment results should take account of the backgrounds and learning experiences of the students. | Assessment results should be interpreted in relation to a student's personal and social context. Among the factors to consider are age, ability, gender, language, motivation, opportunity to learn, self-esteem, socio-economic backgrounds, special interests, special needs, and "test-taking" skills. Motivation to do school tasks, language capability, or home environment can influence learning of the concepts assessed, for example. Poor reading ability, poorly developed psychomotor or manipulative skills, lack of test-taking skills, anxiety, and low self-esteem can lead to lower scores. Poor performance in an assessment may be attributable to a lack of opportunity to learn because required learning materials and supplies were not available, learning activities were not provided, or inadequate time was allowed for learning. When a student performs poorly, the possibility that one or more factors such as these might have interfered with a student's response or performance should be considered. |
| 9. Assessment results that will be combined into summary comments and grades should be stored in a way that ensures their accuracy at the time they are summarized and interpreted. | Comments and grades and their interpretations, formulated form a series of related assessments, can be no better than the data and information upon which they are based. Systematic data control minimizes errors, which would otherwise be introduced into a student's record or information base, and provides protection of confidentiality. |
| 10. Interpretations of assessment results should be made with due regard for limitations in the assessment methods used, problems encountered in collecting the information and judging or scoring it, and limitations in the basis used for interpretation. | To be valid, interpretations must be based on results determined from assessment methods that are relevant and representative of the performance assessed. Administrative constraints, the presence of measurement error, and the limitations of the frames of reference used for interpretation also need to be accounted for. |
| Principle | Commentary |
| 1. The reporting system for a school or jurisdiction should be guided by a written policy. Elements to consider include such aspects as audiences, medium, format, content, level of detail, frequency, timing, and confidentiality. | The policy to guide the preparation of school reports (e.g., reports of separate assessments; reports for a reporting period) should be developed by teachers, school administrators, and other jurisdictional personnel in consultation with representatives of the audiences entitled to receive a report. Cooperative participation not only leads to more adequate and helpful reporting, but also increases the likelihood that the reports will be understood and used by reporting, but also increases the likelihood that the reports will be understood and used by those for whom they are intended. |
| 2. Written and oral reports should contain a description of the goals and objectives of instruction to which the assessments are referenced. | The goals and objectives that guided instruction should serve as the basis for reporting. A report will be limited by a number of practical considerations, but the central focus should be on the instructional objectives and the types of performance that represent achievement of these objectives. |
| 3. Reports should be complete in their descriptions of strengths and weaknesses of students, so that strengths can be built upon and problem areas addressed. | Reports can be incorrectly slanted towards "faults" in a student or toward giving unqualified praise. Both biases reduce the validity and utility of assessment. Accuracy in reporting strengths and weaknesses helps to reduce systematic error and is essential for stimulating and reinforcing improved performance. Reports should contain the information that will assist and guide students, their parents/guardians, and teachers to take relevant follow-up actions. |
| 4. The reporting system should provide for conferences between teachers and parents/guardians. Whenever it is appropriate, students should participate in these conferences. | Conferences scheduled at regular intervals and, if necessary, upon request provide parents/guardians and, when appropriate, students with an opportunity to discuss assessment procedures, clarify and elaborate their understanding of the assessment results, summary comments and grades, and reports, and, where warranted, to work with teachers to develop relevant follow-up activities or action plans. |
| 5. An appeal process should be described to students and their parents/guardians at the beginning of each school year or course of instruction that they may use to appeal a report. | Situations may arise where a student and his/her parents/guardian believe the summary comments and grades inaccurately reflect the level of performance of the student. A procedure by which they can appeal such a situation should be developed and made known to them (for example, in a school handbook or newsletter provided to students and their parents/guardians at the beginning of the school year). |
| 6. Access to assessment information should be governed by a written policy that is consistent with applicable laws and with basic principles of fairness and human rights. | A written policy, developed by teachers, administrators, and other jurisdictional personnel, should be used to guide decisions regarding the release of student assessment information. Assessment information should be available to those people to whom it applies ? students and their parents/guardians, and to teachers and other educational personnel obligated by profession to use the information constructively on behalf of students. In addition, assessment information might be made available to others who justify their need for the information (e.g., post-secondary institutions, potential employers, researchers). Issues of informed consent should also be addressed in this policy. |
| 7. Transfer of assessment information from one school to another should be guided by a written policy with stringent provisions to ensure the maintenance of confidentiality. | To make a student's transition from one school to another as smooth as possible, a clear policy should be prepared indicating the type of information to go with the student and the form in which it will be reported. Such a policy, developed by jurisdictional and ministry personnel, should ensure that the information transferred will be sent by and received by the appropriate person within the "sending" and "receiving" schools respectively. |
Part B applies to the development and use of standardized assessment methods used in student admissions, placement, certification, and educational diagnosis, and in curriculum and program evaluation. These methods are primarily developed by commercial test publishers, ministries and departments of education, and local school systems.
The principles and accompanying guidelines are organized in terms of four areas:
I. Developing and Selecting Methods for Assessment
II. Collecting and Interpreting Assessment Information
III. Informing Students Being Assessed
IV. Implementing Mandated Assessment Programs
The first three areas of Part B are adapted from the Code of Fair
Testing Practices for Education (1988) developed in the United States.
The principles and guidelines as modified in these three sections are intended
to be consistent with the Guidelines for Educational and Psychological
Testing (1986) developed in Canada. The fourth area has been
added to contain guidelines particularly pertinent for mandated educational
assessment and testing programs developed and conducted at the national,
provincial, and local levels.
|
|
|
| Developers of assessment methods should strive to make them as fair as possible for use with students who have different backgrounds or special needs. Developers should provide the information users need to select methods appropriate to their assessment needs. | Users should select assessment methods that have been developed to be as fair as possible for students who have different backgrounds or special needs. Users should select methods that are appropriate for the intended purposes and suitable for the students to be assessed. |
|
|
|
| Developers should: | Users should: |
| 1. Define what the Assessment method is intended to measure and how it is to be used. Describe the characteristics of the students with which the method may be used. | 1. Determine the purpose(s) for assessment and the characteristics of the students to be assessed. Then select an assessment method suited to that purpose and type of student. |
| 2. Warn users against common misuses of the assessment method. | 2. Avoid using assessment methods for purposes not specifically recommended by the developer unless evidence is obtained to support the intended use. |
| 3. Describe the process by which the method was developed. Include a description of the theoretical basis, rationale for selection of content and procedures, and derivation of scores. | 3. Review available assessment methods for relevance of content and appropriateness of scores with reference to the intended purpose(s) and characteristics of the students to be assessed. |
| 4. Provide evidence that the assessment method yields results that satisfy its intended purpose(s). | 4. Read independent evaluations of the methods being considered. Look for evidence supporting the claims of developers with reference to the intended application of each method. |
| 5. Investigate the performance of students with special needs and students from different backgrounds. Report evidence of the consistency and validity of the results produced by the assessment method for these groups. | 5. Ascertain whether the content of the assessment method and the norm group(s) or comparison group(s) or comparison group(s) are appropriate for the students to be assessed. For assessment methods developed in other regions or countries, look for evidence that the characteristics of the norm group(s) or comparison group(s) are comparable to the characteristics of the students to be assessed. |
| 6. Provide potential users with representative samples or complete copies of questions or tasks, directions, answer sheets, score reports, guidelines, guidelines for interpretation, and manuals. | 6. Examine specimen sets, samples or complete copies of assessment instruments, directions, answer sheets, score reports, guidelines for interpretation, and manuals and judge their appropriateness for the intended application. |
| 7. Review printed assessment methods and related materials for content or language generally perceived to be sensitive, offensive, or misleading. | 7. Review printed assessment methods and related materials for content or language that would offend or mislead the students to be assessed. |
| 8. Describe the specialized skills and training needed to administer an assessment method correctly, and the specialized knowledge to make valid interpretations of scores. | 8. Ensure that all individuals who administer the assessment method, score the responses, and interpret the results have the necessary knowledge and skills to perform these tasks (e.g., learning assistance teachers, speech and language pathologists, counsellors, school psychologists, psychologists). |
| 9. Limit sales of restricted assessment materials to persons who possess the necessary qualifications. | 9. Ensure access to restricted assessment materials is limited to persons with the necessary qualifications. |
| 10. Provide for periodic review and revision of content and norms, and, if applicable, passing or cut-off scores, and inform users. | 10. Obtain information about the appropriateness of content, the regency of norms, and, if applicable, the appropriateness of the cut-off scores for use with the students to be assessed. |
| 11. Provide evidence of the comparability of different forms of an instrument where the forms are intended to be interchangeable, such as parallel forms or the adaptation of an instrument for computer administration. | 11. Obtain information about the comparability of interchangeable forms, including computer adaptations. |
| 12. Provide evidence that an assessment method translated into a second language is valid for use with the second language. This information should be provided in the second language. | 12. Obtain evidence about the validity of the use of an assessment method translated into a second language. |
| 13. Advertise an assessment method in a way that states it can be used only for the purposes for which it was intended. | 13. Verify advertising claims made for an assessment method. |
|
|
|
|
|
|
| Developers should provide information to help users administer an assessment method correctly and interpret assessment results accurately. | Users should follow directions for proper administration of an assessment method and interpretation of assessment results. |
|
|
|
| Developers should: | Users should: |
| 1. Provide clear instructions for administering the assessment method and identify the qualifications that should be held by the people who should administer the method. | 1. Ensure that the assessment method is administered by qualified personnel or under the supervision of qualified personnel. |
| 2. When feasible, make available appropriately modified forms of assessment methods for students with special needs or whose proficiency in the original language of administration is inadequate to respond in the anticipated manner. | 2. When necessary and feasible, use appropriately modified forms of
assessment methods with students who have special needs or whose proficiency
in the original language of administration is inadequate to respond in
the anticipated manner.
Ensure that instruments translated from one language to another are administered by persons who are proficient in the translated language. |
| 3. Provide answer keys and describe procedures for scoring when scoring is to be done by the user. | 3. Follow procedures for scoring as set out for the assessment method. |
| 4. Provide score reports or procedures for generating score reports that describe assessment results clearly and accurately. Identify and explain possible misinterpretations of the scores yielded by the scoring system (grade equivalents, percentile ranks, standard scores) used. | 4. Interpret scores taking into account the limitations of the scoring system used. Avoid misinterpreting scores on the basis of unjustified assumptions about the scoring system (grade-equivalents, percentile ranks, standard scores) used. |
| 5. Provide evidence of the effects on assessment results of such factors as speed, test-taking strategies, and attempts by students to present themselves favorably in their responses. | 5. Interpret scores taking into account the effects of such factors as speed, test-taking strategies, and attempts by students to present themselves favorably in their responses. |
| 6. Warn against using published norms when the prescribed assessment method has been modified in any way. | 6. Interpret scores taking account of major differences between the
norm group(s) or comparison group(s) and the students being assessed. Also
take account of discrepancies between recommended and actual procedures
and differences in familiarity with the assessment method between the norm
group(s) and the students being assessed.
Examine the need for local norms, and, if called for, develop these norms. |
| 7. Describe how passing and cut-off scores, where used, were set and provide evidence regarding rates of misclassification. | 7. Explain how passing or cut-off scores were set and discuss the appropriateness
of these scores in terms of rates of misclassification.
Examine the need for local passing or cut-off scores and, if called for, reset these scores. |
| 8. Provide evidence to support the use of any computer scoring or computer generated interpretations. The documentation should include the rationale for such scoring and interpretations and their comparability with the results of scoring and interpretations made by qualified judges. | 8. Ensure that any computer administration and computer interpretations of assessment results are accurate and appropriate for the intended use. If necessary, ensure that relevant information not included in computer reports is also considered. |
| 9. Observe jurisdictional policies regarding storage of and subsequent access to the results. Ensure that computer files are not accessible to unauthorized users. | |
| 10. Ensure that all copyright and user agreements are observed. | |
|
|
|
1. Develop materials and procedures for informing the students being assessed about the content of the assessment, types of question formats used, and appropriate strategies, if any, for responding.
2. Obtain informed consent from students or, where applicable, their parents/guardians in the case of individual assessments to be used for identification or placement purposes.
3. Provide students or, where applicable, their parents/guardians with information to help them decide whether to participate in the assessment when participation is optional.
4. Provide information to students or, where applicable, their parents/guardians
of alternate assessment methods where available and applicable.
1. Provide students or, where applicable, their parents/guardians with information as to their rights to copies of instruments and completed answer forms, to reassessment, to rescoring, or to cancellation of scores and other records.
2. Inform students or, where applicable, their parents/guardians of the length of time assessment results will be kept on file and of the circumstances under which the assessment results will be released and to whom.
3. Describe the procedures that students or, where applicable, their
parents/guardians may follow to register concerns about the assessment
and endeavor to have problems resolved.
1. Inform all persons with a stake in the assessment (administrators, teachers, students, parents/guardians) of the purpose(s) of the assessment, the uses to be made of the results, and who has access to the results.
2. Design and describe procedures for developing or choosing the methods of assessment, selecting students where sampling is used, administering the assessment materials, and scoring and summarizing student responses.
3. Interpret results in light of factors that might influence them. Important factors to consider include characteristics of the students, opportunity to learn, and comprehensiveness and representatives of the assessment method in terms of the learning outcomes to be reported on.
4. Specify procedures for reporting, storing, controlling access to, and destroying results.
5. Provide reports and explanations of results that can be readily understood by the intended audience(s). If necessary, employ multiple reports designed for different audiences.
References
Code of Fair Testing Practices for Education. (1988). Washington, DC: Joint Committee on Testing Practices.
Guidelines for Educational and Psychological Testing. (1986). Ottawa, ON: Canadian Psychological Association.
Standards for Teacher Competence in Educational Assessment
of Students. (1990). Washington, DC: American Federation
of Teachers, National Council on Measurement in Education, and National
Educational Association.
| Marvin Betts
Gary Broker Clement Dassa (WG) Dick Dodds Tom Dunn Bob Gilchrist Nicholas Head Douglas Hodgkinson Barbara Holmes (WG) |
Michael Jackson
Michel Laurier (WG) Tom Maguire (WG) Romulo Magsino Linda McAlpine Allan McDonald Stirling McDowell Craig Melvin Kathy Oberle (WG) Frank Oliva |
Jean Pettifor
Sharon Robertson Don Saklofske Marvin Simner Marielle Simon (WG) Ross Traub (WG) Sue Wagner Kim Wolff Todd Rogers (Chair, Working Group and Joint Advisory Committee) |
There are large, well organized programs to set standards at the provincial and national levels. Provincial-level benchmarks and standards are set through Saskatchewan Education’s Provincial Learning Assessment Program (PLAP). National-level benchmarks and standards are set through the Council of Ministers of Education, Canada, School Achievement Indicators Program (SAIP).
School divisions may also wish to establish benchmarks and set standards.
A benchmark is a “snapshot” of student achievement at a particular level
at a particular moment in time. Benchmarks allow comparison of student
achievement from one year to the next within the school division.
They also allow school divisions to compare their students’ achievement
with provincial and national achievement. Standards describe what
student achievement should be in a particular skill area at a particular
grade. Comparing students’ achievement to the standard tells us whether
students are achieving above or below expectations.
Part I - Establishing Benchmarks
Several Saskatchewan school divisions are establishing benchmarks at a variety of grade levels and subject areas. The description of the benchmarking process that follows is adapted from one developed by Regina Public School Division for its Writing Benchmarks project. However, the benchmarking process can be used for any subject area at any grade level(s).
The benchmarking process produces detailed descriptions of student performance at several levels of achievement and allows school divisions to determine the percentage of students who achieve at various levels.
The benchmarking process has five steps:
1. Collect samples of students’ work;
2. Determine the range of students’ work;
3. Train teacher-scorers;
4. Score students’ work; and,
5. Report the results.
Each of these steps in the benchmarking process is described in more detail below.
Benchmarking - Step 1: Collect samples of students’ work
The nature of the work collected will vary with the subject area and the grade level. It is important that the same math question, science experiment or writing prompt be used with all students participating in the benchmarking activity, so that student performance can be fairly assessed.
Below is a description of the process that was used by Regina Public
Schools to collect samples of students’ writing.
|
Alfred was looking out the window while his parents were preparing supper. A gust of wind started to shake the house, making the bedroom curtains flap. Alfred began to shiver. Suddenly, all the lights in the house went out… Source: Nicholson, L. L. (1989). Test de rendement. Immersion Journal, 13(1), 7-13. |
|
Before you hand in your story, please answer the following questions:
|
Benchmarking - Step 2: Determine the range of students’ work
Range finding is the process of developing descriptions of the characteristics
of several levels of student work (rubrics) and then locating examples
that illustrate student work at each level (exemplars). The specific
steps are:
Each of these steps is described in the sections that follow:
Develop a Rubric
The rubric describes the characteristics of student achievement at several
(usually 4 or 5) achievement levels. The rubric makes it possible
for scorers to assess student work against objective descriptions of work
at various levels, rather than comparing students to each other.
Rubrics for informational or persuasive writing, and for informational
or narrative writing appear on the next two pages.
|
Upper-half papers make clear a definite purpose, pursued with varying degrees of effectiveness. They also have a structure that shows evidence of some deliberate planning. The writer’s control of the conventions of standard written English (spelling, punctuation, grammar, word choice and sentence structure) ranges from fairly reliable at 4 to confident and accomplished at 6. The 6 paper offers sophisticated ideas within an organizational framework that is clear and appropriate for the topic. The supporting statements are particularly effective because of their sense of voice, substance, specificity, or illustrative quality. The writing is vivid and precise, although it may contain an occasional flaw in the conventions of standard English. The 5 paper is clearly organized with effective support for each of the writer’s major points. While the writing offers substantive ideas, it lacks the fluency and clarity of voice found in the 6 paper. Although there are some errors, the conventions of standard English are consistently under control. The 4 paper shows evidence of the writer’s organizational plan, even when listing is evident. Support, though adequate, tends to be less extensive or effective than that found in the 5 paper. The writer generally observes the conventions of standard English. The errors that are present are not severe enough to interfere significantly with the writer’s main purpose. Lower-half papers either fail to convey a purpose sufficiently or lack one entirely. Consequently, their structure ranges from rudimentary at 3, to random at 2, to absent at 1. Control of the conventions of standard written English tends to follow this same gradient. The 3 paper usually shows some evidence of planning, although the development may be insufficient The supporting statements may be limited to a listing or a repetition of ideas, often without a consistent plan or with poor transitions. The 3 paper often demonstrates repeated weaknesses in the conventions of standard English. The 2 paper is characterized by a marked lack of organization or inadequate support for ideas. The development may be superficial or unfocused. Voice is confused or unclear. Errors in the conventions of standard English may seriously interfere with the overall effectiveness of this paper. The 1 paper lacks purpose or development. The dominant feature is the absence of control of structure or the conventions of standard English. Little effort has been invested in addressing the topic. The deficiencies are so severe that the writer’s ideas are difficult or impossible to understand. An asterisk code is reserved for papers that are blank, illegible, or written on a topic other than the one assigned. Because these papers cannot be scored, a Writing Skills Test composite score cannot be reported. |
|
Papers will show some or all of the following characteristics: Upper-half papers make clear a definite purpose, pursued with varying degrees of effectiveness. They also have a structure that shows evidence of some deliberate planning. The writer’s control of the conventions of standard written English (spelling, punctuation, grammar, word choice and sentence structure) ranges from fairly reliable at 4 to confident and accomplished at 6. The 6 paper offers sophisticated ideas within an organizational framework that is clear and appropriate for the genre of story. Interesting and effective use is made of point of view, dialogue, or other narrative techniques. Description is vivid and precise, creating a sense of voice, a mood or atmosphere. Both language and storyline are fluent. Particular conventions of standard English may not yet be mastered, but in general the use of conventions is correct. In no case, however, would the few errors impede understanding. The 5 paper is clearly organized with effective support for each of the writer’s major points. The storyline is developed with substantive ideas, though it lacks the fluency and clarity of voice found in the 5 paper. It uses dialogue and descriptive language much more extensively and effectively than a 4 paper does. Although there are some errors, the conventions of standard English are consistently under control. The 4 paper shows evidence of the writer’s organizational plan, even when listing of events is evident. The plot line is adequate but mundane and lacks the extensive or effective support that is found in the 5 paper. Dialogue, while present, is not incorporated into the descriptive language. Simple modifiers such as adjectives and adverbs are used, and there may be simple embedding of subordinate clauses. The writer generally observes the conventions of standard English. The errors that are present are not severe enough to interfere significantly with the writer’s main purpose. Lower-half papers either fail to convey a purpose sufficiently or lack one entirely. Consequently, their structure ranges from rudimentary at 3, to random at 2, to absent at 1. Control of the conventions of standard written English tends to follow this same gradient. The 3 paper usually shows some evidence of planning, although the development may be insufficient The storyline exists, but may be vague, and consists of a listing of events with little logical connections. The connections between ideas are basic and strung together with then and so. The 3 paper often demonstrates repeated weaknesses in the conventions of standard English. The 2 paper is characterized by a marked lack of organization or inadequate support for ideas. There may be a thread of a storyline, but not much more. The development may be superficial or unfocused. Voice is confused or unclear. Errors in the conventions of standard English may seriously interfere with the overall effectiveness of this paper. The 1 paper lacks purpose or development. The storyline is absent, negligible or cannot be discerned. The dominant feature is the absence of control of structure or the conventions of standard English. The deficiencies are so severe that the writer’s ideas are difficult or impossible to understand. An asterisk code is reserved for papers that are blank, illegible, or written on a topic other than the one assigned. Because these papers cannot be scored, a Writing Skills Test composite score cannot be reported. |
Select Exemplars
Using the rubric, expert scorers read a cross-section of the student’s
writing samples and select representative writing (exemplars) at the various
achievement levels. Examples of exemplars are:
|
This is the beginning of a story. Read it. Then continue writing the story and end it: Alfred was looking out the window while his parents were preparing supper. A gust of wind started to shake the house, making the bedroom curtains flap. Alfred began to shiver. Suddenly, all the lights in the house went out… |
| Grade 4 Exemplar - Level 1
… The house was empty and everything was on the floor the t.v. was broken everything was was broken it was scary to nite the curtains flap aroand and aroand the trees brok and fall. Comments
|
| Grade 4 Exemplar - Level 2
… Alfred always opens the window in his room But he doesn’t know ghosts
come in from The barn. The floors in the barn were about to cavein.
About a month later hestarted to shud the windows.
Comments
|
| Grade 4 Exemplar - Level 3
… Because a gust of wind nocked over a power post. I looked all over the house, but mom and dad were not home. So Alfred want over to his frnds but nobody was home. So Alfred went back hom. And looked in the garage to see if the car was home. It was. Alfred sed to himself Mom and dad never leve without the car. The light came back on and my mom and dad came back to. She was over at her frends house. I sed to my mom. “I was scared.” Comments
|
| Grade 4 Exemplar - Level 4
… Alfred was frightened. Alfred’s dad went to check the small power box outside. Mean while Alfred’s mom tried to find our tiny brown lanturn. Alfred’s dad was about to step in when he saw a gigantic twister! They all worked quick to get everything ready. Alfred’s mom said, “Hurry to the cellar.” Alfred brought lots of fruit with him to eat. When they were down there they ate and talked to each other. Alfred was terribly horrified! Soon all the food was gone. Alfred’s dad and mom polked their heads through the large door. “The twister was still going.” They said to Alfred. After awhile Alfred got really bored, so he fell fast asleep! Zzz. In afew hours Alfred was woken up by his parents. They whispered gently in his ears. “The twister is finally over.” All of them went up stairs. Nothing was broken at all, exept the outside! They decided to fix it up, when they when they got enough money. Alfred and his folks were happy. Then they all went out to find a job! Comments
|
| Grade 4 Exemplar - Level 5
… Alfred waited, then ran downstairs. He called “Mom Dad!” But no reply. Alfred thought they might be downstairs. He walked down the stairs ever so quietly. Creak Creak was the only noise. Alfred tried to turn on the lights. “SHOOT” He Whispered. The lights were burnt out. Alfred picked up a flashlight and clicked it on. They were not down there. Alfred decided to check the attic. So Alfred ever so quietly walked up the stairs, through the kitchen, and up the attic ladder. It was going nuts in the attic. Music was playing, lights were flashing, and all the boxes were opened. Suddenly a ghost bride appeared in front of Alfred. He ran down the ladder, through the kitchen, out the back door and down the hill. AAAAAAHHHH! There was a ghost picking potatoes. “Alfred!” “Dad!” “Here let me explain. When the lights went out we went up to the attic to fix the lights. But we found all this neat stuff. Your mother has her old wedding gown and I am wereing this old coat.” Alfred and his dad went into the house, had supper, and went to bed with a stomach full of potatoes. Comments
|
| Grade 4 Exemplar - Level 6
Fresh air flooded in through the open windows.
Alfred’s parents were going out that night and they told him
to take care of himself.
Little did their parents know that they were planning to go to
the attic that night. They said their goodbys and left.
“Mmmmm!” she shouted. The baby food tried to run but couldn’t.
There was a great slurp and the baby food was gone.
Comments
|
Benchmarking - Step 3: Train teacher-scorers
Scorers are selected from those teachers who work at the grade level from which the writing sample was taken. They participate in a two-day training session. At the training session, the rubrics are introduced and the exemplars are read together, so teacher-scorers can see where the exemplars fall on the rubric. The teachers practice scoring on training pieces. They compare the training pieces to the rubrics and the exemplars and do not compare one paper to another.
The focus during training is on consistency, so that teachers’ scoring is consistent with the rubrics and exemplars every time, and consistent with each other’s scoring every time.
Benchmarking - Step 4: Score students’ work
Teacher-scorers score the writing samples submitted by students. They work in pairs and change partners every half-day. Thus, two people score each paper. When there is a difference between the two scorers’ marks for a paper, the head scorer helps them negotiate the score. The head scorer also reviews a sample of the scored papers to make sure all scored papers are consistent with the rubrics and exemplars.
Benchmarking - Step 5: Report the Results
The reporting process used by Regina Public Schools is as follows:
Data from a benchmarking activity can also be reported to the public. For example, it may be appropriate to report to the public the percentage of students achieving at each level and to provide the public with examples of students’ work at each level.
Advantages and disadvantages of benchmarking
The benchmarking process has many advantages. Benchmarking:
The chief disadvantage of benchmarking is cost. Because it requires
that teachers be released from the classroom to participate in developing
rubrics and identifying exemplars, in training, and in scoring, there will
be costs for substitute teachers.
Part II - Setting Standards
Establishing benchmarks for achievement for any specific subject or skill area will provide baseline data, so that future years’ achievement can be compared. It will allow schools and school divisions to assess whether they are making progress in specific areas.
While it isn’t necessary to set standards for achievement, some school
divisions may choose to build upon the benchmarking process by doing so.
Setting standards is the process of specifying the percentage of students
who are expected to reach each level of achievement. For example:
| As part of the 1997 Provincial Learning Assessment in Mathematics,
standards were set for achievement in performance-based mathematics tasks.
These included items such as calculating volume and surface area of wood blocks and calculating the ratio of the length, surface, area and volumes of these small blocks. A five-level scale was used to describe student performance. Level 5 represented the highest level of performance; Level 1 the lowest. The percentage of students who were expected to achieve each level was: Source: Saskatchewan Education. (1998). 1997 Provincial Learning Assessment in Mathematics. Regina, SK. |
1. Establish a standard-setting panel;
2. Set the standard; and,
3. Report the results.
Standard-setting - Step 1: Establish a standard-setting panel
The ideal standard-setting panel includes about 50 percent teachers who are presently teaching at the grade levels and subject area for which standards are being set. The other 50 percent of the panel can be made up of students, parents, educational administrators and community representatives.
If you are setting standards for one subject area at one grade level,
a panel of 10-12 people is appropriate. If you are setting standards
for several grade levels, establish a panel that is large enough so that
it can split up into grade level panels of 9-12 people for each grade.
Standard-setting - Step 2: Set the standards
The standard-setting process is as follows:
Standard-setting - Step 3: Report the results
Results can be reported in a variety of ways ranging from a formal indicators
report, to a page in the school division’s annual report, to a page in
the school newsletter.
Advantages and disadvantages of setting standards
It may be useful to build on benchmarking by setting standards in school divisions where there is public pressure and concern about standards.
In other school divisions, setting standards may have disadvantages.
Standards replace the expectation for slow and steady progress with pressure
to achieve to a specific level. The cost of standard setting is also
a factor, since substitute teachers will be required and travel and honorariums
are usually paid to non-teachers.
Council of Ministers of Education, Canada. (1999). SAIP: School achievement indicators program: 1998 report on reading and writing assessment. Toronto, ON: Author. (URL unavailable at date of writing.)
Council of Ministers of Education, Canada. (1996). SAIP: School achievement indicators program: 1996 report on science assessment. Toronto, ON: Author. (www.cmec.ca/saip/sci96/)
Saskatchewan Education. (1999). 1998 provincial learning assessment in English language arts (listening and speaking). Regina, SK: Author. (www.Sasked.gov.sk.ca/k/p_e/eval/plap/listening_speaking/1998pdf)
Saskatchewan Education. (1998). 1997 provincial learning assessment in mathematics. Regina, SK: Author. (www.Sasked.gov.sk.ca/k/p_e/eval/plap/math/1997pdf)
Saskatchewan Education. (1997). 1996 provincial learning assessment in language arts (reading and writing). Regina, SK: Author. (www.Sasked.gov.sk.ca/k/p_e/eval/plap/language/1996pdf)
Saskatchewan Education. (1996). 1995 provincial learning assessment in mathematics. Regina, SK: Author. (www.Sasked.gov.sk.ca/k/p_e/eval/plap/math/1998pdf)