Setting Standards in Education: Saskatchewan Standards Symposium
Compiled and edited by Loraine Thompson Information Services Limited.
SSTA Research Centre Report #96-02: 71 pages, $14
Table of Contents

This report provides a review of the literature and a summary of symposium discussions in Saskatchewan regarding educational standards. Participants agreed that more attention must be given to the development and implementation of educational standards. Efforts to make education standards more explicit and public is a shared responsibility that requires provincial coordination. The process of setting standards should be a vehicle for improving the quality of education. Futher efforts are required to develop shared understandings and support for the continued development of education standards in Saskatchewan.

Why do we need education standards?  
Why do we need education standards?
Setting Standards in Education: Some Principles and Practices  
Setting Standards in Mathematics  
Problem Solving (K-9) in Saskatoon Schools
What kind of education standards are desirable for Saskatchewan?  
Perspectives from Saskatchewan Education  
Perspectives from Regina Public School Division  
Perspective from Eston-Elrose School Division
What do we need to know? Next Steps?
Back to: Evaluation and Reporting

The SSTA Research Centre grants permission to reproduce up to three copies of each report for personal use. Each copy must acknowledge the author and the SSTA Research Centre as the source. A complete and authorized copy of each report is available from the SSTA Research Centre.
The opinions and recommendations expressed in this report are those of the author and may not be in agreement with SSTA officers or trustees, but are offered as being worthy of consideration by those responsible for making decisions.

Setting Standards in Education Saskatchewan Standards Symposium


The Saskatchewan Education Standards Network

The Education Standards Network is an informal group of school divisions and educational organizations that have shared interests in issues relating to standards development. The network has been active for about 18 months. The present members of the Network are Eston-Elrose School Division, Moose Jaw Public Schools, Regina Public Schools, Regina Catholic Schools, Saskatoon Public Schools, Saskatoon Catholic Schools and the Saskatchewan School Trustees Association which also acts as coordinator of the Network. Recently, a representative of Saskatchewan Education has begun attending Network meetings. The Network has regular meetings to share information and to work on joint projects. Members of the Network invite other school divisions and organizations to join them. New ideas, greater sharing and increased collaboration will enhance the quality of education in Saskatchewan.

The Education Standards Symposium was held January 24-25, 1996 in Saskatoon. The purpose of the Symposium was to provide a forum for examining education standards in Saskatchewan and to share perspectives regarding emerging issues and future directions. The Symposium agenda appears in Appendix A. About 100 people participated in the Symposium. They represented educational organizations and about 33 different boards of education. A list of Symposium participants appears in Appendix B.

This document includes the proceedings of the Symposium and additional information. The proceedings constitute the main body of the document. Supplementary and explanatory material appear in boxes throughout. Main themes that emerged at the Symposium are summarized in the "What do you think?" boxes that appear throughout this document.

A standard is:

Ø something established by authority, custom or general consent as a model or example; or,

Ø something established for use as a rule or basis of comparison in measuring quantity, quality, value, etc. (Ravitch, 1995, p. 7).

Discussions at the Symposium acknowledged that there are three separate and distinct types of education standards.

Ø Content standards (or curriculum standards) describe what teachers are supposed to teach and students are expected to learn.

Ø Opportunity to learn standards define the availability of programs, staff and other resources that schools, school divisions and governments provide. They also determine whether the education system is fair to all students.

Ø Student performance standards define degrees of student mastery or levels of attainment. They describe the kind of performance that represents inadequate, acceptable or outstanding accomplishment (Ravitch, 1995).

The questions raised at the Symposium and the main themes that emerged were similar to questions and themes in much of the research and writing relating to standards.

Questions raised included:

Ø What type of standards are to be set?

Ø Who is to be involved in standard setting?

Ø How valid and reliable are standards?

Ø How can we set standards that measure thinking skills rather than memorization of subject matter content?

Ø How can we ensure equity in our educational system - for both genders and across all social classes, neighbourhoods and groups?

Ø How can we set standards that accommodate diversity in teaching styles?

Ø How are standards to be used?

Ø Who will control how standards are used?

Ø How can we set standards that provide for individual differences and acknowledge that students have different abilities and interests?

Main themes that emerged at the Symposium included:

Ø The concept of what a standard is differs from one person to another.

Ø An individual's view of what standards are and how they should be used is closely linked to that individual's philosophy of education and world view.

Ø The development of standards should be a collaborative process.

Ø Teachers need to be involved in the development of the standards that they will be expected to implement in their classrooms.


Al Klaussen - President, Saskatchewan School Trustees Association

Good evening ladies and gentleman. Welcome! I certainly want to convey our appreciation for your participation in this discussion of educational standards in Saskatchewan.

The Saskatchewan School Trustees Association is pleased to sponsor this Symposium. A year ago our Association was contacted by several school divisions with a request to assist in addressing questions regarding clarification of educational frameworks to guide classroom practice. On behalf of the Association, the SSTA Research Centre called several meetings to network these school divisions and to prepare for this Symposium.

It is our view that this Symposium is an important step in the process of making educational standards in Saskatchewan more public and more explicit.

We wish to express our appreciation to the school systems that have worked to prepare for this Symposium and who have offered to share their efforts with the broader education community. It is evident by the number of people here this evening that there is a good deal of interest in educational standards in this province. These discussions will help us to be better informed and to better understand what we mean by educational standards in Saskatchewan. I look forward to our work together and I trust that you will find this evening and tomorrow informative and productive.

Table of Contents

Presentation: Why do we need education standards?

Dale Botting - Executive Director, Canadian Federation of Independent Business
What Does the Public Think?

The Saskatchewan School Trustees Association regularly commissions public opinion polls on a variety of educational issues. In a poll conducted in fall 1995 (Saskatche- wan School Trustees Association, 1995a, 1995b) respondents were asked whether students should be tested to determine how well they are learning compared with the performance of other students in Saskatchewan and Canada. As Figure 1 shows, 76 percent either strongly agreed or agreed with comparative testing.

In addition, 91 percent of respondents said that there should be clearly defined expectations of what is taught and tested in school.

I want to thank the conference organizers for inviting me to this Symposium, and especially for asking me to help lead off this opening discussion.

The question posed this evening is, "Why do we need education standards?" Those who know me well will be aware that I wear several hats - as an outspoken advocate of the Saskatchewan business community, as an advocate of general taxpayers and consumers, as a parent of three teenagers (this explains my grey hair), as somewhat of a futurist of longer term provincial trends, and lastly, as a professionally trained educator myself, with a B.Ed. degree from the University of Saskatchewan among some of my other academic credentials.

In helping to contribute to the discussion, I would like to answer tonight's question from each of these three perspectives.

First, there is no question that Saskatchewan business wants to see clear educational standards according to all of our own member surveys. My organization, the Canadian Federation of Independent Business, has over 5,000 members in Saskatchewan - our members are women and men who independently own and operate their own firms. This makes for an excellent cross section of Saskatchewan business opinion.

In a 1992 survey, 85 percent expressed support for Saskatchewan participation in the national Student Achievement Indicators Program. In several member surveys in the 1980s, well over 80 percent were in favour of more standardized, province-wide testing. As recently as June 1995, in a separate CFIB survey, 49.5 percent of our Saskatchewan members indicated they were either somewhat or very dissatisfied with the high school system in preparing workers for employment in their firm. This may be partly why our further research reveals that small business owners in Saskatchewan spend an average of $2,000 per employee in additional orientation and on-the-job training costs when initially hiring every new employee in their business.

What Does the Public Think?

The Saskatchewan School Trustees Association regularly commissions public opinion polls on a variety of educational issues. In a poll conducted in fall 1995 (Saskatche- wan School Trustees Association, 1995a, 1995b) respondents were asked whether students should be tested to determine how well they are learning compared with the performance of other students in Saskatchewan and Canada. As Figure 1 shows, 76 percent either strongly agreed or agreed with comparative testing.

Figure 1 Public responses to, "Should students in this school be tested to compare how well they are learning with the performance of other students in Saskatchewan and Canada?"

Educational standards are important to Saskatchewan business to ensure that minimum competencies, understandings and skills are consistently assured by graduates of our school system as part of a quality labour force in the province. Bluntly put, business people do not want to do some of the most basic "product recall work" on behalf of our "educational factories".

Business also wants to know how the graduating work force in Saskatchewan compares when judged by various measurements, with the graduating students of other competitive jurisdictions and labour pools. In a global economy, the consistent quality of a labour force - measuring quality in all of its dimensions - is becoming one of the most important factors in industrial location and business investment. Without question, the Department of Education is far more important to us than the Department of Economic Development.

As a parent of three children, one of whom is about to graduate from Grade 12 later this year, I also know that I am not alone in my desire to see more educational standards applied uniformly across the province. In a public opinion poll just released by the provincial government, a large number of people wanted to see good standards and 40 percent were not sure that our students are prepared for entering the competitive work force. In the SSTA poll released in November, 1995, 91.1 percent of the people agreed there should be clearly defined expectations of what is taught and tested in our schools and 76.3 percent agreed that Saskatchewan students should be tested to determine how well they are learning in comparison to the performance of other students in Saskatchewan and Canada. Only one in five disagreed.

Like other parents, I am anxious that my children are being properly prepared, compared with those in other schools, to compete for placements in post-secondary schools or for jobs in other parts of Canada and the world. Will they be literate and skilled enough to eventually also create their own jobs, if they want to? And once they have been accepted by technical schools, universities or employers, will they have had the proper basic training to keep learning and to succeed? These are valid questions which need some clear standards for reassurance.

As a taxpayer, standards are important to do cost-benefit analyses to determine if our large public expenditures are providing value for money, and if we are making progress. Whether it be through the use of portfolios, performance assessments or other measurements - taxpayers see themselves as consumers of educational services and will either move elsewhere, go underground, or demand alternative services if they perceive that certain standards of accountability are not being met. The very existence of a continued system of public education, as opposed to U.S. voucher systems or school tax exemption zones, may be at stake. Our aging baby boomers may backlash and withdraw their tax dollars if accountability becomes ignored and our society becomes increasingly overtaxed.

As someone concerned about social justice, I am also an advocate of various standards to ensure that there is equity in our educational system - across all social classes, neighbourhoods, and groups. Regardless of location or background, can all children in Saskatchewan's educational system be guaranteed some minimum level of standard care and attention, as well as minimum learning and outcomes? Universality in education is every bit as important as universality in health care, and standards are a tool to help define whether social equity is really being achieved.

Finally, as an ex-professional educator I would not fear standards, particularly if they help to win back the public's respect and support for my profession. The public system of education is under growing competition with more and more private schools being established and home schooling increasing. Comparing the public school experience with a properly designed system of standards will help our school boards fight back - not just against other consumer choices, but also to preserve and increase your share of the fiscal pie compared to other competing budget priorities of government. But it is a lot harder to fund a road if you don't know where it's going, or to keep pouring money down a mysterious black hole with no measurable results.

Once again, I know that many other fellow teachers also share these concerns. Sixty percent of teachers in a February 1993 survey conducted for the High School Review Committee indicated that graduating students should be evaluated using some combination of Departmental and school division exams, along with teachers' input. Only 30 percent felt that evaluation should be the exclusive domain of teachers. They have nothing to fear but fear itself.

In summary, these are just a few of the reasons why we need standards. It all has to do with building partnerships and shared ownership of our educational system. It means decoding the "edubabble" and knocking down the aloof and mysterious walls of our professional fortress. Once we decide what is a fair basis of comparison in measuring the quantity, quality and value of education, our school boards can be better equipped to claw back society's tax dollars and trust. But the longer we keep procrastinating and getting ourselves tied up in definitional knots, the greater the risk of eroding this good will.

I wish to conclude by applauding SSTA, LEADS, STF, and the Department of Education for holding this Symposium - but I also sincerely hope that soon our timetable for action will speak even more loudly than just workshops and words.

The good news is that you have a lot of business, parent and community partners who want to constructively work with you to develop the most desirable system of education standards possible for a small province of one million people competing in a regional, national and global economy. The bad news, however, is that you may not have much time before our patience runs out!

Table of Contents

Meera Patel - Student, University of Saskatchewan

Hello and thank you. My name is Meera Patel. I graduated from Walter Murray Collegiate in Saskatoon last year, and am presently enrolled at the University of Saskatchewan in the College of Arts and Science. I was asked to speak about what educational standards mean from a student's perspective. There are so many standards in a school setting that in the brief time that I have it would be impossible to name them all, let alone talk about them. Therefore, I'll focus on the standard that is most important to students pursuing a higher education, and that is the standard of academic learning.

Having finished a term of university, I've heard from many first-year students that it would be beneficial for the whole province to change to Departmental exams for courses like Algebra, Science and Calculus. This would give students a level ground to start from when entering university. It would also let our school system compare standards interprovincially and, eventually, with the rest of Canada and other geographic locations. Presently students from different schools have different levels of knowledge. Even at Walter Murray, students had different levels of knowledge depending on the teacher they had.

This brings me to another point. Recently there has been much media coverage in magazines and journals about how North American children fare against Japanese in educational learning and intelligence tests. I was lucky enough to travel to Japan this past summer to take part in an International Science Summer School. Before leaving, I was quite nervous. Everything I had heard in the media had me thinking that I would not be able to keep up with the Japanese students. I didn't know if my Canadian education would serve me when compared to the knowledge of the Japanese. When I got there, I realized that they may be able to do things like math faster, but I was still able to do it. I was more surprised when I found out that my foundation in biology and chemistry pretty much equalled theirs. On the way home, I realized that I was proud not only to be Canadian, but to be a Canadian student.

When reflecting upon our Canadian standards, we all have a lot to be proud of. Our curriculum is broad and able to prepare students well for future education. More important, we have other kinds of standards too. In my thirteen-and-a-half years of school, I have never ever been afraid of violence at school. No guns, or knives, or even any teachers with meter sticks. And I truly believe that teachers are there for the students. Whenever I needed help, be it academically or otherwise, I have always found it. So, when reviewing our education standards, we should focus on the good and build from there. After all, tasks are always easier when it feels like something already has been accomplished.

Thank you.

Table of Contents

Trish McCormick - Teacher, Nutana Collegiate, Saskatoon

Twenty years ago I attended high school in a small city in Saskatchewan. I grew up in a middle-class home in affluent economic times. My parents believed in education and the old adage that, if you got into trouble at school it would be twice as bad when you got home; was a rule to be believed. My classmates seemed to be, for the most part, like me. There were no visibly handicapped students and no visible minorities. My vision of the future was bright - university and then employment. Classmates who did not go on to post-secondary education would have little difficulty finding employment.

Departmental Exams

A Departmental exam is an exam that is set and marked by Saskatchewan Education. At one time, all Grade 12 students in the province were required to write Departmental exams in certain basic subjects.

At present, evaluation of most 30-level (Grade 12) academic courses takes place in one of two ways.

Ø Students whose teachers are accredited in the 30-level course receive 100 percent of their final mark from the teacher.

Ø Students of teachers who are not accredited write a Departmental final exam in that subject area. This exam usually comprises 50 percent of a student's mark and the teacher determines the other 50 percent. (In the case of the new Core Science courses the teacher determines 60 percent of the final mark.)

Teachers become accredited by meeting certain provincial requirements involving academic and professional training and experience. In addition, school divisions may develop further specifications to the provincial policy. Teachers who have been accredited in a subject must renew their accreditation standing once every five years through involvement in professional development activities related to evaluation.

Adapted from: High School Review Advisory Committee. (1994). Final report. Regina, SK: Saskatchewan Education, Training and Employment.

In Grades 10 and 11, I was delighted by the study and discussion of literature and revelled in writing lengthy (and, I thought, scathingly brilliant) editorials and risque poetry and short stories (where I thought my use of the extended metaphor quite brilliant). In my Grade 12 year I had to face Departmentals - as I imagine many of you did. I was lucky - I was in Mr. "White's" class - and no one could get you through Departmentals like Mr. White. He was legendary for his high standards, his intolerance for undone homework, and his ability to make Grade 12 students and interns cry. Students who couldn't "cut it" didn't last long in Mr. White's class.

Early in the year, our task became clear - preparation for the Departmental exam. Mr. White assured us that he had successfully navigated the course of countless Grade 12 English students in the past and we would be no different. And he was right. We never lost sight of our goal. It seemed we no longer had time to glory in the beauty of a perfectly constructed metaphor or discuss (with the moral vigour and certainty of youth) the moral dilemma of a protagonist. Instead, I learned and memorized the 32 points Milton wished to make in his poem Il Penseroso and paraphrased Hamlet almost line by line. I wrote a multitude of old Departmentals. I learned to identify and parse even the trickiest compound, complex sentence and wrote brief "exercises" emulating the style of a sample in an antique composition text, addressing topics I felt sure had been popular in Victorian times but which inspired me not.

I didn't mind at all the time I spent with Mr. White. Unlike some of my classmates, I did not question what or how we learned. And I did not want to risk White's wrath - those who did often never returned. I knew two things: I really wanted to do well on this departmental, and rule #2 - "The teacher is always right". By the end of the class those of us who remained were well prepared. The Departmental held no fear or mystery for us. And, lo and behold, Mr. White was right - it was time for an essay question on whether Hamlet was truly insane to rear its ugly head again.

In many ways Mr. White was a good teacher and an accountable one. His task was clear and his standards were easily measured and I would imagine his success rate (among the survivors) was high. As consumers, my parents and I were happy. I wonder how Mr. White would fare today. If he taught in my classroom he would be confronted by a substantially different group of students and parents, facing an entirely different set of socioeconomic realities. His curriculum would just be beginning to undergo some radical changes but his teaching style would not have kept up with the new processes and learning styles we have recognized as important. I hope his goal, like mine, would be to keep kids in school, to try to make their learning personal, meaningful, and relevant to the goals they aspire to beyond high school, and in the broadest sense, to try and help them find "a life worthwhile".

We find ourselves in very different times. Critics who say the system of education must change must recognize there has been a tremendous revolution during the past 20 years, and it isn't over yet. Although the type of teenager I was in the 70s still exists to some extent, we all know the impact changing family structures, difficult economic times and the changing workplace have had for our students. Educators have tried to meet the changing needs of our clientele but more and more problems - serious problems, outside of school - determine how successful students can be. I firmly believe that many of my students are well prepared for any variety of post-secondary situations. However, they face economic and employment realities that were unheard of for me. As our world has become increasingly complex and challenging, I believe we continue to respond effectively to our students and their needs. Teachers are proud of what they do and they want to be accountable to their consumers. But this accounting must be accurate. It cannot be based on what was - those times are long gone. To honestly account to the public, any description of educational standards must be clearly defined and encompass the true nature of the education we try to provide.

Indicators of academic performance have not kept pace with the changes in education. In Mr. White's day, it was simple: the Departmental exam was based almost exclusively on knowledge of the content of the curriculum which was largely gained through lectures and rote memorization. As well, most students were highly motivated to do well on this exam and the exam accurately reflected how I had spent my year. My results were pretty good, and I would imagine any feedback given to Mr. White was also very positive.

So what might an instrument designed to assess the performance of students in my classroom look like? How would it include their competence on the computer - including surfing the net to find Brad Pitt's website and exchanging writing with students in other countries? Could it capture the values gained through a comparison of prairie writers' positions on the nature of humanity? Would it be at all interested in the presentation of an I Search project deciding whether or not a student should act as a surrogate mother so the child she gave up for adoption can have a sibling? Would it test primary and secondary source research skills? And where does this system recognize the accomplishment of a student who is the first person in her family to achieve a high school diploma? These, I think, are things the public might like to know about.

Would an instrument that can describe the reading levels of my students somehow consider factors which are impacting upon their performances, for example, the motivation level of the young man I know who will only attend as long as it is a condition of his parole? How would it explain their dysfunctional families, drug and alcohol addictions, and poverty? How would it account for the pregnant teenager who spends most of her time worrying she will miscarry for the third time; or the one who lives with 10 others in a small house on a reserve just out of town? How can such factors not be a consideration when we attempt to describe the performance of students within our educational system?

I am well aware of the high cost of education and of the desire of educators and boards to be accountable, but applying indicators which cannot encompass the scope of education and learning today, nor the broader spectrum of students and their needs beyond high school, is potentially dangerous. Saskatchewan has undergone tremendous change in its approaches to curriculum and instruction, in its population and their needs - any attempt to quantify measures of student performance must be geared to these changes. We need a fresh and contemporary plan to adequately describe our educational goals, consumers and accomplishments. Participating in national and international comparisons is not the answer. We are a unique region with a specific audience and our own beliefs about curriculum. This is what we must describe and evaluate.

What do teachers think?

In 1991, Vivian Hajnal surveyed Saskatchewan educators. One of the questions she asked was:

Should all high school students in the province be required to pass a standard provincial set of examinations to get a high school diploma?

Teachers' answers were as follows:

Yes - 56 percent

No - 42 percent

No Opinion - 2 percent

These answers were similar to a 1989 American poll described by Hajnal in which 54 percent of teachers supported standard exams in order to get a high school diploma.

Some would argue that the imposition of indicators of language and math skills might allow the education system to simplify - to stop trying to be all things to all people - but I doubt any of us wish to take a giant step back to what I affectionately call the "White days" without the social and economic conditions that accompanied that time 20 years ago. Some would argue that educational standards would create a level playing field for Grade 12 students. My students are well aware that life is not a level playing field - let's not simply move the game to artificial turf. Still others feel that individual teachers are immune from accountability. This must be addressed through a supervision process by the employing boards. An instrument designed to account to the public should not be designed nor used to evaluate teachers. If it is, teachers will surely be less inclined to address the real needs of their students and more inclined to seek schools where students are likely to do well on indicators of achievement.

I am a teacher who would be more than pleased to account for the education that is occurring in my classroom every day. Ideally this would include some recognition of the students I teach, the educational endeavours we undertake and their achievement. To truly know how well I have served, it would be nice to know the course of my students' lives and whether they were educated to "live a life worthwhile". I'm a practical person - I know some of this is not possible. But I would urge you to consider the merits of an assessment for Saskatchewan consumers which describes the Saskatchewan system of education and accomplishments of its students.

Table of Contents

Discussion Focus #1: Why do we need education standards?

Why Standards?

Education literature reflects a growing sense that education and schooling must change. Policy makers and the public are ready to consider new initiatives related to educational standards because (Ravitch, 1995, p. 9):

1. The belief that "academic performance needs to improve" has been established by media repetition.

2. Canadian students did not perform as well as expected on international assessments.

3. A fundamental shift has emerged suggesting schools should be judged not only by their inputs (resources, facilities, teacher qualifications, etc.) but also by their outcomes or results (student performance).

4. Social equality requires a strengthening of the educational achievement of minority students.

5. A well educated population is essential as the shift from a resource-based to an information-based economy continues.

6. Educators, public officials, parents and business leaders are realizing that there is no clear agreement about what elementary and secondary students are supposed to learn and that there are no reliable measures of individual student performance.

7. The role of schools has been confused in trying to be all things to all people - the priority of educating students must be reestablished.

When answering the question, "Why do we need education standards?", some Symposium participants began their discussions by emphasizing that the term "standards" means different things to different people. They viewed the Symposium as an opportunity to arrive at a common understanding of the different meanings of this term and said that the first step in this process is listening to each other's interpretation of standards.

Most participants said that standards are a way to convey to parents and the public what is going on in the education system - a way of telling the public what is being accomplished. They said that one of the major reasons for establishing standards is to increase public knowledge of and support for the education system. Some groups expressed a concern that parents and members of the public tend to evaluate the present-day education system in terms of the past - in terms of their own experiences of schooling 20, 30 or even 40 years ago. They said that this is not a valid way of evaluating the system, because social and economic situations are so different today than in the past. They said that standards must be developed within the context of today's realities and must inform the public about those realities. Some linked standards to increasing public demands for accountability and asked what the relationship is between standards and accountability. Others were more definite on this issue and said that standards are a way of providing for accountability.

How Should Standards be Used?

The High School Review Advisory Committee discussed at some length issues related to standards and accountability without reaching complete agreement. The following types of educational standards were considered:

1. Norm-referenced standards: "How does this student's performance compare to that of other students?"

2. Criterion-referenced standards: "How does this student's performance compare to an exemplary standard?"

3. Performance benchmark: "How does this student's performance compare to a minimum standard?"

4. Individual potential: "How does this student's performance compare to what the student is capable of?"

5. Growth in the individual: "How does this student's performance compare to her/his earlier performance?" (High School Review Advisory Committee, 1994)

For many of the participants, their discussion raised more questions than it answered. These questions included:

Ø To what degree should Saskatchewan Education be involved in the development of standards?

Ø What is the relationship between standards and the existing curriculum?

Ø How will standards be translated into specific assessment tools?

Ø What are the standards for the subjects currently being taught in schools.

Ø Do we believe in competition in schools and universities?

Ø Should there be a minimum standard for all?

Ø Why is secondary education geared to university entrance? Let the universities give an entrance exam.

Ø What changes do we have to make to help students achieve standards?

Ø What role does the government play in providing funding for opportunity to learn standards?

Ø What about retention versus passing everyone no matter what?

Ø What happens to students who don't meet the standard?

This last question was seen by those who raised it to be critical because it is intertwined with our whole philosophy of education.

One of the major themes that became obvious at the Symposium was that the concept of a "standard" differs from one person to another. To some people, standards are numerical test score cut-offs, to others they are descriptions of desired student learnings. To still others, they are descriptions of different levels of student performance or school programs and policies:

Ø Why are there so many different views of what a "standard" is?

Ø What does the word "standard" mean to you?

Ø Is it desirable that we all speak the same language when we talk about standards?

Ø If it is, what can we do to develop that common language?

Some participants said that developing standards to measure all the things that are important to us won't be easy. For example, love of books and reading is an attitude that we want to cultivate, but also an attitude that is difficult to measure. One group reminded us that in these days of diminishing resources, there is limited money and time available to collect data and present it in ways that are easy to understand.

Others said that regardless of what is done, standards should provide for individual differences and acknowledge that students have different abilities. Standards should also:

Ø create a balance between prescriptive requirements and opportunities for students to pursue individual interests.

Ø allow for different teaching styles.

Ø provide for evaluation that is fair to all students.

Among many participants, there was a fear that standards means the same as standardization. As the comments above suggest, most participants felt that standards should be structured to accommodate differing student interests and abilities, and to recognize that a variety of teaching styles are valid and appropriate.

Table of Contents

Setting Standards in Education: Some Principles and Practices

Dr. Brian Noonan - Superintendent of Education, Saskatoon Catholic Schools

Dr. Noonan's presentation was based on the following literature review which was distributed to all Symposium participants.

A. Introduction

Standards Aren't New

There have always been standards in education. What is different today is that there is pressure to make standards explicit and public.

The concept of standards in education is not new; however, the concept has been subject to different interpretations over time and across countries. For example, in the 1950s and 60s provincial examinations for all Canadian students represented a type of curriculum standard. Through the 1960s and 70s developments such as curriculum reforms, continuous progress of students, and accreditation for teachers emphasized more school and classroom control over curriculum and evaluation practices. One of the consequences of these changes was decreased emphasis on explicit external (i.e., provincial) standards.

Recently, there has been renewed interest in the role of standards in education, as shown by the increasing number of international, national, and local achievement programs and studies. Over the last few years, Canadian students have been involved in both the International Assessment of Education Progress (IAEP) and the Canadian School Achievement Indicators Project (SAIP), and some provinces have introduced various types of achievement standards. Indeed education systems around the world have focused more on measuring the student outcomes of school learning.

This is not to say there is a strong consensus that explicit standards are valued by everyone. Lewis (1995), in describing the "standards movement" makes the point that there always have been standards in education. What may be different today is that there is pressure to make standards explicit and public. The controversial nature of the emerging public interest in standards is clear and, as a result, educators are taking more and more interest in presenting various views on the issue. For example, the Phi Delta Kappan recently devoted a whole issue to standards and the Canadian Journal of Education published an issue devoted to the idea of accountability in Canadian education.

Renewed Interest in Standards

Factors that have revived public interest in standards:

Ø general concern about accountability in public education; and,

Ø changes in classroom teaching and learning.

The reasons for this renewed interest in student achievement and educational progress are manifold, but there are at least two basic factors which have influenced the standards "movement". First is the general concern about accountability in public education and second are changes in classroom teaching and learning.

Recent economic conditions and fiscal restraint have raised calls for increased accountability in education, partly in consideration of the cost-benefit of the education system. Whether or not it is accurate, or fair, some elements of the public see student achievement as the measurable "benefit" of educational costs. As such, student achievement must be measured against explicit and public standards. McEwen (1995) has described the framework of educational accountability in Canada and provided a description of accountability programs in various provinces. She emphasizes that accountability programs are intended to both improve education and to allay the fears of the public that schools are failing. The primary instrument to achieve these two goals are provincial assessment projects and national indicators programs such as the SAIP. Earl (1995), Fagan (1995), Hodgkinson (1995), Maheu (1995), and McEwen (1995) describe provincial accountability programs which typically focus on student achievement as a measure of success. At the same time, others such as Meaghan and Casas (1995) caution against such programs which may be perceived simply as large-scale standardized testing that only serve to sort students and may place no real value on the teaching/learning process.

A second factor which may have revived interest in standards has to do with changes in curriculum, instruction, and assessment practices. Contemporary curricula focus on process skills, more integrated subject areas, and common "core" intended learning outcomes. Instructional practices emphasize both cooperative and independent learning which better address the learning needs of students. Classroom assessment today includes a range of observational and performance assessment measures as well as traditional paper-pencil tests, projects and assignments.

Student diversity is also a characteristic of education today. Special education and inclusive education practices have integrated a fuller range of student abilities in all classrooms. Diversity in classroom practices, assessment techniques, and student population have changed the context of educational measurement. To some, it seems that education has become increasingly and unnecessarily complex and that standards may somehow help to simplify teaching and learning. The combination of diversity of student characteristics and new classroom practices have led to a sense among educators and the public that there no longer are clear and common expectations of what constitutes educational attainment. Biemitter (1993) makes the point that diversity is a characteristic of schools today and presumably needs to be recognized in any discussion of measurement or standards in education.

Accountability issues aside, there can be educational benefits to the development and use of standards. Salinger (1995) summarizes these benefits as: providing a model for thinking about teaching, support for contemporary practices in assessment, providing equity for learning opportunities, and professional development of teachers. Given the apparent increasing interest in standards in education the purposes of this paper are:

(1) to identify the basic principles which underlie current practices of setting standards in education and to review current concepts of standards;

(2) to review some of the ways in which jurisdictions are developing and setting standards; and,

(3) to pose some questions which might help focus discussion and decision making on standards in Saskatchewan education in the 1990s and beyond.

B. Principles of Standard Setting

Criterion-Referenced and Norm-Referenced Measurement

A popular Canadian book (Lewington & Orpwood, 1993, p. 129-130) describes the two types of measurement in the following way:

Ø Criterion-Referenced Test: The Driving Test - Most adults remember the terror (or joy) of taking a provincial drivers' test. One had to meet pre-determined standards to demonstrate driving competence. Individual performance was judged against absolute standards, not by comparison with other drivers. In the same way, criterion-referenced tests measure a student's achievement against certain educational standards or objectives, such as writing a paragraph free of spelling mistakes. For testing experts, criterion-referenced tests are increasingly attractive tools because they measure student performance against an external standard.

Ø Norm-Referenced Test: The World Series - In the final days of the baseball or hockey season, fans are eager to know where their favourite team stands in relation to others in the division or the league. Thus, the team's winning percentage is less significant than its lead over the nearest rival. In the same way, norm-referenced tests measure a student's relative position in the class or in some larger group. As a result, student marks are adjusted to fit a predetermined notion of the "normal" distribution of A's to F's, a practice known as "marking to the curve." Norm-referenced tests are less popular for educational purposes today because they do not paint an accurate picture of individual student achievement.

1. Background - defining school achievement.

The current conceptualization of standards was derived from the principles of criterion-referenced measurement (CRM) introduced in the 1960s and 1970s. Criterion-referenced measurement represented an alternative approach to norm-referenced measurement (NRM), typically associated with standardized testing. Norm-referenced measurement has its origins in psychometric theory and practice such as intelligence testing or other forms of attribute measurement. In essence, such measurement assumes that certain human traits are normally distributed in a population and valid tests are developed to measure these traits reliably. An important element of such norm-referenced testing is its use in measuring latent traits, as opposed to specific performance. For example, intelligence was conceived of as "ability to learn" - a type of potential, a general factor which could predict student performance. NRM is sometimes characterized as a form of indirect measurement which implies a level of inference in an interpretation. To help interpret the results of NRM, statistical procedures and forms of standardization were developed. One purpose of CRM was to help determine accurately the extent to which student performance was demonstrated. Thus, CRM is considered a more direct form of measurement appropriate for assessing student achievement of particular learning objectives. In general, NRM is an indirect form of measurement, whereas CRM may be considered a more direct form of measurement.

The technical principles of norm-referenced test development such as determining validity or reliability coefficients have been applied to developing standardized achievement tests, such as the Canadian Test of Basic Skills or the Canadian Achievement Test. The application of norm-referenced technical procedures to developing achievement tests raised the question as to what it is that such tests are measuring. Keeping in mind that norm-referenced tests were originally developed to measure latent traits (e.g., intelligence), was it reasonable to apply those techniques to school achievement (e.g., student learning outcomes)? This is an important question, because the focus on standard setting today is in part an outgrowth of the concern for accurately defining school achievement.

Over the past 20 or 30 years researchers have attempted to describe the underlying constructs for what is called, in general, school achievement. Achievement applies to a broad range of objectives including traditional academic achievement, encouraging positive attitudes and developing personal aptitudes or talents. Cole (1990), Paris et al (1991) and Masters and Mislevy (1991) are among those who have expressed the view that there is a need to be more explicit with respect to the definitions and assumptions about student learning and school achievement. There does not seem to be a clear consensus as to what is meant by the term achievement. Most likely it is a general construct, a combination of academic progress, positive attitudes, development of natural abilities, and consistent attendance.

2. Criterion-Referenced Measurement

CRM emerged in the 1970s and early 1980s as an alternative to norm-referenced tests in educational measurement. Glaser (1963, 1994) is generally credited with introducing the idea that some aspects of a student's status (e.g., knowledge, skill or attitude) could be measured with respect to specific criteria. Glaser used the term "performance standard" to describe the criteria. The introduction of the idea of performance (e.g., demonstrable learning) helped set the stage for subsequent development of education standards. CRM was seen as an alternative to NRM in the search for improved ways to measure school achievement.

Conceptually, CRM was further refined by Popham (1981), Hambleton and Eignor (1978) and others. Typically, CRM is defined as the extent to which some form of student performance matched specific objectives (or some domains of behaviours) described as intended learning outcomes. Part of the process of describing CRM involved categorizing the large number of objectives-based instruments which were considered as forms of CRM. Gray (1978) and Nitko (1980) provided overviews of the types of instruments considered to be criterion-referenced; in each case dozens of types of instruments were generally categorized as forms of CRM. Berk (1986) provided one of the most extensive overviews of the application of CRM which outlined the purposes of such measurement and the technical qualities of good CRM instruments. CRM then offered the promise of meaningful assessment of student achievement in a significantly different way than norm-referenced measurement. CRM required that specific performance outcomes be specified as criteria to assess students' achievement. This fundamental principle became a basis for the contemporary understanding of "standards" as they are defined and used in educational measurement today.

3. Application of CRM

During the past 30 years, CRM has been applied to a large number of educational policy and curriculum developments. Mastery learning and minimum competency testing utilized the idea of criteria, or domains of behaviours, as a way of making decisions about student learning. Competency-based instruction and various forms of computer-aided instruction were also based on evaluation of student learning on specific intended learnings. As well, licensing bodies in trades and professions adapted CRM to develop tests for certification. Subsequently, performance standards became integral to instruction and evaluation in many forms of child and adult learning. Glass (1978) presented one of the earliest attempts to link standards with criteria. He described six classes of techniques for determining the criterion score in CRM, including minimum competency, performance of others, and decision-theoretic approaches. Glass's concept of the standards-criteria relationship implies the role of expert judgement in describing competency or performance; a key element of the standard-setting process is the role of expert judges. This is in contrast to the purely technical (e.g., quantitative) processes of NRM. In general though, techniques outlined by Glass may be considered technical in the sense that there is little requirement for expert judgement in establishing criteria for success. The role of expert judgement is a more recent addition to the place of CRM in education.

Recently, CRM has been applied to the principles of curriculum-based assessment. Concomitant with the development of new curricula, there has been a renewed interest in more direct forms of assessing student learning. New curricula are often based on cognitive principles of learning which emphasize procedural knowledge (e.g., learning strategies and applications) rather than declarative knowledge (facts and understandings). Measuring student outcomes for curricula which emphasize these different types of learning requires different instruments and techniques. Conventional, norm-referenced or standardized tests are viewed as inadequate for most of these purposes. Consequently, efforts were being made to develop instruments to better measure curriculum objectives. Portfolios, performance assessments, and use of observation techniques are seen to be alternatives to paper-pencil tests as measures of student progress based on a specific local or provincial curricula. Such instruments purport to have greater content validity than other forms of assessment because they measure a broader range of knowledge and strategies, and are a more direct form of assessment. On the other hand, these techniques may be criticized for the lack of reliability which is characteristic of norm-referenced, standardized measures. The technical qualities (validity evidence and reliability) of any type of measurement are important factors to be considered in setting standards. However, it is beyond the scope of this paper to review, in any comprehensive manner, the literature on validity evidence.

4. Standards-Referenced Measurement.

One of the important steps in the standards movement was the application of CRM to classroom learnings which emphasized the role of expert judgement. Sadler (1987) has described an alternative interpretation to the original, more technical understanding of CRM by distinguishing between criterion and standards. The former is considered to be a characteristic which can be judged, the latter as a level of attainment or degree of any measurable quality. Sadler defined this interpretation as standards-referenced assessment. Such assessment may be characterized as one of four types:

Ø Numerical cut-offs, for example, typing words per minute (wpm) or scales which translate percentages into letter grades (e.g., 80% = A).

Ø Tacit knowledge is the judgement of experts, that is "in the head evaluations" which characterize the connoisseur. Appraisals in the arts or in some athletics use this model.

What's an Exemplar?

An exemplar is an example of student work which is typical of a specific level of quality. Below is an exemplar taken from Saskatchewan Education's (1996) 1994 Provincial Language Arts Learning Assessment which characterizes the work of a Grade 5 student at level 3 on a 5-point scale.

Ø Exemplars are key examples which characterize a specific level of quality. They are typically concrete and explicit, and are most useful if only one or a small number of criteria are involved. For example, the hardness scale of a mineral or the standard metre length are exemplars used in the physical sciences. Assessing a performance with multiple criteria, however, makes the use of exemplars more complex.
Exemplar - Grade 5 - Holistic Writing - Level 3 (out of 5)

Fred the Banana

Once upon a time there was a banana. It's name was Fred. Fred lived in a bunch of bananas in the African Jungle. All his friends got eaten by monkeys. The next day some men came to catch monkeys they cot about five and put them in the back of the plain. They grabbed a bunch of bananas for the monkeys to eat on the trip to the zoo. Then the men hopped in the air plain and flewe out of the jungle and in to florida. they put the monkeys in the back of a truck and drove to the zoo. Fred lived under the monkey cage. He explored the zoo for about a month. Then he got bored so he just stayed home. Then one day a ornge truck drove by and a creat fell out. One of the orgnes came over to fred and told him his name is Barney. Fred shod Barney the hole zoo. Then he looke for a home for a home for Barney. They found one under the lion cage. They had lots of fun running around the zoo and playing hide and go seek.

What's a Rubric?

When a Symposium participant asked, "What the heck's a rubric?", several others nodded their heads to show that they were wondering too.

A rubric is the criteria that describes student standards of performance by which a product, performance or demonstration will be developed and assessed.

Below is a rubric developed by a team of teachers representing the Eston-Elrose and Regina Public School Division for Response to Reading.

Ø Verbal descriptions, sometimes called "rubrics" are written statements of the quality under consideration. They have the benefit of making the standard more objective and public.

In reviewing the use of the four types of standards Sadler concludes that a combination of exemplars and verbal descriptions holds the most promise for the future of educational standards setting. However, more work needs to be done with respect to the actual process of setting standards. Section D of this paper reviews some techniques for setting standards.

Figure 3: Rubrics for Response to Reading - Grades 6 to 9

Explanation of own reaction to text with multiple reasons
Explanation of own reaction to text
Makes simple reaction with logical support
Apathy - little interest or involvement in text

Making Connections
Makes logical connections to outside sources and applies to other situations (literary experiences, other subject areas, own writing, others' writing)
Makes logical connections to and beyond personal experiences
Makes surface connection to personal experiences
Has difficulty making logical connections

Asks questions to identify key issues and make inferences as to author's intent
Values others' perspectives Asks questions to identify various key issues
Occasionally includes others' perspectives as a means of increasing interpretive possibilities
Asks questions to clarify meaning
Recognizes author's perspective
Asks unrelated questions
Not able to tell author's intent

Insight into Elements of the Writer's Craft (Style)
Compares and contrasts functions of the elements of the author's style
Able to see implications for own work
Analyzes the elements of the author's style
Recognizes and discusses the elements of the author's style
Elements of the author's style not apparent

In summary, standard setting in education is based on criterion-referenced measurement, an approach to assessing learning outcomes that is conceptually different from norm-referenced assessment. Because there is a need to establish standards against which student performance may be measured, it is generally accepted that criterion-referencing should be the basis for standard setting. Following is a discussion of different types of standards which may be employed in education. C. Types of Educational Standards
Three Types of Education Standards

Ø Content standards (or curriculum standards) describe what teachers are supposed to teach and students are expected to learn.

Ø Opportunity to learn standards define the availability of programs, staff and other resources that schools, school divisions and governments provide. They also determine whether the education system is fair to all students.

Ø Student performance standards define degrees of student mastery or levels of attainment. They describe the kind of performance that represents inadequate, acceptable or outstanding accomplishment (Ravitch, 1995).

The term "standards" has taken on various meanings and interpretations. Indeed, the term has been applied to a large number of statements, both general and specific, about educational expectations. For example, standards have been referred to as basic goals of education, as minimum competencies, as curriculum objectives, as measured student achievement and so on. More recently there has been a move to develop a common language on standards which has been helpful in furthering the debate as to the desirability of setting and using standards. Linn (1994) has identified three types of educational standards:

Ø content standards;

Ø opportunity to learn standards, and,

Ø performance standards.

Following is an overview of each type of these standards.

1. Content standards.

These are also called "curriculum standards" and, in general, they represent intended learning outcomes for a particular content area. One of the best known of these standards is the one developed by the National Council of Teachers of Mathematics (NCTM) (1989). These standards detail explicitly the mathematical knowledge that should form the curriculum for school students at various development levels. NCTM standards are derived from the body of knowledge which forms Mathematics (algebra, geometry, numeration, etc.). Other subject areas such as Science and English Language Arts also use this form of standards and have defined what students should know in a particular domain in a field of study. With the development of such standards it seems clear that the basic rationale and principles of standard setting may be common across subject areas. However, it is less clear if there should be a common format for standards across subject areas.

Curriculum developers are faced with a dilemma related to the extent to which their work should be considered as content standards - and how those should be expressed. For example, should science curricula be based on discipline knowledge (life sciences, physical sciences, etc.) or should it be based on dimensions of scientific literacy - a quite different framework. Presumably, content standards would be quite different for the two approaches to the curricula. The development of the writing dimension in English language arts presents a similar problem for curriculum writers. Are content standards to be related to the general principles of literacy or the traditional rules of grammar and syntax, or a combination of the two? Establishing the appropriateness of content standards involves expert professional judgements prior to implementation in the classroom. As well, if curricula are to considered as a standard, that fact should be made clear in the curriculum documents.

2. Opportunity to learn (OTL) standards.

Porter (1995) views OTL standards as the "criteria for assessing the efficiency or quality of the resources, practices, and conditions necessary at each level of the educational system ... with an opportunity to learn the material ..." (p. 23). This definition of standards addresses the problem of equity in education, that is, ensuring fairness and equality of results for all students. It would not be fair if some students, for whatever reason, had an unearned advantage over others, such as more equipment, better trained teachers, or access to more resources. OTL standards are intended to determine whether all students had access to learning opportunities. This extends the notion of curriculum standards by emphasizing the role of the school and school jurisdiction in being accountable to students in the curriculum offered in classrooms.

Porter (1995) and Howe (1994) have examined the problem of OTL standards from the perspective of school delivery standards and the accountability of schools and school jurisdictions. Porter (1995) suggests that OTL standards include:

Ø availability of curricula;

Ø instructional materials and technology;

Ø quality of instruction;

Ø extent of safe, secure facilities; and,

Ø policies of non-discrimination.

Although these may appear, on the surface, as simplistic or self-evident, OTL standards may have greater implications than is first thought. As described earlier, diversity of student population has become the norm in many schools and school jurisdictions. Policies of inclusion, mainstreaming, education equity, retention initiatives, and other efforts have produced school systems characterized by diversity. Equity demands not only equality of opportunity but equality of results. This raises the important policy question as to how OTL standards are to be implemented in an increasingly diverse population.

Clearly OTL standards have to do with school improvement, school finance, and school governance. As time passes, OTL standards may be the most difficult to establish, to measure, and to implement. Howe (1994) related OTL standards to compensatory education, egalitarianism versus utilitarianism, and redefinition of the very nature and purpose of publicly-supported education. A careful review of OTL standards suggests that they may demand serious restructuring of current views of schooling if they are to be considered fundamental to contemporary education.

3. Student performance standards.

If one thinks of content standards as the input to the educational enterprise, and OTL standards as the process, then student performance is the output of the system - or at least one major output. If there is confusion as to some goals and objectives of schools, there is a consensus that assessing student outcomes is important. There is, however, room for debate as to how performance standards should be expressed.

The concept of performance standards probably comes closest to a "pure" application of CRM described earlier. Taylor (1994) expands on this by describing performance assessment as based on a standards model versus a measurement model. A measurement model is based on comparative, norm-referenced assessments which emphasize reliability or consistency of measurement. A standards model, on the other hand, emphasizes validity, the "public" nature of the standard, and that there may be a number of ways in which students can demonstrate their mastery of particular knowledge or skills. This last point is important in reinforcing the principle that performance standards are judged on what a student does, not only on what he/she knows.

Because performance standards are based on demonstrated learning they are (or should be) less inferential. As was described earlier this is one of the principles which differentiates criterion-referenced measurement from norm-referenced measurement. For example, if one sets a standard saying that a successful student must write an error-free, five-sentence paragraph, then the student must show that he/she can actually do so. The idea of demonstrable learning is the basis of the application of CRM to licensing, mastery learning, or competency-based instruction. In each case, the candidate must be able to demonstrate the skill, knowledge, or other attribute at stake; these standards are sometimes called benchmarks or indicators. Many school jurisdictions have become involved in establishing various forms of such standards, typically for basic skills such as language or mathematics. For example, the North York Board of Education (1992) has established benchmarks as part of their standard-setting process. The benchmarks, developed by teachers and administrators, are public expressions of the intended achievement outcomes of students.

An integral part of such performance standards are clear descriptions of how such standards are established and scored. Some procedures for establishing performance (or other) standards are described in Section D. Scoring performance differentiates the standards model from the measurement model (Taylor, 1994). Standards represent expectations not only for the performance criteria (skills, knowledge, processes) but also for the quality of the performance. Consequently performance standards include scoring rules called rubrics (Sadler, 1987) which guide teachers in using the standards. A scoring rubric enables one to rate a student's performance against exemplars at several levels. Typically, the scorer uses an ordinal level scale (i.e., 0 to 5, or 0 to 6) with clear performance criteria and a description of the exemplary performance at each level.

All standards, but particularly performance standards, must, in part, be judged on their validity and reliability. This is important, because performance standards often are measured using instruments other than, or in addition to, paper-pencil tests. Performance tests, portfolios, observations or other forms of direct assessment are commonly used in assessing student learning because such instruments are known to provide better validity evidence. Performance standards need to be high on measures of validity - that is, they need to assess what is intended. Linn (1994) and Messick (1994) have reviewed various dimensions of validating performance standards. In standard setting, validity refers to the legitimacy of any interpretation or reference about a student's performance; in principle, validity evidence helps to lower the level of inference in an assessment. To help ensure that standards are valid, there must be evidence that supports the reasonableness of the standard. Traditionally, validity evidence has been related to face and content validity, predictive validity, or construct validity. Moss (1995) and others extend this argument by suggesting that validity must also include consideration of the consequences of an assessment. In high stakes assessments one would want assurance as to the validity of the measurement instrument.

Reliability (i.e., consistency, stability) is one of the more troublesome technical qualities of performance standards (Linn, 1994; Reckase, 1995). One of the perceived benefits of NRM has been the increased reliability of results; CRM has typically been viewed as less reliable. If CRM is to be used for high stakes purposes (such as for entrance to university or to license a trade), then reliability is an increasingly important quality which needs to be made public. More research is needed to examine (and perhaps satisfy) some of the criticisms relating to CRM's reliability and its application to standards in education.

In summary, three types of standards have been identified - each has a different use and interpretation. Content standards help identify the framework for curriculum development, and opportunity to learn standards can help to address systemic (school- or division-level) factors which can affect student achievement and which are beyond the control of students. Performance standards provide demonstrable measures of student learning. The next section reviews the general process for developing any of three types of standards - content, OTL, and student performance.

D. The Process of Setting Standards

As described earlier, standards are intended to provide a clear set of statements against which student learning can be measured. Interestingly, one of the aspects of standards which is not commonly reported concerns the actual decision-making process which results in the statement which becomes the standard. Two questions which need to be addressed are who set the standards? and how did they do so? Berk (1986), Linn (1994), and Taylor (1994) have identified two elements that may be used to determine standards - namely judgemental information and empirical information. In principle, the process may be entirely judgemental, entirely empirical, or a combination of the two. Fundamental to the standard-setting process is the role of the expert judge.

The Expert Judge

The expert judge is often used in both norm-referenced and criterion-referenced measurement. The expert judge excels in an area and has expertise born of training and experience which has been shown to be fair, accurate and consistent. Expert judges often select items or tasks which test learning in their area of expertise.

1. The Expert Judge.

The use of expert judges in both norm-referenced and criterion-referenced measurement has been common practice. Jaeger (1991) describes the expert judge as one who excels in the domain, acts rapidly in decision making, sees deeper levels of problems in the domain, and has self-monitoring skills. Typically, it has been the role of the judge to select items which test the prescribed objectives, knowledge, processes or other dimension of learning which is under examination. The judge selects from some domain of learning, general and/or explicit statements, which accurately reflect that domain. The expert judge holds what Sadler (1987) describes as tacit knowledge, expertise born of training and experience which has been shown to be fair, accurate, and consistent.

Depending upon the task, expert judges typically would be curriculum or instructional specialists. However, given training in the standard-setting process, classroom teachers may possess the knowledge to set standards. One of the skills of standard setting is making tacit knowledge explicit.

In general terms, judgement in standard setting refers to the ability to make fair and accurate decisions based on one's knowledge and experience. Taylor (1994) suggests that expert judges have "internalized" the domain in which they work which is similar to what Sadler (1987) called tacit knowledge. Reid (1991) has described how one might further train the expert judge to assure (as much as it is possible) the best possible decision in the standard-setting process.

Although the concept of professional judgement is not new to education (or other social sciences) a discussion about standards must involve, at some point, clarification of the nature of professional judgement. For example, does current teacher training develop this skill in classroom teachers? What technical knowledge (i.e., curriculum development, statistics) is necessary for expert judges? How does one refine, develop and practice skills in making judgements? How does one lower the level of inference in professional judgements to enhance reliability? Answers to these types of questions will help develop a standard-setting process based on professional competence and confidence.

2. Standard-Setting Procedures

As described earlier, Berk (1986) has described the techniques used to set standards as a combination of judgemental and empirical procedures. Extending Berk's description of standard-setting procedures, four possible techniques emerge as shown in Figure 4.

Figure 4: Standard-Setting Techniques

judgemental {} judgemental-empirical {} empirical-judgemental {} empirical

Ways of Setting Standards

Ø Judgemental techniques - use only the results of decisions made by panels of expert judges.

Ø Judgemental-empirical techniques - use empirical data such as student scores to help judges set standards.

Ø Empirical-judgemental techniques - empirical information is obtained through student testing which is then used by judges to set a standard.

Ø Empirical techniques - decisions are made on the basis of a statistical norm which reflects average student performance.

Each of these four techniques is described in more detail below.

(i) Judgemental techniques. This procedure involves using only the results of decisions made by panels of expert judges. Most curricula are developed using this approach, as are most teacher-made tests or commercial tests which sometimes accompany teaching-learning materials. One of the technical considerations is the number of judges required to assure inter-judge reliability.

(ii) Judgemental-empirical techniques. This procedure incorporates empirical data (student scores, survey results, etc.) to help judges set standards. In practice this might mean that first standards are set for some domain of learning, then students' test scores are used to confirm the standard. For example, if a standard for identifying gifted students was being developed, the standard may be set first through teacher judgements as to what classroom behaviour might constitute "giftedness". Subsequently scores on ability tests would be used to confirm the accuracy and fairness of the teacher judgement. The empirical information may convince the judges to confirm or modify the judgement-based standard before it is used in decision making.

(iii) Empirical-judgemental techniques. In this procedure, a process opposite to the one described above is used. First, empirical information is obtained from students through testing, then a judgement is applied to set the standard. For example, a standardized achievement test might provide information to judges on the group performance of students on a set of, for example, math computation skills. Subsequently, this may help teachers determine a standard for a particular developmental level. However, the judges may be of the view that the standardized test, though reliable, has missed an important skill which, in their opinion, should be included in the standard. Ultimately, then, judgements are used to modify and refine the standard originally based on empirical evidence.

(iv) Empirical techniques. Empirical techniques alone do not, by definition, fulfill the requirements of standard setting because a purely empirical standard is, in fact, a norm. A popular view of non-educators is that such purely empirical 'standards' (e.g., norms) are the best measures of student learning. Norms, because of their relatively simple arithmetic derivation, do have appeal; however, the use of norms as standards contradicts the concept of standards-based measurement (Taylor, 1994; Sadler, 1987) insofar as standard setting requires professional judgement. This emphasizes the importance of developing expertise in teachers and others in the process of standard setting.

Other writers have proposed similar, though less technical, descriptions of standard-setting procedures. Linn (1994) identified two procedures - armchair reviews (tantamount to the judgement techniques) and - technical reviews (similar to empirical-judgemental techniques). Taylor (1994) outlines a four-step procedure for standard setting.

Ø using expert judges to develop standards;

Ø establishing benchmarks;

Ø developing performance criteria which operationalize the standard; and,

Ø clarifying scoring rules (rubrics) as described earlier.

In general, it is probably accurate to say that standard setting is a process which emphasizes various forms of expert judgement, taking into consideration various types of empirical information. Whatever process is used, it should be emphasized that standard setting can be a time-consuming (and costly) process. The process of developing, testing, refining and confirming is an iterative one that probably has no shortcuts.

3. Some Issues in Standard Setting

Several concerns arise from a review of the literature on standard-setting procedures.

Issues in Standard Setting

Ø What type of standard is to be set?

Ø Who sets the standard?

Ø How valid and reliable is the standard?

Ø How is the standard scored?

Ø How can we ensure that the public understands the processes used to develop and apply standards?

Ø How do we set standards that measure thinking skills rather than subject matter content?

Ø How are standards to be used?

First, there are issues with respect to types of standards and standard setting. For example,

Ø What type of standard is under consideration (content, OTL, or performance)?

Ø Who sets standards and how are they set (the role of judges)?

Ø What is known about the technical qualities of a standard (validity and reliability)?

Ø How are standards scored and reported (rubrics, exemplars, etc.)?

Second, standards, in principle, are public statements and it is quite possible that the public may not appreciate the technical and professional skills needed to develop standards. There is a need, therefore, to ensure that the processes used to develop and use standards are well understood. A demand for accountability is not in itself a rationale for using standards, which may lack meaning, accuracy or appropriateness.

Third, many curricula today are based on the principles of cognitive learning - as opposed to behaviourial approaches to learning. Cognitive learning principles emphasize the use of one's memory and the development of metacognition (that is, thinking about one's learning) to help improve the learning process by making learning experiences more meaningful. This stress on the "processes" of learning rather than the "products" of learning may make it more difficult to clearly articulate standards. How does one set standards for learning outcomes described as tactics or strategies rather than subject matter content? It is not clear how such problems may affect the way standards are expressed. Masters and Mislevy (1991) have explored some conceptual and technical matters in CRM under what they call "new views of learning", in essence the application of cognitive psychology. Kendall and Marzano (1994) have also described how cognitive psychology can be applied to standard setting using the framework of declarative, procedural, and contextual knowledge in a number of different subject areas. More research on the application of cognitive learning to classroom practices is needed before standard setting can be generally applied.

A final simple question is, "how are standards to be used"? McEwen (1995) suggests that accountability is intended to both improve public education and to allay public concerns. How can standards serve these two, somewhat conflicting, purposes? The use of standards implies a degree of public revelation of the results. Therefore, the purposes of setting and using standards must be made clear at the outset. Further, if the only use of standards is to compare groups or to rank performance, then there will be little motivation for standard setting. There is a need to clarify if the purpose is to help set public educational goals, to identify preferred forms of instruction, to make assessments fair and more accurate, or some other purpose.

In summary, a number of procedures are available to those who establish standards. In essence, those procedures rely on expert judges and, to varying degrees, empirical information. Whatever process is selected, those who establish standards need to be aware of the technical qualities of their standards - that is their validity and reliability. Setting standards requires a high degree of both professional expertise and public transparency.

E. Practices in Standard Setting

1. The American Experience

The processes and products of standard-setting projects across North America have been well documented. The National Assessment of Educational Progress (NAEP) provides examples of the types of instruments that may be used in large-scale assessments. Subject specific groups such as the National Council of Teachers of Mathematics (NCTM) and professional organizations such as the National Council of Measurement in Education (NCME) have documented their efforts to apply the processes of standard setting. The NCTM has prepared comprehensive standards for mathematics which have been used in curriculum projects across North America. The NCME has concerned itself with professional and ethical standards for its members. These initiatives demonstrate the widespread applicability of the principles of standard setting in education.

2. The Canadian Experience

School jurisdictions across Canada also have been embarking on standard setting in response to provincial and local needs. For example, the province of Ontario has established a "benchmark" system as a form of indicator of student achievement. The North York Board of Education (1992) has produced Student Outcomes in Reading and Writing - and other forms of standards which are expectations for all students in reading and writing. The North York Benchmarks program is based on Sadler's (1987) view of standards-based assessment. Using exemplars and verbal descriptions, standards (benchmarks) were established by panels of expert teachers. With respect to the development process, it should be noted that the literacy profiles developed by the division made use of standardized tests and provincial standards (exemplars). This is not necessarily typical of standards-based assessment, however. Perhaps this approach offers promise by using both criterion- and norm-referenced measurement.

The national School Achievement Indicators Project (SAIP) (Council of Ministers of Education, 1994) is an interprovincial initiative which uses forms of both direct and indirect assessment in the performance assessments. Much can be learned from the process which the instrument developers used to create and administer a large-scale assessment project using alternative forms of assessment. SAIP also provides a glimpse of what can be a costly, time-consuming process which requires a high level of expertise and technical competence to administer, analyze, and report. The project shows that unless the results of such efforts are seen to be useful to educators, the efforts may not be worthwhile. SAIP may not become a true standards project unless the results are used, in a meaningful way, to help set expectations for student performance.

F. Summary

The purpose of this paper has been to identify some aspects of the evolution of educational standards. Standards-referenced measurement is a distinct approach to assessing educational attainment; it is an adaptation of criterion-referenced measurement first introduced over thirty years ago. The use of standards emphasizes curriculum-based rather than norm-referenced measurement. Standards are based on the belief that it is possible, and desirable, to fairly and accurately determine what students know and are able to do. Doing so will improve education because it will both improve classroom teaching and learning, and enhance accountability and public confidence in education. Standards incorporate the expertise of professional educators with generally accepted public goals and expectations. Based on a general consensus and implemented consistently and fairly, standards provide the opportunity to serve individual students and to engender the necessary support by the community.

Table of Contents

Setting Standards in Mathematics 
Problem Solving (K-9) in Saskatoon Schools

B. Noonan, B. Hanson - Saskatoon Catholic Schools

D. Holum - Saskatoon Public Schools

A. Introduction

One of the assumptions of standard setting in education is that classroom teachers possess the expert knowledge to establish meaningful performance standards. Following is a description of a process undertaken by the Saskatoon Public and Catholic school boards to involve classroom teachers in setting standards in math problem solving (K-9) based on the new Department of Education curricula for Mathematics.

Problem solving is one of five strands which form the basis for the intended learnings (objectives) for students and teachers. The other strands are: data management/analysis, numbers and operations, geometry, and measurement.

Problem solving was chosen for this experiment because it was seen to be central to the new curricula. In retrospect, the Development Committee probably would have chosen a different strand, one less "cognitively complex"; however, the standard-setting process would be similar, despite the topic under consideration.

The general process, which was perhaps unique to this exercise, was to present the performance standards as "exemplary" problems. The assumption was that it is possible, in principle, to have a math problem (or set of problems) which would indicate the level of performance for any student. This is similar to the concept of exemplary writing skills used in Language Arts standard setting.

B. The Standard-Setting Process


Fifteen classroom teachers representing Kindergarten through Grade 9 were asked to participate in the process. Typically, these were teachers involved to some extent in implementing the new Mathematics curriculum. Consequently, they possessed knowledge of the organization and goals of the curriculum. This was important because the focus of the project was on setting student performance standards based on the content standards presented in the curriculum document.


The process involved teachers working in teams over three half-day working sessions (a total of about 12 hours). Following is an outline of the three sessions.

Ø Session 1 - Presentation of the principles of standard setting and involvement in an activity in which the concept of performance standards (demonstrable learning) was clarified.

Ø Session 2 - The participants were organized into teams based on five grade "bands" (Kindergarten and Grade 1, Grade 2 and 3, Grade 4 and 5, Grade 6 and 7, Grade 8 and 9). Each team was asked to review a bank of problems which would indicate progressively higher performance within each grade band. Somewhat arbitrarily, it was decided that there would be four levels of performance (e.g., levels 1-4) for each grade band.

The specific task was to identify one exemplary problem at each of the four performance levels for each of five grade bands. In addition, because the curriculum presents three types of problems within the strand (understanding/planning, executing, reflecting) there were a total of 60 problems in Kindergarten to Grade 9.

Ø Session 3 - Once standards had been set (problems identified, in this case) teachers were asked to prepare verbal descriptions for each of the four levels of performance. These verbal descriptions became rubrics, which are central to all contemporary standard-setting procedures. This last step was important because it is consistent with the principle that standards (and rubrics) are the tacit knowledge of experts, made public.

C. The Next Step

The next phase of this process will be to field test the exemplary problems, which will provide empirical evidence concerning the efficacy of the standards. This will be done by selecting classrooms of students to actually attempt the problems for their grade "band". Once information on the validity and reliability of the standards is gained, the process may be extended to involve other Mathematics curriculum strands.

D. Summary

The standard-setting process used here provided information which may be of use in future work. Following is a brief review of the process:

Ø Teachers reported that they found the process to be an excellent professional growth experience. It provided a different view of curriculum and implementation.

Ø The process is time-consuming and requires training even for the most expert teacher.

Ø There is a need to clarify different types of standards when conducting the process (e.g., content, opportunity-to-learn, performance).

Ø The process for empirically testing standards needs to be clarified. That, too, may be a time-consuming (and costly) process.

Ø It is not clear what the most appropriate use might be for such standards. Should they be used for:

(1) modifying instruction?

(2) reporting to parents?

(3) adapting the curriculum?

(4) other?

Answers to those questions and others will help determine the future of standard setting in Mathematics.

Table of Contents

Discussion Focus #2: What kind of education standards are desirable for Saskatchewan?

Another theme that emerged at the Symposium was the close link between an individual's view of what standards are and how they should be used and that individual's philosophy of education and world view. For example, people who value competition and see the world as an intensely competitive place see standards as a tool to determine who "succeeds" and who "fails". People who believe that the education system must ensue that all students achieve to a certain basic level see standards as a way of ensuring that teachers and schools have the resources needed to bring students to that level.

How does your view of what standards are reflect your philosophy of education and world view?

Will the current emphasis on standards allow people with different philosophies of education to better understand each other or will it create further divisions?

Types of Standards

Participants in this discussion were asked to consider the three types of standards: curriculum content, opportunity to learn and student performance and to address how each of these types of standards should be developed and who should develop them. Their remarks about each of the three types of standards appear below, but there were numerous comments about the interrelated nature of the three standards. One group pointed out that curriculum content standards must be clearly defined before student performance standards can be formulated. Another said that we've done a better job on curriculum content and student performance standards than on opportunity to learn standards. This group said that opportunity to learn standards may be more difficult to develop than the other two types because curriculum content and student performance standards are implicit in curriculum, and because there are so many things we can't control when implementing opportunity to learn standards.

Curriculum Content Standards

Participants agreed that the development of curriculum content standards should be a collaborative process with involvement of parents, teachers, boards of education, parent councils and Saskatchewan Education. Most said that Saskatchewan Education should take a leadership role in this process and should ensure that all stakeholders, particularly teachers and others who are responsible for implementation, are consulted.

Opportunity to Learn Standards

Most participants said that creating a level playing field for all students is important but also pointed out some of the complexities associated with creation of that level playing field. For example:

Ø It is difficult to define what "fair" is.

Ø Flexibility is essential - how do you ensure that a system is both fair and flexible?

Ø We are retaining students longer which increases the range of diversity.

Ø Integrating challenged students represents many problems for the challenged students, their classmates and their teachers.

One group took a more pessimistic view saying that inequalities exist and that we are unable to create an entirely level playing field.

Participants said that boards of education should play a major role (some said the lead role) in developing opportunity to learn standards. One group emphasized that parent support for this type of standard is crucial. Another said that we must learn from teachers who have a wealth of experience in this area. Several groups mentioned policies of Saskatchewan Education such as the Adaptive Dimension, gender equity and Indian and Métis education and said that these policies are a step in the right direction. One group said that developing opportunity to learn standards is a broad social issue, and that there is a need to involve a range of stakeholders and groups.

Several of the groups pointed out that implementing opportunity to learn standards requires adequate supports and resources. Some groups did not address the issue of how the resources to support implementation of these standards should be provided; some said that the provincial government has a major responsibility in this area. One group said that the reality is that boards of education need to rely less on provincial revenue to address these issues and to do more local fundraising.

Student Performance Standards

Several groups said that before we can set student performance standards we need to determine what a graduating student should look like and asked whether we can agree on this. Participants indicated that curriculum documents shed some light on the skills and abilities that students must have, but that the expectations in curricula must be more explicitly stated if they are to guide teachers in student evaluation. Many participants emphasized that student performance standards must assess the full range of student knowledge, skills and abilities and emphasize problem solving and creative thinking rather than memorization.

Developing Standards

During the second group discussion, participants were also asked, "What areas of student learning do we want standards for? How might we develop common understandings for appropriate classroom assessment practices based on Saskatchewan standards?"

Some participants had comments about the context within which standards are developed. One group said that only after we have examined our purpose for developing standards can we effectively proceed with that process. They went on to emphasize that in the process of standards development, we must always focus on the main purpose of public education - to serve the needs of students. There was also an emphasis on curriculum. Participants said that standards should be consistent with the philosophy and objectives of new curricula. Another group expressed a fear that the effort to measure results in a standard way will rob funds and time from the implementation of opportunity to learn standards, especially in these times of dwindling resources.

Participants had varying views concerning the areas for which we want standards. One group said that we need to determine the minimum standards that apply to all students. Another said that standards should reflect the breadth and depth that is required in each subject area. Expressive, conceptual, attitudinal and knowledge elements need to be addressed. Yet another said that we need standards that take into account academic performance, attendance and aptitude.

There was agreement that standards should be developed collaboratively and that students, teachers, trustees (both division and local boards), other professionals (university faculty), business, industry and Saskatchewan Education should be involved. Development should be structured so that activity takes place on the local, divisional, regional and provincial levels.

Table of Contents

Presentation: Perspective from Saskatchewan Education

Ken Horsman - Assistant Deputy Minister, Saskatchewan Education
The Saskatchewan Education Indicators Program

The Saskatchewan Education Indicators Program is a collaborative initiative of Saskatchewan Education, the SSTA, the STF, LEADS and other stakeholders. The program describes progress towards achieving the Goals of Education in Saskatchewan. Through context, process and outcomes indicators, the program enables decision-makers to monitor the provincial education system in a comprehensive manner. A three-year reporting cycle is being followed with full reports every third year and updates in the subsequent years.

The Indicators Program draws on information from a variety of sources, including:

The Provincial Learning Assessment Program: determines skill levels, attitudes and knowledge of Grades 5, 8 and 11 students in Language Arts and Mathematics, in alternate years.

The Curriculum Evaluation Program: studies the effectiveness of Core Curriculum by examining the extent or degree of implementation, and the impact of curriculum on student learning.

Provincial Student Records System: keeps records of trends in student enrollment patterns and Grade 12 marks.

Pan Canadian Education Indicators Programs: an initiative of the Council of Ministers of Education Canada, it includes cross-country information on student flows, citizenship behaviours, achievement and satisfaction with the education system.

Statistics Canada: this federal agency conducts periodic surveys, census reviews and studies of labour market conditions and youth attitudes and behaviours.

It is certainly a pleasure to receive such a generous introduction. I think the people here today, and last night, have already said a lot of what I have to say. There is such great potential in this audience and in the education system of this province, that no one person can really claim to be an expert. We must involve all of our colleagues.

I want to thank all the organizers of this Symposium. It is very important and very timely that we have taken on the task of dealing with this subject. I would also like to thank Brian Noonan, because I read his paper thoroughly in preparing my remarks, and I am going to draw on it fairly heavily to reinforce those things I feel are important.

The Saskatchewan Education Indicators Program (Continued)

The mission of the Saskatchewan Education Indicators Program is: to collect relevant and appropriate data to support decision making, planning, and policy development at all levels of the education system and to demonstrate public accountability by providing information about the education system to the education community and the public at large.

The Indicators Program, as a matter of principle, will:

provide a broad range or set of relevant and appropriate indicators tied to the Goals of Education;

assist in assessing and evaluating the strengths and weaknesses of the Kindergarten to Grade 12 education system;

support decision making, planning and policy development at all levels of the education system; and,

contribute to informed dialogue about the education system.

Today, I want to do three things. First, I want to review where we've been on this issue of standards, because often we charge off on new directions without having a clear understanding of where we've been. Although we may not recognize it, we have been a great distance already. Second, I want to review some of the basic positions of the Department of Education and to describe what we have done in the past and are currently doing, because a lot of our work is not well known and, perhaps, is not seen in context. Third, I want to discuss with you some other issues and challenges, as I see them. The educational community in this province knows where it's going. We are very highly regarded and we need to continue the task of improving quality and building on what we have.

I want to refer you to the foundation document, the Saskatchewan Education Indicators Program, that was produced by the Department of Education in 1993. It sets out what we call our Indicators Program. What is described in that foundation document is very similar to what Brian described in his paper. I won't read the mission statement of the Indicators Program or the guiding principles, but I want to remind you that we do have a set of Goals of Education, we have a mission statement to guide us in identifying how well our system is doing, and we have a set of principles upon which we base specific activities. We do have things in place, and we can build upon what is commonly accepted by all of us. We have established the framework, even though the language is sometimes imprecise and I have, in the few moments that I have spoken, confused the words "accountability", "standards" and "objectives". This is not confusion on the part of us as educators, but, rather, a reflection of the variety of perspectives that are brought to bear upon the educational system. It's important that we do not think of standards only in terms of student performance.

One of the strongest points in Brian's paper was his summary of the research which described different types of standards, because, for me, that summarizes our Departmental policy. There is more to measuring or establishing standards than looking at one small aspect of the education system. The three types of standards that represent our education system are opportunity to learn, curriculum content and student performance. It is not just how well our students do on Grade 12 exams that represents our system. Grade 12 exams are an important point, but not the only indicator.

Opportunity to Learn Standards

The first type of standard Brian mentioned is opportunity to learn. Let me remind you that we have legislation and policies in place which take care of, for example, the number of days in a school year. This is a very important component of our legislation which provides opportunities for learning. Our legislation establishes a Board of Teacher Education which ensures the quality of teacher training in our province. We currently have major initiatives underway to promote equity, including guidelines to promote gender equity and to reduce racism. These equity initiatives are another aspect of opportunity to learn.

We are not prepared to accept the Swiss education system, for example, which gives women a different role. Or the German system which refuses to use the word multiculturalism. I speak first hand because some of you may know that I have just had the opportunity to represent Canada at the International Bureau of Education, which is the comparative education institute for the United Nations (UNESCO). I spent some time with the German and Swiss delegations and I learned that education is truly a social institution. We can achieve whatever we want in our education system as long as we consider our social structure. We are not prepared to do the things that the Japanese are doing. We have Goals of Education which say we want a broad, general education for all, which is what the Germans and the Swiss do not have. Their students take a test at age 12, at which time about one-third of the students get screened off into an academic stream. The parents I spoke to hire tutors for up to two years to make sure their children get through the test.

I provide this information, not saying it is bad. I provide it saying that education is a social institution, and it must occur within our social structure. But it must also occur within the global context. In Switzerland the 16-to-20-year-olds in the upper stream spend 25 to 40 hours a week in class instruction, when we spend 25. They get three years of calculus before they get out of high school, while our students may not get any and, on the calculus standard, won't do as well. So, how do we deal with that issue?

Curriculum Content Standards

The second component that Brian discussed is the content of the curriculum. Over the last 10 years, we have established an entirely new framework for curriculum in this province, and most of us are very proud of the system we have. Now this is not to say that pride should be blind pride, because our presence here today tells us we must move forward as diligently as possible to make the system even better. We have developed Goals of Education, Required Areas of Study, Common Essential Learnings, and we have established more specific learning objectives. We have curriculum foundational objectives describing what we want to do. One of the issues we have to address is whether our curriculum is appropriate for the developmental ages of children. Do we have expectations for children at certain age levels that aren't really reasonable for children at those levels?

I have talked a little bit about the opportunity to learn and have cited a few examples of what we are doing. I talked a little bit about the content of the curriculum and how we need to debate what's right and what's wrong with it.

2Student Performance Standards

The third component of our system is performance objectives. We have established some performance standards that we are currently reviewing in terms of our Grade 12 exams. We are looking at the courses for which there should be a Grade 12 exam, at which students should take the exam, and at the objectives and processes the exams should be based upon. We have been involved in the School Achievement Indicators Program at the national level, in terms of assisting in the development of more authentic means of evaluation. Saskatchewan has contributed a great deal to a program that we are still monitoring. We should reject the concept that we are doing only a paper and pencil assessment; we are into performance assessment. I think, as well, that in terms of performance we will need to increase our efforts at follow-up. We do all this testing; we get all this information; we've been doing this for three or four years around the provincial learning assessment and curriculum evaluation, and I don't think we've done enough with the results.

All of us are very interested in students' performance and, again, I encourage you not to see performance as the only vehicle for improving the quality of our system or setting standards. Let me remind you that three or four things are happening right now in this province. We have Grade 12 exams for non-accredited teachers. We have implemented a provincial learning assessment program which tests students on a sample basis at Grade 5, 8 and 11 in Language Arts and Mathematics in alternate years. We have begun the process of not only giving the test - measuring where we are - but of establishing standards of performance - determining where we should be.

We also have a curriculum evaluation system. Some of you may know that we've recently published results of the science curriculum evaluation. One of the main conclusions of that study is that our students are not doing well at problem solving. The Department of Education is not trying to deal with that finding alone, rather we are trying to engage our partners in education, the Saskatchewan Teachers Federation, the Saskatchewan School Trustees Association, and the universities, in developing a strategy for improving students' problem-solving abilities.

Improving the System

Here in Saskatchewan, we have addressed the three different types of standards that Brian described. But does this mean we have in place what we need to have? The answer is clearly "NO" - for several reasons! We aren't fully meeting the needs of students. We are trying very hard to do the best we can and we should compliment ourselves, but we aren't doing enough. Many students are dropping out; many students are alienated from the system; many students are not achieving their potential. So it's important that we increase standards, not just in terms of the performance of the student, but also in terms of the performance of the system. I guess that it is when we acknowledge these limitations, we sometimes feel our system isn't doing well.

I remind you that it is doing very well. The polls done recently show that over 80 percent of Saskatchewan people who have children in school approve of the system. I don't know of a profession or institution that has such a high rating. In polls done over a three-year period, from 1992 to 1995, Saskatchewan was the only province in which approval for the education system stayed steady or increased. So our system is being well received, but it needs improvement!

Before I talk about the challenges before us, let me summarize some of the things we have done. We have established guidelines around bias in education, we have increased the number of credits required for high school graduation from 21 to 24. We have a policy of a broad, general education. So let us be clear that we have, over the last number of years, since Directions and Core Curriculum, implemented a number of measures which have, in the Canadian and international contexts, tightened the wrench to increase standards in a traditional sense. These include higher requirements for credits, more testing and so forth.

Challenges for the Future

I'd like to conclude by identifying four challenges I see before us. I think we need to extend our efforts concerning the opportunity to learn. We need to ensure that we have greater equity. We must look toward financial fairness and we must work together to build a better system.

In terms of opportunity to learn, we need to consider the question of consistency. We know that at the university, for example, we can go from one class to another and find less than we would like in terms of consistency. We have to ask ourselves as a K-12 system, "Do we have the kind of consistency we need to have?" We know we have to meet the diversity of students, but do we have the consistency we will need to do that?

So those are some of the challenges around opportunities to learn. We have to continually ask ourselves, "Are the objectives we are setting within our curriculum specific enough and are they appropriate? Are they simple enough without being simplistic"" We must challenge ourselves and be careful we do not look upon the Saskatchewan system in a parochial fashion. We must examine what is happening outside our borders and we must deal with national and international issues. That doesn't mean that we reject them; that doesn't mean that we accept them. We need to analyze them; we do need to be part of the national, international and the global situation. Because I know my two children won't be competing only in Regina, and they won't be competing only in Fort Qu'Appelle; they'll be competing on the national and international levels. Now that's not the only way to consider their needs, but we must recognize that it is one of the realities.

We have made considerable progress in working toward equity, but there is still a ways to go. Equality of opportunity and of outcomes for all students is a major goal of Saskatchewan's education system.

Financial fairness is an important consideration. The Foundation Operating Grant, for many years, has been based on the assumption that we should provide resources in an equitable fashion. We are definitely looking at ways to improve the way the Foundation Grant is structured. We have to look at our time on task. I don't just mean time on task just within the context of the school year and the school day, but as a society - are we putting enough effort into the quality of our education system?

Finally, a fourth point that I would summarize is the challenge to work together to build a better system. It's very easy for us to take positions that are entrenched and it's very easy to move ahead blindly. It is so very important that we move ahead together with cooperation and with understanding.

Let me conclude by saying that we have a quality system and that we definitely need to do more work on the issue of standards. This is important. It is virtually impossible to argue that we shouldn't be involved in the issue of standards. This must be done by considering all points of view and with a broad perspective on the standards that we are talking about.

We must move ahead with the assurance that we know what we are doing. We're not doing this for the first time. We have many years of experience; we have a sound foundation and so we can move ahead deliberately and with confidence. We must move ahead with the student in mind because that is who we are here for. But, there isn't just one view of what the student needs. And, so, our efforts must be aimed at building the strongest system, based on the broadest possible knowledge of what students' needs are.

I hope that my comments have contributed to your discussions today and that we continue through this day and the next few years with very productive activity. Thanks very much!

Table of Contents

Presentation: Perspective from Regina Public School Division

Barbara Young - Superintendent, Curriculum and Support Services

Brian Malley - Assistant Superintendent, Curriculum and Support Services

Myra Froc - French Immersion Consultant

Lori Rog - Language Arts Consultant

Differing Perspectives on Standards

First, let's look at why there is so much talk about standards, both in the literature and in general conversations about what is taught in schools and what can be expected of students at any point in their school career. Standards are important to many individuals and groups and can be interpreted from the perspective of the student, the teacher, the parent, the education system and the community.

From the student's perspective, standards help create independent learners. They tell the student what is expected of him or her. This makes it easier for students to map out a course to achieve the goal and to self-evaluate - to assess whether they have achieved the goal.

From the teacher's perspective, standards serve four functions:

Ø Interpretation and Demonstration ... a constant search. You always look for some sort of standard so that you know where to go with your students.

Ø Consistency When I first saw this sheet on holistic scoring, it was the first time I'd seen the so-called standards ... the sooner we get this implemented, the sooner we can start becoming more consistent as a system and report on a more accurate basis to parents and to the students.

Ø Communication ... we need to get more feedback from all the grades by teachers so that we can get some agreement on whether the exemplars we've chosen represent each of the scoring levels ....

Ø Confidence ... it's given me more confidence in my classroom teaching ... because despite all the pressure you get from the outside on what your students should be capable of doing, I know by the benchmarks that my students are where they should be.

From the parents' perspective, standards provide guidance in three areas:

Ø The meaning of marks - Many parents look at standards in terms of marks. They are used to seeing letter grades or percentages on reports, but are now asking what those grades mean.

Sure, my son got a B - and I used to think I knew what that meant, but does that tell me what he knows? or what he should know? or what he needs to do better or change so that he can get an A? Even if he gets an A, does it mean that he remembers well for a test, or does it mean that he has learned the processes of learning? or both? With his B, does he know how to get the answers in math, but makes careless mistakes? does he only know how to do 70 percent of the questions? or a combination of both?

Ø Comparisons with other students

What should be expected of a child at a certain level? What is the normal range? How does my child compare with others? Are comparisons possible?

Ø How can I help my child learn?

Do I need to help my children at home? If my children are meeting expectations, is help necessary? How do I know if they have fallen behind; or are ahead?

From the education system's perspective, standards help us:

Ø Monitor our progress - How well are we doing? What are our strengths? What are our weaknesses?

Ø Be accountable - How can we be more accountable to our stakeholders?

Ø Set priorities for resource allocation - How do we know where to put our resources? In what area should we offer inservice? What print and non-print resources do we need? What human resources?

Ø Provide meaningful information - How can we give parents more and better information?

Ø Implement Core Curriculum - Where does this fit in the grand scheme of implementing Core Curriculum?

From the community's perspective, standards help us answer question such as:

Ø What does an average of 70 percent mean?

Ø If a student has an A in English, what can the student do?

Ø If I hire a student who has passed Grade 12, what does he/she know?

In the Regina Public school system, we have worked on many fronts to develop standards or to make use of information about standards. For example, we have prepared an education indicators report for our system, we are active in the Standards Network and we offer the International Baccalaureate program. Standards for this program are set in Europe and apply whenever the program exists.

One of the areas of which we are proudest is our work in elementary English and French Language Arts. We have established benchmarks or standards for several grades in this area. A description of this process and its outcomes begins on the next page.

At Regina Public, we have accomplished a great deal, but in a sense our work is just beginning. In the future, we see ourselves addressing issues such as:

Ø working toward common definitions of terms relating to standards;

Ø developing different types of standards - curriculum content standards, opportunity to learn standards and student performance standards are all important;

Ø identifying levels of standards - standards exist at different developmental levels and change through time;

Ø determining who sets standards - this involves issues such as determining when and how to involve parents;

Ø establishing ethical guidelines around the development and use of standards - what is appropriate use of standards? What is misuse of information?;

Ø becoming more precise about the ways in which different types of standards are used for accountability to evaluate student progress, to improve instruction; and,

Ø educating the public about what standards are and what they tell us.

The Development of System-Wide Standards for Writing in Regina Public Schools

Written by Sandra Falconer Pace - Regina Public Schools

Edited by Loraine Thompson


Experienced teachers naturally develop and internalize standards for evaluating student abilities. As teachers become more knowledgeable about writing instruction, they will carry the criteria for writing in their own minds. However, based on the students they have, and their own knowledge and values, each teacher's standards will differ from those of others. Every student has had the experience of getting a grade that differs by 10 percent or more from two different teachers for work of similar quality.

Regina Public Board of Education has begun the process of establishing standards for student writing at various grade levels in both English and French. The purpose of the program is threefold:

Ø to answer questions from parents and the general public about the quality of student writing;

Ø to provide baseline data about the quality of student writing prior to the implementation of new Language Arts curricula which include the teaching of writing processes; and,

Ø to provide a set of benchmarks of student progress in writing skills to guide teachers in their instruction.

An Overview of the Project

Students participating in the project were asked to write a composition in response to a story starter or question. Grade 3, 6 and 7 students wrote narrative compositions, Grade 9 and 12 students persuasive compositions. Students were given several sessions over a period of about a week to draft, revise and proofread their compositions. Then the completed compositions were sent to Central Office for holistic marking by a team of trained markers. During holistic marking, the piece of writing is judged for its overall effectiveness. No single factor is considered more than another in determining a score. Content, grammar, spelling, punctuation, organization, syntax and vocabulary are all judged simultaneously. Each piece of student writing is scored independently by two markers. This ensures that each marker is checked by another person. It is not possible to holistically score students' work without this check, because it is too likely that a single marker will drift off scale.

The markers who are scoring student work have received special training so that they give a similar mark to similar work. Their training focussed on criteria and exemplars. Criteria detail the expectations for each level of writing. Exemplars are pieces of student work illustrating how the criteria will be shown in a piece student work.

Results to Date

Our program began with testing Grade 7 French Immersion students writing in English. Parents had frequently asked if students in the French Immersion program wrote as well in English as English program students when they went on to high school.

The results were clear: students in the French Immersion program wrote in English as well or better than students in the English program. Although there can be many reasons for this finding, it is worth noting that French Immersion students, besides being a somewhat selected population by Grade 7, also have almost double the amount of Language Arts instruction compared to students in the English program. French Immersion students receive Language Arts instruction in both languages. Many skills in Language Arts transfer from one language to another.

The results of testing all of the division's Grades 6, 9 and 12 students (in English) were as follows:

Ø At Grade 6, the median or middle score in the whole range of scores was 3.5 on a six-point scale. Over three-quarters of the students (79.7 percent) scored between 3.5 and 6.0 (the upper half).

Ø At Grade 9, the median or middle score was 4.0 on a six-point scale. Almost three-quarters of the students (74.1 percent) scored between 3.5 and 6.0 (the upper half).

Ø At Grade 12, the median score was 4.0 on a six-point scale. More than half of the students (57.2 percent) scored between 3.5 and 6.0 (the upper half).

In French, Grades 3, 6, 9 and 12 were tested. These results are expected to be released shortly by the school division.

Development of the Program

The first part of the process of setting up a program such as this is to create the scoring guide that will be used to grade papers. The scoring guide is a set of criteria which describe the characteristics of each of six levels of writing quality. They include descriptions of organization, ideas or content, syntax, vocabulary choice and mechanics.

Next, a range-finding process is used to choose examples of student writing at each of the six levels of quality. In the range-finding process, many pieces of writing are read and analyzed in relation to the criteria. Some of the examples chosen are set as exemplars or particular examples of the mid-point range of each level of student writing. These exemplars are constantly referred to during the process of making judgements about student writing in the marking process. Other examples of student writing are also chosen to be used to train the markers. These training sets often contain examples of student compositions which are on the borderline between levels.

Next, teachers experienced at the grade levels being tested are recruited to score student writing. The markers are trained to mark using the exemplars and the criteria. Samples of the writing are read, marked and discussed until the identifying characteristics of the levels of quality are internalized by each marker. During both the training process and the marking process, checks are made to ensure that the marking remains consistent with the criteria.

Markers are trained to avoid bias. Bias can be the result of characteristics of the paper itself or of characteristics of the person doing the marking.

Characteristics of the paper which can generate bias:
Ø handwriting;
Ø lightness or darkness of the writing;
Ø "cute" aspects of the writing (turning the dots on the "i" into hearts);
Ø neatness;
Ø format (skipping lines, unusual margins, etc.);
Ø word processed papers (markers are biased against these, surprisingly); and,
Ø presence or absence of a title or title page.

Characteristics of the marker which can generate bias:
Ø expectations for writing performance which are developmentally or instructionally inappropriate for the students being scored;
Ø personal reactions to the voice, persona or tone of the composition;
Ø reactions to the content of the composition;
Ø idiosyncratic preferences or prejudices in style or usage; and,
Ø assumptions about the writer or testing conditions.

Each piece of writing is read by two markers who independently assign a mark from 1 to 6 according to the quality level of the piece. If the marks from two markers differ more than one point, the head marker reads the piece and discusses the discrepancy between the marks.

Benefits of the Program

The largest benefit of this testing program has been for teachers. Many teachers had already become more knowledgeable about the teaching of writing and were incorporating more instruction in writing techniques into their classroom instruction. For these teachers, the criteria and exemplars provide a standard by which they could explain more clearly to students and to parents the quality of student writing. For teachers beginning to implement the new curricula, the standard created by the criteria and exemplars provide signposts or benchmarks to guide their development of expectations for student writing.

Table of Contents

Presentation: Perspective from Eston-Elrose School Division

Improving Student Learning: An Emphasis on Quality

Jim Gunningham - Director of Education

Let me begin by asking you two questions:

1. Do you want your children to receive As and Bs in their classes?

2. Are you satisfied when your children receive Cs, Ds and Fs?

If this doesn't satisfy you, why should your school be content with it? The sorting and selecting of students according to the bell curve is what many schools still accept as the norm. In today's Information Age, all people must be skilled. Student achievement in a school should reflect the "J" curve.

The research of Benjamin Bloom, which has been replicated many times, showed us a way in which 95 percent of all students can receive As and Bs in classes of 30 or more students. To achieve significant results schools and teachers must:

1. Teach the prerequisites first;

2. Enhance cueing and checking for student understanding;

3. Infuse the higher order thinking skills; and,

4. Increase student time on task.

As a result of this study and subsequent educational research, the focus of our Division's professional development and actions has been on instructional improvement.

An important set of assumptions or basic beliefs underlies our actions to improve student learning.

1. Communities expect schools to fulfill three functions. They provide custodial care, sort and select students, and teach for learning. All the functions are important but the Eston-Elrose School Division believes the primary mission of the division and all of its schools is to focus on teaching for learning.

When the primary mission of the school is teaching for learning, your beliefs and actions must demonstrate that:
(i) All students can learn; and,
(ii) All students will learn.

Motivational studies have all come to similar conclusions. Students attribute their success or failure in learning to their effort rather than to their ability. In other words, kids who learn better do better in school. The key is to have students focus on the task of mastery rather than their comparative abilities. This becomes self-evident watching people play video games.

2. If a school system can clearly specify what it wants to produce, then it is possible to conceive and design a delivery system to produce the intended outcomes. What is worth knowing? When your students leave school, what should they be able to do? Teaching is not a random act; success in improvement hinges on what happens in the classroom. Our Board has defined the outcomes and assessment tools to measure student achievement.

3. In the future, even more than the past, schools will be held accountable for measured results. People judge schools according to results. There are many expectations of schools, but our focus is, "Did students learn what we taught them in our program of curriculum and instruction?"

The area that our school division has committed itself to first, is developing assessment strategies that improve teaching and learning.

Assessment and instruction are an intertwined process. Assessment is not an add-on after instruction, but is part of all phases of instruction, if one expects high standards of learning for each student. Instructionally sound assessment enhances the opportunity to learn by:

(i) guiding instruction by providing data on what students are ready to do and pinpointing what they still need to learn; and,

(ii) building the capacity of teachers and students to improve their work.

What would you accept as evidence of high quality work - 70 percent, 80 percent, 90 percent?

How would you feel the next time you have surgery, if the doctor achieved 75 percent on that procedure? Or if the next time you fly you are told that the pilot received a 60 percent grade for takeoff? What is quality? Quality is the commitment to continuous improvement. Percentages and quality do not mix. What is high quality student work? Individual teachers, when asked to describe high quality work, had difficulty. In most cases the criteria have not been written down before the marking starts. In a research project, a group of teachers was asked to grade writing. The range of grades varied from A to F for the same paper.

In our Division, samples of student work which was considered to be of high quality were collected. Teachers met and reviewed student work to begin the process of developing performance standards and rubrics. Rubrics are the quality criteria and standards by which a product, performance or demonstration will be developed and assessed.

The focus on assessment has brought the importance of accountability clearly to centre stage. We are concentrating on the idea that the responsibility of teachers is to create conditions where students can succeed if they choose to be actively engaged in the learning process. The problem is how to make students accountable for their own success. It is very easy for teachers, because of their concern and strong desire for students' success, as well as parental pressure for high marks, to accept excuses and poor work from students. We believe that young people are in the process of "becoming" responsible. We recognize the problem, but have continually nurtured irresponsible behaviour by giving poor grades, but the same credit to the responsible as to the irresponsible. Thus, a student who knows 50 percent of the work gets the same credit as the one who knows 95 percent of it.

Performance-based learning reports poor performance on a student's part as "incomplete" with zero credit. Students are held accountable for incomplete learning but are invited to improve by demonstrating that learning at a later date. Credit will only be given when learning at an acceptable level is achieved.

Students who understand the content and how to apply it can demonstrate that they have actually learned something. This instills confidence in the students and their parents and shows the credibility of the school division.

Eston-Elrose Student Learning Improvement Plan

Staff of Eston-Elrose School Division

The Eston-Elrose School Division has developed five exit behaviour outcomes:
Ø community contributor;
Ø quality worker;
Ø complex thinker;
Ø self-directed learner; and,
Ø effective communicator.

To be prepared to meet the challenges of the twenty-first century, our students are expected to demonstrate these behaviours.

These educational outcomes become the criteria by which materials are selected, and content, instructional procedures and evaluation are aligned. All aspects of the educational program are means to accomplishing these educational purposes.

Essential skills will be taught and reinforced across all disciplines.

Required Areas of Study

All students are expected to have knowledge of seven Required Areas of Study: Language Arts, Mathematics, Science, Social Studies, Health Education, Arts Education, and Physical Education. Secondary level students shall complete a minimum of 24 courses for graduation. The required program offers all students an education that will serve them well regardless of their choices after leaving school. In each of these areas, it is expected that the student shall convey a deep understanding of foundational and learning objectives using generalizations, concepts, facts and strategies.

The following processes and skills will be taught and reinforced in all subjects and grades as outlined within the curriculum guide of each discipline.

Learning Outcome #1: Effective Communicators

Effective communicators are literate, confident users of language who are able to express their ideas with fluency and clarity.

Effective Communicators:

_ Convey thoughts, feelings and/or beliefs to others with fluency and clarity.

_ Consider the thoughts, feelings and/or beliefs of others.

The following skills will be taught and reinforced in all subjects and grades: reading, writing, listening and speaking.

Learning Outcome #2: Complex Thinkers

Complex thinkers identify, access, integrate and apply available resources and information to reason, make decisions and solve problems in a variety of contexts.

Complex Thinkers:
_ Develop a range of thinking skills or abilities.
_ Display significant reasoning and problem-solving skills.
_ Select thinking processes appropriate to the resolution of a complex issue.

The following processes and skills will be taught and reinforced in all subjects and grades:
Analyzing Part/Whole Relationships
Generating Alternative Solution
Making Sound Inferences
Reasoning Categorically
Supporting Deductions
Uncovering Assumption
Using Analogies and Metaphors
Predicting Consequences
Clarifying Ideas
Applying Conditional Reasoning/Structural Analysis
Supporting Induction

Learning Outcome #3: Quality Workers

Quality workers demonstrate responsibility and quality work in the creation of intellectual, artistic, practical, and physical products which reflect originality, high standards, and the use of advanced technologies.

Quality Workers:
_ Consistently produce quality products that achieve their purposes.
_ Initiate and follow through to task completion.
_ Apply the best possible resources, technologies and organizational strategies to the task.
_ Continually assess, evaluate and adjust to maintain excellence.

The following processes and skills will be taught and reinforced in all subjects and grades:
Organizing and Using Time
Using Computers/Communicating
Assessing/Evaluating Information

Learning Outcome #4: Community Contributors

Community contributors are ethical citizens who direct their time, energies, and talents to improving the welfare of others as an individual within the family, the community and Canada.

Community Contributors:
_ Provide time, energy and talent to improve the welfare of others in the local and larger community.
_ Demonstrate consideration for individual differences.
_ Employ participatory skills necessary to be an effective group member.

The following processes and skills will be taught and reinforced in all subjects and grades:
Intra/Inter-Personal Relating

Learning Outcome #5: Self-Directed Learners

Self-directed learners are independent, life-long learners who value learning as an empowering activity of great personal and social worth.

Self-Directed Learners:
_ Set priorities and achievable goals.
_ Take responsibility for actions.
_ Develop and implement a plan for self-improvement.
_ Apply learning to a new situation of their own.

The following processes and skills will be taught and reinforced in all subjects and grades:
Career Planning

Assessing Student Performance

The Eston-Elrose School Division has explicitly defined learning outcomes and will focus on authentic demonstrations of learning rather than on educational inputs and processes. The chief indicator of the education system's success is demonstrable student learning, therefore, the tools used to measure that learning become very important. The following principles guide schools within the division in responding to the question, "How will student performance be assessed?"

1. Expectations and performance standards are clearly defined. Teachers, students and parents have a clear understanding of what all students are expected to do well. Students participate in the assessment of their work, and progress reporting to parents is ongoing.

2. Students are provided with sufficient opportunities to demonstrate learner outcomes. Assessment information is collected in a variety of ways (written criteria-referenced examinations, performance assessments, and portfolios). Students understand what is expected of them, what assessment information is being collected and how this information will be used. All students are ensured the instructional support and the opportunities necessary to achieve the desired high standards of achievement. Assessment methods are appropriate for, and compatible with, the purpose and context of the assessment.

3. Assessment is based on authentic demonstrations of learning. Authentic assessments require students to perform specific behaviours in a real-life situation. For example, to test a student's essay-writing ability a teacher would ask the student to write one or more essays on topics jointly selected by teacher and student, not to answer questions about essay writing. In science, students' ability to use specific pieces of equipment would be tested by requiring students to perform a series of tasks using the equipment, rather than by asking students to answer written questions about the equipment.

The authentic task itself should provide students with the opportunity to demonstrate what they have learned by using the knowledge in a meaningful context. In this way, the task facilitates further learning and, at the same time, provides the student and educator with data for assessing what has been learned. The goal is to complement, not replace, traditional assessment.

Table of Contents

Discussion Focus #3: What do we need to know? Next steps?

Most participants agreed that the current context of education means that more attention must be given to the development and implementation of standards.

The public has a perception of fog and confusion in the education system. A standards development process may help educators, students and parents clear the fog, increase consistency and instill confidence in our education system.

Standards and Standardization

Standards are never the result of imposed standardization ... standards relate to jobs done well by individuals, as judged within a context of particular purpose and effect (Wiggins, 1993, p. 282).

A person with standards is a person with a passion for excellence and habitual attention to detail in all work done .... High standards, whether in people or institutions, are revealed through reliability, integrity, self-discipline, and craftsmanship - as character in short. Raising standards locally (through changes in the kinds of work and conduct tolerated) will raise test scores. But to raise standards locally, faculties will have "to agree to agree" and benchmark their work against appropriate wider-world standards linked to their (many) institutional customers. This can happen only when teaching is defined to include assessing and when structures are created to compel teachers to talk to one another and agree about performance outcomes and how to assess them (Wiggins, 1993, p. 284).

Most groups talked about the purpose of standards. They said that the primary purpose of standards development must always be improved student learning and improved instruction. Several emphasized that standards are not synonymous with standardization and should not be interpreted to mean only standardized testing. Several also emphasized that the purpose of standards should not be just to compare or sort students.

Most groups listed principles that should guide the development of standards. Principles mentioned frequently are:


Ø should reflect the diversity of the student population;

Ø should reflect the full range of student knowledge and abilities and not be just academic;

Ø must reflect the Saskatchewan and Canadian context;

Ø should be based on the developmental needs of students (Is this appropriate for this age/grade?);

Ø must be linked to the curriculum;

Ø should be designed in ways that get students to take more responsibility for their own learning; and,

Ø must be expressed in non-technical language that everyone can understand.

Most groups talked about the need to develop standards on a provincial basis. They said that it doesn't make sense for each school division to be developing its own standards. Provincial standards were seen both as a way to ensure consistency and to reduce the costs that would arise if school divisions were to work independently.

All groups agreed that the process used to develop provincial standards should involve all stakeholders and several said that this process should be led by Saskatchewan Education. Several groups said that teachers need to be involved in a very significant way because they are the ones who are closest to the students and the ones who will have to implement any standards developed. It was emphasized that communication about the process is the key to creating student, parent and community involvement. Specific ideas for standards development included:

Ø using technology as much as possible to share ideas;

Ø developing tools (like rubrics) that are available to all teachers and shared among teachers;

Ø creating networking opportunities (including more symposiums and workshops) that allow people to share ideas - parents should be included in this network;

Ø establishing a clearing house of ideas relating to standards that are available to everyone; and,

Ø placing greater emphasis on alternate assessment tools (portfolios, performance assessments) perhaps through half-day regional workshops.

Some participants suggested that the development of provincial standards should begin with an area where there is a high level of agreement, like student attendance, and then move on to areas that are more complex. All agreed that developing standards is a process of ongoing dialogue and interaction as well as a program of concrete activities.

Participants at the Standards Symposium agreed that the development of standards should be a collaborative process. Parents, educators, trustees, students, Department of Education and members of the public should all play a role. However, the process used to develop standards and the roles that should be played by each of these groups was not clearly defined?

What role do you personally want to play in the development of standards?

How do you feel that the group of which you are a part should relate to other interested groups?

How can we ensure that the interests and needs of all stakeholders are represented in the standards development process.

How can we ensure that standards are used in a way that reflects the interests of all Saskatchewan stakeholders?

Are some interests and needs of some stakeholders more valid than the interests and needs of others?

Some participants addressed the economics of standards development and questioned whether it will be possible to begin a province-wide process without substantial input of money.

One group pointed out that standards development is really only the beginning of the process. We also have to ask ourselves, "Who is responsible for ensuring that standards are met?"

Several groups said that the next steps should be concrete action to begin the process of coordinated province-wide standards development. Some said that a committee of all interest groups should be struck; others said that SaskEd should assume a leadership role; still others said that the SSTA and SaskEd should jointly be responsible for assuming leadership.

Symposium Synopsis

Rita Priestley - Executive Assistant, STF representing Saskatchewan Teachers' Federation

Preparation of this Symposium began several days ago. Last night I went home pensive: I travelled some distance this morning but came full circle this afternoon.

A couple of lines from a book on futures come to mind:

The farmer sees a very different world from the hunter and gatherer.
The industrialist sees a very different world from the farmer.
The teacher sees a different world from the industrialist.

I went last night to the dictionary to remind myself of the meaning of "epistemology", the part of philosophy which deals with the origin, nature and limits of knowledge. Is knowledge out there or is meaning constructed internally?

I live in both worlds. In some ways, we're struggling here between worlds. We have different ways of knowing - different ways of learning. We live different realities, shaped by gender, family, culture, religion, occupation, and other diverse influences.

I am a teacher. My father is a farmer.

He retired six years ago at the age of 70; sold the farm which had been my grandfather's. His greatest joy and satisfaction was in farming; his greatest sorrow, largely unspoken, was that the had no son, no son-in-law, no daughter who wished to take over the family farm.

I realize I owe much of what I know to be important about teaching and learning to the example my father set as a farmer.

Farming to my father was more than a job, more than work, more than a business, more than a way of life - it was a calling, a vocation.

I have images of him walking the land, reaching down to grasp the soil in his large, callused hand - savouring the scent, the texture, the sweetness, the moisture - assessing the needs and the readiness of the land, making decisions of what work would be done next. His relationship to the land was close to sacred - his prayers - those times which he, in his words, "talked to God," were said in the fields.

He did not look to Japan or to any other Asian or European country to tell him what crops would grow best on his land. He did not list precise outcomes for his crops. He had no predetermined performance goals for the land. He farmed without rubrics.

But he had vision, he had hopes, he had dreams, he had faith. He had commitment and perseverance and integrity. His knowledge came not from books and university courses, but from lived experience, from dialogues with his father and brothers and his friends and neighbours. Through experience, he had constructed a body of tacit knowledge.

He knew the benefits of crop rotation and diversification. He knew the benefits and hazards of burning the stubble, using chemical fertilizers, pesticides. But that's not what I remember.

I remember how each spring he waited in anticipation - ready to get on the land. He watched the sky, listened to the weather, read the forecasts, talked to neighbours. He read the soil - selected the best seeds, and when conditions were right, he seeded. Then he watched the skies, and prayed for rain and cheered the sun and watched his hopes spring into shoots and blossom, head out, and ripen. He was attuned to the rhythms of the seasons.

He brought in the bumper harvests and the disappointing ones. He coped with frost and hail and drought, and looked to the next year.

His standards were internal. His knowledge, tacit.

His cattle were more than assets; regardless of pedigree, each had a name. And the neighbours marvelled that right up until he came to sell off the herd, he could go out in the yard and call the lead cow and she would bring the herd in for milking.

When he came to sell the farm, there were lots of takers; the soil was good, his crops consistently good. He sold not to the highest bidder, but to the young man he thought would best look after the land - after all it represented the investment of my father's life - and his father's before him.

And before he moved, to the amazement and puzzlement of many, he painted and repaired all the outbuildings.

My mother said to me, "Some think he's crazy to be painting and repairing when we're leaving in a few months. Do you understand, Rita?"

"Yes I do," I said.

It was an act of love, an act of homage, an act of respect. It was a handing over from one generation to the next, in better shape than when it had come to him.

To the end he was a steward of the land, steward of the future, understanding his responsibility to future generations.

Teaching is a lot like stewardship, like farming. We grow human beings. The relationship between teachers and students, at its best, is a marvellous, even sacred thing. It cannot be captured in lists of outcomes, in scope and sequence charts, in taxonomies of standards, in rubrics.

There was strong agreement at the Symposium that teachers need to be involved in the development of standards used in Saskatchewan schools. Parents, trustees, educational administrators and teachers themselves all agreed that because teachers are the people who have to actually implement standards, they should play a major role in standards development.

What roles should teachers play in the development of local, regional and provincial standards?

What role does teachers' extensive knowledge of their students and their community play in the application of standards?

How can teachers learn about standards development and application from each other?

Students come to us in different shapes and sizes, with different language and cultural backgrounds, different experiences, different abilities and aptitudes, different readiness and supports for learning. These are the factors which Brian referred to earlier as "unearned advantage and undeserved disadvantage". These differences have a profound impact on student learning. Classrooms which nurture growth are vibrant, robust, alive - sensitive to the differences among the learners.

Accountability is an extremely complex topic. In an environment where the "mantra", "competition in a global economic marketplace" is not challenged, it can be seductive to reduce accountability to standards and standards to performance outcomes for students. Indeed, there is considerable pressure to do so. It was exciting, therefore, to see the holistic approach and the emergent design used by Regina.

As stewards of the future generation, we must ask the tough questions about content standards, we must enlarge upon those which Brian has described as "opportunity to learn standards", for that list is minimal. We need high "standards" (expectations) around the kinds of resources and structures and teaching environments which enable teachers to do their best for all children, regardless of their race, culture, ability, or socioeconomic background, and which enable children to learn.

Children have very little control of their "racial" or cultural background, the language they learn at home, the opportunities for learning prior to school, the income of their family, or the school they attend - over many of the factors which impact on their learning.

Teachers have very little control over their school placement, grade and teaching assignment, class size and composition, classroom and facility assignment, budget, or resources - over many of the factors which impact on their teaching.

To focus on student performance outcomes without adequate attention to opportunity to learn indicators, to other indicators of system performance, is to place the responsibility where there is the least power and to shirk our collective community responsibility to our children and their teachers.

At this Symposium, there has been some mention of content and opportunity to learn standards, but the presentations have primarily addressed student performance standards. To overemphasize the focus and compare students, has the real potential to marginalize those who come with different ways of knowing, different ways of learning - those with backgrounds different from the culture imbedded in the standards. Standards are neither value free nor culture free.

I am heartened by the call to attention to ethics by Barbara, the call for further dialogue in this area. We would welcome the opportunity to engage in that dialogue.

One of the many things I learned from my work with gifted learner education is that it takes an environment for giftedness to show itself.

In my work at the Federation I have come to realize that it takes an environment for professionalism to manifest itself - professionalism grounded in competency, autonomy and responsibility; with professional competency balanced with professional virtue. I am invigorated by the Regina experience - the acknowledgement of the power of professional development reflective groups. The process allows the formation of a community of learners who construct meaning from experience. It promotes ownership and professionalism.

The value for teachers is to engage in the process as an ongoing way of making meaning of the lived experience in the classroom. The product cannot simply be transplanted. How do we create equity in opportunity for teachers to engage in the process, to learn? If we are able to value teachers' tacit knowledge and to make it explicit, we must put in place appropriate processes and structures and provide the time and the resources for teachers to work reflectively and collaboratively to make them explicit. It speaks loudly to budgets in times of tight resources.

If the implicit standards are made explicit, we must recognize the role of culture. Standards are not value free or culture free.

We must attend to different world views, different ways of learning and knowing. We must acknowledge the behavioral and cognitive approaches and the implications of adopting them either as a framework or for curriculum and instruction.

Thomas McGreal at the ASCD Conference in 1994 said teaching is the most complex professional act on the planet. Each day a teacher makes in excess of 2,000 decisions. To do so, she calls upon content knowledge, content pedagogy, general pedagogy, (knowledge of teaching and learning styles, child development, classroom management strategies), and takes into account many contextual factors. Teachers make decisions in action; they learn from reflection in action. Like my father, they are guided by an internal compass - their standards are internal.

It is important for us to remember that the further we get from the classroom, the easier it is to see teaching and learning in terms of charts and grids and matrices.

The child and the teacher in the learning environment is a living system.

Biology offers us different models and lessons than do engineering and physics.

Once you dissect a butterfly and lay its parts on the table, you no longer have a butterfly.

Dennis Tetu - Director of Education, Northern Lakes School Division representing League of Educational Administrators and Directors and Superintendents

One of the major points made at this Symposium is that there is considerable diversity of opinion and thought regarding the issue of standards.

What Did the Provincial Auditor Say?

The spring 1995 report of Saskatchewan's Provincial Auditor makes several recommendations concerning "value for money" accounting and accountability within the education system.

The report recommends:

.19 The Department [of Education] should obtain written agreements with the education organizations it funds setting out clearly the Department's expectations in terms of financial, operating and compliance with authorities objectives.

.20 The Department [of Education] should ensure the education organizations it funds maintain the systems and procedures necessary to meet the Department's objectives and expectations.

.21 The Department [of Education] should receive reports from the education organizations it funds showing the extent to which the Department's objectives and expectations have been met. To the extent practicable, those reports should be verified.

I heard that those within schools do not see the same need for standards for the same reasons as those from outside. Perhaps it is a matter of communication. I heard that parents and other publics are really not aware of what is going on in schools and with students. Maybe the call for standards is a call for an indicator of what our students are learning, and how they are learning it. Maybe standards would do this for us - since parents apparently don't know what is occurring in our schools. As evidence of this, the call for standards often suggests that we base our standards on what was, not on today's realities. The implication for L.E.A.D.S. is that we should work with our teachers and various publics to ensure that an awareness of what is going on in schools today is provided and promoted.

Secondly, I heard that there are many questions regarding what standards really mean. There appears to be a lack of common language about this subject. Some of our external publics are asking for one thing (primarily academic achievement standards) and are hearing something else, or perhaps not even listening. I'm hearing, "let's measure, if indeed we need to measure, the right things for the right reasons," the process of learning versus the products or outcomes.

The Saskatchewan Way has been to do what is right for all students as individuals. I'm hearing that we must remain collaborative and involved when looking at the question of standards.

From Saskatchewan Education (Ken Horsman) I heard comments regarding our Goals of Education for this province, the Common Essential Learnings and other aspects of Core Curriculum. Ken said that we should not focus only on performance outcomes and academic achievement. I'm encouraged by this; yet the standards work to date in elementary mathematics and science appears to have curriculum- and performance-based outcomes. I see government that is very concerned with economics and with the productivity of our province. Within this situation, the role of L.E.A.D.S. is to continue to remind Saskatchewan Education and this government not to slide into a domain where only performance standards matter. Our Saskatchewan Way has been a collaborative way; one in which our education system, with Directions as the map, has reinvested itself in the process of learning, with process skills the focus as opposed to product skills.

With this focus, we are presently quite successful in competing on the global world scene. Therefore, perhaps L.E.A.D.S. and others need to ensure that Saskatchewan Education remains loyal to standards development that stresses the processes of learning.

Lastly, in the presentations, I heard many exciting examples of standards development that focus on the broad perspective - projects that include very well articulated performance outcomes of achievement, blended with processes that help us determine how well children are learning. This activity needs to be communicated to parents and the business community - to make them aware. I believe that this knowledge would be very comforting to parents and others and would satisfy, to a large extent, the drive for performance and achievement standards of accountability.

Regina Public has shown us how to gauge students' work against a benchmark and Eston-Elrose has shown us how improving student learning is the most important issue relating to standards.

Mary Anderson - Trustee, Yorkton Board of Education, Saskatchewan School Trustees Association

Trustees have a very special interest in the issue of standards and accountability, because we're in the middle between parents and schools. Schools, the public and the Department of Education all expect us to be accountable both fiscally and educationally.

I think that most of us here today have come to recognize that we must agree upon accountability measures that will tell us how well students are meeting objectives. If we don't develop such measures, outside pressures will force them upon us. Sometimes, we feel threatened or defensive when the topic of standards is raised; yet we know that we are doing a very good job providing a high equality education. Standards will help us answer the question, "How do we know we are doing a good job?"

Any accountability measures that we adopt must be developed through a collaborative process. They must also be developed with a focus on the student.

In all of our work concerning standards, we need to reaffirm that the students come first. We want equality of opportunity and equality of results. Anything less is a disservice to our students. The last two days we've talked a lot about expectations - how we expect that standards will influence the education system, and what each of us expects our role in standards development to be.

Expectations vary depending on whether you're a parent, student, teacher, trustee or member of the public. We need to share these expectations with each other.

We need to communicate on the issue of standards. We need to talk about what our expectations are and we also need to share information about what's happening in each of our communities. In standards development, the process is as valuable as the product and communication is an important part of the process.

The issue of standards represents an opportunity for dialogue among all those with a commitment to the education of children. For teachers, parents and trustees, it represents an opportunity to ensure continuity, consistency and quality. For students, it represents an opportunity to assume greater ownership over their own learning.

On behalf of the school trustees of Saskatchewan, I would like to thank you for your work, ideas and enthusiasm over the past two days.

Ken Horsman - Assistant Deputy Minister, Saskatchewan Education

These observations are a summary of the major points made at this Symposium.

1. There has been an expression of great interest in the issue of standards as a vehicle to improve quality.

2. Therefore, it is clear that further work needs to be initiated. However, there is less consensus and clarity regarding what should be done next.

3. It is clear, however, that everyone is interested in being involved - professionals and non-professionals.

4. There is considerable tension about what and how to make the next step. For example some want more specifics, less process; others want more process, less specifics.

5. However, there is a willingness to move ahead and we need to do that.

6. Whatever is done, it must be matched against the diverse needs of students.


Berk, R. A. (1986). A consumer's guide to setting performance standards on criterion-referenced tests. Review of Educational Research. 56(1), 137-172.

Biemitter, A. (1993). Lake Wobegan revisited: On diversity and education. Educational Researcher, 22(9), 7-12.

Cole, N. S. (1990). Conceptions of educational achievement. Educational Researcher, 19(3), 2-7.

Council of Ministers of Education, Canada. (1994). SAIP: Reading and writing. Toronto, ON: Author.

Curriculum and Instruction Review Committee. (1989). Directions. Regina, SK: Saskatchewan Education.

Earl, L. M. (1995). Assessment and accountability in education in Ontario. Canadian Journal of Education, 20(1), 45-55.

Fagan, L. P. (1995). Performance accountability in the Newfoundland system. Canadian Journal of Education, 22(1), 65-76.

Glaser, R. (1994). Criterion-referenced tests: Part 1. Origins. Educational Measurement: Issues and Practices, 13(4), 9-11.

Glaser, R. (1963). Instructional technology and the measurement of learning outcomes: Some questions. American Psychologist, 18, 19-52.

Glass. G. V. (1978). Standards and Criteria. Journal of Educational Measurement, 15(4), 237-261.

Gray, W. M. (1978). A comparison of Piagetian theory and criterion-referenced measurement. Review of Educational Research, 48(2), 223-249.

Hajnal, V. (1991). Teachers' attitudes toward elementary and secondary schools: A Saskatchewan perspective. Regina, SK: Saskatchewan School Trustees Association.

Hambleton, R. K. (1994). The rise and fall of criterion-referenced measurement. Educational Measurement: Issues and Practices, 13(4), 21-26.

Hambleton, R. K., & Eignor, D. (1978). A practitioner's guide to criterion-referenced test development, validation, and test score usage. (Draft edition). Amherst, MA: University of Massachusetts.

High School Review Advisory Committee. (1994). Final Report. Regina, SK: Saskatchewan Education, Training and Employment.

Hodgkinson, D. (1995). Accountability in education in British Columbia. Canadian Journal of Education, 20(1), 18-26.

Howe, K. R. (1994). Standards, assessment and equality of educational opportunity. Educational Researcher, 23(8), 27-33.

Jaeger, R. M. (1991). Selection of judges for standard-setting. Educational Measurement: Issues and Practices, 10(2), 3-6, 10, 14.

Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64(3), 425-461.

Kendall, J. S., & Marzano, R. J. (1994). The systematic identification and articulation of content standards and benchmarks. Update. Aurora, CO: Mid-Continent Regional Educational Laboratory. Unpublished manuscript.

Lewington, J., & Orpwood, G. (1993). Overdue assignment: Taking responsibility for Canada's schools. Toronto, ON: John Wiley & Sons.

Lewis, A. C. (1995). An overview of the standards movement. Phi Delta Kappan, 76(10), 744-750.

Linn, R. L. (1994). Performance assessment: Policy promises and technical measurement standards. Educational Researcher, 23(9), 4-14.

Maheu, R. (1995). Education indicators in Quebec. Canadian Journal of Education, 20(1), 56-64.

Masters, G. N., & Mislevy, R. J. (1991). New views of student learning. Princeton, NJ: Educational Testing Service.

McEwen, N. (1995). Accountability in education in Canada. Canadian Journal of Education, 20(1), 1-17.

Meaghan, D. E., & Casas, F. R. (1995). On the testing of standards and standardized achievement testing: Panacea, placebo, or pandora's box. Interchange, 26(1), 33-58.

Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23.

Moss, P. A. (1995). Themes and variations in validity theory. Educational Measurement: Issues and Practice, 14(2), 5-13.

National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards for school mathematics. Reston, VA: Author.

Nitko, A. J. (1980). Distinguishing the many varieties of criterion-referenced tests. Review of Educational Research, 50(3), 461-485.

North York Board of Education. (1992). Student outcomes in reading and writing. North York, ON: Author.

Paris, S. G., Lawton, T. A., Turnes, J. C., & Roth, J. L. (1991). A development perspective on standardized achievement testing. Educational Researcher, 20(5), 12-20.

Popham, J. (1981). Modern educational measurement. Englewood Cliffs, NJ: Prentice-Hall Inc.

Porter, A. C. (1995). The uses and misuse of opportunity-to-learn standards. Educational Researcher, 24(1), 21-27.

Ravitch, D. (1995). National standards in American education: A citizen's guide. Washington, DC: The Brookings Institution.

Reckase, M. (1995). Portfolio assessment: A theoretical estimate of score reliability. Educational and Measurement: Issues and Practices, 14(1), 12-14, 31.

Reid, J. B. (1991). Training judges to generate standard-setting data. Educational Measurement: Issues and Practices, 10(2), 3-6, 10, 14.

Sadler, D. R. (1987). Specifying and promulgating achievement standards. Oxford Review of Education 13(2), 191-209.

Salinger, T. (1995). IRA, standards, and educational reform. The Reading Teacher, 49(4), 290-297.

Saskatchewan Education. (1993). Saskatchewan Education Indicators Program: A foundation document. Regina, SK: Author.

Saskatchewan Education. (1996). 1994 Provincial learning assessment in Language Arts (reading and writing). Regina, SK: Author.

Saskatchewan Education, Training and Employment. (1994). Science 1-5 curriculum evaluation project report. Regina, SK: Author.

Saskatchewan Provincial Auditor. (1995). 1995 spring report. Regina, SK: Author.

Saskatchewan School Trustees Association. (1995a). Great expectations: Opening doors to dialogue. Regina, SK: Author.

Saskatchewan School Trustees Association. (1995b). Public opinions about education in Saskatchewan. Regina, SK: Author.

Simon, E., Sangster, S., Doctorow, R., Breckmar, J., & Hooker, S. (1995). The literacy benchmarks process. North York, ON: North York Board of Education.

Taylor, C. (1994). Assessment for measurement or standards: The peril and promise of large-scale assessment reform. American Educational Research Journal, 31(2), 231-262.

Wiggins, G. P. (1993). Assessing student performance: Exploring the purpose and limits of testing. San Francisco, CA: Jossey-Boss.

Appendix A: Symposium Agenda

Education Standards Symposium
Saskatoon Travelodge Hotel
January 24-25, 1996

Purpose: To provide a forum for examining education standards in Saskatchewan and to share perspectives regarding emerging issues and future directions.

Wednesday, January 24, 1996

7:30 p.m. Registration and Informal Sharing

8:00 p.m. Welcome and Overview

8:15 p.m. Presentation: "Why do we need education standards?"
Dale Botting - Canadian Federation of Independent Business
Meera Patel - Student
Trish McCormick - Teacher
Discussion group focus #1: "Why do we need education standards?"

9:40 p.m. Informal Sharing

Thursday, January 25, 1996

8:45 a.m. Welcome and Overview

8:55 a.m. Presentation: "Setting Standards in Education: Some Principles and Practices"
Brian Noonan, Saskatoon Catholic Schools
Discussion group focus #2: "What kind of education standards are desirable for Saskatchewan?"

10:40 a.m. Presentation: Perspective from Saskatchewan Education, Ken Horsman, Assistant Deputy Minister

11:00 a.m. Presentation: Perspective from Regina Public School Division
Regina Public School Division Staff

12:00 p.m. Lunch

1:00 p.m. Presentation: Perspective from Eston-Elrose School Division
Eston-Elrose School Division Staff

2:20 Discussion group focus #3: "What do we need to know? Next steps?"
Symposium Synopsis
Saskatchewan Teachers' Federation - Rita Priestley
Saskatchewan League of Educational Administrators, Directors and Superintendents - Dennis Tetu
Saskatchewan School Trustees Association - Mary Anderson
Saskatchewan Education - Ken Horsman

3:30 Adjourn

Appendix B: Symposium Participants

Education Standards Symposium
January 24-25, 1996

Teams made up of a teacher, an educational administrator and a trustee from the following boards of education:
Arcola S.D. No. 72
Blaine Lake S.D. No. 57
Eston-Elrose S.D. No. 33
Estevan Rural S.D. No. 62
Gull Lake S.D. No. 76
Ile A La Crosse No. 112
Indian Head S.D. No. 19
Kamsack S.D. No. 35
Last Mountain S.D. No. 29
Lloydminster R.C.S.S.D.
Long Lake S.D. No. 30
Moose Jaw S.D. No. 1
Moose Jaw R.C.S.S.D. #22
North Battleford S.D. #103
North Battleford Cath. #16
Northern Lakes S.D. No. 64
Outlook S.D. No. 32
Prince Albert Comp
Prince Albert S.D. No. 3
Prince Albert Rural S.D. #56
Prince Albert R.C.S.S.D. #6
Regina East S.D. No. 77
Rosetown S.D. No. 43
Regina RCSSD No 81
Regina S.D. No. 4
Saskatoon No. 13
Shaunavon S.D. No. 71
Spiritwood RCSSD No. 82
St. Paul's RCSSD No. 20
Wadena S.D. No. 46
Weyburn RCSSD No. 84
Yorkton S.D. No. 93

Representatives from:

Canadian Federation of Independent Business

Federation of Saskatchewan Indian Nations

Saskatchewan Education

Saskatchewan Federation of Home and School Associations

Saskatchewan Instructional Development and Research Unit

Saskatchewan League of Educational Administrators, Directors and Superintendents

Saskatchewan Professional Development Unit

Saskatchewan Teachers' Federation

University of Regina, Faculty of Education

University of Saskatchewan, College of Education

Table of Contents

Back to: Evaluation and Reporting