On Evaluation

On Evaluation

Educational Solutions Worldwide Inc.

Caleb Gattegno

Newsletter

vol. IV no. 2

December 1974

First published in 1974. Reprinted in 2009.

Copyright Š 1974-2009 Educational Solutions Worldwide Inc. Author: Caleb Gattegno All rights reserved ISBN 978-0-87825-279-4 Educational Solutions Worldwide Inc. 2nd Floor 99 University Place, New York, N.Y. 10003-4555 www.EducationalSolutions.com

The material in this Newsletter falls into three parts, although each is related in our mind to the topic of evaluation, understood as the device to know more objectively whether some educational activities have actually served the students.

At the beginning we print a group of short articles by several colleagues who were recently given the new task of introducing our new product, called Mini-tests, to school administrators. We believe these short pieces deserve publication as human documents that say something about the adventure of marketing a new activity, and also because they add something valuable to the picture of the schools in one city.

We reproduce two pages of a flyer on the Mini-tests to inform readers about the new instrument and to serve as an introduction to the last article of Part 1, written by the person in charge of this project at Educational Solutions. All this material can be seen as a vivid presentation of a new product through the eyes of those selling it while recognizing the approach of the buyers.

The following two articles attempt to state the problem of evaluation in terms of learning. The first concentrates on the evaluation of progress in reading and so links with Part 1; the second, more general, refers to the role of continuous feedback in the evaluation of the acquisition of skills.

We end, as usual, with some news items.

Table of Contents

Chapter 1 ............................................................................. 1 Chapter 2 ............................................................................. 3 Chapter 3 ............................................................................. 5 Chapter 4.............................................................................. 7 Chapter 5 ............................................................................. 9 Mini – Tests ......................................................................... 11 In Each Kit........................................................................................ 12 Chapter 6.............................................................................15 Looking Afresh At Evaluation ............................................ 19 Continuous Feedback And Evaluation ............................... 29 News Item .......................................................................... 35

Chapter 1

During the past month I have learned two important things from my visits to acquaint schools with our Mini-test program: first, that “selling� could be a rich and variegated human experience and, second, that a product which embodies the solution to a perceived need virtually sells itself. Both of these realizations came as a surprise, for my pre conception was one of having to break down walls of indifference and resistance and of feeling unarmored and ill-equipped to carry out such an aggressive task.

District 4 in East Harlem was my first target — though I soon dropped the rather militaristic imagery when I found that, from calls to twentyone schools, I had appointments to visit eighteen. It was not that my salesmanship had been so good that these appointments were relatively easy to come by; it was rather that principals or their assistants were willing to act by hunch or on intuition from scanty information and let the unknown reach them. In fact, of the three schools where I could not get an appointment, only in one was the principal so committed to past plans that nothing new was allowed to enter. The other two principals felt so overwhelmed by financial difficulties that they did not want to be tantalized by anything that would cost them more money. The openness I met in these initial telephone contacts encouraged me to loosen up and be more prepared to meet the unknown too, from my end.

Although each place had its unique style and atmosphere, every school I visited, except one, received me in a friendly and courteous manner.

1

On Evaluation

Once an interview began, it was interesting to see the varied responses to my initial presentation of the purpose and character of the Minitests. For some this broad overview elicited straightaway the response that this was exactly what was needed to meet the needs for preparing students to take standardized tests. For others, especially school reading coordinators, a fairly thorough analysis of the workings of the program was wanted in order to see what would actually be involved in terms of logistics and time. (In some cases I was asked to return and give a demonstration for groups of teachers. ) Many thought that the Mini-tests were a godsend in the light of the recent directives about acceptable practices for preparing students for standardized tests, and several wondered if we had a similar program for mathematics. Those who themselves had been involved in school or district-wide practice material saw immediately how much time, hard work and precision had been required to develop the Mini-tests. They respected the efforts we had made and saw the program as a vast improvement over previous haphazard and incomplete attempts.

Especially appreciated was the continuous feedback on all levels built into the program.

In summary every school but one I visited in District 4 and the few schools I visited in other districts thought the Mini-tests idea had merit and would be useful. Many schools were hampered by lack of funds from ordering as many kits as they would like, but some principals thought it worthwhile to approach the P. T. A. for money. One school even proposes to raise money by having a bake sale.

Every visit has shown me some new aspect of the experience of selling Mini-tests — either a better understanding of what the program offers, or some new detail that had escaped my notice, or most important other dimensions of meeting new people in new situations — and I am grateful for what each one has taught me.

Zulette Catir

2

Chapter 2

One needn’t be a marketing research analyst to be struck by a remarkable fact: in the thirty visits I’ve made during the past several weeks to show people the Mini-tests program, the response from all thirty has ranged from interested to enthusiastic. These visits, with teachers, assistant principals, principals, reading coordinators on all levels, as well as two deputy district superintendents, have taken place in (six) school districts in three boroughs (and one Long Island community). The neighborhoods have ranged from impoverished to wealthy, the schools from those which show lowest on standardized reading tests to those which show highest.

The fact that the Mini-tests have been universally well received raises an interesting and, perhaps, important question: Why does this product sell so well?

It is easy to assume that Mini-tests are popular because teachers and administrators on all levels are evaluated (at least in the minds of many) according to the results on reading achievement tests by the students in their charge. And the Mini-tests seem to be a viable approach for elevating test scores in a way which doesn’t violate officially condoned procedures for preparing students in test-taking skills. No doubt, this advantage of using the “Minis” does account for a portion of their popularity.

Nonetheless, a number of statements by several of the persons with whom I met point to deeper and more far-reaching concerns which

3

On Evaluation

seem to be widespread, yet easily overlooked. One deputysuperintendent said that the materials and suggested activities would offer teachers an excellent opportunity to work with their students on finely-tuned analytic thinking. His district has always scored at or near the top on the standardized reading tests, so his primary concern was clearly not with raising scores. And yet, even in areas where scores were closer to the other end of the scale, these same sentiments were often expressed. One principal in such an area told his staff that the “Minis� would allow them to foster important cognitive skills, a chance they shouldn't pass up. People who stated flatly that the reading scores were not important to them pointed out that, since standardized tests appear to be a permanent fixture in our way of life, the work on basic skills accomplished by the program could prepare children to be relaxed and confident in a challenge they would have to meet often in their lives. Also noted by many were the detailed analyses of the misleading aspects of questions that occur frequently on all types of standardized instruments. At last teachers would be clear enough to work with great precision on the problems at hand.

Clearly, in our efforts to serve the educational community, we have developed in the Mini-tests program another tool which meets, in a more adequate way, a number of persistent challenges teachers have to face. What’s especially nice is the fact that the solution, in this case, is relatively obvious, making it immediately attractive to a large number of those who could be served by it.

Ted Swartz

4

Chapter 3

For the past two weeks I have met with principals, assistant principals and reading coordinators in nine schools in Upper Manhattan’s District 6 and 5 schools in District 11 in the Bronx. My purpose was to acquaint the school staff with our new product, the Mini-tests. These tests evolved from our work in reading with teachers who felt that even those children who were reading on or above grade level were being jeopardized by the other-than-reading demands of the standardized tests. The Mini-tests were designed to give children practice in the skills of test-taking and to reduce the trauma of the particular standardized test.

On the average, I spent one half hour per meeting with the school administrators. The variety of responses has given me some insight into the diversity within the school system. Here’s what I mean. On my first visit, for example, I met a principal who declared, “I’m not interested in preparing children to take the standardized tests. The scores mean nothing to me and I’m neither impressed nor threatened by the New York Times assessment of which school has the best readers based on these scores.”

This was the exception, for in the other schools most of the reading coordinators were very interested in what we had to offer, and some so concerned with the advent of the standardized tests that they had already taken steps to produce their own materials. While it is true that the standardized test is an inadequate measure of a child’s reading ability, it is a reality. We at Educational Solutions know that there is no

5

On Evaluation

reason why children cannot successfully master the challenges presented by the tests once they know what the demands are.

What became most apparent to the staff members when I showed them the Mini-tests was that while their preparatory materials were merely shorter versions of the actual tests, ours were educational instruments.

Through a careful analysis of the format, as well as the types of errors children make, we have reduced the variables within any one test.

Because each test is restricted in this way, the children can focus on specific skills and actually learn from the test.

Aside from the test itself, what most impressed the majority of administrators, were our suggestions for ways in which the teachers could work with the children on the awarenesses required by each question. Although one school’s staff members felt that their children could not develop this kind of “discerning eye,� our emphasis on making children aware that tests use such devices as: equivalent expressions, circumlocutions, opposites and synonyms, to trick them, was felt by most to be the essence of test preparation. Finally, the provisions for self-evaluation which allow children to take full responsibility for their own progress was considered the single most valuable contribution to their education outside the demands of any specific test.

Sally Koiker

6

Chapter 4

Because I had some acquaintance with District 12, while serving as a teacher-of-teachers for three years at the Twin Park School, it seemed a good place for me to begin introducing principals to the Mini-tests.

Identifying myself on the telephone as a member of Dr. Gattegno’s staff, I was warmly received and all but three principals gladly agreed to set up an appointment with me.

The administrators' responses to the Mini-tests were so varied. In one school the principal arranged for me to meet with a team of 12 specialists in his school. Only after one-and-a-half hours of questioning did they agree unanimously to order the Mini-tests for their entire school population. At the next meeting, the principal thumbed through the notebook filled with examples of the Mini-tests and asked for the price list after three minutes. She recognized at once that the kit would serve her students and did not ask for any explanation from me.

The professionals who examined the Mini-tests expressed their delight in such different ways. At one school, a reading specialist said: “I see that you’ve followed a diagnostic-prescriptive formula. That is very compatible with how we work here.” Another corrective reading teacher commented: “Your teacher’s guide states the objective and then gives suggestions for instruction. That’s good for us because we are an objective-oriented school.” At another school a principal commented: “That's good that you include transparencies in the kit, our overhead projectors have just been gathering dust!” While at the next school the assistant principal suggested that they purchase replacement kits since

7

On Evaluation

the transparencies were intended for instructional purposes and he was primarily concerned with giving students practice.

While the administrators responded differently to the details of the Mini-tests, they all expressed their perception that it is a thorough, organized treatment of a particular task which they must meet in their schools. Many volunteered the comment that they had worked on preparing similar materials but that they had not managed such a careful, detailed approach to preparing students for standardized tests.

In one school, the principal introduced me to his staff of specialists in the following way: “Miss Mitchell, you have two strikes against you. We don't like Gattegno and we think the Twin Parks School received special treatment (i.e. more money) in this District. * But let us see what you have.” After examining the Mini-tests and questioning me for about an hour, they concurred that the Mini-tests provided a more precise preparation than they had managed in the past. It turned out to be the biggest order I’ve had. The Mini-tests, indeed, sell themselves.

Katherine Mitchell

*

Educational Solutions has been working at the Twin Parks School for three years and Dr. Gattegno was chief consultant.

8

Chapter 5

I’ve spent part of the last month visiting thirteen public junior high schools, twenty-two public elementary schools and one Title I District reading coordinator, in communities in the Bronx and Brooklyn including Riverdale, Bedford-Stuyvesant, Park Slope, Brooklyn Heights and one school on Manhattan’s upper east side.

Following a telephone conversation with the principal about the “Minitests,” a visit was arranged with him or his assistant. In every case except two I was well received by one or more of the school’s administrators and spoke to them about “Mini-tests” and other Educational Solutions services for fifteen to forty-five minutes.

Although one or two in fact turned out not to be interested in looking at the “Mini-tests,” the remainder had their eyes open and knew in a few moments that they were looking at a thorough solution to the problem of preparing children to do as well as possible on standardized reading tests. In general they were aware of the limitations of these tests and glad to find materials which directly address these points. Many told me of, or showed me their own efforts at a solution and were very appreciative of the completeness and reasonable price of the “Minitests.”

On several occasions I felt I might have made a bigger sale if I had given a slick, higher pressure presentation, but since the “Mini-tests” really sold themselves I was happy to watch my customers evaluate them and decide themselves whether to buy and how many. About a

9

On Evaluation

third of these people were further interested in hearing about “Words in Color,” usually as a “reading system” they had heard of, or briefly seen, but had not been moved to investigate themselves. Often someone passing through the room or past the door would recognize the Mini-charts and comment on them.

Steve DeGuilio

10

Mini – Tests

. . . . . . for understanding Achievement Tests in reading and gaining accuracy and speed In their concern about the progress of great numbers of students, school systems generally mandate some kind of district-wide testing, using tests from which it is possible to derive statistical comparisons of students, classes, special programs, schools or districts.

All such standardized reading achievement tests have marked limitations. Their validity as tests of reading competence per se is limited because they test reading competence along with several other factors affecting the student's reading comprehension which have to do with thinking and speaking rather than reading, such as-. the extent of his spoken English vocabulary the extent of his life-experience his facility in test-taking skills his ability to make inferences from incomplete data Their reliability is limited because they base their measurement on the student’s performance under unnatural circumstances on one or two days during the year. They are not even useful tools for teaching testtaking because the student never gets an opportunity to look at the corrected test and learn what he can from his errors.

11

On Evaluation

Many teachers and administrators who understand the limitations of these tests, but accept that for now they must be given until we have something better, have asked insistently for a way of preparing their students for these tests which will lessen the possibility of frustration 0r trauma, free them to reveal as accurately as possible their actual level of reading competence on the test, and make accessible to them whatever benefits such tests can have.

To this end Educational Solutions, Inc. has designed the MINI-TEST program for assisting teachers in working on student awareness of and facility in meeting the types of demands presented by all achievement tests: specific skills needed only for test-taking special types of comprehension focused on by most tests basic vocabulary expected at their grade-level (not learned before) the format used by most tests (currently mini-tests reflect the format of the Metropolitan Achievement Test used by so many city public schools) In addition the Mini-tests can: assist teachers in diagnosing difficulties in reading enable a student through feedback from his own tests, to compare performance on different days and thus become a better test-taker.

In Each Kit NOTES FOR THE TEACHER providing a precise step-by-step guide for those activities which (1) generate awareness of test-taking skills, and (2) develop facility in test-taking.

PLANNING SHEETS indicating the order that the tests should be given depending on the level of competence revealed as the tests progress.

12

Mini – Tests

TRANSPARENCIES of all “starter” and some “practice” mini-tests to facilitate class discussion either before or after taking the tests, plus a TRANSPARENCY PEN. MINI-TESTS ON SPIRIT STENCILS to produce enough copies of each test for the students in the class. Each is a self-contained one page test which takes most students less than five minutes to complete. In every kit, there are four “starter” and four “practice” mini-tests for each type of question the students will meet on the achievement test for their level.

STUDENT ANSWER SHEETS (on spirit stencil) for recording separately answers to the Intermediate and Advanced tests.

ANSWER KEYS for all tests, for use by students or teacher.

STUDENT SCORE CARDS (on spirit stencil) for students to monitor their own progress.

CLASS SUMMARY SCORE SHEETS for each level of test.

Guide for introducing BASIC VOCABULARY used in achievement tests (“Explorer Plan”).

GRADE

LEVEL

BASIC KIT

REPLACEMENT KIT

1-2

Primary

34 stencils/20 transp. 35.00

33

stencils

22.00

1-3

PrimaryElementary

43 stencils/22 transp. 40.00

41

stencils

27.00

3-4

Elementary

17 stencils/10 transp. 20.00

17

stencils

13.00

5-6

ElementaryIntermediate

25 stencils/12 transp. 26.00

25

stencils

19.00

5-6

Intermediate

20 stencils/10 transp. 21.00

20

stencils

14.00

7-8

Intermediate- 25 stencils/12 transp. 26.00 Advanced

25

stencils

19.00

7-9

Advanced

20

stencils

14.00

20 stencils/10 transp. 21.00

On Evaluation

NOTES:

1

The number of mini-test stencils and transparencies varies according to the levels represented and how many types of questions the various levels include.

2

Any of the Basic Kits can be shared by up to four teachers if each has his own set of stencils.

14

Chapter 6

It has been a good experience to visit with principals and reading supervisors — and in some cases their teachers — about our Mini-tests which assist teachers in preparing students for standardized achievement tests in reading. One thing was clear in meeting all of them: they recognize already that such tests present a special problem to students that is not solved by their gaining greater and greater competence in reading. Only one principal said that the problem was solved in their school and that the children scored third in the district, while one principal (in a different district) told me he refused to waste the children’s time on such preparation as he did not care how they scored on the yearly test and that he was not judged by his district superintendent on the basis of these yearly scores. But with all the others the most frequent comments — after looking over our Mini-tests — were •

we’ve tried to do something like this but our effort is much less thorough and precise than what you have done;

•

this seems to be just what we need;

•

these can save a dedicated teacher hours of work she would otherwise give to preparing similar, though less adequate, materials;

•

I can see that the children could enjoy working on these.

At first the main concerns that most principals and reading supervisors voiced were (1) how can I find money to buy some kits, and (2) how 15

On Evaluation

many teachers can share one kit and still get the benefit, since money is so scarce?* Then after the new directives were issued by the Director of Evaluation for New York City some raised questions about whether our materials •

respected these guidelines which allow one to work on the mechanics of test-taking without working on any of the actual content of any of the existing achievement tests;

•

would be helpful in preparing the students for the new New York City Achievement Test to be given in 1975 about which there is very little information so far, since our Mini-tests follow the format of the Metropolitan Achievement Test formerly given.

One principal questioned if there would be a problem because her students were being taught reading by the Distar method which employs a script somewhat different than the usual.

When I said to this last principal that in any case her students had to learn the regular script in order to take the test, she began to see the Mini-tests as a vehicle for taking care of the transfer to the regular script at the same time they allowed the students to be prepared for the test-taking skills, recognizing that for the slowest moving children the work on them could be concentrated near the actual testing time in order not to require the transfer until it was actually necessary.

To the principal whose students score well I was able to suggest that some might even score better if they were clearer about how the tests were deliberately constructed — especially since the curve on which the grade-level assessments are made is such that one mistake can make a grade-level difference with competent students. To the principal who was not concerned about how his students scored I was able to say that for as little as a half hour a week starting in January these students might be made to feel much more at ease on the testing day and have

*

Three teachers can share a kit.

16

Chapter 6

also a higher score on their permanent records which might sometime matter to them and their family.

To the principal concerned about putting too much pressure on his students already being tested weekly by the reading program they were using, I was able to say that the Mini-tests would become intellectual games which the whole class enjoyed together and looked forward to playing — and that the time required was as little as a half hour a week if the work was spread over two or three months.

In answer to the more basic questions asked by a number of principals, I was able to say the following: 1 In our material the “acceptable practices” outlined by the New York City Board of Education are implemented and none of the “unacceptable practices” are suggested.

2 So far as the change to a new test, in New York City the awarenesses needed for the test–taking that we have worked on through the Mini-test are those common to all of the achievement tests that have so far been developed even though the format differs slightly from one test to another. For instance, all achievement tests •

Work on vocabulary by asking for the matching of words with pictures, or with phrases or other words;

•

Work on reading comprehension through having student read paragraphs and find answer to questions on the basis of explicit, implicit or inferred data and/or have students read a story and choose words to complete it in some way.

This means it is only the few differences in the format for these types of questions on the “new” test which may need to be presented once they are known.

Some achievement tests (like the Stanford at some levels) add sections that require awareness of the component sounds or syllable of words.

We are now preparing some additional Mini-tests and brief notes for 17

On Evaluation

teachers on these types of question so that the awareness of the students may be focused on the demands of these types of questions such as: •

reading a word quietly to oneself and hearing its component sounds well enough to recognize them in other words read in the same way;

•

Saying two or three syllables in varying orders and recognizing if they from an English word or not.

Dorothea E. Hinman

18

Looking Afresh At Evaluation

The ease with which educators all over the world, and more particularly in North America, have accepted the notion of objective testing as a means of evaluating what goes on in schools tells us that 1

they are not in contact with their students’ learning, which may be going on all the time, and,

2 that they cannot conceive of their own way of assessing students’ progress in learning.

They feel the need for help and it seems reasonable to them that outsiders should deal with the matter, provided that no blame be put on them for the bad scores some students will make.

The conception that only students must be blamed for their lack of achievement, and that only they should fail, was accepted by educators when they were themselves students. Since the students were asked to acquire those parts of the knowledge accumulated over the centuries that qualified them to join the managers in their culture, it seemed clear that the effort must be the students' and that it was their responsibility to show their success in retaining as much as possible of what they were exposed to. Tests betrayed philosophies of education as well as the standards erected by societies. Tests were door-openers, justified because the Establishments could use them to recruit their servants (civil and trade). When the democratic principle became more widespread it became obvious that tests were biased in favor of certain groups and could not be used universally yet fairly. Rather than abandon them, testmakers embarked on amending them to reflect 19

On Evaluation

social conditions more accurately. But they did not think of embarking on the production of tests connected to the demands of learning on every child so that they would evaluate the points that children had reached on the way to mastery and force teachers to do the right things.

Perhaps educators even doubt that this is possible. Historically we entered upon teaching without studying learning in detail. Efforts to make teaching more functional resulted in many bright ideas, such as the invention of chalkboards, notebooks, books, etc., that relieved the memory and made it possible to illustrate and exemplify.

Still, skills are only acquired by practice and by the extension of mastered skills to conquer new ones. There is no point in learning to use a pair of compasses if no circles have to be drawn, or to hold a pencil if one neither writes nor draws!

As soon as we take the point of view of the learner we find that it is very easy to know exactly where anyone is in terms of determining 1

what cannot yet be done,

2 what can be done with some difficulty, and 3 what can be done with ease.

It is therefore possible to design and produce instruments that indicate for all to see that this person is precisely at this point in the study in question.

For example, in the study of reading, since learners must first make sense of the conventions of writing, we can associate with this level of reading (which we will call RO) an instrument (called EO) which tells us which of the conventions have been mastered and which remain to be mastered. The learners must know that sounds are represented by signs, different sounds by different signs and, in the case of English, with a double ambiguity: the same sign can be sounded in several ways, and the same sound can be represented by several signs. These awarenesses are not yet sufficient for the correct writing of a “nonphonetic� language such as English. Hence a third requirement must 20

Looking Afresh At Evaluation

be laid on the learners, that of being able to recognize all the words of English, or even perhaps to write correctly all the words of English.

We call R1, R2, R3 the three levels (and therefore the meanings) of reading that are covered by these multiple awarenesses and facilities.

R1 is established when, with a very restricted language (part of written English), learners know that the sounds they utter when they respond to sequences of subsets of these signs, belong to the language they know, and also trigger a meaning for them. For example, if the choice of a restricted language consists of a as in at, u as in up, i as in it, e as in pet, o as in pot, p as in pup, t as in tat, s as in is, s as in us, followed by ss as in pass (a choice made in Words in Color), the learners can read any of the few hundred statements that can be made with these signs, of which pat is upset, step it up, stop it, tess tests spot, are a few.

To R1 corresponds an evaluation instrument E1 which will inform us 1

whether the learner sees each word as a combination of signs-for-sounds, the signs starting from the left while the sounds melt into a continuum;

2 whether he utters just the proper sounds, neither omitting nor adding any; 3 whether he can utter a succession of words, written from left to right, preserving the sound of each but running some together and separating others with a pause, as required by the conventions of English speech; 4 whether a sequence of written words each of which he knows, becomes a statement carrying meaning as it would if he heard it spoken.

To be able to assert the learner’s mastery in E1, we present him with a page of statements like the above upside down. If each statement is read as if it were presented normally we can say that E1 permits us to conclude that the level R1 has been conquered. We can then proceed to level R2 whose end is reached when all the sounds of the language have been met.

21

On Evaluation

E2 is formed by extending E1 from the restricted language of R1 to that which covers all the sounds of English. There could strictly be a succession E1, E , E . . . . . . E1 (x) — the last being also named E2 — each differing from the previous one by including one new sound.

Clearly as we follow closely the advances in reading that result from 1

the extension of vocabulary following the introduction of a new sound and a new sign, and

2 the corresponding extension of the possible statements in number, length, variety, we are offered opportunities to include in the instruments of evaluation new components that were not easily distinguished earlier while maintaining all the exercises that can automatically be extended.

At the end E2 will cover very different ground from that of E1 and indeed appear to be a test of a higher mastery in the field of reading.

For example in E2 we can include any number of statements in which one and the same letter represents different sounds, as for example “all was dark as many hares raced around the village swamp,” or “a meringue-machine signed a girl up to give away onions,” or “the pretty officer said to her, “hello there” or “in the future pure language will be put into us and busily buried in further rules for its use” or “women come home to pots and pans, work for a doctor, sing in a choir once in a while; a solitary woman is a rarity.” But we leave for E3 statements that cover all the spellings of one sound.

Since hearing has been present so prominently in R2 and E2 we can test its mastery by exercises 1

that prove that listening is used as well now as it was in early childhood, and

2 that the mind of the student functions with respect to the demands of the written language in the way that this requires in addition to the already mastered demands of the spoken language.

22

Looking Afresh At Evaluation

The oral dictation, with no repetition, of sentences of increasing length will verify 1

that the student can take in a number of words,

2 hold them in the order received, 3 evoke them again in the same order, and 4 transmute these words into the proper written sequences by involving imagery, its translation into the muscular orders that guide the marks made by the writing tool on the paper, a check by the eye to see that what has been written is what was required.

E2 therefore is a test of mastery in detail of the functionings involved in reading up to the level of an encounter with all the sounds of English.

If the test is not concerned with the choice of vocabulary there are no grounds for being concerned with whether or not students comprehend what they read. But if the vocabulary has been screened to cover the many thousands of words of the spoken language of the testees then it is fair to see to what extent the written language is already an equivalent system to the spoken one.

We must therefore consider how to be fair to the students and how to be sure that comprehension is being tested and nothing else. There are obviously two comprehensions: that used by people in their oral intercourse and covering all sorts of subjects of everyday life, mostly understood by most people, and the one that is involved in scanning a written text. Since intonation, the presence of voices, generally makes the first quite easy, we can test comprehension of a text that is read aloud with respect for the phrasing and the melody of English. A further test can start with silent reading of a text that would yield its meaning or meanings at once if read aloud, and treat it in the same way as the first; and a third test would require testees to study a text silently and answer questions about it in writing.

Similar treatments will be used in E3 for R3 with the one difference that all the irregularities of spelling will be allowed in the presented texts.

23

On Evaluation

The comprehension of texts that cover everyday experiences of the testees comes from the actual reading powers of the students since reading transmutes the printed texts into spoken statements.

Comprehension is often confused with another process (which we call R4 and for which there is an evaluation E4) that concerns itself with acquiring new experience from reading a text. Such an activity implies R3 but also assumes that writers have done their part to make everything they write capable of being deduced from the context or able to generate unambiguous images. Since we must consider R4 separately to do justice to it we shall not enter this field here. Testing this kind of comprehension has to follow a test that shows that students recognize all spelling variants and do not utter mute letters, thus missing a meaning that their spoken language would have yielded.

In our way of working, in order to be fair to the students, we must see to it that they have access to all the spellings of English words organized in such a way that they see them not as a list to be memorized but as a display of writings for definite sounds. My English Fidel gives this awareness.

In order to gain facility a number of exercises have been designed as part of my Words in Color scheme. In E3 we use, among others, the following two kinds of tests.

1

Find examples of English words showing 8 or 9 different phonetic interpretations for ea or ou, say. Or: How many vowel sounds can be associated with ough? Give all the examples you can think of.

2 Each column of the Fidel presents a number of spellings for one and the same sound. To imagine statements containing more than one is a challenge for anybody and therefore says something about the familiarity the students have with the English language. For example, taking all the spellings: “pat laughs under a plaid eating a meringue” “John’s honor was not acknowledged”

24

Looking Afresh At Evaluation

“the sergeant in the bazaar said “sweetheart” my father replied “ah!” “the girl put the pearls over her fur saying courageously “my word!”

As the two fundamental ambiguities of English have now been thoroughly examined the rest of the evaluation of R3 consists in estimating how much of the student’s familiarity with spoken language has become part of his written language.

It is clear that most speakers can use their speech to express most of their experiences that can be expressed in words. But because speech has become second nature its users rarely note things about it. For instance, in speech we readily pause for a short time between clauses but do not necessarily know which pauses are to be represented by a comma, a semi-colon or a colon. What does one need in order to convey (in written English) a question, an exclamation, a hesitation, silence in the middle of a sentence, the reference to an agreed definition that is needed to clarify meaning? How does one convey that a statement is finished?

Other conventions, such as capitalizations in the case of English — which may differ from German or French — are part of the written language but cannot be linked with an awareness of what one does when speaking. These can easily be included as incidental learnings that are not concerned with the mental qualities of the students but only with items deposited in the memory.

R3 may include exercises in the language that are connected with awarenesses of how English behaves even if the comprehension of texts is restricted to what the students can understand from their spoken language. For instance it is possible to test that students 1

know whether a number of given sentences can be considered as “belonging,” i.e. making sense together, or as “not belonging,” i.e. not generating any meaning if said one after the other in any order; 25

On Evaluation

2 know that words in some sentences can be omitted without altering the main meaning. This can serve as a basis for ensuring the deliberate cultivation of brevity, precision, conciseness of expression, increasing the chances of verbal communication; 3 know that in a paragraph not all sentences are equally necessary to convey the main meaning. This can serve in the production of “precis,� first of paragraphs, then of statements of various lengths; 4 know that it is also possible to increase meaning by adding words or sentences to existing ones. That it is almost always possible to re-open a subject that was apparently closed by adding a statement which requires a sequel.

Testing all of these cannot be done within the format of existing reading tests. A new format will be necessary. This we shall examine somewhere else when we offer our solutions for doing justice to students’ real share in their education in contrast to that of the many other components which are not paying for their failure as the students have to.

Comprehension in the context of our proposal for tests up to the level of R3 will be covered if students write texts that are then submitted to their peers (with some editing by specialists) and about which questions are asked that are meaningful to them with their interests at their age.

By reducing the weight of vocabulary, by distinguishing various familiarities with the medium under the headings of sound, stress, phrasing and intonation; spelling, punctuation, capitalization; the structure of sentences, the need for precision, for the right number of words to convey ideas, we are stressing the linguistic education of students. Indeed we can now see that the awarenesses of phonology, of syntax, of spelling, that are required to transmute the written text into a spoken discourse and to extract from it the meanings associated with it, will ensure comprehension at the R3 level while we are preparing the ground for R4 as the way to acquire information via the printed word.

26

Looking Afresh At Evaluation

Hence rather than test how students score on selected texts and sets of chosen words that may have little to do with their actual life, we now propose to reach an exact picture of how deeply each student has penetrated into the written language whose components have been sorted out and made accessible in exercises linked with his use of the spoken language. We thus guarantee that students’ experiences and awarenesses are given at least as much weight as current tests give to the a priori level of achievement represented by lists of words and selected passages. By restoring this balance we have at least stressed the right of testees to their own experience.

Caleb Gattegno

27

Continuous Feedback And Evaluation

It seems so obvious, when we want so much to individualize instruction that we should look for processes of evaluation which can tell us all the time where each student is. The usual once-a-year or even once-a-term test may suggest too late what needs to be done to put things right in individual cases.

Of course the real way to meet the challenge is to see to it that each student can, and does, chart his progress all the time for himself. To achieve this we have three tasks: 1

to make each student sensitive to his own learning and motivated to consider it objectively;

2 to prepare an instrument that can be easily used and can easily display what is important in the everyday course of learning and that automatically produces features that take account of weeks and months of continuous involvement in that learning; 3 to make students competent very quickly in the use of the instrument so that they have a set of immediate feedbacks reflecting the learning as it goes on.

At the level of the elementary school, as observation of children’s games shows at once, the personal stress in spontaneous learning is on action. Therefore if we can present our exercises as action-andperception games, students will be motivated spontaneously and know 29

On Evaluation

that they are concerned with their progress every time they play. Hence there is an enhanced chance that they will watch themselves while learning and be ready to record a score if it is easily recorded. If we do not know how to present our exercises as games of this kind it is much harder to achieve what we want as then the continuous feedback can only be obtained from outside, with all the hazards this entails.

Fortunately in some areas we have been able to offer games that are of perception and action and to develop the instrument for continuous feedback. We shall consider this when we sketch the answer to (3).

We can now introduce the actual recording instrument. Reminiscent of the behavior-frequency chart developed by Ogden Lindsley, it is a sheet of graph paper with time marked along the horizontal axis and frequency along the vertical axis (using a logarithmic scale). Although used in the field of the exact sciences for a long time, graphs produced by observers recording particular phenomena have not been used extensively in education before Lindsley introduced “precision teaching.”

We intend to give students the responsibility of becoming precisely aware of what they are doing and to define a certain “pinpoint” through it. By watching their work on this pinpoint on successive occasions and recording on the graph the number of times they deal successfully with it, the shape and properties of the resulting graph will indicate whether their facility is increasing or not.

This fact is very important: on the one hand the student proves he is aware by knowing what it is that he wants to take into account. Hence the “pinpoints” are indices of awareness. Without such specific awarenesses all things are undifferentiated and confused. But on the other hand even with them the student has only a hint of what to do unless he also follows the history of how the awarenesses shape him and relate this particular skill to other skills.

We look at learning as a four-stage operation. First the learner must come into contact with a new challenge. This mere contact translates itself into his hesitancy, his clumsiness, his claim for help. Second 30

Continuous Feedback And Evaluation

when enough stepping stones have been located in the field (this is an analysis of the challenge) a move to integrate them leads to an understanding of their respective relationships so that a sense of the whole emerges which now appears to be the essence of the acquired skill. When enough practice has been obtained a sense of mastery results. To prove that such a third stage has been achieved, the learner gives himself a new challenge to apply his advance to, and the sequence of contact, analysis, mastery, application begins again, but on a more complex challenge. Our instrument will tell us at which level in the sequence a student finds himself. The first is displayed by a large number of random responses. To be in the second is shown by a reduction in the number and a coordination of one’s functionings to include the new one. The third by a smooth and fast growing curve through the pinpoints.

Thus it is possible to discern objective properties of the graph and interpret them in the general study of the acquisition of a skill.

When a number of graphs can be correlated, by the nature of the pinpoints, it is possible to gain further knowledge not visible on one graph. Lindsley utilizes two simultaneous graphs to indicate counts of errors (hopefully diminishing) and counts of correct movements (hopefully increasing). In complex activities there may be a number of components for which graphs (distinguished by color or by type of line) may reveal not only something valuable with respect to the learner as the activity is being plotted, but also with respect to the field studied.

For instance, how could one know without such data whether reading is independent of, or helped by, good awareness of correct spelling?

The role of the logarithmic scale is to allow considerable growth to be shown without shooting off the graph. The subdivisions of the horizontal axis represent successive days (in groups of 5 or 7) over the months of plotting.

We must let each student, whenever possible, mark his own sheet with the value of the pinpointed components in his study. This makes it possible for the instrument to cater for large groups of students

31

On Evaluation

simultaneously so long as each observes the rules for graphing specific awarenesses.

In order to teach the students graphing we can select as simple an activity as jumping and use two linear scales on the graph paper. Since there is either a height at which to place a bar, or a distance from a given line at which one can land in a long jump, we can easily use either to start an activity that will be plotted for a succession of jumps.

There is certainly a lower bound for each jump and the student aims to find an upper bound, even if this changes over time. For a number of jumps in one session or for jumps spread over a few days a simple graph will emerge indicating whether the upper bound is moved up after practice.

We can propose games such as recognizing which word, out of a set of 30, say, written on a chart hanging on the wall, has been covered by the teacher’s hand. Students can tally their performance and end the game when they know how long it has taken them to be sure every time. A test can be the student’s reproduction of the set on a piece of paper, getting the location and the spelling right without access to the original chart. Learning to look carefully, to retain properly (i.e. re-location and spelling), to be sure of the extent of the set, can all be made conscious awarenesses and will stimulate students to attempt even harder tasks — with sets of signs covering all the spellings of English, words with exceptional spellings only justified by their etymology (such as sapphire and phthisic), etc.

In spontaneous learning, such as the learning required by games, the involvement of the players results from the perception of the relevance of the exercise to their growth in some activity. This quality of relevance can be maintained in the exercises suggested for reading or writing or spelling, where the materials can show an expansion of the previously mastered demands to the conquest of new ground. There is no need for artificial motivation through extrinsic rewards. Learning is a reward in itself. By charting the whole area of a skill to be mastered it now becomes possible to find the various pinpoints that need to be followed to take learners from contact to mastery. Each field of study proposes its own sets. The sets can tell us both the curriculum that

32

Continuous Feedback And Evaluation

needs to be presented and how to take students through it with a continuous feedback that can be made visible on a sheet of graph paper. We therefore have an integrated teaching-testing approach: teaching being expressed in the sequence of proposed activities, testing being the concomitant enhancement of what is being done at the time with full awareness that it is moving towards reducing hesitancy and clumsiness, and that it is increasing understanding of the tasks in a way that will lead to mastery.

Let it be clear that we are dealing here only with learnings that yield skills.

There are other learnings that may end up with owning a new sensitivity in some areas — for example, noticing social components, aesthetic components, chronological components, etc. These cannot be said to follow the four-stage approach to acquiring a skill; awareness and application are sufficient. We therefore leave them out of our study which refers to enough varied skills to occupy years of activity for each student and each teacher.

In the teaching we have advocated for a quarter of century, teaching that is subordinated to learning, it is clear that the techniques that are used together with the specially designed materials make it possible for student and teacher to follow learning as it takes place. Hence we can consider the sets of exercises given in these materials as the instruments for the continuous feedback we are after. The charts will display the summary conclusions reached each day on the way to mastery of the pinpointed components; their collection on a graph will produce the trends and furnish the rates if required in the cases of individual learners.

In brief, readers can find in re-examining the materials I have created for the various skills studied in schools that they, with the techniques proposed, allow teachers and students to remain in close contact with their activities as they unfold in time and therefore yield the continuous feedback needed to be objective in knowing where students are and what remains to be done.

33

On Evaluation

The use of charting will give a concrete form to this feedback that will make it available to each individual and to the public. This is its main function and value and its service to social evaluation.

Caleb Gattegno

34

News Item

The first workshop on English as a second language for the Association of Users of the Silent Way took place on December 7, 1974 at our headquarters. Seven students from Community Colleges part of CUNY came to be involved in three kinds of lessons watched by thirty-two instructors from City College and other institutions. From 9-12, the lessons were given by Dr. Gattegno who showed first what he means by “working on the students while they work on the language.” A second lesson showed how to use the materials (Fidel, charts, rods) for specific purposes connected with oral work. A final lesson on request showed how to produce unambiguous situations, using rods, to force the discrimination of verb forms.

The afternoon was used to convey the way that observation of lessons can serve teacher and observers alike.

Because observation is a difficult art requiring a suspension of judgment, an alertness to details, a control of the observer’s automatic reaction systems, the interplay of several sensitivities, a number of hours were needed with no certainty at the end that all the participants met the demands.

Further such one-day workshops were considered to be of use and some are planned. Consult with Cecilia Perrault at 212/924-1744.

35

About Caleb Gattegno Caleb Gattegno is the teacher every student dreams of; he doesn’t require his students to memorize anything, he doesn’t shout or at times even say a word, and his students learn at an accelerated rate because they are truly interested.

In a world where memorization, recitation, and standardized tests are still the norm, Gattegno was truly ahead of his time.

Born in Alexandria, Egypt in 1911, Gattegno was a scholar of many fields. He held a doctorate of mathematics, a doctorate of arts in psychology, a master of arts in education, and a bachelor of science in physics and chemistry. He held a scientific view of education, and believed illiteracy was a problem that could be solved. He questioned the role of time and algebra in the process of learning to read, and, most importantly, questioned the role of the teacher. The focus in all subjects, he insisted, should always be placed on learning, not on teaching. He called this principle the Subordination of Teaching to Learning.

Gattegno travelled around the world 10 times conducting seminars on his teaching methods, and had himself learned about 40 languages. He wrote more than 120 books during his career, and from 1971 until his death in 1988 he published the Educational Solutions newsletter five times a year. He was survived by his second wife Shakti Gattegno and his four children.

www.EducationalSolutions.com