Assessing Assessment with Anne Katz and Scott Thornbury: Two perspectives on the state and future of assessment
Anne Katz’s work is centered around assessment, curriculum design, and standards development. Her latest publication about assessment, published in Celce-Murcia, Brinton, and Snow’s (2014) Teaching English as a Second or Foreign Language, examines the role of assessment in contributing to learning in the English language classroom.
Scott Thornbury is the author of several award-winning books, papers, and book chapters for teachers on language and methodology. Most recently, Scott has published a new edition of AnA-to-Z of ELT (2017) with both new and revised entries that “reflect the development in linguistic theory (and) language education.”
Anne and Scott are both faculty members of The New School’s MA TESOL Program in New York City.
I had the good fortune to interview Anne Katz and Scott Thornbury and to explore their views on assessment in language classrooms. Anne, a self-proclaimed “Assessments person”, provides insight into how we misuse (and perhaps sometimes misunderstand) assessment terminology. In our interview, Anne talks about the importance of aligning assessment and learning and emphasizes that assessment is a process that must be done, “in multiple ways and at multiple times to give an overall (and more complete) picture of students’ learning.”
In his blog An A-to-Z of ELT: O is for Outcome, Scott Thornbury writes, “I have to say at the outset that I have an almost pathological horror of testing and assessment.” Unfortunately, his fear is not unique. In this interview, Scott reminds us that, “Doing tests for the sake of them… is a fairly sterile activity, especially if this takes time away from learning and use” and argues that there is, “no learning without feedback – but there is learning without formal testing.”
While coming from seemingly different viewpoints, both Scott and Anne emphasize the importance of linking assessment (in whatever form given) back to students’ learning needs and goals.
Many teachers use the word ‘assessment’ synonymously with ‘test’. Yet, these words’ meanings are vastly different. How would you differentiate the two?
Scott Thornbury: I wouldn’t say that the meanings are vastly different; rather that assessment subsumes testing. To quote from the new edition of An A-to-Z of ELT: “assessment refers to the different ways of collecting information about a learner’s progress and achievement. One of these ways is by testing the learner, but testing and assessment are not necessarily the same thing. Assessment is the more general term, and is nowadays preferred to testing, given the latter’s somewhat formal and mechanical connotations.”
Anne Katz: We are generally not careful how we use the terms score, grade, test and assessment. I would define them as follows:
- Score: Is a descriptor (be it a number, letter or other) measuring a student’s performance for a specific test/ task/ activity or otherwise at a specific point in time.
- Grade: The grade is the sum/average/product of many different course elements including, but not limited to, tests. Grading is typically done at end of course/term to see where students are in reference to their goals. To avoid subjectivity, grades should tie back to the course’s grading policy. The grading policy is an explicit guide defining the scoring procedure for each task/ lesson/ test. Having a grading policy lets students know exactly what they need to do to obtain a certain score (and overall grade). Its detail gives students certain control over their ‘grading destiny’.
Tests and Assessment are ways of gathering information that feed into the overall grade.
- Tests are just one way of gathering information about a student, therefore, only one type of assessment. They are usually one point in time and often thought of in summative terms.
- Assessment is a broader plan for collecting information about students’ performance so that teachers, students and other stakeholders can make decisions about and reflect on learning. It can include testing, but also might include items like: teacher observation, portfolio work, classroom interaction, checklists, scoring rubrics, etc. At the heart of assessment is finding many ways of collecting data about the student. If assessment if not aligned with the learning process, that is, if it does not provide input for learning and feedback to get to the next stage, then it is not useful.
Some teachers/students believe that testing (standardized testing) benefits the testers more than the students. Yet, others (students included) see the value in taking exams/ exam courses. What types of tests (standardized or not) really help our learners/ and which hurt learners? Explain.
ST: Testing/assessment can be an important motivator for some learners, especially those who do not get immediate feedback on their learning e.g. through interaction with other speakers in real life-like circumstances. Doing tests for the sake of them, however is a fairly sterile activity, especially if this takes time away from learning and use. The tests that really help learners are probably those that encourage them to keep learning – rather than discourage them. These are the kinds of tests that provide useful information as to the gap between the learner’s current competence and the target one – ideally a target that has been negotiated with the learners themselves, rather than being some arbitrary measure of native-like proficiency.
Perhaps the best kind of testing is that which is indistinguishable from teaching: sociocultural theorists make the case for what is called ‘dynamic assessment’ where the learner is set a task and the teacher intervenes where necessary to ‘scaffold’ the task – the learner’s capacity to respond to, and build on, this scaffolding is what is assessed, not some predetermined measure of proficiency. In this sense, assessment is embedded in instruction, rather than being separate from it.
AK: We need to return to the purpose of a test or any assessment. The key is figuring out the relationship between the test and the learning aim of the student. A useful test needs to provide information about whether students have achieved their learning aim.
There is certainly a testing industry, but it exists in collaboration with educational institutions that want to standardize how student learning is documented. When these institutions utilize quality standardized tests that have been piloted and tested, they can help students and schools make decisions about learning. Returning to the notion of students’ purposes for learning, if a student registers for an exam course and wants to take this type of test, then it does not necessarily help if the teacher disagrees or doesn’t help the student prepare for that test.
As teachers who need to administer standardized tests as part of school requirements, we need to look at these standardized tests’ criteria and ask if these criteria will serve students’ learning aims and prepare them for when they leave school. If the answer is no, then it becomes a political issue which often can be better handled by a teacher association that collectively can argue for change.
Many teachers feel, “slaves of the testing industry.” One teacher wrote, “I exist in part of the industry where the standardized tests are treated as God”. How can teachers become liberated from this industry or do tests liberate teachers from the daunting task of test making?
ST: Yes, unfortunately we live in a positivist educational paradigm, where the content of learning is made as granular as possible so that it can be taught using ‘delivery’ methods of instruction (rather than being jointly constructed through interaction, for example), and where it can be tested as discrete items of knowledge, or ‘factoids’. This might work for some subjects, but it does not really work for language –which is a complex and dynamic whole – and resists being broken down into its molecular elements.
Put another way, knowledge of the discrete items of language does not predict proficiency in being able to use them in real-time communication and social contexts. How teachers liberate themselves from this paradigm is a political and ideological question, more than an educational one. Within their own institutions, perhaps they can resist the push to granularity and the constant testing of the granules – but it will take some resisting!
AK: It probably will depend on the institution within which teachers work. Teachers can collect their own information about student learning while also using tests required by their institutions. The issue is whether the learning environment is aligned with the required testing. If a school’s required tests are in line with its course books, expected instructional methodology and overall philosophy, then these tests can serve as a valuable learning tool. However, when (standardized) tests are used as proxy for the evaluation of a school’s success, teachers (and I speak for myself) become resentful because this is not the only way we can measure a learning outcome.
Whether we subscribe to or use standardized tests or not, we need to think of our students as stakeholders in their own learning and we need to continue collecting information to see whether students are developing the skills they need beyond the classroom.
Many teachers feel they are not ‘good test makers’. Should teachers choose between being a ‘teacher’ or ‘examiner’? On that same note, why is assessment not a bigger feature of training courses / teacher education?
ST: My experience suggests that there are two things that teachers know very little about even after in-service training, and that is pronunciation teaching, and testing! I think testing is neglected because it is such a contested area – it brings to the surface values and beliefs that often remain submerged. There is nothing like a meeting to decide on a new end-of-course test to really get teachers fired up – but it can be a very constructive, if sometimes contentious, process, because it forces teachers to articulate their principles and their often covert theories of learning and teaching. Choosing between a multiple-choice grammar test and an interactive speaking task can really set the cat among the pigeons!
AK: At one point or another, most of us have been traumatized by tests. In the past, assessment research and instruction was highly theoretical and focused mostly on testing theories and statistics and not on students. For a classroom teacher, this was not what he / she needed to know to document classroom learning. Only more recently has there been more focus on ‘Assessment for Learning’.
Good teaching should include interaction with student learning, not just the delivery of what is assumed to be good instruction. Are students benefiting from instruction? Are they meeting learning targets? How do we know? Assessment provides a good source of this feedback.
Sir Ken Robinson says that standardized testing hurts our students’ creativity. Some teachers believe that testing and assessment are still primordial and dictate often what happens in the classroom. Teachers feel they have to ‘teach the test’ rather than the language. How best can we move away from this towards a learning centred approach?
ST: Well, one way of moving towards a learning centred approach might be to focus exclusively on formative, rather than summative, testing. That is, testing is folded into the learning process as a kind of feedback loop, and there are no formal end-of-course tests as such. But few institutions are prepared to take this on, since it creates the dilemma of: ‘Who is to move on a level and who is not to?’ In an ideal world, this is a decision that should be negotiated between the learner, the teacher, and other stakeholders, taking into account all the available evidence. But we do not live in an ideal world.
AK: This certainly happens quite a good deal, in part because of accountability pressures. Are schools serving the needs of their students, meeting the expectations of stakeholders, funders, parents, and so forth? The question to ask is whether the tests are aligned with what everyone, including the student, wants to see in terms of learning outcomes.
If we want a learner/learning-centered approach, then we first need to define what the learning outcomes and experiences we want our students to have. If we don’t do this, we run the risk of having our tests dictate what needs to be learned.
There are an increasing number of reports that students undergo major stress because of tests. Some argue this stress hinders learning, while others suggest this stress can be motivational. How can (or should we) eliminate testing stress?
ST: Again, we can eliminate testing stress to a certain extent by focusing on the formative rather than the summative, and building in forms of counselling where learners get clear messages about their progress and can negotiate their subsequent learning trajectory.
AK: Learners are different, as we know. Some respond positively to testing stress and are motivated to achieve. Others feel that tests rarely if ever show what they know and so are demoralized by testing. Part of the stress comes from the idea that ONE test can provide answers to all of our questions about learning. A better idea is to develop an assessment plan that includes multiple types of assessment administered at multiple points of time.
If we really believe everyone should succeed, then there many alternatives that can provide support for students to help them succeed. Ultimately, it’s not just about assessment, it’s really about our educational belief system. As educators, we should be wanting to help as many students as possible.
Where do we go from here? Can we really do away with testing completely? If you could design the perfect test, what would it look like?
ST: I think that if we re-construe assessment as just another form of feedback, we can take some of the toxicity out of it. There is no learning without feedback – but there is learning without formal testing.
AK: If I had to design the ‘perfect test’ it would be a blank page. There is no ‘perfect test’. A ‘perfect test’ would be have to be completely in line with both instructional and students’ needs. This is not possible. Therefore, as teachers we need to provide multiple tests (and forms of assessment) over multiple times to gather and sort information. It’s like creating a mosaic. Each individual tile may be beautiful in itself, but only when we put everything together can we see the complete picture.
On behalf of ETAS, I’d like to thank both Anne and Scott for their time, thoughts and wisdom.
Katz, A. (2014). Classroom assessment. In M. Celce-Murcia, D. M. Brinton, & M. A. Snow (Eds.), Teaching English as a second or foreign language (4th ed., pp. 340-361). Boston, MA: National Geographic Learning/Cengage Learning.
Thornbury, S. (2017). The new A – Z of ELT: A dictionary of terms and concepts used in English language teaching. Oxford: 8Macmillan.
Thornbury, S. (April, 2017). An A-to-Z of ELT Blog: N is for New Edition. Retrieved from https://scottthornbury.wordpress.com/2017/04/16/n-is-for-new-edition
Thornbury, S. (February, 2013). An A-to-Z of ELT [Blogpost]: O is for Outcomes. Retrieved from https://scottthornbury.wordpress.com/2013/02/10/o-is-for-outcomes
Emilia Siravo is a freelance ESL teacher in Zürich, Switzerland. In addition to the CELTA, DELTA I, DELTA III, and SVEB certifications, Emilia received her Master’s in TESOL from The New School University. Emilia loves teaching and finds that second language learning enables students to open new doors, to discover new cultures and to explore new perspectives. Follow Emilia online on Twitter: @esiravo or read her blog @: https://emiliasiravo.com