By Matri Lamb
On a Monday morning in July, after one of the most challenging school years in any teacher’s memory, I find myself sitting with a surprisingly cheerful group of Language Arts educators convening on Zoom for the first day of the National Writing Project’s weeklong scoring conference.
NWP’s approach to large-scale assessment—combining a high degree of rigor with intensive peer-to-peer interaction—is unique. It’s not hard to see why teachers, so accustomed to working in isolation, under immense pressure to meet standards, would look forward to coming together in this way to hone their evaluation skills and learn from each other.
The Analytic Writing Continuum (AWC) was first developed by the NWP’s research and evaluation team in 2004. There are now three additional AWC-based rubrics—Analytic Writing Continuum for Source-Based Argument (AWC-SBA), for Literary Analysis (AWC-LA), and for Civically Engaged Writing (CEWAC). All told, they have been used to assess nearly 100,000 student papers from grades 3-12.
In contrast to most large-scale writing assessments, which offer limited information about how improvements in student writing may be achieved, all four versions of the AWC are designed to provide an accurate assessment of important performance attributes of writing, which teachers can use to guide instruction.
As I speak with scorers both new and seasoned, a picture emerges of an assessment model that is a philosophy as much as it is a methodology. At its foundation is a profound appreciation for the living conversation in which all writing participates. The job of assessment within this always evolving context is to respond to the “call” of an act of writing, honoring it for what it is while evaluating, as objectively as possible, the moves it’s making. The feature that really sets it apart is a determined focus on what a piece of writing is doing, instead of what it is failing to do.
The deficit / task fulfillment mentality that prevails in most academic standards and assessment has no place here—and that’s a big relief, teachers tell me. AWC scorers say that the method gives them greater confidence in their ability to honor students’ efforts while pointing them toward improvement and eventual mastery, with the Continuum marking the way. As such, it offers a clear future-focused map for both assessment of student writing and for instructional development.
AWC scoring has taken place remotely since 2016, allowing a broad cohort of educators from around the country to participate. The custom online platform is designed around a supportive, synchronous structure that lends itself to learning and collaborating. Events are organized with multiple "table" units nested within "rooms,” allowing for ongoing review of scorings and immediate individual scorer support. Four to five scorers at each table relate closely with each other and with their more experienced table leader throughout the conference. Room leaders in turn support table leaders.
The system has proven remarkably effective for assessing writing. Multiple layers of checks and balances ensure that rubrics don’t rigidify into formulas or bend to the influence of cultural and linguistic biases. The result is an inter-rater reliability ranging from 89 to 93 percent across attributes, with similar results for test-retest reliability.
Over years of refining this human scoring process, the National Writing Project (NWP) has mobilized a highly trained and diverse network of educators from all over the country. In the process, it has also contributed a wealth of information to the field of writing pedagogy. Thanks to the scoring continuum, we have a better understanding of how student writers are making use of the instruction teachers are giving, and what can be done to better support equitable outcomes for all students.
“You can’t look at writing standards without looking at actual student writing. All of it has to go together,” says Aija Simmons, longtime Bay Area Writing Project teacher-leader and scorer. “That was such an aha for me. I bring that line of thinking into my classroom, to my students, so that when we write, and when we decide what quality writing is, we're not doing that in the absence of age-appropriate examples. I justify the language of the standard with evidence from the students' work over and over and over again.”
The continual dialogue between standards and student work is fundamental to the AWC approach. In advance of a scoring conference, a group of experienced scorers will gather to read hundreds of student papers drawn from the same population as the writing that is going to be formally scored, in order to identify “anchor papers” that are representative of the different possible scores along the continuum. These anchor papers are a key element in both the training of new scorers as well as the maintaining of integrity and accuracy among more experienced scorers.
At this stage in the process—known as “range-finding”—scorers engage in animated interrogations of the rubric. Is it accurately representative of the strengths that students are bringing to the table? Does it allow and celebrate students’ home literacies and multicultural experiences? Is it expansive enough? Precise enough? Is there something we’re missing? Anchor papers are then paired with less straightforwardly representative “calibration” papers, which are chosen to elicit discussion among scorers at regular intervals throughout the scoring conference to ensure that everyone stays in sync.
“The whole scoring process is really respectful of students and honoring the effort they put in,” says Kim Douillard, director of the San Diego Area Writing Project. Douillard has been involved with NWP in a leadership capacity since the late 90’s and has had her hands in every aspect of assessment over the years. She appreciates that the AWC is a scoring system as opposed to a rubric in isolation.
“A lot of times in schools and classrooms you’re simply handed a rubric and it’s supposed to work magically all by itself. The AWC is a system that works together. That includes the training NWP provides. It’s the rubric, the anchor papers, the calibration papers, the training—all this comes together to allow us to score accurately. It also works as powerful professional development, because teachers leave saying ‘I can do this in my classroom.’”
As an NWP teacher-leader, Mark Dziedzic from the Greater Madison Writing Project has seen dozens of teachers return from NWP scoring conferences over the years.
“Consistently across the board they have talked about it being one of the best experiences they have ever had for their own growth. I had sent people to learn about the scoring process, and everybody came back really impressed with that, but each time people came back talking about how they really grew as teachers in their work with students.”
This summer was the first time Mark participated himself. As we’re talking, he lands on a word that sums up the experience for him: generosity.
“There’s a generosity that as teachers we don’t often recognize. To be generous with our feedback is not how I would have thought of it at many points in my life. Previously I would have thought that it was generous to name all the things that were wrong. But that never actually felt very generous to me. Being generous in terms of being able to name what the writing is doing and what a writer could do next to elevate it—those really do feel generous.”
The culmination of the scoring process, as Aija, Kim, and Mark describe, happens close to home—back in the classroom, in the relationship between teacher and student. By offering a map forward for both teachers and students, a good assessment system gives back something more than a score. It helps classrooms become places where developing writers can feel proud of what they’ve accomplished, and motivated and empowered to go on always improving.