Local Site Research Initiative: Results and Reflections
Summary: Five NWP sites conducted locally defined studies involving program and comparison groups to investigate how key aspects of their programs contributed to changes in teaching practices and student achievement in writing; the results consistently favored the NWP.
In 2005–2006, the National Writing Project expanded its national portfolio of rigorous evaluation studies through the Local Site Research Initiative (LSRI). By participating in LSRI, writing project sites continue to develop their capacity to engage in research and evaluation.
Through a process involving a request for proposals and independent, external review, five NWP sites were selected to participate in Cohort II of LSRI. Reviewers evaluated the proposals for focus, content, methodology (including some form of comparative reference), feasibility, and the capacity of the site to successfully execute its research plan. Each site received $20,000 and technical assistance from NWP's Research Unit.
The five studies, all locally defined and conducted, investigated how key aspects of sites' programs contributed to changes in teaching practices and student achievement in writing.
I'm a practitioner first; but I've always believed in the power of research to validate good work. . . . In LSRI, we came together and learned about how other sites were conducting research. It made me say, "We can do that!" With the help of the [NWP's] Research Unit and the LSRI we shaped a research project that is demonstrating the success of our inservice programs.
—Diane Scollay, Gateway Writing Project
Collectively, the studies covered a range of grade levels, from elementary through high school. The studies included teachers and students in rural, urban, and suburban areas with diverse economic, language, racial, and ethnic backgrounds. These studies captured data about teacher participation in the writing project in a variety of settings and kinds of programs.
Research Design and MethodsAll five studies employed a quasi-experimental design to permit comparative reference in the interpretation of results, as suggested by the Principles for Scientific Research in Education (National Research Council 2002). Comparison groups were formed at a level appropriate to the investigating sites' programmatic focus and research. One study focused on a whole-school intervention program and compared participating schools with demographically similar schools; other studies focused on individual classrooms, matching individual teachers and their classrooms; and yet another study focused on an after-school program for students, matching individual students.
All five studies analyzed pre/post student writing samples. The writing, with identifying information removed, was independently evaluated at NWP's national scoring conference. Building on a long tradition of writing assessment, the NWP defined a rigorous framework for evaluating student writing, using "anchor papers"—samples exemplifying each level of achievement—along with descriptive commentary. More than 7,500 writing samples were scored using a rubric adapted from the 6+1 Traits Writing Model (Culham 2003). The four-point scale was extended to six points in order to increase sensitivity, and language was clarified to enhance reliability. The rubric addressed six specific attributes of writing: ideas, organization, voice, sentence fluency, word choice, and conventions (Bellamy 2005). Each writing sample also received an overall holistic score.
The scorers participated in six hours of training at the beginning of the scoring conference; their scoring was calibrated to a criterion level of performance and then recalibrated following major breaks (meals and overnight). Twenty-three percent of the papers were double scored, with overall reliabilities (measured as interrater agreement, when two scores are identical or within one point of each other) ranging from 90% to 95%, with an aggregate across all scores of 92%.
With reliable student outcome data as well as documentation of teacher practice in hand, site research teams gathered for a three-day "data festival" to analyze the data and write up what they learned. Sites collaborated with each other and with senior Research Unit staff members to apply qualitative as well as quantitative analytic methods to their data.
Student OutcomesThe results, taken across sites, consistently favored the NWP. For each site, the NWP group is compared with the comparison group on six attributes of writing as well as an overall holistic score.
The results, taken across sites, consistently favored the NWP.
For every one of the measured attributes, the improvement of students taught by NWP-participating teachers exceeded that of students whose teachers were not participants. In 20 of the 35 contrasts (57%) the differences between NWP participants' students and comparison students were statistically significant. Four studies demonstrated statistically significant differences favoring the NWP group on at least one measure of student writing.
The results were especially favorable for the attributes of ideas, organization, and voice. These results signal effectiveness in areas that are emphasized in many writing project activities. At the same time, students in writing project classrooms outperformed their peers in the area of conventions, suggesting that even these basic skills benefit from the NWP approach to the teaching of writing, which usually includes addressing conventions in the context of ideas and their expression—not as a separate topic. The findings for conventions support the validity of this approach. These studies uniformly indicate positive effects for the students of teachers who participated in writing project programs. Brief synopses of all five studies, along with researcher reflections, follow. An extended report on LSRI Cohort II is available on the NWP website (Summary Report of National Research Results) at www.nwp.org/cs/public/print/doc/about/press_room.csp.
Looking to the FutureThe first two years of LSRI have demonstrated the effectiveness of this approach in building a national portfolio of evaluation studies for the NWP as well as in strengthening sites' capacity for conducting rigorous research into the effects of their work. If your site is interested in applying for the next round of LSRI funding, see the NWP website for the request for proposals. The Research Unit will offer preproposal support for interested sites during the Annual Meeting; full proposals will be due in January along with the Continuing Funding Applications. We anticipate that this new round of support will expand the network of LSRI sites and further develop the capacity of our network to investigate its work.
Summary of Results by SiteGateway Writing Project, University of Missouri, St. Louis:
Leadership in Writing Institute: An Evaluation of Professional Development Impact
Nancy Robb Singer and Diane Scollay
The Gateway Writing Project (GWP) study investigated a yearlong professional development partnership with a midsized school district. The program encouraged teachers to take an inquiry stance toward writing pedagogy by collecting and analyzing data from their own classrooms as well as trying and testing new classroom strategies. The study focused on seven of these teachers, third through fifth grade, and the students in their classes; seven matched teachers (third through fifth grade) and their students provided comparison data. Teacher outcomes included knowledge of writing-process pedagogy, classroom practices, and attitudes toward writing and writing instruction.
Three differences between the classrooms of program and comparison teachers were observed: 1) program teachers engaged students in a wider range of writing tasks; 2) program teachers designed writing instruction to extend over multiple class periods; and 3) program teachers modeled reading/writing connections for their students. In contrast, writing instruction in comparison teachers' classrooms largely centered on completing tasks in shorter blocks of time, mirroring the type of timed writing found on state tests.
Program students' writing scores increased more than those of comparison students on the holistic assessment and on all six analytic traits. The differences were statistically significant on all measures but one, word choice. Furthermore, program students showed greater gains on standardized reading assessments, suggesting (at a minimum) that professional development based on writing pedagogy will not necessarily detract from—and may even support—instruction in reading.
Additionally, Gateway Writing Project used its involvement in the LSRI to help expand its capacity in two ways: 1) The LSRI data helped the Missouri Writing Projects Network (MWPN) secure a long-term contract with the Missouri Department of Elementary and Secondary Education. In the summer of 2006 the MWPN conducted its first round of Literacy Academies in all nine Missouri Regional Professional Development Centers. 2) Twenty GWP consultants have scored papers at the LSRI-sponsored scoring conferences. As a result, GWP has developed the capacity to assess writing samples as part of the array of services it can offer school partners.
Mississippi Writing/Thinking Institute, Mississippi State
The Effect of Mississippi Writing/Thinking Institute Professional Development on the Writing Achievement of Ninth-Graders
Sherry Swain, Richard Graves, Liz Townsend, and Linda Irby
Our study probed the effects of Mississippi Writing/Thinking Institute (MWTI) partnerships in high schools. The groups, 298 students in two program schools and 157 in two comparison schools, were closely matched demographically, with 83% of each group receiving free or reduced-price lunches. Ethnically, students were 97% African American and 3% Caucasian in the program group compared with 76% and 24% in the comparison group. There were five teachers of writing in each group.
Program teachers were experiencing their first year of inservice work; however, MWTI had also served teachers of the program students during their eighth grade year. Professional development focused on process approaches, including peer response and teacher conferences. Specific strategies for helping students develop sentence fluency, elaborate details, and organize ideas were also offered in workshops, classroom demonstrations, and study groups.
The study confirmed that program teachers implemented strategies from MWTI professional development more than comparison teachers, although statistical differences in their practices were evidenced in only two areas: encouragement of student choice in their writing and use of peer response. The fact that program students outperformed comparison students to a statistically significant degree in all areas, coupled with the two-year effect of MWTI professional development, suggests to us that MWTI approaches should be practiced at consecutive grade levels.
These results affirming the impact of our work inspire us, yet we are even more inspired by the NWP analytic rubric and our increased capacity for scoring writing. Our cadre of teacher-consultants with scoring conference experience is conducting a writing assessment for a number of school districts in the state. Hurricane Katrina left Mississippi without the funds for a writing assessment this year, but LSRI, like all NWP programs and initiatives, continues to help MWTI respond to the needs of Mississippi schools and teachers.
New York City Writing Project, Lehman College, City University
of New York:
Teacher and Student Outcomes of a Professional Development Model for Improving the Teaching of Writing
Anne G. Campos and Roger F. Peach
The increase in writing scores of students, even in history and science classes, suggests that writing across the curriculum can have a positive effect on students' overall writing achievement.
This study investigated a professional development partnership between the New York City Writing Project (NYCWP) and six high schools in a large urban district with high poverty, low student achievement, a large number of inexperienced teachers, and an increasing emphasis on high-stakes testing. Both groups—program and comparison—were more than 90% African American or Hispanic; ELL students made up 38% of the program group and 46% of the comparison group.
Program teachers adopted views and practices that resulted from NYCWP professional development, which included interactions with NYCWP teacher-consultants who fostered use of new strategies in their classrooms. The program teachers treated writing as a process, employing prewriting and peer-response techniques. Program teachers in content areas other than language arts reported incorporating writing into their instructional practice. Comparison teachers, on the other hand, felt that the professional development they received was less useful to them, and they employed writing strategies less consistently.
Program students' writing scores increased more than comparison students' on the holistic and all six analytic measures. In six of the seven measures the differences were statistically significant. Subgroup analyses for ELL students demonstrated a similar pattern: The increase in ELL program students' scores, even when these students were taught by less-experienced teachers than those of comparison students, suggests that focusing on writing as a process can benefit all students, including those for whom writing in English is a challenge. The increase in writing scores of students, even in history and science classes, suggests that writing across the curriculum can have a positive effect on students' overall writing achievement.
South Coast Writing Project, University of California,
Evaluating IIMPaC: Teacher and Student Outcomes Through a Professional Development Program in the Teaching of Writing
Sheridan Blau, Rosemary Cabe, Tim Dewar, and Anne Whitney
Our LSRI study examined effects of our professional development program IIMPaC (an acronym for Inquiry, Inservice, Modeling, Practice, and Coaching) on low-performing schools serving low-income populations with a high proportion of ELL students. The program encourages teachers to adopt an inquiry stance; teachers collaborate to experiment with new strategies for teaching writing and to modify existing ones. The study focused on fourth- through eighth-graders and their language arts teachers.
Teachers in both the program and the comparison groups used "process" terms and strategies; however, we saw striking differences in the ways teachers understood these strategies and framed writing for their students. Program teachers more frequently showed student-written models of writing, gave students more opportunity to develop their own topics, and employed a broader range of prewriting strategies. Program teachers encouraged students to develop ideas by revising more substantially, and engaged students more deeply in writing as communication. They also facilitated student engagement by providing opportunities for students to share their writing, publish it beyond the classroom, and collect it over time.
Program students' writing performance improved more than comparison students' on all of the measures, holistic and analytic; however, the differences between the two groups were not statistically significant, perhaps because of the general sophistication of the comparison teachers and the relatively high achievement of the comparison students at the outset of the study. Program students did increase their use of prewriting strategies, such as annotating a text, to a statistically significant degree.
Our research activities directly influenced our work in the IIMPaC program, helping us to make program goals more explicit both to ourselves and to the teachers we worked with. Just as importantly, our LSRI participation helped us to build research capacity at our site and forged new bonds between our writing project site and our university's graduate programs in education.
Southern Nevada Writing Project, University of Nevada,
Through the Lens of the Family Writing Project: The Southern Nevada Writing Project's Impact on Student Writing and Teacher Practices
Marilyn McKinney, Saralyn Lasley, Rosemary Holmes-Gull, and Michael Nussbaum
This study investigated the effects of the Family Writing Project (FWP), a family literacy program offered by the Southern Nevada Writing Project (SNWP). The FWP serves students, parents, and teachers in a school district with a rapidly growing and increasingly diverse and mobile population. FWP teacher-facilitators engage students and families in writing and art activities, publishing, and community service projects.
Program students demonstrated greater growth than comparison students on each of the six writing traits as well as the holistic score. On word choice the difference was statistically significant. Furthermore, FWP students' attitudes toward writing held steady or improved, while the attitudes of comparison students became less positive.
Program teacher-facilitators reported changes in their classroom practice that included improving their relationships with students and their families, deepening their understanding of how writers write, and developing their leadership and teaching skills. This study suggests that strong personal relationships between teachers, students, and families can support student achievement.
The FWP was a powerful way to provide meaningful continuity to teacher-consultants and enhance the relevance of the site to teacher-consultants and the community. LSRI interviews promoted opportunities for teachers to make explicit connections for themselves between their classroom practice and their work as FWP facilitators. By exploring characteristics of this successful continuity program, we were able to document the FWP's impact on keeping several teacher-consultants in the profession as well as the importance of supporting teacher-consultant-generated continuity ideas. We have also been using the research results to define and strengthen aspects of teacher leadership at our site.
Bellamy, P.C., ed. 2005. Seeing with New Eyes. Portland, OR: Northwest Regional Educational Laboratory.
Culham, R. 2003. 6+1 Traits of Writing. New York: Scholastic Professional Books.
National Research Council. 2002. Scientific Research in Education. Washington, D.C.: National Academy Press.