Research Studies Continue to Show Impact of NWP Programs
Summary: The latest research conducted at writing project sites in five states shows significant gains in writing achievement for students of teachers who have participated in NWP programs.
In the fall of 2003, the first group of Local Site Research Initiative (LSRI) sites embarked on a journey to develop their capacities to conduct formal evaluations of local programs. Since that time, fourteen sites have traveled what is becoming a well-established LSRI route, completing research and providing results of local and national interest. Tom Meyer, director of the Hudson Valley Writing Project, a member of LSRI Cohorts II and III, explains the local impact of the research:
LSRI has provided opportunities for members of our site to engage in research about our site . . . . Moreover, LSRI has consistently provided us with terrific support. On the one hand, the support has been technical, as we have been the beneficiaries of site visits during which we have worked on research design, instrument design, piloting of analytic strategies, and planning of our team’s work. The site visits have, in turn, created opportunities for our local research team and campus colleagues to meet with members of NWP’s national office. In many cases, these meetings have fortified support for NWP while also building understanding of our site’s general goals and research agenda.
Today, an expanded LSRI remains true to its mission to assist sites with efforts to conduct evaluation studies examining the impact of local programs. (A more complete description of the LSRI can be found in a 2005 Voice article.) Conceived in part as a response to growing interest in documenting the effects of NWP activities and in part as an effort to develop and support sites’ research capacities, the LSRI has created a unique structure to address the two biggest challenges of evaluating something like the NWP: the range of activity types at any particular site, and the fact that while adhering to common values and models of activity, sites are encouraged to adapt those models appropriately to be most effective in the local context.
In Missouri, LSRI provided important data that led to the collaboration of the Missouri state network with their state department of education to design and deliver professional development.
This flexibility to adapt the NWP models makes each instance of programming (even common programming like the summer institute) different from place to place in subtle but important ways. Add in the fact that programs at the local sites are incredibly varied (ranging from summer institutes to professional development partnerships to direct services provided to English language learning children of migrant families) and it becomes apparent that no one singular research design can appropriately fit all these circumstances.
To respond to this challenge, the LSRI encourages sites to develop research designs that are locally focused and locally responsive, therefore more appropriate to local contexts and circumstances. While also engaging in a variety of activities (e.g., participation in “sharing conferences” in which sites learn from and contribute to the learning of each other; the provision of technical assistance around research design, measurement, and analysis issues, etc.) to ensure rigor in terms of research design and implementation, the LSRI is developing a body of evidence that describes NWP impact and effects.
Rigorous Comparative Studies
Every one of the LSRI studies involves the examination of NWP programming through a pre/post measure design comparing the performance of students of teachers participating in NWP programming with that of students of teachers who have not. Every one of the LSRI studies involves the direct performance assessment of student writing. Every one of the LSRI studies had their student writing independently scored at a national scoring conference at which a specially developed rubric was applied with considerable reliability and technical rigor.
Table 1 summarizes the findings of the LSRI research conducted by Cohorts II and III and scored at NWP’s national scoring conference. On that table, upward-facing triangles represent positive findings in which students in the classes of NWP participants outscored their counterparts in comparison classes. If those triangles are solid, the differences were large enough to be considered statistically significant. Although there are none reported, circles represent comparisons with no differences between the groups, and downward-facing triangles would indicate comparisons favoring the comparison groups.
Table 1: Summary of LSRI Results (2006–2007)
Table 1 documents the consistently positive findings in this growing body of evidence. One can readily see that with seven independent measures of writing performance tested across nine separate studies, students in NWP-led classes outperformed their non-NWP counterparts in all 63 contrasts. In 31 of these, the differences were so great as to be statistically significant. The consistency of these findings (that in no case did the comparison group outperform the students in NWP classes) demonstrates the effectiveness of NWP’s work.
Findings Indicate NWP Is Effective
There are at least two interesting patterns in these data as well. The first is the strong results in the areas of Content, Organization, and, especially, Stance (which is very similar to Voice). These elements concern the quality of thought and the manner in which it is expressed in the writing—what many would argue are the most important elements. To the extent that these attributes are a considerable focus of many NWP efforts, these strong results are quite gratifying.
And second, it is noteworthy that Conventions also exhibits strong positive results favoring students in the NWP participants’ classes. In four of nine studies (44%), the results were significant. For those who wonder if an emphasis on ideas and written expression hinders the development and mastery of conventions, we see here evidence that it does not—that these skills can be learned as a product of attention to content, organization, expression, and the like.
Practices That Help Students Improve Their Writing
In the recently released report Writing Next, authors Steve Graham and Doris Perrin (2007) synthesize the available research and identify thirteen practices that consistently demonstrate a positive relationship to improved student writing. NWP’s LSRI results are certainly consistent with Graham and Perrin’s meta-analysis of the research. Moreover, Graham and Perrin identify one practice that the research suggests is actually associated with poorer writing performance. That practice is the explicit addressing of grammar and conventions in a manner disassociated from writing itself.
However, while their synthesis of the research suggests that we do not help writing quality by focusing on conventions in the abstract, it does not comment on the reverse: whether there are any means other than an exclusive focus on them by which conventions can be successfully taught. The LSRI data suggest that there is: a mastery of conventions can be realized through an emphasis on them in the context of writing.
Two Benefits of LSRI
The LSRI has two important benefits: Not only are we able to generalize and document some of the effects of writing project principles and practices across LSRI sites; the individual sites also benefit from their locally focused program evaluations. Anne Whitney, then of the South Coast Writing Project, shared some of the influences of the site’s research on its programs:
Our LSRI research had an immediate effect on the way we carried out the inservice program we were studying. Identifying specific areas to focus on in our study helped us to prioritize among the goals of the program. And as data came in, we were able to see with new clarity how our practice lined up with those goals—or didn’t.
In Missouri, LSRI provided important data that led to the collaboration of the Missouri state network with their state department of education to design and deliver professional development. Diane Scollay, director of the Gateway Writing Project, described conversations with the state director of assessment where the importance of local impact data was discussed:
He said, “I am going to need some research that backs up your claims.” Without hesitation, I could say, “We have it.”
Scollay then explained the results of Gateway's LSRI study conducted with a local Missouri school district and provided him a copy of the full report sent to the U.S. Department of Education. A week later, the director of assessment made the decision to sponsor the Missouri Writing Project’s Network Literacy Academies.
Site by site across the LSRI community these stories are repeated—local translations of research and evaluation influencing not only the work of the sites, but also the schools and state systems in which they work to improve the teaching of writing. As the Local Site Research Initiative expands and deepens efforts to research and evaluate the effects of NWP programs, the local writing project researchers will no doubt continue to build a research base for writing improvement programs and a portfolio documenting their effectiveness.
Graham, S., and D. Perin. 2007. Writing Next: Effective Strategies to Improve Writing of Adolescents in Middle and High Schools – A Report to Carnegie Corporation of New York. Washington, DC: Alliance for Excellent Education.