Earlier this week CREDO, the education research outfit at Stanford led by Macke Raymond, released another in its series of city, state, and national evaluations of charter school performance. This one was on Ohio (pdf). The studies are an amusing Rorsarch test for charter critics. The ones about places where charters are underperforming are widely cited and CREDO is presented as an august institution to be heeded in a Solomon-like fashion. When one comes out showing a city or state where charters are dramatically outpacing other schools it’s crickets or suddenly CREDO is another front group for “corporate reform.”

Actually, CREDO is none of those things but it’s a good research shop offering a great analytic view into how charters are playing out in different places. This week’s Ohio analysis, in broader context, offers some important lessons.

First, beware the ecological fallacy. Not every charter in Ohio is dreadful and there are some quite good ones. That said, overall the state is a charter debacle. If your only experience with charter schools was Ohio it would be understandable if you thought the entire idea was essentially flawed. Within Ohio there are cities doing a better or worse job. For instance Cleveland, the site of some interesting charter innovation, is an outlier high within in the state. Also pay attention to the different impact on different socioeconomic, racial, and ethic groups. Still, the overall story remains discouraging.

Second, this isn’t new. Ohio has been a laggard for some time and despite multiple evaluations pointing this out for more than a decade (Sara Mead and I included it in multi-state charter evaluation we led in the early part of the 2000s and things were not good then). More importantly, the state has missed numerous opportunities to improve its policies and by extension its charter operations. Policy mistakes in the early going of chartering were par for the course, that’s what innovation looks like. But Ohio has failed to learn from its own experience and the experience of other states that are higher performing. That’s inexcusable. The CREDO analysis says that more recent reform efforts are only, “dimly discernible” in the charter data. Bellwether is working with some charter leaders in Ohio on ways to use policy to accelerate the pace of improvement.

Third, policy and practice matter, this is not a crapshoot. There are places around the country that are much higher performing. CREDO analyses of Los Angeles, New Jersey, Massachusetts (and specifically Boston), New York City, and elsewhere show that charters can dramatically change achievement and outcomes for students if policymakers get the balance between freedom and accountability right. What does dramatically look like? In Los Angeles the effect sizes are equivalent to 50 days of additional reading instruction and 79 days of additional math instruction. On Massachusetts and Boston CREDO reports,

…on average, charter students in Massachusetts gain an additional one and a half months of learning in reading over their [traditional public school] counterparts. In math, the advantage for charter students is about two and a half months of additional learning in one school year. Charter students in Boston gain an additional 12 months in reading and 13 months in math per school year compared to their [traditional public school] counterparts.

Extended learning time anyone?

Fourth, other jurisdictions are learning lessons Ohio is not. In the public back and forth about charters critics frequently cite CREDO as evidence that most charters aren’t any better or are worse than other public schools. But that data has been superseded by more recent CREDO analyses (pdf) that show (a) the aggregate performance picture is better and (b) it’s improving as non-renewals, closures, and better authorizing decisions and practices on the front-end, start to have an effect. CREDO describes the landscape this way in its most recent (2013) report:

In the aggregate, both reading and math results show improvement compared to the results reported in Multiple Choice. The analysis of the pooled 27 states shows that charter schools now advance the learning gains of their students’ more than traditional public schools in reading. Improvement is seen in the academic growth of charter students in math, which is now comparable to the learning gains in traditional public schools. On average, students attending charter schools have eight additional days of learning in reading and the same days of learning in math per year compared to their peers in traditional public schools. In both subjects, the trend since 2009 is on an upward trajectory, with the relative performance of the charter sector improving each year. Related results for different student groups indicate that black students, students in poverty, and English language learners benefit from attending charter schools. 

That’s a little different from the boilerplate descriptions of charter performance one usually hears.

Fifth, charter supporters shouldn’t declare mission accomplished just yet. There are still too many low-performing charters and quality is not yet where it should be. CREDO’s 2013 report also states plainly that, “…charter school quality is uneven across the states and across schools.“ Meanwhile, on the horizon there are complicated questions about what sort of obligations the charter sector overall should bear as it becomes a larger part of the education sector. There are 2.7 million students in charters now and just this month the National Alliance for Public Charter Schools reported that 150 school districts have at least 10 percent of their students in public charter schools and in 12 districts more than a third of students are now educated in charters. Parental demand and charter growth are increasing.

Finally, an often-overlooked piece of context here. Most charter schools are still authorized by school districts. So the quality problem is at least as much a problem of school district decision-making as it is of charters themselves. Something to think about for those who think the current system design works fine and just needs more of various inputs.