Why Evidence-Based Practices Don’t Work: Part I

To be clear, I am in favor of building a strong education R & D sector. However, it’s important to acknowledge the serious shortcomings of the current system. It is because of this current state that I am arguing that evidence-based practices don’t work.

I’m making two claims.

Claim #1: The current evidence-base in education research is extremely thin at best and completely misleading at worse.
Claim #2: By their very design, studies that result in evidence-based practices discount externalities instead of solving for them.

Probably Wrong, Definitely Incomplete

On its face, probably wrong, definitely incomplete doesn’t inspire confidence. However, we put this phrase in the footer of almost all of the documents we create at School Performance Institute (SPI) to remind us of its value as a core ethos of our improvement work. When people first see this phrase they typically have a reaction something like, “Wow, that doesn’t inspire confidence!”

We disagree.

Launching Improvement Work

We’ve spent the last year learning how to use improvement science to increase the high school readiness rates of our 8th graders. Currently, we're gearing up to spread this improvement work from one middle school improvement team to a second middle school team at United Schools Network. During our upcoming launch day, we'll spend time creating team norms, studying improvement science, and digging into each school's on-track indicator system data. Thoughtful planning and a strong start to this work makes it much more likely that we’ll achieve our ambitious goals this school year.

Lessons in Improvement

Earlier this year, I kicked off our Learning to Improve blog series by discussing seven early lessons from our improvement science work at United Schools Network. The good thing is that I think that all of those lessons were on point. But, we’ve also learned a great deal about using improvement science methodology through our 8th Grade On-Track project this school year. In this post, I will expand on those early lessons and present some of our new learning.

Learning to Improve at United Schools Network

James’ reading grade dropped from a B in 7th grade to a D during the first trimester of his 8th grade year. The rest of his grades were a C or higher, his attendance rate was above 96%, and he had never been in serious trouble. Most people would look at James’ academic, attendance, and behavior stats and not see a student that is in need of intervention. We disagree. At the very moment that his reading grade dropped, James was in need of extra support.

Partners in Improvement

This school year, School Performance Institute (SPI), the learning and improvement arm of United Schools Network, served as a partner with Columbus Collegiate Academy (CCA) to work on a tough-to-solve problem. SPI fills a number of critical roles including project management, strategic communication, human capital management, improvement research & advising, and data analytics. At our core is a commitment to continuous learning and improvement and a focus on using data to predict students’ learning, progress, and ultimately their success.

A Formula for Improvement

In my last post, I provided an analysis for the educational funding strategy shift at the Gates Foundation.

The gist of the shift is that Gates is funding organizations that are marrying on-track indicator systems with improvement science in order to improve important student outcomes. I know this because I went after a Gates Grant last year and didn’t get it. But, I’m glad I did it because of the learning that came out of the process. In fact, it led to the current 8th Grade On-Track project I’m running at Columbus Collegiate Academy this school year.

Why the Gates Foundation Shifted Its Education Funding Strategy

The marrying of on-track indicator systems with improvement science has caught the attention of the Bill & Melinda Gates Foundation. Over the next five years, Gates plans to direct 60 percent of its $1.7 billion in education funding towards networks of schools working together to identify solutions to local problems and using data to drive improvement.

Don’t Use Accountability Goals in School Improvement Work

Goals for accountability and goals for improvement are two very different things.

I’m not sure if I should be embarrassed or if most of the folks reading this are right there with me. Either way, this was an important revelation as I’ve worked to bring improvement science methodology to my work as a school leader at United Schools Network. Far too often, these two types of goals get conflated during school improvement projects and this can have unintended consequences.

What is Good Attendance?

If you asked your colleagues, the answers you get may surprise you.

It’s something I’d encourage you to do- run an experiment at your school or district- and see what happens. A few months ago, I did just that.

Is Professional Development Effective?

Despite spending billions on professional development (PD) in the United States, well-regarded studies by organizations such as American Institutes for Research (AIR) and TNTP show no measurable results from this spending. This was the case even when the training was considered rigorously aligned to the tenets of best practice for educator professional development. At the same time, student outcomes, especially for students of color and students living in poverty, are unacceptable.

Know-How.

The good news is this may very well be the much sought after silver bullet we’ve long searched for in educational improvement circles. The bad news is that this elusive magic elixir is very hard to come by. Simply put, know-how is the detailed practical knowledge necessary to get good ideas to actually work in classrooms, schools, and districts.