June 12, 2018

The NBA Playoffs, Match-Ups, and Teacher Fit: An Interview with Steve Glazerman

By Bellwether

Share this article

The NBA Finals wrapped up on Friday, but they offered a nice reminder about the importance of how employees fit within an organization and context. Players who looked like All-Stars in one round of the playoffs became unplayable in the next due to match-up problems. In other cases, players who washed out in other contexts were suddenly relevant again.

Victor Oladipo

photo of Victor Oladipo via Flickr user Keith Allison

On a recent podcast, Malcolm Gladwell used the NBA playoffs as an entry point to talk about worker fit. Gladwell gave the example of NBA player Victor Oladipo. Oladipo was the second pick in the 2013 draft, made the All-Rookie team his first year, and then…didn’t quite live up to his perceived potential. He was traded once but didn’t mesh well with his new teammates, and then he was traded again, this time to the Indiana Pacers. Suddenly he looked like a different player. He made the All-Star team this year and almost took down LeBron James and the Cleveland Cavaliers in the first round of the playoffs.

Gladwell’s point is that we should think much more carefully about employee fit. A successful worker somewhere may not be successful everywhere. Individuals are dependent on their teammates and organizational supports; context matters.

Steve Glazerman

How much does fit matter for teachers? Is it as important in schools as it seems to be in basketball? To learn more, we reached out to Steve Glazerman, a Senior Fellow at Mathematica Policy Research. We talked to him at the end of last month, just as the NBA Finals were starting. What follows is an edited transcript of our conversation.  
Chad Aldeman: You helped lead an evaluation of a federal program called the Talent Transfer Initiative. Can you tell us about what the program tried to do, whether it was successful, and what we can learn from it?
Steve Glazerman: The report came out in 2013. It tested the idea that one way to improve low-performing schools is to give them greater access to high-performing teachers. It acknowledged that while it’s always good to be able to develop your teachers into high-performing teachers, sometimes the high-performing teachers are already there in the district, and you just need to align the incentives so that they want to work with the students who need them the most. So, the intervention was designed to identify and offer fairly large incentives for high-performing teachers — typically the top 20 percent of performers in the district — to transfer into a set of schools that were identified as low-achieving.
Our study measured the impact the teachers had on their new schools. We found that on average, those teachers did have a positive impact on their new schools, as you would expect if two things were true: one, that the measure we were using to identify high-performing teachers, which was a value-added measure of teachers’ impact on student test scores, was a good measure and was measuring something real; and two, the skill that teachers had demonstrated in their high-performing schools would transfer to low-performing schools.
A reasonable conclusion might be that based on the study findings — and the study was conducted as a randomized, controlled trial — those two conditions were met.
Aldeman: Summarizing that study as well as others in the literature, what do we know about teacher fit and whether teaching ability translates across different contexts, in terms of schools as well as subjects (for elementary school teachers who might be teaching multiple things)?
Glazerman: My friend Peter Youngs has helped lead other work on person-organization fit in schools. In our study, when we broke down the findings between elementary and middle school, the results were driven by elementary schools. All the positive impacts were generated by the transfers, and that’s what most of the sample was. We didn’t really have a large sample at the middle school level anyway,  so there are a lot of possible explanations for why, when we looked at grade-spans separately, there were different results for elementary school and middle school.
One possible explanation — which is one of many — is that skill and teacher effectiveness transfer is more of an elementary school phenomenon than a middle school phenomenon. There are a lot of ways in which middle schools and elementary schools differ. Middle schools are departmentalized, and in elementary schools, some are and some are not. In middle school you’re teaching a subject, as opposed to a group of students to whom you’re teaching multiple subjects. There are also other factors, having to do with school size, the degree of contrast between the types of schools that teachers were transferring from, and the types of schools that teachers were transferring to.
One thing that was unique about our study was that previous studies generally looked at teacher movement as it existed in the wild, and what we know is that effective teachers tend to move from lower-performing to higher-performing schools. They don’t go in the other direction. By providing incentives — in this case $20,000 over a two-year period — we effectively reversed some of that flow in order to observe that kind of movement. But one thing that is true is that there was variation in the degree of contrast between the sending schools and the receiving schools, and that might be relevant for interpreting the findings.
Kaitlin Pennington: What do we know about teacher fit as it pertains to various stages in the teacher pipeline? Let’s start with teacher preparation. How should a teacher preparation program think about teacher fit?
Glazerman: I wouldn’t call myself an expert on this topic. But you can actually look at teacher preparation programs or universities and observe the kinds of schools they place teachers into. A lot of times there is a direct flow. In an urban area, a university based in that area will send their teachers to that city’s public school district. You’re starting to see an increase in residency programs and other teacher prep programs. These programs try to make sure that prospective teachers will have opportunities to practice in the same environment where they are going to teach, either as interns, residents, or student teachers, more than for just a token few weeks of student teaching but instead something like a semester or a school year.
Even though these programs are growing, we don’t have a lot of evidence either on the impact of teacher residency programs, the impacts of the residency component of teacher prep programs, or for particular types of placements. So that’s a good question, and I’m afraid we probably don’t have a good answer to it.
Aldeman: What about other stages of a teacher’s career, like the teacher’s classroom assignment, professional development, evaluation, etc.? What are the implications for teacher fit as it interacts with other parts of a teacher’s job?
Glazerman: It would be relevant to look at the various rules that govern the placement and assignment of teachers, particularly within larger school districts.
The matching of teachers to classrooms and to teaching assignments has varying degrees of mutual consent. The notion of mutual consent is important. If you have a system where teachers don’t have a say over where they’re placed, that strikes me as a formula for potentially ending up with poor fit. When you look at the factors that teachers cite as a reason why they would or would not work in a particular school — and this came up a lot when we were focusing on the recruitment side of the Talent Transfer Initiative — the number one factor that is mentioned is the principal.
Maybe the fit question here is the fit between the teacher and the school leader. A really good teacher, we were told repeatedly, would never work for someone who isn’t a strong principal, and they would follow a strong principal anywhere, somewhat irrespective of the other kinds of factors.
It would be naive to think that pay and other kinds of working conditions aren’t relevant, but this fit between teacher and principal seems to be important. I’m not sure who’s documented that well, so that may be somewhat anecdotal, but that comes up a lot and was cited a lot as a key factor — one we needed to design that intervention around.
Aldeman: You mentioned earlier that in the Talent Transfer Initiative, you got more confident in the tools used to evaluate a teacher. Can you say more about that and about the broader implications for teacher evaluation systems?
Glazerman: There’s been a long-running controversy about test-score-based measures of teacher performance. Value-added measures, growth percentiles — they get called different things, but they’re all statistical estimates of the impact of a teacher on student test scores.
A lot of that debate doesn’t go anywhere. We actually tend to agree on certain things, specifically that those measures are often noisy. The disagreement is what to do with that noise. One group would say they’re noisy so we can’t use them; another group would say they’re noisy so we should use them wisely or use them in a certain way that informs our thinking.
Our study was somewhat unusual in that value-added was the only criterion for screening teachers. They had to have been teaching in the tested grade for 3 years in most cases, and they had to have achieved value-added scores that were in the top 15 or 20 percent of the district. It varied slightly by the district, and there were 10 districts in the study. We haven’t had a whole lot of evidence on the validity of these estimates. Do they actually measure what they’re trying to measure? Because again they’re just estimates.
The most persuasive evidence in favor of using value-added estimates comes from Chetty, Friedman, and Rockoff in their famous two-part study where they looked at teacher switching as strategies for assessing whether exposure to a high-value-added teacher or to a low-value-added teacher affects the outcomes of students, both in the short-term and long-term. It was fairly convincing, but we haven’t had much where we have a random assignment of students to teachers or teachers to classrooms, which gives us some ability to compare similar students while the teachers vary in their value-added scores. So we sort of have experimental evidence on whether value added is associated with true impact, as measured by random assignment. It’s somewhat indirect for a number of reasons. One example was a re-analysis of data from the MET study, and the other example that I’d point to was a precursor to that, in a study that was done in Los Angeles Unified School District. And then finally there was our study.
I have an unpublished paper on this topic that actually looks at our results side-by-side with the L.A. results and the MET results. They are all basically consistent with the same conclusion, which is that the forecast bias associated with value-added estimates is not detectable. That’s not a strong conclusion, because in the case of our study, while the sample was perfectly adequate to do what it was meant to do, which is evaluate the impact of a particular intervention, it wasn’t designed for this analysis. So it’s not a strong conclusion, but it’s certainly consistent with the idea of value-added estimates being unbiased.
That’s sort of a quantified version of what I said earlier, which was that in order for a program like this to be effective, it presumes that there’s a signal embedded in value-added scores, even if they’re measured with error. That was comforting, not only in that it provided us some assurance that there is some useful signal in that measure, but also in that it was consistent with similar studies that used random assignment in other settings. Each of those studies has strengths and weaknesses, and so I think what our paper does is show that even though each of these studies has their own strengths and weaknesses, they all sort of offset each other and they all reach the same conclusion, which is that forecast bias is not significant in each case.
Aldeman: Can you explain “forecast bias” in a way that a generalist would understand? What would that mean for a teacher or a principal?
Glazerman: Bias is just the difference between what we’re trying to measure and what we actually measure. Forecast bias is basically saying, is there a difference between what we’re trying to measure, which is impact of teachers on student test scores, and what we actually measure, which is the value-added score of a teacher, and can we forecast the results of a randomized experiment? Or, in the case of Chetty, Friedman, and Rockoff’s paper, can it forecast the prediction based on a transfer? For example, if a teacher transfers into a certain grade level, their value-added score would imply a certain impact on those students subsequently. For the high value-added teacher, you expect certain scores to go up. That’s the forecast, and then they ask whether the results matched the forecast, and the answer is yes.
Pennington: What do we know about teacher transfers and retention? Did the teachers who were a part of your study have to stay in their schools for a certain number of years? Also, did you track job satisfaction while they were there?
Glazerman: No one can be forced to stay anywhere. We did measure retention over time. On purpose, the $20,000 payments weren’t conditional on subsequent performance, but they were paid out over a 2-year period. We found that retention of teachers who were part of the program was greater than retention in the control group after that first year and in the second year. But going into the third year, the retention rate was back down to a point where there was no longer a difference. One way to interpret that is that the payout delayed teacher exits that would have happened anyway, but otherwise teachers are going to make their career plans the way they’re going to make their career plans.
Aldeman: I’m going to over-generalize a little bit, but it almost seems like there’s probably some value in having fit, and teachers definitely value having principals that they like working with (and I’m sure there’s some spillover of effective colleagues). But what the literature suggests is that there’s actually quite a bit of definable, generic teaching ability that’s transferable across situations, and that we can somewhat identify that trait and follow it over time. Does that sound about accurate to you?
Glazerman: Yes, but an important qualification would be that whenever we look at any of this research (that I did or that you might be citing that looks at teachers in different settings), it is always voluntary. In the case of the incentive, it induced more teachers to move than would have, but it’s still voluntary. Many more teachers were offered incentives to transfer than actually transferred, because that’s how the program was designed. We knew that it would be a hard sell. We don’t know — nor would we necessarily want to know — what would happen if we randomly plucked a high-performing teacher in a very high-performing school and placed that teacher in a very different setting. You could say that the evidence shows that teacher skills transfer and a good teacher is a good teacher, but that might not be true. The job might be so completely different that this randomly plucked teacher isn’t good at it.
It’s pretty clear, particularly going into large, diverse, urban school districts or a county district that’s big and diverse, that you could meet two different teachers who are teaching ninth grade math and they do not have the same job. One of them is teaching students who are coming in ready to take ninth grade math and who need the standard curriculum that is inscribed in the state standards for ninth grade mathematics. The students are well-supported at home, and the teacher might be very good at doing that.
Somebody else is dealing with students who are three or four grade levels behind. They can’t even access the material that’s a prerequisite for being in that class, and with several years of instruction, they might still not be able to access it. That teacher has to adapt to a classroom full of students who are in those kinds of circumstances, and that is a very different job. They are facing all kinds of trauma or homelessness and numerous challenges outside the classroom that make it difficult for the students to succeed inside the classroom.
Aldeman: Okay, last question. Who’s going to win, the Cavs or the Warriors?
Glazerman: You’re asking a Celtics fan. I’m going to have to abstain. I’m bored, I don’t want to see this matchup again! 
[Editor’s note: Steve was right.]

More from this topic

Thank you! Your subscription has been confirmed. You'll hear from us soon.