Does it matter what gender your child's teacher is? (Read Our New Piece At Time.com)
From Po and Ash:
As kids file back into school this week, news media have been pronouncing that there’s a new hidden enemy in our children’s classrooms. The teachers. A just-released Hoover Institute journal, Education Next, published “new and convincing evidence” that teachers’ gender has “large effects” on student performance. With 80% of teachers being women, it seemed like this was the answer for the "boy crisis" – claims that boys are increasingly behind in school and disappearing from college campuses.
The trouble is that the evidence is neither new nor convincing. We've written a new piece for Time.com with an overview of our problems with the study and the coverage. But we've got more specifics we want you to be aware of.
Our Time essay mentions how small this teacher-gender-effect is, (it might raise or lower your child's score by 1 point on a hundred point test), and laughs at how small that is compared to other things that affect a child's test scores. But in the Time essay we didn't get a chance to fully critique this study's methodology. We'll do that here. We're putting in the extra effort to do so because we believe this study will be used as a political tool by every politician looking to create more single sex schools. We're not opposed to single sex schools, just to using faulty research to claim their superiority.
The report is by Thomas Dee, normally a fairly well respected professor at Swarthmore, who is at Stanford's Hoover Institution for the year.
He makes the following errors in his report's wording and methodology:
1. He says teacher gender has never been studied as a potential reason that boys often lag girls in reading, while girls lag boys in math and science. But this has actually been studied quite a bit, and Thomas Dee knows it - he even cited some of those papers in earlier articles he had written.
There are a number of studies on elementary school children's achievement and the possible impact their teachers' genders may have on it. A 2005 British study of 9,000 children determined teachers’ gender did have an effect: female teachers were always more effective than males, regardless the sex of the students. A study of urban African-American students had a similar result: students did better in science with female teachers than with male teachers.
But Dee's claim is particularly egregious since, in 1995, researchers, lead by Cornell University professor Ronald G. Ehrenberg, studied the exact same NELS data Dee is using. And the question they asked was the same: if teachers' gender affect students' academic achievement.
Dee is well-aware of this earlier study, because he himself has written about it in at least two papers (that we found). So why would Dee suddenly act as if it didn't exist? The only possible explanation we can come up with is that, if he had mentioned it, then he would have had to acknowledge its findings. Which would be a problem for Dee, because they are diametrically opposite to his own.
Ehrenberg's team found that teachers' gender did not affect their students' achievement.
They found teachers' race, not gender, did impact their assessment of the children’s work – which might cause problems for future academic placement – but teachers' race didn't impact the children's achievement, either.
Since Dee didn't explain how the results could be so different, we called Ehrenberg, to see if he could explain it to us.
When we asked about the new report, Ehrenberg hadn't had the opportunity to examine it closely, and he remarked that he'd generally respected Dee's scholarship. But he was "a little uncomfortable," having observed it was unclear on a several points.
For example, Ehrenberg’s earlier research compared students’ achievement over a period of years. But it appears that Dee may have only examined the students’ eighth grade test results alone. Without a history of subsequent achievement, there’s no way to know if a child did poorly because of the teacher’s gender, or because he simply struggled in that subject at that point in time.
In fact, Ehrenberg reminded us, we really need to know the child's prior experiences in school as well: if the key to achievement is teacher gender, then we should know sex of the teachers before the children ever got to the eighth grade, too.
We felt like Ehrenberg's brief assessment essentially gutted the value of Dee's report.
But it turned out that's just the tip of the iceberg.
2. We also spoke to Leslie Scott, a Principal Researcher at American Institutes for Research. Scott was a manager and designer of the NELS survey. She's been working on collecting and analyzing this very data since 1987.
Concurring with Ehrenberg that Dee problematically seemed to be looking at a single point in time, Scott explained that was particularly concerned her because the original survey never found a gender gap of achievement to begin.
So she finds it perplexing how Dee can use this data to determine causation of a phenomenon they didn’t find in the first place.
3. If all that wasn't bad enough, Dee’s statistical calculations don’t apply to this data at all.
Dee relied upon a statistical method that charts grades on a 100-point-wide curve. The result is a chart that has a 30 percent of a standard deviation between a grade – say an "A" and a "B." Using this method and his data, Dee determined that there is a “four percent of a standard deviation” difference between boys and girls’ grades, if they have a teacher of the opposite sex.
That sounds ominous if you’re a parent getting your kindergartner ready for their First Day. But if you’re a statistician, that’s laughable. (And we know, because we called a statistician, and he literally laughed at us.)
As we said at the beginning, if Dee’s methodology is correct, for a boy who might have scored a 85% on an math test if he had a male teacher, if he has a female teacher might see his test score plummet to . . . 84%. It's about one to two percentage point swing in results. That’s it.
But Scott says that the whole premise of the standard deviation test doesn't even apply to NELS. Dee transferred that 30-point scale from another study to hers. According to Scott, NELS didn't calculate its data that way, so you simply cannot do that transfer and come up with accurate results.
Questionable at best scholarship is bad enough. But the media coverage of it is a real problem. Here's why.
The danger of reports such as these, says Ehrenberg, is they are used to advocate much stronger policy positions than the findings warrant. And, he asks, do we want to use data like this to champion segregating our kids by sex and race?
(That's already happening. A local ABC affiliate in Denver contacted state and local officials about the report, and they replied that the issue needed to be taken seriously and studied further.)
No doubt – without examining the data or, more significantly, what that sort of division that may mean for society as a whole – there will be people waving this study around for years.
Parents will now be all-too-sure the answer to their sons’s education is the gender of teachers. Will those parents ever hear that reading is the activity linked to improved test scores? But the average kid spends 14 and a half hours a week watching television and just 90 minutes with a book?
How many supervisors will complain how few men are teaching – saying that’s the problem reaching troubled kids? In so doing, they'll avoid the real problem: not a lack of a Y-chromosone, but those without teaching certificates and expertise in the fields they teach.
How many hours will a school board waste blaming the taxpayers for refusing to give them the extra funds needed to establish single-sex classrooms – when there is no debate about the lack of funds in our inner city schools. The textbooks out of date years before the Solar System shrunk last week, the children hungry and shivering in classrooms without heat.
The real problem here is a policy wonk-pushed, media-driven desperation to find the Ginsu knives that slices the budget, dices complex issues into soundbites, magically saves our kids, fixes our schools, and more.
We distract ourselves with a few news cycles about how a teacher’s gender may explain all that is wrong with teaching our children. But it is just that. A distraction. We’re as distracted by such headlines as little kids are by the humming of a lightbulb. And we use that distraction to point fingers instead of getting our hands dirty, doing the real work that needs to be done.
As kids file back into school this week, news media have been pronouncing that there’s a new hidden enemy in our children’s classrooms. The teachers. A just-released Hoover Institute journal, Education Next, published “new and convincing evidence” that teachers’ gender has “large effects” on student performance. With 80% of teachers being women, it seemed like this was the answer for the "boy crisis" – claims that boys are increasingly behind in school and disappearing from college campuses.
The trouble is that the evidence is neither new nor convincing. We've written a new piece for Time.com with an overview of our problems with the study and the coverage. But we've got more specifics we want you to be aware of.
Our Time essay mentions how small this teacher-gender-effect is, (it might raise or lower your child's score by 1 point on a hundred point test), and laughs at how small that is compared to other things that affect a child's test scores. But in the Time essay we didn't get a chance to fully critique this study's methodology. We'll do that here. We're putting in the extra effort to do so because we believe this study will be used as a political tool by every politician looking to create more single sex schools. We're not opposed to single sex schools, just to using faulty research to claim their superiority.
The report is by Thomas Dee, normally a fairly well respected professor at Swarthmore, who is at Stanford's Hoover Institution for the year.
He makes the following errors in his report's wording and methodology:
1. He says teacher gender has never been studied as a potential reason that boys often lag girls in reading, while girls lag boys in math and science. But this has actually been studied quite a bit, and Thomas Dee knows it - he even cited some of those papers in earlier articles he had written.
There are a number of studies on elementary school children's achievement and the possible impact their teachers' genders may have on it. A 2005 British study of 9,000 children determined teachers’ gender did have an effect: female teachers were always more effective than males, regardless the sex of the students. A study of urban African-American students had a similar result: students did better in science with female teachers than with male teachers.
But Dee's claim is particularly egregious since, in 1995, researchers, lead by Cornell University professor Ronald G. Ehrenberg, studied the exact same NELS data Dee is using. And the question they asked was the same: if teachers' gender affect students' academic achievement.
Dee is well-aware of this earlier study, because he himself has written about it in at least two papers (that we found). So why would Dee suddenly act as if it didn't exist? The only possible explanation we can come up with is that, if he had mentioned it, then he would have had to acknowledge its findings. Which would be a problem for Dee, because they are diametrically opposite to his own.
Ehrenberg's team found that teachers' gender did not affect their students' achievement.
They found teachers' race, not gender, did impact their assessment of the children’s work – which might cause problems for future academic placement – but teachers' race didn't impact the children's achievement, either.
Since Dee didn't explain how the results could be so different, we called Ehrenberg, to see if he could explain it to us.
When we asked about the new report, Ehrenberg hadn't had the opportunity to examine it closely, and he remarked that he'd generally respected Dee's scholarship. But he was "a little uncomfortable," having observed it was unclear on a several points.
For example, Ehrenberg’s earlier research compared students’ achievement over a period of years. But it appears that Dee may have only examined the students’ eighth grade test results alone. Without a history of subsequent achievement, there’s no way to know if a child did poorly because of the teacher’s gender, or because he simply struggled in that subject at that point in time.
In fact, Ehrenberg reminded us, we really need to know the child's prior experiences in school as well: if the key to achievement is teacher gender, then we should know sex of the teachers before the children ever got to the eighth grade, too.
We felt like Ehrenberg's brief assessment essentially gutted the value of Dee's report.
But it turned out that's just the tip of the iceberg.
2. We also spoke to Leslie Scott, a Principal Researcher at American Institutes for Research. Scott was a manager and designer of the NELS survey. She's been working on collecting and analyzing this very data since 1987.
Concurring with Ehrenberg that Dee problematically seemed to be looking at a single point in time, Scott explained that was particularly concerned her because the original survey never found a gender gap of achievement to begin.
So she finds it perplexing how Dee can use this data to determine causation of a phenomenon they didn’t find in the first place.
3. If all that wasn't bad enough, Dee’s statistical calculations don’t apply to this data at all.
Dee relied upon a statistical method that charts grades on a 100-point-wide curve. The result is a chart that has a 30 percent of a standard deviation between a grade – say an "A" and a "B." Using this method and his data, Dee determined that there is a “four percent of a standard deviation” difference between boys and girls’ grades, if they have a teacher of the opposite sex.
That sounds ominous if you’re a parent getting your kindergartner ready for their First Day. But if you’re a statistician, that’s laughable. (And we know, because we called a statistician, and he literally laughed at us.)
As we said at the beginning, if Dee’s methodology is correct, for a boy who might have scored a 85% on an math test if he had a male teacher, if he has a female teacher might see his test score plummet to . . . 84%. It's about one to two percentage point swing in results. That’s it.
But Scott says that the whole premise of the standard deviation test doesn't even apply to NELS. Dee transferred that 30-point scale from another study to hers. According to Scott, NELS didn't calculate its data that way, so you simply cannot do that transfer and come up with accurate results.
Questionable at best scholarship is bad enough. But the media coverage of it is a real problem. Here's why.
The danger of reports such as these, says Ehrenberg, is they are used to advocate much stronger policy positions than the findings warrant. And, he asks, do we want to use data like this to champion segregating our kids by sex and race?
(That's already happening. A local ABC affiliate in Denver contacted state and local officials about the report, and they replied that the issue needed to be taken seriously and studied further.)
No doubt – without examining the data or, more significantly, what that sort of division that may mean for society as a whole – there will be people waving this study around for years.
Parents will now be all-too-sure the answer to their sons’s education is the gender of teachers. Will those parents ever hear that reading is the activity linked to improved test scores? But the average kid spends 14 and a half hours a week watching television and just 90 minutes with a book?
How many supervisors will complain how few men are teaching – saying that’s the problem reaching troubled kids? In so doing, they'll avoid the real problem: not a lack of a Y-chromosone, but those without teaching certificates and expertise in the fields they teach.
How many hours will a school board waste blaming the taxpayers for refusing to give them the extra funds needed to establish single-sex classrooms – when there is no debate about the lack of funds in our inner city schools. The textbooks out of date years before the Solar System shrunk last week, the children hungry and shivering in classrooms without heat.
The real problem here is a policy wonk-pushed, media-driven desperation to find the Ginsu knives that slices the budget, dices complex issues into soundbites, magically saves our kids, fixes our schools, and more.
We distract ourselves with a few news cycles about how a teacher’s gender may explain all that is wrong with teaching our children. But it is just that. A distraction. We’re as distracted by such headlines as little kids are by the humming of a lightbulb. And we use that distraction to point fingers instead of getting our hands dirty, doing the real work that needs to be done.
posted by Ashley Merryman at 9:30 AM
3 Comments:
Hurrah!! Thank you so much for actually critiquing the study and not just taking it for granted that it must be right. This kind of thing happens all the time in education where politics and money are so important. It is so hard for the public to sift through the good, bad, and ugly of educational research when a politician gets behind "one" study and makes that study the sole decision making tool for everyone. Now if we could only look at reading research in this way.
Thanks very much!
This comment has been removed by a blog administrator.
Post a Comment
<< Home