Quantum Forest

notes in a shoebox

I like the idea of having data on school performance, ask not to directly rank schools—hard, to say the least, at this stage—but because we can start having a look at the factors influencing test results. I imagine the opportunity in the not so distant future to run hierarchical models combining Ministry of Education data with Census/Statistics New Zealand data.

At the same time, there is the temptation to come up with very simple analyses that would make appealing newspaper headlines. I’ll read the data and create a headline and then I’ll move to something that, personally, seems more important. In my previous post I combined the national standards for around 1,000 schools with decile information to create the standards.csv file.

Up to this point we have read the data, removed special schools and created variables that represent the proportion of students that al least meet standards. Now I’ll do something very misleading: calculate the average for reading.OK for each school decile and plot it, showing the direct relationship between socio-economic decile and school performance.

Scatterplot of average proportion of students at least meeting the reading national standards for each socioeconomic decile.

Using only the average strengthens the relationship, but it hides something extremely important: the variability within each decile. The following code will display box and whisker plots for each decile: the horizontal line is the median, the box contains the middle 50% of the schools and the vertical lines 1.5 times the interquantile range (pretty much the range in most cases):

Box and whisker plot for proportion of students at least meeting the reading national standard for each decile. Notice the variability for each decile.

A quick look at the previous graph shows a few interesting things:

• the lower the decile the more variable school performance,
• there is pretty much no difference between deciles 6, 7, 8 and 9, and a minor increase for decile 10.
• there is a trend for decreasing performance for lower deciles; however, there is also a huge variability within those deciles.

We can repeat the same process for writing.OK and math.OK with similar trends, although the level of standard achievement is lower than for reading:

Box and whisker plot for proportion of students at least meeting the writing national standard for each decile.

Box and whisker plot for proportion of students at least meeting the mathematics national standard for each decile.

Achievement in different areas (reading, writing, mathematics) is highly correlated:

Scatterplot for proportion meeting mathematics and reading national standards. Dark points are deciles 1 to 5, while light points are deciles 6 to 10. Notice the large overlap for performance between the two groups.

All these graphs are only descriptive/exploratory; however, once we had more data we could move to hierarchical models to start adjusting performance by socioeconomic aspects, geographic location, type of school, urban or rural setting, school size, ethnicity, etc. Potentially, this would let us target resources on schools that could be struggling to perform; nevertheless, any attempt at creating a ‘quick & dirty’ ranking ignoring the previously mentioned covariates would be, at the very least, misleading.

Note 2012-09-26: I have updated data and added a few plots in this post.

1. Great post! I saw this data come out and thought something interesting could be done with it.

• Luis

2012/09/25 at 10:06 am

Thanks a lot for the positive comment. I think that besides demonizing particular schools we could combine lots of different data sources to try to understand what is driving not only the averages but the underlying variability.

2. “Using only the average strengthens the relationship, but it hides something extremely important: the variability within each decile.”

This was something Hans Rosling pointed out in one his first TED lectures wasn’t it? Can’t remember exactly which one, changed the way I looked at global/national statistical data though.

• Luis

2012/09/25 at 10:04 am

You are right Ben. Rosling showed in his 2006 TED talk that the variability within Africa spanned from receiving humanitarian aid (Sierra Leone) to places where one would go to a resort for holidays (I think this was Mauritius). We are at the same stage in the conversation on education, we have some ‘African deciles’, where we are grouping highly successful schools with struggling ones and pointing out that ‘this decile is a problem’.