Quantum Forest

notes in a shoebox

Page 2 of 18

A tad of novelty and adventure

Traveling for swimming competition—kiddo competing while I officiate—and staying in a motel. There is some novelty and adventure for a kid in jumping in a car and appearing in a slightly different landscape a few hours later. That sense of adventure has often been lost or drowned in adults, probably the product of too much business travel to (almost) pointless meetings. I should say brief adventure, as tonight will be early bed as Saturday starts at 7:30am sharp on the pool deck.

Driving down Canterbury on a Friday evening is an easy and unexciting drive: flat, straight roads with the odd narrow bridge and no much traffic. Farmland, milk trucks, some crops here and irrigators over there. The clouds are always impressive though. This time they look mildly menacing, grey but well defined, with a hint of yellow and ocre nearing the end of the day.

We arrived at the motel, checked-in and moved our luggage to the room. On the plus side the place is spotless and not terribly noisy (despite being next to the road); however, it suffers of internet scarcity: room includes 50MB for two nights. My phone’s farts use more bandwidth! Of course I spoke too soon, as I can now hear boggans driving down the road with their modified cars: very low, one can imagine their bottoms almost touching the street, with huge exhausts to compensate for their [insert complex here] and bass boosting stereos.

It was just over a 2 hour drive but I feel a tad tired. I have lost practice driving around and most of my ‘commute’ (if 15 minutes can be called that) is by bicycle. Better wrap up this post here before I blow away all my fifty megabytes of restricted browsing.

No one can become really educated without having pursued some study in which he took no interest—for it is a part of education to learn to interest ourselves in subjects for which we have no aptitude—T.S. Eliot

For a better-sampling Quantum Forest

I have been writing in internet on and off—perhaps mostly off—for near 20 years, including various blog stints since July 2003. This is my fifth or sixth iteration for a blog and I figured out that one element that makes it difficult to keep going in its current form is how skewed is the sampling of topics I covered. I mean all this quantitative, coding, etc. is like looking through a prism that only lets through a tiny portion of life.

Prism used to set 'prism plots' in forest inventory, where the distance to the tree and its size determines if it is inside the plot. (Photo: Luis).

Prism used to set ‘prism plots’ in forest inventory, where the distance to the tree and its size determines if it is inside the plot (Photo: Luis, click to enlarge).

I am loosening my mental definition of what should be in this site because as much as I like programming and numbers, it becomes tiring to always be switched on for those topics.  Some times this change will go unnoticed while others will represent a big departure from what is (or used to be) the core of this blog’s content.

I am hoping to try different topics (perhaps more common in a previous blog incarnation), angles and media. We will see how it works out.

…All right, but apart from the sanitation, medicine, education, wine, public order, irrigation, roads, the fresh water system and public health, what have the Romans ever done for us?—Reg in ‘Life of Brian’, Monthy Python.

Keeping momentum

I don’t believe in New Year resolutions: the idea that people have to wait until this particular position of the orbit around the sun to start doing something. At the same time, I am not great at sticking with changes of behavior: I get distracted easily and I trace it back to losing momentum. Once I stop I find it very difficult to restart.

Yesterday—Tuesday 27 January—I decided to fix at least one thing at home every day, so I can one day be up to date with all the DIY and TLC the house needs. It can be working in a big project or as simple as hanging a picture, but I have to do something. In reality I have been doing this since mid-December, but I prefer to formalize it just to keep momentum. Keep on moving forward; that simple.

Gratuitous picture: cabling my head (Photo: Luis, click to enlarge).

Gratuitous picture: cabling my head (Photo: Luis, click to enlarge).

Jetsam 26: accidental knee

Accidental phone camera knee (Photo: Luis, click to enlarge).

Accidental phone camera knee (Photo: Luis, click to enlarge).

Mucking around with maps, schools and ethnicity in NZ

I’ve been having a conversation for a while with @kamal_hothi and @aschiff on maps, schools, census, making NZ data available, etc. This post documents some basic steps I used for creating a map on ethnic diversity in schools at the census-area-unit level. This “el quicko” version requires 3 ingredients:

  • Census area units shape files (available from Statistics New Zealand for free here).
  • School directory (directory-school-current.csv available for free here).
  • R with some spatial packages (also free).

We’ll read the school directory data, aggregate ethnicity information to calculate the Herfindahl–Hirschman Index of diversity and then plot it.

Then I moved to create a map in R, for the sake of it:

And we get a plot like this:

Ethnic diversity in schools at the Census Area Unit level (0 very diverse, 1 not diverse at all).

Ethnic diversity in schools at the Census Area Unit level (0 very diverse, 1 not diverse at all).

Just because it is Monday down under.

P.S. Using the diversity-index-census-area-unit.csv and joining it with the shapefile in QGIS one can get something prettier (I have to work on matching the color scales):

Similar map produced with point and click in QGIS.

Similar map produced with point and click in QGIS.

Map rendering is so much faster in QGIS than in R! Clearly the system has been optimized for this user case.


When I write code I’m often amazed by the direction of the statements. I mean, we read and write left-to-right except when we assign statements to a variable. So here we are, doing our business, slowly working in a sentence and, puff!, we get this insight from the future and we assign it to our past, on the left. I commented this on Twitter and @fadesingh pointed me to this artistic creation of right-to-left programing language, except that the assign was left-to-right.

R has a right-assign operator (->) and I was thinking, how would it look if I wrote all the assigns to the left, using an adaptation of my previous post.

Not too alien; now using the magrittr package with right assign would make a lot more sense.

Funding change per student (NZ$) on total school roll (click to enlarge).

Funding change per student (NZ$) on total school roll (click to enlarge).

Back of the envelope look at school decile changes

Currently there is some discussion in New Zealand about the effect of the reclassification of schools in socioeconomic deciles. An interesting aspect of the funding system in New Zealand is that state and state-integrated schools with poorer families receive substantially more funding from the government than schools that receive students from richer families (see this page in the Ministry of Education’s website).

One thing that I haven’t noticed before is that funding decisions are more granular than simply using deciles, as deciles 1 to 4 are split into 3 steps each. For example, for Targeted Funding for Educational Achievement in 2015 we get the following amounts per student for decile: 1 (A: $905.81, B: $842.11, C: $731.3), 2 (D: $617.8, E: 507.01, F: 420.54), 3 (G: $350.25, H: $277.32, I: $220.59), 4 (J: $182.74, K: $149.99, L: $135.12), 5 ($115.76), 6 ($93.71), 7: ($71.64), 8 ($46.86), 9 ($28.93) and 10 ($0).

The Ministry of Education states that 784 schools ‘have moved to a higher decile rating’ while 800 ‘have moved to a lower decile rating’ (800 didn’t move). They do not mean that those numbers of schools changed deciles, but that information also includes changes of steps within deciles. Another issue is that it is not the same to move one step at the bottom of the scale (e.g. ~$63 from 1A to 1B) or at the top (~$29 from 9 to 10); that is, the relationship is not linear.

I assume that the baseline to measure funding changes is to calculate how much would a school would get per student in 2015 without any change of decile/step. That is, funding assuming that the previous step within decile had stayed constant. Then we can calculate how a student will get with the new decile/step for the school. I have limited this ‘back of the envelope’ calculation to Targeted Funding for Educational Achievement, which is not the only source of funding linked to deciles. There are other things like Special Education Grant and Careers Information Grant, but they have much smaller magnitude (maximum $73.94 & $37.31 per student) and the maximum differences between deciles 1 and 10 are 2:1.

Steps are in capital letters and need to be translated into money. Once we get that we can calculate differences at both student level and school level:

If we look at the 50% of the schools in the middle of the distribution they had fairly small changes, approximately +/- $22 per student per year or at the school level +/- 3,000 dollars per year.

An interesting, though not entirely surprising, graph is plotting changes of funding on the size of the school. Large schools are much more stable on deciles/step than small ones.

Change of funding per student per year (NZ$) on size of the school (number of students).

Change of funding per student per year (NZ$) on size of the school (number of students).

Change of funding per school per year (thousands of NZ$) on school size (number of students).

Change of funding per school per year (thousands of NZ$) on school size (number of students).

Overall, there is a small change of the total amount of money for Targeted Funding for Educational Achievement used in the reclassified school system versus using the old deciles ($125M using 2014 deciles versus $132M using 2015 deciles) and for most schools the changes do not seem dramatic. There is, however, a number of schools (mostly small ones) who have had substantial changes to their funding. Very small schools will tend to display the largest changes, as the arrival or departure of only few pupils with very different socioeconomic backgrounds would have a substantial effect. An example would be Mata School in the Gisborne area, which moved 13 steps in decile funding (from A to N) with a roll of 11 kids. How to maintain a more steady funding regime seems to be a difficult challenge in those cases.

One consequence of the larger variability in small schools is that rural areas will be more affected by larger changes of funding. While overall 34% of the schools had no changes to their decile/step classification in rural areas that reduces to 22%; on top of that, the magnitude of the changes for rural schools is also larger.


Data files used for this post: DecileChanges_20142015 and directory-school-current.

Operational school funding is much more complex than deciles, as it includes allocations depending on number of students, use of Maori language, etc.

P.S. Stephen Senn highlights an obvious problem with the language the Ministry uses: there are 9 deciles (the points splitting the distribution into 10 parts). We should be talking about tenths, a much simpler word, instead of deciles.

Paying for a job well done

At the moment I am writing R code that involves a lot of simulation for a project. This time I wanted to organize the work properly, put a package together, document it,… the whole shebang. Hadley Wickham has excellent documentation for this process in Advanced R, which works very well as a website. Up to this point there is nothing new; but the material is also available as a book.

At this point in my life I do not want to have a physical object if I can avoid it. On top of that, code tutorials work a lot better as a website, so one can copy, paste and experiment. PDF or ebooks are not very handy for this subject either. Here enters a revolutionary notion: I like to pay people who do a good job and, in the process, make my job easier but sometimes I do not want an object in exchange.

One short term solution: asking Hadley for his favorite charity and donating the cost of a copy of the book. That gets most people happy except, perhaps, the publisher. I then remembered this idea by Cory Doctorow, in which he acts as a middleman between people who wish to pay him for his stories (but don’t want a physical copy of books) and school libraries that wish to have copies of the books.

Wouldn’t it be nice to have an arrangement like that for programming and research books? For example, we could get R learners who prefer but can’t afford books and people willing to pay for them.

Paying for intangibles (Photo: Luis, click to enlarge).

Paying for intangibles (Photo: Luis, click to enlarge).

« Older posts Newer posts »

© 2016 Quantum Forest

Theme by Anders NorenUp ↑