Evolving notes, images and sounds by Luis Apiolaza

Category: programming (Page 1 of 6)

Old dog, etc

Yesterday I attended an introduction to “Artificial Intelligence in Forestry”* workshop at the School of Forestry, University of Canterbury. There were plenty of industry people, including some of I taught “a few” years ago.

I am mostly a quantitative genetics/stats person, but I keep my eyes open for anything that could be helpful dealing with my genetics trials, and have dabbled on using spatial things, LiDAR, etc. As I have pointed out in the past: 1. I am a sucker for entertaining problems and 2. old dogs can learn, albeit more slowly than 30 years ago, new tricks.

It was a nice opportunity to dip my toes in applications of machine learning for image analysis (detection and classification), mostly using ArcGis Pro and a short exercise using R. Funnily enough, my biggest barrier in the workshop was dealing with Windows + ArcGis. I am a Mac and Linux person, generally dealing with coding statistical analyses writing commands in text files. Most of my GIS related work is done via R (which I’ve been using since 1997) and any interactive work with maps using QGIS. Finding anything with point and click took me ages, until we got to the R part, which was hard for most people but kind of easy for yours truly.

Most importantly, having a chat with people and getting ideas to try with my data were the coolest part of the day. Thanks to Vega Xu, Justing Morgenroth and Ning Ye for organising the course!

*Confession: I dislike attaching the AI label to everything these days. Perhaps a better name would have been “Machine Learning in Forestry”. Small detail.

Classroom view from the back.

How old is your favourite language?

We often forget for how long we’ve been writing code in specific languages. For example, I started using SAS in 1992 for the analysis of progeny trials, Python to control Fortran sampling simulations in 1997, and R for general statistics in 1998. Your favourite language could be fairly old:

Fortran: 66 years old
COBOL: 64 yo
Lisp: 63 yo
BASIC: 59 yo
C: 51 yo
SAS: 51 yo
SQL: 49 yo
MATLAB: 44 yo
C++: 38 yo
Python: 32 yo
R: 30 yo
Java: 28 yo
Ruby: 28 yo
Javascript: 27 yo
Clojure: 16 yo
Julia: 11 yo
Elixir: 11 years old

Anyone using other than RStudio?

I asked both in Mastodon and Twitter “Anyone using other than #RStudio as their main #rstats IDE?” and—knowing that some programmers are literal and would probably reply ‘Yes’—I added “What is it?”

Of course I got a few replies like “I only have used RStudio” (Why reply?) or “I use RStudio but in docker containers” (Still RStudio). I also received mostly helpful answers, with some of the usual suspects and a more esoteric option:

  • The most popular alternative was Visual Studio Code using the R Extension for Visual Studio Code plugin together with the languageserver R package.
  • Neovim together with the Nvim-R plugin.
  • Emacs (or one of its variants) + ESS (Emacs Speaks Statistics).

On the esoteric side Spacemacs (Emacs + layers of configuration), or an unholy combination of Emacs with Vim keybindings.

Any of these options will let you use Rmarkdown or Quarto if you are into that.

P.S. I focused on crossplatform options (I tend to move a lot), but a comment in Mastodon mentioned a Windows-only option that could be useful for a few people: Notepad++ together with the NppToR utility.

Some love for Base R. Part 3

It seems a few people have found useful the reminders of base-R functionality covered in “Some love for Base R” Part 1 and Part 2. So I will keep on mentioning a few bits and pieces that you may find handy when going back to Base or even visiting it for the first time.

A reminder: the fictional setting is that you are revisiting legacy code or developing new code under strong constraints: minimal use of packages. The latter could be because you are using webR or you’re keen on having few dependencies. I am assuming R 4.1 when mentioning native pipes, but not the existence of the _ placeholder yet.

In this post I play a little with variable names. None of this would be “production code”, but it would work fine in your analyses. I have similar code (except without pipes) that is almost 20 years old and still running.

A rose by any other name would smell as sweet

William Shakespeare

Changing variable names, renaming, does not work quite like in the tidyverse, in which it can be one more step in a list of transformations with rename(). In base R we rely on the names() function, which is used for both for listing the names of an object and changing them. Usually people would either change all names, providing a vector with names, or replacing one or more names by referring to their position, as in:

names(warpbreaks)
#[1] "breaks"  "wool"    "tension"

names(warpbreaks) <- c("bre", "woo", "ten")
names(warpbreaks)
#[1] "bre" "woo" "ten"

names(warpbreaks)[3] <- "tension"
names(warpbreaks)
#[1] "bre"     "woo"     "tension"

names(warpbreaks)[1:2] <- c("breaks", "wool")
names(warpbreaks)
#[1] "breaks"  "wool"    "tension"

One problem is relying on the position of the variable, which may change with different datasets. One option—although a bit wordy—is to use a regular expression to rename a specific variable with the base sub() function:

# General use
# names(data_frame) <- sub("old_name", "new_name", names(data_frame))

names(warpbreaks) <- sub("wool", "woolly", names(warpbreaks))
names(warpbreaks)
#[1] "breaks"  "woolly"  "tension"

Inside sub() we get a list of all the names for the data frame, look for the one that matches “wool” and replace it by “woolly”.

Cleaning names

A typical problem when receiving datasets is that the authors followed weird naming conventions or, more likely, no convention at all. There are shouting ALLCAPS, names separated by dots, or by spaces, or whatever. I usually work with lowercase names separated by underscores if more than one word. The easiest way to convert names is using janitor‘s clean_names() function.

Our data set’s names could look like this:

names(bos)
#[1] "BLOCK_NO"  "TREE_NO"   "FAM_CODE"  "age core"  "site.code"

I could write a bare bones clean names function in base (covering most of my cleaning needs) using the following code:

names(data_frame) |> tolower() |> {\(x) gsub("[\\. ]", "_", x)}()

For a data frame:

1. get names for the data frame names(data_frame)

2. pass them to the next function |> (using native pipe)

3. convert names to lowercase tolower()

4. replace dots and spaces with underscore using regular expression gsub()

The Klingonian part is a lambda function wrapped by { }() so it works with the native pipe and it is the same as {function(x) gsub("[\\. ]", "_", x)}(). One could perfectly write the code without pipes and using less Klingon.

Applying the function we’d get:

names(bos) |> tolower() |> {\(x) gsub("[\\. ]", "_", x)}()
#[1] "block_no"  "tree_no"   "fam_code"  "age_core"  "site_code"

Of course you’d need to assign the names, so they overwrite the existing ones. Easiest way would be to add -> names(bos) at the end of the line. A right-side assign (wink).

names(bos) |> tolower() |> {\(x) gsub("[\\. ]", "_", x)}() -> names(bos)

Some love for Base R. Part 2

Where were we? Giving some love to base-R and putting together the idea that it is possible to write R very clearly when using base. Two sets of typical issues:

Subsetting rows and columns

When running analyses we often want to work on a subset of all cases (rows) or variables (columns). People are used to filter() (for rows) and select() (for columns) in the tidyverse but then search how to do that in base and get ugly responses. For example, if we had a number of trials in a data frame called all_trials and we wanted to keep only a single one located in Christchurch we could try using sort of matrix notation, keeping the rows that meet the criterion, and all variables, as we don’t specify criteria for them:

my_trial <– all_trials[all_trials$location == "Christchurch", ]

# better, by using with(). More below
my_trial <- with(all_trials, all_trials[location == "Christchurch", ])

You could have been tempted to use all_trials[location == "Christchurch", ] by itself, but R wouldn't have known to look for location inside all_trials. Much clearer, though, would have been to use the subset() function from base R, which does the job of both filter() and select() in the tidyverse. It works like this:

subset(data_frame, conditions_for_rows, select = conditions_for_columns)

# we keep all columns, as we aren't using select
my_trial <– subset(all_trials, location == "Christchurch")

It is way clearer and pipe ready, as the first argument is the data frame name!

This code can easily be expanded to more complex conditions; for example to include all trees from Christchurch and (&)that are also taller than 10 m:

my_trial <– subset(all_trials, location == "Christchurch" & height > 10)

The dataset contains multiple variables but we only want to keep, say, location, block, height and diameter:

my_trial <– subset(all_trials, location == "Christchurch" & height > 10,
                   select = c(location, block, height, diameter))

with() and pipes

Another one. In the tidyverse functions are designed to receive the name of the data frame as the first argument, as in some_function(data = ..., other arguments). Most of the time in base R data is not the first argument and, in some cases, the functions do not take data = ... as an argument. The first case is not a problems, unless we want to use the base pipe |>. The second leads to either going for $ notation or, god helps us, using attach() to make our variables global. Note: never do this.

Argh! What to do? Here is where with() comes to life, being very useful for these two problematic cases. In essence, with(data_frame, function) is saying "look for the function arguments in the specified data_frame".

For example, this blog post gives a lengthy comparison of the %>% and |> pipes but, in my opinion, it complicates things a lot because is missing the use of with(). The post starts "When I am feeling lazy, I use base R for quick plots plot(mtcars$hp, mtcars$mpg)".

As a start, if I were feeling lazy I would've used plot(mpg ~ hp, mtcars), highlighting that the plot function already takes the data argument. In fact, I'm using it as plot(formula, data). If I needed data in the first place I could have simply used with(), which defaults to a data frame as the first argument:

mtcars |> with(plot(mpg ~ hp))

# This is simply calling
with(mtcars, plot(mpg ~ hp))

Instead, the author chooses to use anonymous (lambda) functions, which do have their place in R, but ends up with nasty looking code:

mtcars |> (\(x) plot(x$mpg ~ x$hp))()
# vs
mtcars |> with(plot(mpg ~ hp))
# or even
mtcars |> with(plot(hp, mpg))

I'm partial to using a formula in plot because I can easily visualise the underlying model in my head.

Some functions, mean() for example, don't take a data frame argument. Again, with() is your friend.

# This produces an error
mtcars |> mean(mpg)

# while this one works
mtcars |> with(mean(mpg))

Both within() (used in part 1) and with() will make your base code mucho moar readable (pun intended) and pipe ready.

« Older posts

© 2024 Palimpsest

Theme by Anders NorenUp ↑