Sun, 20 Aug 2017

RcppArmadillo 0.7.960.1.1

armadillo image

On the heels of the very recent bi-monthly RcppArmadillo release comes a quick bug-fix release 0.7.960.1.1 which just got onto CRAN (and I will ship a build to Debian in a moment).

There were three distinct issues I addressed in three quick pull requests:

  • The excellent Google Summer of Code work by Binxiang Ni had only encountered direct use of sparse matrices as produced by the Matrix. However, while we waited for 0.7.960.1.0 to make it onto CRAN, the quanteda package switched to derived classes---which we now account for via the is() method of our S4 class. Thanks to Kevin Ushey for reminding me we had is().
  • We somehow missed to account for the R 3.4.* and Rcpp 0.12.{11,12} changes for package registration (with .registration=TRUE), so ensured we only have one fastLm symbol.
  • The build did not take not too well to systems without OpenMP, so we now explicitly unset supported via an Armadillo configuration variable. In general, client packages probably want to enable C++11 support when using OpenMP (explicitly) but we prefer to not upset too many (old) users. However, our configure check now also wants g++ 4.7.2 or later just like Armadillo.

Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language--and is widely used by (currently) 382 other packages on CRAN---an increase of 52 since the CRAN release in June!

Changes in this release relative to the previous CRAN release are as follows:

Changes in RcppArmadillo version 0.7.960.1.1 (2017-08-20)

  • Added improved check for inherited S4 matrix classes (#162 fixing #161)

  • Changed fastLm C++ function to fastLm_impl to not clash with R method (#164 fixing #163)

  • Added OpenMP check for configure (#166 fixing #165)

Courtesy of CRANberries, there is a diffstat report. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

/code/rcpp | permanent link

#10: Compacting your Shared Libraries, After The Build

Welcome to the tenth post in the rarely ranting R recommendations series, or R4 for short. A few days ago we showed how to tell the linker to strip shared libraries. As discussed in the post, there are two options. One can either set up ~/.R/Makevars by passing the strip-debug option to the linker. Alternatively, one can adjust src/Makevars in the package itself with a bit a Makefile magic.

Of course, there is a third way: just run strip --strip-debug over all the shared libraries after the build. As the path is standardized, and the shell does proper globbing, we can just do

$ strip --strip-debug /usr/local/lib/R/site-library/*/libs/*.so

using a double-wildcard to get all packages (in that R package directory) and all their shared libraries. Users on macOS probably want .dylib on the end, users on Windows want another computer as usual (just kidding: use .dll). Either may have to adjust the path which is left as an exercise to the reader.

The impact can be Yuge as illustrated in the following dotplot:

This illustration is in response to a mailing list post. Last week, someone claimed on r-help that tidyverse would not install on Ubuntu 17.04. And this is of course patently false as many of us build and test on Ubuntu and related Linux systems, Travis runs on it, CRAN tests them etc pp. That poor user had somehow messed up their default gcc version. Anyway: I fired up a Docker container, installed r-base-core plus three required -dev packages (for xml2, openssl, and curl) and ran a single install.packages("tidyverse"). In a nutshell, following the launch of Docker for an Ubuntu 17.04 container, it was just

$ apt-get update
$ apt-get install r-base libcurl4-openssl-dev libssl-dev libxml2-dev
$ apt-get install mg          # a tiny editor
$ mg /etc/R/     # to add a default CRAN repo
$ R -e 'install.packages("tidyverse")'

which not only worked (as expected) but also installed a whopping fifty-one packages (!!) of which twenty-six contain a shared library. A useful little trick is to run du with proper options to total, summarize, and use human units which reveals that these libraries occupy seventy-eight megabytes:

root@de443801b3fc:/# du -csh /usr/local/lib/R/site-library/*/libs/*so
4.3M    /usr/local/lib/R/site-library/Rcpp/libs/
2.3M    /usr/local/lib/R/site-library/bindrcpp/libs/
144K    /usr/local/lib/R/site-library/colorspace/libs/
204K    /usr/local/lib/R/site-library/curl/libs/
328K    /usr/local/lib/R/site-library/digest/libs/
33M     /usr/local/lib/R/site-library/dplyr/libs/
36K     /usr/local/lib/R/site-library/glue/libs/
3.2M    /usr/local/lib/R/site-library/haven/libs/
272K    /usr/local/lib/R/site-library/jsonlite/libs/
52K     /usr/local/lib/R/site-library/lazyeval/libs/
64K     /usr/local/lib/R/site-library/lubridate/libs/
16K     /usr/local/lib/R/site-library/mime/libs/
124K    /usr/local/lib/R/site-library/mnormt/libs/
372K    /usr/local/lib/R/site-library/openssl/libs/
772K    /usr/local/lib/R/site-library/plyr/libs/
92K     /usr/local/lib/R/site-library/purrr/libs/
13M     /usr/local/lib/R/site-library/readr/libs/
4.7M    /usr/local/lib/R/site-library/readxl/libs/
1.2M    /usr/local/lib/R/site-library/reshape2/libs/
160K    /usr/local/lib/R/site-library/rlang/libs/
928K    /usr/local/lib/R/site-library/scales/libs/
4.9M    /usr/local/lib/R/site-library/stringi/libs/
1.3M    /usr/local/lib/R/site-library/tibble/libs/
2.0M    /usr/local/lib/R/site-library/tidyr/libs/
1.2M    /usr/local/lib/R/site-library/tidyselect/libs/
4.7M    /usr/local/lib/R/site-library/xml2/libs/
78M     total

Looks like dplyr wins this one at thirty-three megabytes just for its shared library.

But with a single stroke of strip we can reduce all this down a lot:

root@de443801b3fc:/# strip --strip-debug /usr/local/lib/R/site-library/*/libs/*so
root@de443801b3fc:/# du -csh /usr/local/lib/R/site-library/*/libs/*so
440K    /usr/local/lib/R/site-library/Rcpp/libs/
220K    /usr/local/lib/R/site-library/bindrcpp/libs/
52K     /usr/local/lib/R/site-library/colorspace/libs/
56K     /usr/local/lib/R/site-library/curl/libs/
120K    /usr/local/lib/R/site-library/digest/libs/
2.5M    /usr/local/lib/R/site-library/dplyr/libs/
16K     /usr/local/lib/R/site-library/glue/libs/
404K    /usr/local/lib/R/site-library/haven/libs/
76K     /usr/local/lib/R/site-library/jsonlite/libs/
20K     /usr/local/lib/R/site-library/lazyeval/libs/
24K     /usr/local/lib/R/site-library/lubridate/libs/
8.0K    /usr/local/lib/R/site-library/mime/libs/
52K     /usr/local/lib/R/site-library/mnormt/libs/
84K     /usr/local/lib/R/site-library/openssl/libs/
76K     /usr/local/lib/R/site-library/plyr/libs/
32K     /usr/local/lib/R/site-library/purrr/libs/
648K    /usr/local/lib/R/site-library/readr/libs/
400K    /usr/local/lib/R/site-library/readxl/libs/
128K    /usr/local/lib/R/site-library/reshape2/libs/
56K     /usr/local/lib/R/site-library/rlang/libs/
100K    /usr/local/lib/R/site-library/scales/libs/
496K    /usr/local/lib/R/site-library/stringi/libs/
124K    /usr/local/lib/R/site-library/tibble/libs/
164K    /usr/local/lib/R/site-library/tidyr/libs/
104K    /usr/local/lib/R/site-library/tidyselect/libs/
344K    /usr/local/lib/R/site-library/xml2/libs/
6.6M    total

Down to six point six megabytes. Not bad for one command. The chart visualizes the respective reductions. Clearly, C++ packages (and their template use) lead to more debugging symbols than plain old C code. But once stripped, the size differences are not that large.

And just to be plain, what we showed previously in post #9 does the same, only already at installation stage. The effects are not cumulative.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

/code/r4 | permanent link