Welcome to the fourteenth post in the rationally rambling R rants series, or R4 for short. The last two posts were concerned with faster installation. First, we showed how ccache can speed up (re-)installation. This was followed by a second post on faster installation via binaries.
This last post immediately sparked some follow-up. Replying to my tweet about it, David Smith wondered how to combine binary and source installation (tl;dr: it is hard as you need to combine two package managers). Just this week, Max Ogden wondered how to install CRAN packages as binaries on Linux, and Daniel Nuest poked me on GitHub as part of his excellent containerit project as installation of binaries would of course also make Docker container builds much faster. (tl;dr: Oh yes, see below!)
So can one? Sure. We have a tool. But first the basics.
Packages for a particular distribution are indexed by a packages file for that distribution. This is not unlike CRAN using top-level PACKAGES*
files. So in principle you could just fetch those packages files, parse and index them, and then search them. In practice that is a lot of work as Debian and Ubuntu now have several tens of thousands of packages.
So it is better to use the distro tool. In my use case on .deb
-based distros, this is apt-cache
. Here is a quick example for the (Ubuntu 17.04) server on which I type this:
$ sudo apt-get update -qq ## suppress stdout display
$ apt-cache search r-cran- | wc -l
419
$
So a very vanilla Ubuntu installation has "merely" 400+ binary CRAN packages. Nothing to write home about (yet) -- but read on.
A decade ago, I was involved in two projects to turn all of CRAN into .deb binaries. We had a first ad-hoc predecessor project, and then (much better) a 'version 2' thanks to the excellent Google Summer of Code work by Charles Blundell (and mentored by me). I ran with that for a while and carried at the peak about 2500 binaries or so. And then my controlling db died, just as I visited CRAN to show it off. Very sad. Don Armstrong ran with the code and rebuilt it on better foundations and had for quite some time all of CRAN and BioC built (peaking at maybe 7k package). Then his RAID died. The surviving effort is the one by Michael Rutter who always leaned on the Lauchpad PPA system to build his packages. And those still exist and provide a core of over 10k packages (but across different Ubuntu flavours, see below).
In order to access c2d4u you need an Ubuntu system. For example my Travis runner script does
# Add marutter's c2d4u repository, (and rrutter for CRAN builds too)
sudo add-apt-repository -y "ppa:marutter/rrutter"
sudo add-apt-repository -y "ppa:marutter/c2d4u"
After that one can query apt-cache
as above, but take advantage of a much larger pool with over 3500 packages (see below). The add-apt-repository
command does the Right Thing (TM) in terms of both getting the archive key, and adding the apt
source entry to the config directory.
Now, all this command-line business is nice. But can we do all this programmatically from R? Sort of.
The RcppAPT package interface the libapt library, and provides access to a few functions. I used this feature when I argued (unsuccessfully, as it turned out) for a particular issue concerning Debian and R upgrades. But that is water under the bridge now, and the main point is that "yes we can".
Building on RcppAPT, within the Rocker Project we built on top of this by proving a particular class of containers for different Ubuntu releases which all contain i) RcppAPT and ii) the required apt
source entry for Michael's repos.
So now we can do this
$ docker run --rm -ti rocker/r-apt:xenial /bin/bash -c 'apt-get update -qq; apt-cache search r-cran- | wc -l'
3525
$
This fires up the corresponding Docker container for the xenial
(ie 16.04 LTS) release, updates the apt
indices and then searches for r-cran-*
packages. And it seems we have a little over 3500 packages. Not bad at all (especially once you realize that this skews strongly towards the more popular packages).
A little while a ago a seemingly very frustrated user came to Carl and myself and claimed that out Rocker Project sucketh because building rstan
was all but impossible. I don't have the time, space or inclination to go into details, but he was just plain wrong. You do need to know a little about C++, package building, and more to do this from scratch. Plus, there was a long-standing issue with rstan and newer Boost (which also included several workarounds).
Be that as it may, it serves as nice example here. So the first question: is rstan
packaged?
$ docker run --rm -ti rocker/r-apt:xenial /bin/bash -c 'apt-get update -qq; apt-cache show r-cran-rstan'
Package: r-cran-rstan
Source: rstan
Priority: optional
Section: gnu-r
Installed-Size: 5110
Maintainer: cran2deb4ubuntu <cran2deb4ubuntu@gmail.com>
Architecture: amd64
Version: 2.16.2-1cran1ppa0
Depends: pandoc, r-base-core, r-cran-ggplot2, r-cran-stanheaders, r-cran-inline, r-cran-gridextra, r-cran-rcpp,\
r-cran-rcppeigen, r-cran-bh, libc6 (>= 2.14), libgcc1 (>= 1:4.0), libstdc++6 (>= 5.2)
Filename: pool/main/r/rstan/r-cran-rstan_2.16.2-1cran1ppa0_amd64.deb
Size: 1481562
MD5sum: 60fe7cfc3e8813a822e477df24b37ccf
SHA1: 75bbab1a4193a5731ed105842725768587b4ec22
SHA256: 08816ea0e62b93511a43850c315880628419f2b817a83f92d8a28f5beb871fe2
Description: GNU R package "R Interface to Stan"
Description-md5: c9fc74a96bfde57f97f9d7c16a218fe5
$
It would seem so. With that, the following very minimal Dockerfile is all we need:
## Emacs, make this -*- mode: sh; -*-
## Start from xenial
FROM rocker/r-apt:xenial
## This handle reaches Carl and Dirk
MAINTAINER "Carl Boettiger and Dirk Eddelbuettel" rocker-maintainers@eddelbuettel.com
## Update and install rstan
RUN apt-get update && apt-get install -y --no-install-recommends r-cran-rstan
## Make R the default
CMD ["R"]
In essence, it executes one command: install rstan
but from binary taking care of all dependencies. And lo and behold, it works as advertised:
$ docker run --rm -ti rocker/rstan:local Rscript -e 'library(rstan)'
Loading required package: ggplot2
Loading required package: StanHeaders
rstan (Version 2.16.2, packaged: 2017-07-03 09:24:58 UTC, GitRev: 2e1f913d3ca3)
For execution on a local, multicore CPU with excess RAM we recommend calling
rstan_options(auto_write = TRUE)
options(mc.cores = parallel::detectCores())
$
So there: installing from binary works, takes care of dependencies, is easy and as an added bonus even faster. What's not too like?
(And yes, a few of us are working on a system to have more packages available as binaries, but it may take another moment...)
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The fourth release of littler as a CRAN package is now available, following in the now more than ten-year history as a package started by Jeff in 2006, and joined by me a few weeks later.
littler is the first command-line interface for R and predates Rscript
. In my very biased eyes better as it allows for piping as well shebang scripting via #!
, uses command-line arguments more consistently and still starts faster. Last but not least it is also less silly than Rscript
and always loads the methods
package avoiding those bizarro bugs between code running in R
itself and a scripting front-end.
littler prefers to live on Linux and Unix, has its difficulties on OS X due to yet-another-braindeadedness there (who ever thought case-insensitive filesystems where a good idea?) and simply does not exist on Windows (yet -- the build system could be extended -- see RInside for an existence proof, and volunteers welcome!).
A few examples as highlighted at the Github repo:
This release brings a few new examples scripts, extends a few existing ones and also includes two fixes thanks to Carl. Again, no internals were changed. The NEWS
file entry is below.
Changes in littler version 0.3.3 (2017-12-17)
Changes in examples
The script
installGithub.r
now correctly uses theupgrade
argument (Carl Boettiger in #49).New script
pnrrs.r
to call the package-native registration helper function added in R 3.4.0The script
install2.r
now has more robust error handling (Carl Boettiger in #50).New script
cow.r
to use R Hub'scheck_on_windows
Scripts
cow.r
andc4c.r
use#!/usr/bin/env r
New option
--fast
(or-f
) for scriptsbuild.r
andrcc.r
for faster package build and checkThe
build.r
script now defaults to using the current directory if no argument is provided.The RStudio getters now use the
rvest
package to parse the webpage with available versions.Changes in package
- Travis CI now uses https to fetch script, and sets the group
Courtesy of CRANberries, there is a comparison to the previous release. Full details for the littler release are provided as usual at the ChangeLog page. The code is available via the GitHub repo, from tarballs off my littler page and the local directory here -- and now of course all from its CRAN page and via install.packages("littler")
. Binary packages are available directly in Debian as well as soon via Ubuntu binaries at CRAN thanks to the tireless Michael Rutter.
Comments and suggestions are welcome at the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new version of drat just arrived on CRAN as another no-human-can-delay-this automatic upgrade directly from the CRAN prechecks (though I did need a manual reminder from Uwe to remove a now stale drat repo URL -- bad @hrbrmstr -- from the README in a first attempt).
This release is mostly the work of Neal Fultz who kindly sent me two squeaky-clean pull requests addressing two open issue tickets. As drat is reasonably small and simple, that was enough to motivate a quick release. I also ensured that PACKAGES.rds
will always if committed along (if we're in commit mode), which is a follow-up to an initial change from 0.1.3 in September.
drat stands for drat R Archive Template, and helps with easy-to-create and easy-to-use repositories for R packages. Since its inception in early 2015 it has found reasonably widespread adoption among R users because repositories with marked releases is the better way to distribute code.
The NEWS
file summarises the release as follows:
Changes in drat version 0.1.4 (2017-12-16)
Changes in drat functionality
Changes in drat documentation
- Updated 'README.md' removing another stale example URL
Courtesy of CRANberries, there is a comparison to the previous release. More detailed information is on the drat page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A small maintenance release, version 0.6.13, of the digest package arrived on CRAN and in Debian yesterday.
digest creates hash digests of arbitrary R objects (using the 'md5', 'sha-1', 'sha-256', 'crc32', 'xxhash' and 'murmurhash' algorithms) permitting easy comparison of R language objects.
This release accomodates a request by Luke and Tomas to make the version
argument of serialize()
an argument to digest()
too, which was easy enough to accomodate. The value 2L
is the current default (and for now only permitted value). The ALTREP changes in R 3.5 will bring us a new, and more powerful, format with value 3L
. Changes can be set in each call, or globally via options()
. Other than we just clarified one aspect of raw
vector usage in the manual page.
CRANberries provides the usual summary of changes to the previous version.
For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the thirteenth post in the ridiculously rapid R recommendation series, or R4 for short. A few days ago we riffed on faster installation thanks to ccache. Today we show another way to get equally drastic gains for some (if not most) packages.
In a nutshell, there are two ways to get your R packages off CRAN. Either you install as a binary, or you use source. Most people do not think too much about this as on Windows, binary is the default. So why wouldn't one? Precisely. (Unless you are on Windows, and you develop, or debug, or test, or ... and need source. Another story.) On other operating systems, however, source is the rule, and binary is often unavailable.
Or is it? Exactly how to find out what is available will be left for another post as we do have a tool just for that. But today, just hear me out when I say that binary is often an option even when source is the default. And it matters. See below.
As a (mostly-to-always) Linux user, I sometimes whistle between my teeth that we "lost all those battles" (i.e. for the desktop(s) or laptop(s)) but "won the war". That topic merits a longer post I hope to write one day, and I won't do it justice today but my main gist that everybody (and here I mean mostly developers/power users) now at least also runs on Linux. And by that I mean that we all test our code in Linux environments such as e.g. Travis CI, and that many of us run deployments on cloud instances (AWS, GCE, Azure, ...) which are predominantly based on Linux. Or on local clusters. Or, if one may dream, the top500 And on and on. And frequently these are Ubuntu machines.
So here is an Ubuntu trick: Install from binary, and save loads of time. As an illustration, consider the chart below. It carries over the logic from the 'cached vs non-cached' compilation post and contrasts two ways of installing: from source, or as a binary. I use pristine and empty Docker containers as the base, and rely of course on the official r-base image which is supplied by Carl Boettiger and yours truly as part of our Rocker Project (and for which we have a forthcoming R Journal piece I might mention). So for example the timings for the ggplot2
installation were obtained via
time docker run --rm -ti r-base /bin/bash -c 'install.r ggplot2'
and
time docker run --rm -ti r-base /bin/bash -c 'apt-get update && apt-get install -y r-cran-ggplot2'
Here docker run --rm -ti
just means to launch Docker, in 'remove leftovers at end' mode, use terminal and interactive mode and invoke a shell. The shell command then is, respectively, to install a CRAN package using install.r
from my littler package, or to install the binary via apt-get
after updating the apt indices (as the Docker container may have been built a few days or more ago).
Let's not focus on Docker here---it is just a convenient means to an end of efficiently measuring via a simple (wall-clock counting) time
invocation. The key really is that install.r
is just a wrapper to install.packages()
meaning source installation on Linux (as used inside the Docker container). And apt-get install ...
is how one gets a binary. Again, I will try post another piece to determine how one finds if a suitable binary for a CRAN package exists. For now, just allow me to proceed.
So what do we see then? Well have a look:
A few things stick out. RQuantLib really is a monster. And dplyr is also fairly heavy---both rely on Rcpp, BH and lots of templating. At the other end, data.table is still a marvel. No external dependencies, and just plain C code make the source installation essentially the same speed as the binary installation. Amazing. But I digress.
We should add that one of the source installations also required installing additional libries: QuantLib is needed along with Boost for RQuantLib. Similar for another package (not shown) which needed curl and libcurl.
So what is the upshot? If you can, consider binaries. I will try to write another post how I do that e.g. for Travis CI where all my tests us binaries. (Yes, I know. This mattered more in the past when they did not cache. It still matters today as you a) do not need to fill the cache in the first place and b) do not need to worry about details concerning compilation from source which still throws enough people off. But yes, you can of course survive as is.)
The same approach is equally valid on AWS and related instances: I answered many StackOverflow questions where folks were failing to compile "large-enough" pieces from source on minimal installations with minimal RAM, and running out of resources and failed with bizarre errors. In short: Don't. Consider binaries. It saves time and trouble.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A boring little RVowpalWabbit package update to version 0.0.10 came in response to another CRAN request: We were switching directories to run tests (or examples) which is now discouraged, so we no longer do this as it turns that we can of course refer to the files directly as well. Much cleaner.
No new code or features were added.
We should mention once more that is parallel work ongoing in a higher-level package interfacing the vw
binary -- rvw -- as well as plan to redo this package via the external libraries. In that sounds interesting to you, please get in touch.
More information is on the RVowpalWabbit page. Issues and bugreports should go to the GitHub issue tracker.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the twelveth post in the randomly relevant R recommendations series, or R4 for short. This post will insert a short diversion into what was planned as a sequence of posts on faster installations that started recently with this post but we will resume to it very shortly (for various definitions of "very" or "shortly").
Earlier today Davis Vaughn posted a tweet about a blog post of his describing a (term) paper he wrote modeling bitcoin volatilty using Alexios's excellent rugarch package---and all that typeset with the styling James and I put together in our little pinp package which is indeed very suitable for such tasks of writing (R)Markdown + LaTeX + R code combinations conveniently in a single source file.
Leaving aside the need to celebreate a term paper with a blog post and tweet, pinp is indeed very nice and deserving of some additional exposure and tutorials. Now, Davis sets out to do all this inside RStudio---as folks these days seem to like to limit themselves to a single tool or paradigm. Older and wiser users prefer the flexibility of switching tools and approaches, but alas, we digress. While Davis manages of course to do all this in RStudio which is indeed rather powerful and therefore rightly beloved, he closes on
I wish there was some way to have Live Rendering like with blogdown so that I could just keep a rendered version of the paper up and have it reload every time I save. That would be the dream!
and I can only add a forceful: Fear not, young man, for we can help thou!
Modern operating systems have support for epoll and libnotify, which can be used from the shell. Just how your pdf application refreshes automagically when a pdf file is updated, we can hook into this from the shell to actually create the pdf when the (R)Markdown file is updated. I am going to use a tool readily available on my Linux systems; macOS will surely have something similar. The entr
command takes one or more file names supplied on stdin
and executes a command when one of them changes. Handy for invoking make
whenever one of your header or source files changes, and useable here. E.g. the last markdown file I was working on was named comments.md
and contained comments to a referee, and we can auto-process it on each save via
echo comments.md | entr render.r comments.md
which uses render.r
from littler (new release soon too...; a simple Rscript -e 'rmarkdown::render("comments.md")'
would probably work too but render.r
is shorter and little more powerful so I use it more often myself) on the input file comments.md
which also happens to (here sole) file being monitored.
And that is really all there is to it. I wanted / needed something like this a few months ago at work too, and may have used an inotify
-based tool there but cannot find any notes. Python has something similar via watchdog which is yet again more complicated / general.
It turns out that auto-processing is actually not that helpful as we often save before an expression is complete, leading to needless error messages. So at the end of the day, I often do something much simpler. My preferred editor has a standard interface to 'building': pressing C-x c
loads a command (it recalls) that defaults to make -k
(i.e., make
with error skipping). Simply replacing that with render.r comments.md
(in this case) means we get an updated pdf file when we want with a simple customizable command / key-combination.
So in sum: it is worth customizing your environments, learning about what your OS may have, and looking beyond a single tool / editor / approach. Even dreams may come true ...
Postscriptum: And Davis takes this in a stride and almost immediately tweeted a follow-up with a nice screen capture mp4 movie showing that entr
does indeed work just as well on his macbook.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Another RcppArmadillo release hit CRAN today. Since our last 0.8.100.1.0 release in October, Conrad kept busy and produced Armadillo releases 8.200.0, 8.200.1, 8.300.0 and now 8.300.1. We tend to now package these (with proper reverse-dependency checks and all) first for the RcppCore drat repo from where you can install them "as usual" (see the repo page for details). But this actual release resumes within our normal bi-monthly CRAN release cycle.
These releases improve a few little nags on the recent switch to more extensive use of OpenMP, and round out a number of other corners. See below for a brief summary.
Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language--and is widely used by (currently) 405 other packages on CRAN.
A high-level summary of changes follows.
Changes in RcppArmadillo version 0.8.300.1.0 (2017-12-04)
Upgraded to Armadillo release 8.300.1 (Tropical Shenanigans)
faster handling of band matrices by
solve()
faster handling of band matrices by
chol()
faster
randg()
when using OpenMPadded
normpdf()
expanded
.save()
to allow appending new datasets to existing HDF5 filesIncludes changes made in several earlier GitHub-only releases (versions 0.8.300.0.0, 0.8.200.2.0 and 0.8.200.1.0).
Conversion from
simple_triplet_matrix
is now supported (Serguei Sokol in #192).Updated configure code to check for g++ 5.4 or later to enable OpenMP.
Updated the skeleton package to current packaging standards
Suppress warnings from Armadillo about missing OpenMP support and
-fopenmp
flags by settingARMA_DONT_PRINT_OPENMP_WARNING
Courtesy of CRANberries, there is a diffstat report. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Earlier today Rcpp passed 1250 reverse-dependencies on CRAN as another big milestone. The graph is on the left depicts the growth of Rcpp usage (as measured by Depends, Imports and LinkingTo, but excluding Suggests) over time.
Rcpp cleared 300 packages in November 2014. It passed 400 packages in June 2015 (when I only tweeted about it), 500 packages in late October 2015, 600 packages last March, 700 packages last July, 800 packages last October, 900 packages early January, and 1000 packages in April. The chart extends to the very beginning via manually compiled data from CRANberries and checked with crandb. The next part uses manually saved entries. The core (and by far largest) part of the data set was generated semi-automatically via a short script appending updates to a small file-based backend. A list of packages using Rcpp is kept on this page.
Also displayed in the graph is the relative proportion of CRAN packages using Rcpp. The four per-cent hurdle was cleared just before useR! 2014 where I showed a similar graph (as two distinct graphs) in my invited talk. We passed five percent in December of 2014, six percent July of 2015, seven percent just before Christmas 2015, eight percent last summer, nine percent mid-December 2016 and then cracked ten percent this summer.
1250 user packages is staggering. We can use the progression of CRAN itself compiled by Henrik in a series of posts and emails to the main development mailing list. A decade ago CRAN itself did not have 1250 packages, and here we are approaching 12k with Rcpp at 10% and growing steadily. Amazeballs.
This puts a whole lot of responsibility on us in the Rcpp team as we continue to keep Rcpp as performant and reliable as it has been.
And with that, and as always, a very big Thank You! to all users and contributors of Rcpp for help, suggestions, bug reports, documentation or, of course, code.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the eleventh post in the rarely rued R rants series, or R4 for short. Time clearly flies as it has been three months since out last post on significantly reducing library size via stripping. I had been meaning to post on today's topic for quite some time, but somehow something (working on a paper, releasing a package, ...) got in the way.
Just a few days ago Colin (of Efficient R Programming fame) posted about speed(ing up) package installation. His recommendation? Remember that we (usually) have multiple cores and using several of them via options(Ncpus = XX)
. It is an excellent point, and it bears repeating.
But it turns I have not one but two salient recommendations too. Today covers the first, we should hopefully get pretty soon to the second. Both have one thing in common: you will be fastest if you avoid doing the work in the first place.
One truly outstanding tool for this in the context of the installation of compiled packages is ccache. It is actually a pretty old tool that has been out for well over a decade, and it comes from the folks that gave the Samba filesystem.
What does it do? Well, it nutshell, it "hashes" a checksum of a source file once the preprocessor has operated on it and stores the resulting object file. In the case of rebuild with unchanged code you get the object code back pretty much immediately. The idea is very similar to memoisation (as implemented in R for example in the excellent little memoise package by Hadley, Jim, Kirill and Daniel. The idea is the same: if you have to do something even moderately expensive a few times, do it once and then recall it the other times.
This happens (at least to me) more often that not in package development. Maybe you change just one of several source files. Maybe you just change the R code, the Rd documentation or a test file---yet still need a full reinstallation. In all these cases, ccache
can help tremdendously as illustrated below.
Because essentially all our access to compilation happens through R, we need to set this in a file read by R. I use ~/.R/Makevars
for this and have something like these lines on my machines:
VER=
CCACHE=ccache
CC=$(CCACHE) gcc$(VER)
CXX=$(CCACHE) g++$(VER)
CXX11=$(CCACHE) g++$(VER)
CXX14=$(CCACHE) g++$(VER)
FC=$(CCACHE) gfortran$(VER)
F77=$(CCACHE) gfortran$(VER)
That way, when R calls the compiler(s) it will prefix with ccache
. And ccache
will then speed up.
There is an additional issue due to R use. Often we install from a .tar.gz
. These will be freshly unpackaged, and hence have "new" timestamps. This would usually lead ccache
to skip to file (fear of "false positives") so we have to override this. Similarly, the tarball is usually unpackage in a temporary directory with an ephemeral name, creating a unique path. That too needs to be overwritten. So in my ~/.ccache/ccache.conf
I have this:
max_size = 5.0G
# important for R CMD INSTALL *.tar.gz as tarballs are expanded freshly -> fresh ctime
sloppiness = include_file_ctime
# also important as the (temp.) directory name will differ
hash_dir = false
A quick illustration will round out the post. Some packages are meatier than others. More C++ with more templates usually means longer build times. Below is a quick chart comparing times for a few such packages (ie RQuantLib, dplyr, rstan) as well as igraph ("merely" a large C package) and lme4 as well as Rcpp. The worst among theseis still my own RQuantLib package wrapping (still just parts of) the genormous and Boost-heavy QuantLib library.
Pretty dramatic gains. Best of all, we can of course combine these with other methods such as Colin's use of multiple CPUs, or even a simple MAKE=make -j4
to have multiple compilation units being considered in parallel. So maybe we all get to spend less time on social media and other timewasters as we spend less time waiting for our builds. Or maybe that is too much to hope for...
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
FOAAS upstream is still at release 1.1.0, but added a few new accessors a couple of months ago. So this new version of rfoaas updates to these: asshole()
, cup()
, fyyff()
, immensity()
, programmer()
, rtfm()
, thinking()
. We also added test coverage and in doing so noticed that our actual tests never ran on Travis. Yay. Now fixed.
As usual, CRANberries provides a diff to the previous CRAN release. Questions, comments etc should go to the GitHub issue tracker. More background information is on the project page as well as on the github repo
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The fourteenth release in the 0.12.* series of Rcpp landed on CRAN yesterday after a somewhat longer-than-usual gestation period (and word is it may have been due to some unrelated disturbances from lots of changes within the main r-devel build).
This release follows the 0.12.0 release from July 2016, the 0.12.1 release in September 2016, the 0.12.2 release in November 2016, the 0.12.3 release in January 2017, the 0.12.4 release in March 2016, the 0.12.5 release in May 2016, the 0.12.6 release in July 2016, the 0.12.7 release in September 2016, the 0.12.8 release in November 2016, the 0.12.9 release in January 2017, the 0.12.10.release in March 2017, the 0.12.11.release in May 2017, the 0.12.12 release in July 2017 and the 0.12.13.release in late September 2017 making it the eighteenth release at the steady and predictable bi-montly release frequency.
Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 1246 packages (and hence 77 more since the last release) on CRAN depend on Rcpp for making analytical code go faster and further, along with another 91 in BioConductor.
This release is relatively minor compared to other releases, but follows through on the deprecattion of the old vectors for Date
and Datetime
(which were terrible: I was influenced by the vector design in QuantLib at the time and didn't really understand yet how a SEXP
vector should work) we announced with Rcpp 0.12.8 a year ago. So now the new vectors are the default, but you can flip back if you need to with #define
.
Otherwise Dan rounded a corner with the improved iterators he contributed, and Kirill improved the output stream implementation suppressing a warning with newer compilers.
Changes in Rcpp version 0.12.14 (2017-11-17)
Changes in Rcpp API:
New const iterators functions
cbegin()
andcend()
added toMatrixRow
as well (Dan Dillon in #750).The
Rostream
object now contains aBuffer
rather than allocating one (Kirill Müller in #763).New
DateVector
andDatetimeVector
classes are now the default fully deprecating the old classes as announced one year ago.Changes in Rcpp Package:
- DESCRIPTION file now list doi information per CRAN suggestion.
Changes in Rcpp Documentation:
- Update CITATION file with doi information and PeerJ preprint.
Thanks to CRANberries, you can also look at a diff to the previous release. As always, details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A maintenance release 0.3.3.3.1 of RcppEigen is now on CRAN (and will get to Debian soon). It brings Eigen 3.3.* to R.
The impetus was a request from CRAN to change the call to Rcpp::Rcpp.plugin.maker()
to only use ::
as the function has in fact been exported and accessible for a pretty long time. So now the usage pattern catches up. Otherwise, Haiku-OS is now supported and a minor Travis tweak was made.
The complete NEWS
file entry follows.
Changes in RcppEigen version 0.3.3.3.1 (2017-11-19)
Compilation under Haiku-OS is now supported (Yu Gong in #45).
The
Rcpp.plugin.maker
helper function is called via::
as it is in fact exported (yet we had old code using:::
).A spurious argument was removed from an example call.
Travis CI now uses https to fetch the test runner script.
Courtesy of CRANberries, there is also a diffstat report for the most recent release.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A maintenance release RcppClassic 0.9.9 is now at CRAN. This package provides a maintained version of the otherwise deprecated first Rcpp API; no new projects should use it.
Per a request from CRAN, we changed the call to Rcpp::Rcpp.plugin.maker()
to only use ::
as the function has in fact been exported and accessible for a pretty long time. So now the usage pattern catches up.
Courtesy of CRANberries, there are changes relative to the previous release.
Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The tenth (!!) annual annual R/Finance conference will take in Chicago on the UIC campus on June 1 and 2, 2018. Please see the call for papers below (or at the website) and consider submitting a paper.
We are once again very excited about our conference, thrilled about who we hope may agree to be our anniversary keynotes, and hope that many R / Finance users will not only join us in Chicago in June -- and also submit an exciting proposal.
So read on below, and see you in Chicago in June!
R/Finance 2018: Applied Finance with R
June 1 and 2, 2018
University of Illinois at Chicago, IL, USA
The tenth annual R/Finance conference for applied finance using R will be held June 1 and 2, 2018 in Chicago, IL, USA at the University of Illinois at Chicago. The conference will cover topics including portfolio management, time series analysis, advanced risk tools, high-performance computing, market microstructure, and econometrics. All will be discussed within the context of using R as a primary tool for financial risk management, portfolio construction, and trading.
Over the past nine years, R/Finance has includedattendeesfrom around the world. It has featured presentations from prominent academics and practitioners, and we anticipate another exciting line-up for 2018.
We invite you to submit complete papers in pdf format for consideration. We will also consider one-page abstracts (in txt or pdf format) although more complete papers are preferred. We welcome submissions for both full talks and abbreviated "lightning talks." Both academic and practitioner proposals related to R are encouraged.
All slides will be made publicly available at conference time. Presenters are strongly encouraged to provide working R code to accompany the slides. Data sets should also be made public for the purposes of reproducibility (though we realize this may be limited due to contracts with data vendors). Preference may be given to presenters who have released R packages.
Please submit proposals online at http://go.uic.edu/rfinsubmit. Submissions will be reviewed and accepted on a rolling basis with a final submission deadline of February 2, 2018. Submitters will be notified via email by March 2, 2018 of acceptance, presentation length, and financial assistance (if requested).
Financial assistance for travel and accommodation may be available to presenters. Requests for financial assistance do not affect acceptance decisions. Requests should be made at the time of submission. Requests made after submission are much less likely to be fulfilled. Assistance will be granted at the discretion of the conference committee.
Additional details will be announced via the conference website at http://www.RinFinance.com/ as they become available. Information on previous years'presenters and their presentations are also at the conference website. We will make a separate announcement when registration opens.
For the program committee:
Gib Bassett, Peter Carl, Dirk Eddelbuettel, Brian Peterson,
Dale Rosenthal, Jeffrey Ryan, Joshua Ulrich
A shiny new (mostly-but-not-completely maintenance) release of RQuantLib, now at version 0.4.4, arrived on CRAN overnight, and will get to Debian shortly. This is the first release in over a year, and it it contains (mostly) a small number of fixes throughout. It also includes the update to the new DateVector
and DatetimeVector
classes which become the default with the upcoming Rcpp 0.12.14 release (just like this week's RcppQuantuccia release). One piece of new code is due to François Cocquemas who added support for discrete dividends to both European and American options. See below for the complete set of changes reported in the NEWS
file.
As with release 0.4.3 a little over a year ago, we will not have new Windows binaries from CRAN as I apparently have insufficient powers of persuasion to get CRAN to update their QuantLib libraries. So we need a volunteer. If someone could please build a binary package for Windows from the 0.4.4 sources, I would be happy to once again host it on the GHRR drat repo. Please contact me directly if you can help.
Changes are listed below:
Changes in RQuantLib version 0.4.4 (2017-11-07)
Changes in RQuantLib code:
Equity options can now be analyzed via discrete dividends through two vectors of dividend dates and values (Francois Cocquemas in #73 fixing #72)
Some package and dependency information was updated in files
DESCRIPTION
andNAMESPACE
.The new
Date(time)Vector
classes introduced with Rcpp 0.12.8 are now used when available.Minor corrections were applied to
BKTree
, to vanilla options for the case of intraday time stamps, to theSabrSwaption
documentation, and to bond utilities for the most recent QuantLib release.
Courtesy of CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the RQuantLib page. Questions, comments etc should go to the rquantlib-devel mailing list off the R-Forge page. Issue tickets can be filed at the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A first maintenance release of RcppQuantuccia got to CRAN earlier today.
RcppQuantuccia brings the Quantuccia header-only subset / variant of QuantLib to R. At present it mostly offers calendaring, but Quantuccia just got a decent amount of new functions so hopefully we can offer more here too.
This release was motivated by the upcoming Rcpp release which will deprecate the okd Date and Datetime vectors in favours of newer ones. So this release of RcppQuantuccia switches to the newer ones.
Other changes are below:
Changes in version 0.0.2 (2017-11-06)
Added calendars for Canada, China, Germany, Japan and United Kingdom.
Added bespoke and joint calendars.
Using new date(time) vectors (#6).
Courtesy of CRANberries, there is also a diffstat report relative to the previous release. More information is on the RcppQuantuccia page. Issues and bugreports should go to the GitHub issue tracker.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A maintenance release of our pinp package for snazzier one or two column vignettes is now on CRAN as of yesterday.
In version 0.0.3, we disabled the default \pnasbreak
command we inherit from the PNAS LaTeX style. That change turns out to have been too drastic. So we reverted yet added a new YAML front-matter option skip_final_break
which, if set to TRUE
, will skip this break. With a default value of FALSE
we maintain prior behaviour.
A screenshot of the package vignette can be seen below. Additional screenshots of are at the pinp page.
The NEWS entry for this release follows.
Changes in pinp version 0.0.4 (2017-11-04)
Courtesy of CRANberries, there is a comparison to the previous release. More information is on the tint page. For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A maintenance release of the tint package arrived on CRAN earlier today. Its name expands from tint is not tufte as the package offers a fresher take on the Tufte-style for html and pdf presentations.
A screenshot of the pdf variant is below.
This release brings some minor enhancements and polish, mostly learned from having done the related pinp (two-column vignette in the PNAS style) and linl (LaTeX letter) RMarkdown-wrapper packages; see below for details from the NEWS.Rd
file.
Changes in tint version 0.0.4 (2017-11-02)
Courtesy of CRANberries, there is a comparison to the previous release. More information is on the tint page.
For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Following up on the initial 0.0.1 release of linl, Aaron and I are happy to announce release 0.0.2 which reached the CRAN network on Sunday in a smooth 'CRAN-pretest-publish' auto-admittance. linl provides a simple-yet-powerful Markdown---and RMarkdown---wrapper around the venerable LaTeX letter class; see below for an expanded example also included as the package vignette.
This versions sets a few sensible default values for font, font size, margins, signature (non-)indentation and more; it also expands the documentation.
The NEWS entry follows:
Changes in tint version 0.0.2 (2017-10-29)
Courtesy of CRANberries, there is a comparison to the previous release. More information is on the tint page. For questions or comments use the issue tracker off the GitHub repo.
For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Our pinp package for snazzier one or two column vignette received it second update. Now at version 0.0.3, it arrived on CRAN on Saturday with minimal fuzz as an 'CRAN-pretest-publish' transition.
We added more frontmatter options, documented more, and streamlined some internals of the LaTeX class borrowed from PNAS. A screenshot of the (updated) vignette can be seen below. Additional screenshots of are at the pinp page.
The NEWS entry for this release follows.
Changes in tint version 0.0.3 (2017-10-28)
Section 'Acknowledgements' now conditional on a frontmatter setting, section 'Matmethods' has been removed,
pnasbreak
no longer used which stabilizes LaTeX float formatting. References are now shown in the column just like other content (Dirk in #36).Vignette now uses new numbered sections frontmatter switch which improves the pdf outline.
New front-matter options for title/section header colors, and link colors (Dirk in #39).
YAML frontmater options are now documented in the help page for
pinp
as well (Dirk in #41).
Courtesy of CRANberries, there is a comparison to the previous release. More information is on the tint page. For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Aaron Wolen and I are pleased to announce the availability of the initial 0.0.1 release of our new linl package on the CRAN network. It provides a simple-yet-powerful Markdown---and RMarkdown---wrapper the venerable LaTeX letter class. Aaron had done the legwork in the underlying pandoc-letter repository upon which we build via proper rmarkdown integration.
The package also includes a LaTeX trick or two: optional header and signature files, nicer font, better size, saner default geometry and more. See the following screenshot which shows the package vignette---itself a simple letter---along with (most of) its source:
The initial (short) NEWS entry follows:
Changes in tint version 0.0.1 (2017-10-17)
- Initial CRAN release
The date is a little off; it took a little longer than usual for the good folks at CRAN to process the initial submission. We expect future releases to be more timely.
For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Three years ago I referenced the Seinfeld Streak used in an earlier post of regular updates to to the Rcpp Gallery:
This is sometimes called Jerry Seinfeld's secret to productivity: Just keep at it. Don't break the streak.
and showed the first chart of GitHub streaking
And two year ago a first follow-up appeared in this post:
And a year ago we had a followup last year
And as it October 12 again, here is the new one:
Again, special thanks go to Alessandro Pezzè for the Chrome add-on GithubOriginalStreak.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
We are thrilled to announce a new big RcppArmadillo release! Conrad recently moved Armadillo to the 8.* series, with significant improvements and speed ups for sparse matrix operations, and more. See below for a brief summary.
This also required some changes at our end which Binxiang Ni provided, and Serguei Sokol improved some instantiations. We now show the new vignette Binxiang Ni wrote for his GSoC contribution, and I converted it (and the other main vignette) to using the pinp package for sleeker pdf vignettes.
This release resumes our bi-monthly CRAN release cycle. I may make interim updates available at GitHub "as needed". And this time I managed to mess up the reverse depends testing, and missed one sync()
call on the way back to R---but all that is now taken care of.
Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language--and is widely used by (currently) 405 other packages on CRAN.
A high-level summary of changes follows.
Changes in RcppArmadillo version 0.8.100.1.0 (2017-10-05)
Upgraded to Armadillo release 8.100.1 (Feral Pursuits)
faster incremental construction of sparse matrices via element access operators
faster diagonal views in sparse matrices
expanded
SpMat
to save/load sparse matrices in coord formatexpanded
.save()
,.load()
to allow specification of datasets within HDF5 filesadded
affmul()
to simplify application of affine transformationswarnings and errors are now printed by default to the
std::cerr
streamadded
set_cerr_stream()
andget_cerr_stream()
to replaceset_stream_err1()
,set_stream_err2()
,get_stream_err1()
,get_stream_err2()
new configuration options
ARMA_COUT_STREAM
andARMA_CERR_STREAM
Constructors for sparse matrices of types
dgt
,dtt
amddst
now use Armadillo code for improved performance (Serguei Sokol in #175 addressing #173)Sparse matrices call
.sync()
before accessing internal arrays (Binxiang Ni in #171)The sparse matrix vignette has been converted to Rmarkdown using the pinp package, and is now correctly indexed. (#176)
Courtesy of CRANberries, there is a diffstat report. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
RProtoBuf provides R bindings for the Google Protocol Buffers ("ProtoBuf") data encoding and serialization library used and released by Google, and deployed fairly widely in numerous projects as a language and operating-system agnostic protocol.
A new releases RProtoBuf 0.4.11 appeared on CRAN earlier today. Not unlike the other recent releases, it is mostly a maintenance release which switches two of the vignettes over to using the pinp package and its template for vignettes.
Changes in RProtoBuf version 0.4.11 (2017-10-03)
The
RProtoBuf-intro
andRProtoBuf-quickref
vignettes were converted to Rmarkdown using the templates and style file from the pinp package.A few minor internal upgrades
CRANberries also provides a diff to the previous release. The RProtoBuf page has copies of the (older) package vignette, the 'quick' overview vignette, a unit test summary vignette, and the pre-print for the JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The thirteenth release in the 0.12.* series of Rcpp landed on CRAN this morning, following a little delay because Uwe Ligges was traveling and whatnot. We had announced its availability to the mailing list late last week. As usual, a rather substantial amount of testing effort went into this release so you should not expect any surprise.
This release follows the 0.12.0 release from July 2016, the 0.12.1 release in September 2016, the 0.12.2 release in November 2016, the 0.12.3 release in January 2017, the 0.12.4 release in March 2016, the 0.12.5 release in May 2016, the 0.12.6 release in July 2016, the 0.12.7 release in September 2016, the 0.12.8 release in November 2016, the 0.12.9 release in January 2017, the 0.12.10.release in March 2017, the 0.12.11.release in May 2017, and the 0.12.12 release in July 2017 making it the seventeeth release at the steady and predictable bi-montly release frequency.
Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 1069 packages (and hence 73 more since the last release) on CRAN depend on Rcpp for making analytical code go faster and further, along with another 91 in BioConductor.
This releases contains a large-ish update to the documentation as all vignettes (apart from the unit test one, which is a one-off) now use Markdown and the (still pretty new) pinp package by James and myself. There is also a new vignette corresponding to the PeerJ preprint James and I produced as an updated and current Introduction to Rcpp replacing the older JSS piece (which is still included as a vignette too).
A few other things got fixed: Dan is working on const
iterators you would expect with modern C++, Lei Yu spotted error in Modules, and more. See below for details.
Changes in Rcpp version 0.12.13 (2017-09-24)
Changes in Rcpp API:
Changes in Rcpp Modules:
- Misplacement of one parenthesis in macro
LOAD_RCPP_MODULE
was corrected (Lei Yu in #737)Changes in Rcpp Documentation:
Rewrote the macOS sections to depend on official documentation due to large changes in the macOS toolchain. (James Balamuta in #742 addressing issue #682).
Added a new vignette ‘Rcpp-introduction’ based on new PeerJ preprint, renamed existing introduction to ‘Rcpp-jss-2011’.
Transitioned all vignettes to the 'pinp' RMarkdown template (James Balamuta and Dirk Eddelbuettel in #755 addressing issue #604).
Added an entry on running 'compileAttributes()' twice to the Rcpp-FAQ (##745).
Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A maintenance release of RcppZiggurat is now on the CRAN network for R. It switched the vignette to the our new pinp package and its two-column pdf default.
The RcppZiggurat package updates the code for the Ziggurat generator which provides very fast draws from a Normal distribution. The package provides a simple C++ wrapper class for the generator improving on the very basic macros, and permits comparison among several existing Ziggurat implementations. This can be seen in the figure where Ziggurat from this package dominates accessing the implementations from the GSL, QuantLib and Gretl---all of which are still way faster than the default Normal generator in R (which is of course of higher code complexity).
The NEWS
file entry below lists all changes.
Changes in version 0.1.4 (2017-07-27)
The vignette now uses the pinp package in two-column mode.
Dynamic symbol registration is now enabled.
Courtesy of CRANberries, there is also a diffstat report for the most recent release. More information is on the RcppZiggurat page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A few short weeks after the more substantial 0.0.9 release of RcppAnnoy, we have a quick bug-fix update.
RcppAnnoy is our Rcpp-based R integration of the nifty Annoy library by Erik. Annoy is a small and lightweight C++ template header library for very fast approximate nearest neighbours.
Michaël Benesty noticed that our getItemsVector()
function didn't, ahem, do much besides crashing. Simple bug, they happen--now fixed, and a unit test added.
Changes in this version are summarized here:
Changes in version 0.0.10 (2017-09-25)
- The
getItemsVector()
function no longer crashes (#24)
Courtesy of CRANberries, there is also a diffstat report for this release.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A maintenance update RcppGSL 0.3.3 is now on CRAN. It switched the vignette to the our new pinp package and its two-column pdf default.
The RcppGSL package provides an interface from R to the GNU GSL using the Rcpp package.
No user-facing new code or features were added. The NEWS file entries follow below:
Changes in version 0.3.3 (2017-09-24)
We also check for
gsl-config
at package load.The vignette now uses the pinp package in two-column mode.
Minor other fixes to package and testing infrastructure.
Courtesy of CRANberries, a summary of changes to the most recent release is available.
More information is on the RcppGSL page. Questions, comments etc should go to the issue tickets at the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new version of the RcppCNPy package arrived on CRAN yesterday.
RcppCNPy provides R with read and write access to NumPy files thanks to the cnpy library by Carl Rogers.
This version updates internals for function registration, but otherwise mostly switches the vignette over to the shiny new pinp two-page template and package.
Changes in version 0.2.7 (2017-09-22)
Vignette updated to Rmd and use of
pinp
packageFile
src/init.c
added for dynamic registration
CRANberries also provides a diffstat report for the latest release. As always, feedback is welcome and the best place to start a discussion may be the GitHub issue tickets page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A bug-fix release RcppClassic 0.9.8 for the very recent 0.9.7 release which fixes a build issue on macOS introduced in 0.9.7. No other changes.
Courtesy of CRANberries, there are changes relative to the previous release.
Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A first update 0.0.2 of the pinp package arrived on CRAN just a few days after the initial release.
We added a new vignette for the package (see below), extended a few nice features, and smoothed a few corners.
The NEWS entry for this release follows.
Changes in tint version 0.0.2 (2017-09-20)
The YAML segment can be used to select font size, one-or-two column mode, one-or-two side mode, linenumbering and watermarks (#21 and #26 addressing #25)
If
pinp.cls
orjss.bst
are not present, they are copied in ((#27 addressing #23)Output is now in shaded framed boxen too (#29 addressing #28)
Endmatter material is placed in
template.tex
(#31 addressing #30)Expanded documentation of YAML options in
skeleton.Rmd
and clarified available one-column option (#32).Section numbering can now be turned on and off (#34)
The default bibliography style was changed to
jss.bst
.A short explanatory vignette was added.
Courtesy of CRANberries, there is a comparison to the previous release. More information is on the tint page. For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A rather boring and otherwise uneventful release 0.9.7 of RcppClassic is now at CRAN. This package provides a maintained version of the otherwise deprecated first Rcpp API; no new projects should use it.
Once again no changes in user-facing code. But this makes it the first package to use the very new and shiny pinp package as the backend for its vignette, now converted to Markdown---see here for this new version. We also updated three sources files for tabs versus spaces as the current g++
version complained (correctly !!) about misleading indents. Otherwise a file src/init.c
was added for dynamic registration, the Travis CI runner script was updated to using run.sh
from our r-travis fork, and we now strip the library after they have been built. Again, no user code changes.
And no iterate: nobody should use this package. Rcpp is so much better in so many ways---this one is simply available as we (quite strongly) believe that APIs are contracts, and as such we hold up our end of the deal.
Courtesy of CRANberries, there are changes relative to the previous release.
Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A brandnew and very exciting (to us, at least) package called pinp
just arrived on CRAN, following a somewhat unnecessarily long passage out of incoming. It is based on the PNAS LaTeX Style offered by the Proceeding of the National Academy of Sciences of the United States of America, or PNAS for short. And there is already a Markdown version in the wonderful rticles packages.
But James Balamuta and I thought we could do one better when we were looking to typeset our recent PeerJ Prepint as an attractive looking vignette for use within the Rcpp package.
And so we did by changing a few things (font, color, use of natbib and Chicago.bst
for references, removal of a bunch of extra PNAS-specific formalities from the frontpage) and customized a number of other things for easier use by vignettes directly from the YAML header (draft mode watermark, doi or url for packages, easier author naming in footer, bibtex file and more).
We are quite pleased with the result which seems ready for the next Rcpp release---see e.g., these two teasers:
and
and the pinp package page or the GitHub repo have the full (four double-)pages of what turned a more dull looking 27 page manuscript into eight crisp two-column pages.
We have few more things planned (i.e., switching to single column mode, turning on linenumbers at least in one-column mode).
For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new version of drat arrived earlier today on CRAN as another no-human-can-delay-this automatic upgrade directly from the CRAN prechecks. It is mostly a maintenance release ensuring PACKAGES.rds
is also updated, plus some minor other edits.
drat stands for drat R Archive Template, and helps with easy-to-create and easy-to-use repositories for R packages. Since its inception in early 2015 it has found reasonably widespread adoption among R users because repositories with marked releases is the better way to distribute code.
The NEWS
file summarises the release as follows:
Changes in drat version 0.1.3 (2017-09-16)
Ensure
PACKAGES.rds
, if present, is also inserted in repoUpdated 'README.md' removing stale example URLs (#63)
Use https to fetch Travis CI script from r-travis
Courtesy of CRANberries, there is a comparison to the previous release. More detailed information is on the drat page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new and much enhanced version of RcppMsgPack arrived on CRAN a couple of days ago. It came together following this email to the r-package-devel list which made it apparent that Travers Ching had been working on MessagePack converters for R which required the very headers I had for use from, inter alia, the RcppRedis package.
So we joined our packages. I updated the headers in RcppMsgPack to the current upstream version 2.1.5 of MessagePack, and Travers added his helper functions allow direct packing / unpacking of MessagePack objects at the R level, as well as tests and a draft vignette. Very exciting, and great to have a coauthor!
So now RcppMspPack provides R with both MessagePack header files for use via C++ (or C, if you must) packages such as RcppRedis --- and direct conversion routines at the R prompt.
MessagePack itself is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it is faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.
Changes in version 0.2.0 (2017-09-07)
Added support for building on Windows
Upgraded to MsgPack 2.1.5 (#3)
New R functions to manipulate MsgPack objects: msgpack_format, msgpack_map, msgpack_pack, msgpack_simplify, mgspack_unpack (#4)
New R functions also available as msgpackFormat, msgpackMap, msgpackPack, msgpackSimplify, mgspackUnpack (#4)
New vignette (#4)
New tests (#4)
Courtesy of CRANberries, there is also a diffstat report for this release. More information is on the RcppRedis page.
More information may be on the RcppMsgPack page. Issues and bugreports should go to the GitHub issue tracker.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new minor release of RcppRedis arrived on CRAN last week, following the release 0.2.0 of RcppMsgPack which brought the MsgPack headers forward to release 2.1.5. This required a minor and rather trivial change in the code. When the optional RcppMsgPack package is used, we now require this version 0.2.0 or later.
We made a few internal updates to the package as well.
Changes in version 0.1.8 (2017-09-08)
A new file
init.c
was added with calls toR_registerRoutines()
andR_useDynamicSymbols()
Symbol registration is enabled in
useDynLib
Travis CI was updated to using
run.sh
The (optional MessagePack) code was updated for MsgPack 2.*
Courtesy of CRANberries, there is also a diffstat report for this release. More information is on the RcppRedis page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
An new version 0.0.9 of RcppAnnoy, our Rcpp-based R integration of the nifty Annoy library by Erik, is now on CRAN. Annoy is a small and lightweight C++ template header library for very fast approximate nearest neighbours.
This release corrects an issue for Windows users discovered by GitHub user 'khoran' who later also suggested the fix of binary mode. It upgrades to Annoy release 1.9.1 and brings its new Manhattan distance to RcppAnnoy. A number of unit tests were added as well, and we updated some packaging internals such as symbol registration.
And I presume I had a good streak emailing with Uwe's robots as the package made it onto CRAN rather smoothly within ten minutes of submission:
Changes in this version are summarized here:
Changes in version 0.0.9 (2017-08-31)
Synchronized with Annoy upstream version 1.9.1
Minor updates in calls and tests as required by annoy 1.9.1
New Manhattan distance modules along with unit test code
Additional unit tests from upstream test code carried over
Binary mode is used for
save
(as suggested by @khoran in #21)A new file
init.c
was added with calls toR_registerRoutines()
andR_useDynamicSymbols()
Symbol registration is enabled in
useDynLib
Courtesy of CRANberries, there is also a diffstat report for this release.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A second fix-up release is needed following on the recent bi-monthly RcppArmadillo release as well as the initial follow-up as it turns out that OS X / macOS is so darn special that it needs an entire separate treatment for OpenMP. Namely to turn it off entirely...
Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language--and is widely used by (currently) 384 other packages on CRAN---an increase of 54 since the CRAN release in June!
Changes in RcppArmadillo version 0.7.960.1.2 (2017-08-29)
Courtesy of CRANberries, there is a diffstat report. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new version 0.2.0 of the RcppSMC package arrived on CRAN earlier today (as a very quick pretest-publish within minutes of submission).
RcppSMC provides Rcpp-based bindings to R for the Sequential Monte Carlo Template Classes (SMCTC) by Adam Johansen described in his JSS article.
This release 0.2.0 is chiefly the work of Leah South, a Ph.D. student at Queensland University of Technology, who was during the last few months a Google Summer of Code student mentored by Adam and myself. It was pleasure to work with Leah on this, and see her progress. Our congratulations to Leah for a job well done!
Changes in RcppSMC version 0.2.0 (2017-08-28)
Also use
.registration=TRUE
inuseDynLib
inNAMESPACE
Multiple Sequential Monte Carlo extensions (Leah South as part of Google Summer of Code 2017)
Using Rcpp attributes (#2).
Adding multiple normalising constant estimators (#7).
Static Bayesian model example: linear regression (#10 addressing #9).
Framework for additional algorithm parameters and adaptation (#19 addressing #16; also #24 addressing #23).
Common adaptation methods for static Bayesian models (#20 addressing #17).
Supporting MCMC repeated runs (#21).
Adding adaptation to linear regression example (#22 addressing #18).
Courtesy of CRANberries, there is a diffstat report for this release.
More information is on the RcppSMC page. Issues and bugreports should go to the GitHub issue tracker.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The BH package on CRAN was updated today to version 1.65.0. BH provides a sizeable portion of the Boost C++ libraries as a set of template headers for use by R, possibly with Rcpp as well as other packages.
This release upgrades the version of Boost to the rather new upstream version Boost 1.65.0 released earlier this week, and adds two new libraries: align
and sort
.
I had started the upgrade process a few days ago under release 1.64.0. Rigorous checking of reverse dependencies showed that mvnfast needed a small change (which was trivial: just seeding the RNG prior to running tests), which Matteo did in no time with a fresh CRAN upload. rstan is needing a bit more work but should be ready real soon now and we are awaiting a new version. And once I switched to the just release Boost 1.65.0 it became apparent that Cyclops no longer needs its embedded copy of Boost iterator---and Marc already made that change with yet another fresh CRAN upload. It is a true pleasure to work in such a responsive and collaborative community.
Changes in version 1.65.0-1 (2017-08-24)
Upgraded to Boost 1.64 and then 1.65 installed directly from upstream source with several minor tweaks (as before)
Fourth tweak corrects a misplaced curly brace (see the Boost ublas GitHub repo and its issue #40)
Added Boost multiprecision by fixing a script typo (as requested in #42)
Updated Travis CI support via newer
run.sh
Via CRANberries, there is a diffstat
report relative to the previous release.
Comments and suggestions are welcome via the mailing list or the issue tracker at the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Over the last few days, Rcpp passed another noteworthy hurdle. It is now used by over 10 percent of packages on CRAN (as measured by Depends, Imports and LinkingTo, but excluding Suggests). As of this morning 1130 packages use Rcpp out of a total of 11275 packages. The graph on the left shows the growth of both outright usage numbers (in darker blue, left axis) and relative usage (in lighter blue, right axis).
Older posts on this blog took note when Rcpp passed round hundreds of packages, most recently in April for 1000 packages. The growth rates for both Rcpp, and of course CRAN, are still staggering. A big thank you to everybody who makes this happen, from R Core and CRAN to all package developers, contributors, and of course all users driving this. We have built ourselves a rather impressive ecosystem.
So with that a heartfelt Thank You! to all users and contributors of R, CRAN, and of course Rcpp, for help, suggestions, bug reports, documentation, encouragement, and, of course, code.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
On the heels of the very recent bi-monthly RcppArmadillo release comes a quick bug-fix release 0.7.960.1.1 which just got onto CRAN (and I will ship a build to Debian in a moment).
There were three distinct issues I addressed in three quick pull requests:
is()
method of our S4
class. Thanks to Kevin Ushey for reminding me we had is()
..registration=TRUE
), so ensured we only have one fastLm
symbol.configure
check now also wants g++ 4.7.2
or later just like Armadillo.Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language--and is widely used by (currently) 382 other packages on CRAN---an increase of 52 since the CRAN release in June!
Changes in this release relative to the previous CRAN release are as follows:
Changes in RcppArmadillo version 0.7.960.1.1 (2017-08-20)
Courtesy of CRANberries, there is a diffstat report. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the tenth post in the rarely ranting R recommendations series, or R4 for short. A few days ago we showed how to tell the linker to strip shared libraries. As discussed in the post, there are two options. One can either set up ~/.R/Makevars
by passing the strip-debug
option to the linker. Alternatively, one can adjust src/Makevars
in the package itself with a bit a Makefile magic.
Of course, there is a third way: just run strip --strip-debug
over all the shared libraries after the build. As the path is standardized, and the shell does proper globbing, we can just do
$ strip --strip-debug /usr/local/lib/R/site-library/*/libs/*.so
using a double-wildcard to get all packages (in that R package directory) and all their shared libraries. Users on macOS probably want .dylib
on the end, users on Windows want another computer as usual (just kidding: use .dll
). Either may have to adjust the path which is left as an exercise to the reader.
The impact can be Yuge as illustrated in the following dotplot:
This illustration is in response to a mailing list post. Last week, someone claimed on r-help that tidyverse would not install on Ubuntu 17.04. And this is of course patently false as many of us build and test on Ubuntu and related Linux systems, Travis runs on it, CRAN tests them etc pp. That poor user had somehow messed up their default gcc
version. Anyway: I fired up a Docker container, installed r-base-core
plus three required -dev
packages (for xml2, openssl, and curl) and ran a single install.packages("tidyverse")
. In a nutshell, following the launch of Docker for an Ubuntu 17.04 container, it was just
$ apt-get update
$ apt-get install r-base libcurl4-openssl-dev libssl-dev libxml2-dev
$ apt-get install mg # a tiny editor
$ mg /etc/R/Rprofile.site # to add a default CRAN repo
$ R -e 'install.packages("tidyverse")'
which not only worked (as expected) but also installed a whopping fifty-one packages (!!) of which twenty-six contain a shared library. A useful little trick is to run du
with proper options to total, summarize, and use human units which reveals that these libraries occupy seventy-eight megabytes:
root@de443801b3fc:/# du -csh /usr/local/lib/R/site-library/*/libs/*so
4.3M /usr/local/lib/R/site-library/Rcpp/libs/Rcpp.so
2.3M /usr/local/lib/R/site-library/bindrcpp/libs/bindrcpp.so
144K /usr/local/lib/R/site-library/colorspace/libs/colorspace.so
204K /usr/local/lib/R/site-library/curl/libs/curl.so
328K /usr/local/lib/R/site-library/digest/libs/digest.so
33M /usr/local/lib/R/site-library/dplyr/libs/dplyr.so
36K /usr/local/lib/R/site-library/glue/libs/glue.so
3.2M /usr/local/lib/R/site-library/haven/libs/haven.so
272K /usr/local/lib/R/site-library/jsonlite/libs/jsonlite.so
52K /usr/local/lib/R/site-library/lazyeval/libs/lazyeval.so
64K /usr/local/lib/R/site-library/lubridate/libs/lubridate.so
16K /usr/local/lib/R/site-library/mime/libs/mime.so
124K /usr/local/lib/R/site-library/mnormt/libs/mnormt.so
372K /usr/local/lib/R/site-library/openssl/libs/openssl.so
772K /usr/local/lib/R/site-library/plyr/libs/plyr.so
92K /usr/local/lib/R/site-library/purrr/libs/purrr.so
13M /usr/local/lib/R/site-library/readr/libs/readr.so
4.7M /usr/local/lib/R/site-library/readxl/libs/readxl.so
1.2M /usr/local/lib/R/site-library/reshape2/libs/reshape2.so
160K /usr/local/lib/R/site-library/rlang/libs/rlang.so
928K /usr/local/lib/R/site-library/scales/libs/scales.so
4.9M /usr/local/lib/R/site-library/stringi/libs/stringi.so
1.3M /usr/local/lib/R/site-library/tibble/libs/tibble.so
2.0M /usr/local/lib/R/site-library/tidyr/libs/tidyr.so
1.2M /usr/local/lib/R/site-library/tidyselect/libs/tidyselect.so
4.7M /usr/local/lib/R/site-library/xml2/libs/xml2.so
78M total
root@de443801b3fc:/#
Looks like dplyr wins this one at thirty-three megabytes just for its shared library.
But with a single stroke of strip
we can reduce all this down a lot:
root@de443801b3fc:/# strip --strip-debug /usr/local/lib/R/site-library/*/libs/*so
root@de443801b3fc:/# du -csh /usr/local/lib/R/site-library/*/libs/*so
440K /usr/local/lib/R/site-library/Rcpp/libs/Rcpp.so
220K /usr/local/lib/R/site-library/bindrcpp/libs/bindrcpp.so
52K /usr/local/lib/R/site-library/colorspace/libs/colorspace.so
56K /usr/local/lib/R/site-library/curl/libs/curl.so
120K /usr/local/lib/R/site-library/digest/libs/digest.so
2.5M /usr/local/lib/R/site-library/dplyr/libs/dplyr.so
16K /usr/local/lib/R/site-library/glue/libs/glue.so
404K /usr/local/lib/R/site-library/haven/libs/haven.so
76K /usr/local/lib/R/site-library/jsonlite/libs/jsonlite.so
20K /usr/local/lib/R/site-library/lazyeval/libs/lazyeval.so
24K /usr/local/lib/R/site-library/lubridate/libs/lubridate.so
8.0K /usr/local/lib/R/site-library/mime/libs/mime.so
52K /usr/local/lib/R/site-library/mnormt/libs/mnormt.so
84K /usr/local/lib/R/site-library/openssl/libs/openssl.so
76K /usr/local/lib/R/site-library/plyr/libs/plyr.so
32K /usr/local/lib/R/site-library/purrr/libs/purrr.so
648K /usr/local/lib/R/site-library/readr/libs/readr.so
400K /usr/local/lib/R/site-library/readxl/libs/readxl.so
128K /usr/local/lib/R/site-library/reshape2/libs/reshape2.so
56K /usr/local/lib/R/site-library/rlang/libs/rlang.so
100K /usr/local/lib/R/site-library/scales/libs/scales.so
496K /usr/local/lib/R/site-library/stringi/libs/stringi.so
124K /usr/local/lib/R/site-library/tibble/libs/tibble.so
164K /usr/local/lib/R/site-library/tidyr/libs/tidyr.so
104K /usr/local/lib/R/site-library/tidyselect/libs/tidyselect.so
344K /usr/local/lib/R/site-library/xml2/libs/xml2.so
6.6M total
root@de443801b3fc:/#
Down to six point six megabytes. Not bad for one command. The chart visualizes the respective reductions. Clearly, C++ packages (and their template use) lead to more debugging symbols than plain old C code. But once stripped, the size differences are not that large.
And just to be plain, what we showed previously in post #9 does the same, only already at installation stage. The effects are not cumulative.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The bi-monthly RcppArmadillo release is out with a new version 0.7.960.1.0 which is now on CRAN, and will get to Debian in due course.
And it is a big one. Lots of nice upstream changes from Armadillo, and lots of work on our end as the Google Summer of Code project by Binxiang Ni, plus a few smaller enhancements -- see below for details.
Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language--and is widely used by (currently) 379 other packages on CRAN---an increase of 49 since the last CRAN release in June!
Changes in this release relative to the previous CRAN release are as follows:
Changes in RcppArmadillo version 0.7.960.1.0 (2017-08-11)
Upgraded to Armadillo release 7.960.1 (Northern Banana Republic Deluxe)
faster
randn()
when using OpenMP (NB: usually omitted when used fromR)faster
gmm_diag
class, for Gaussian mixture models with diagonal covariance matricesadded
.sum_log_p()
to thegmm_diag
classadded
gmm_full
class, for Gaussian mixture models with full covariance matricesexpanded
.each_slice()
to optionally use OpenMP for multi-threaded executionUpgraded to Armadillo release 7.950.0 (Northern Banana Republic)
expanded
accu()
andsum()
to use OpenMP for processing expressions with computationally expensive element-wise functionsexpanded
trimatu()
andtrimatl()
to allow specification of the diagonal which delineates the boundary of the triangular partEnhanced support for sparse matrices (Binxiang Ni as part of Google Summer of Code 2017)
Add support for
dtCMatrix
anddsCMatrix
(#135)Add conversion and unit tests for
dgT
,dtT
anddsTMatrix
(#136)Add conversion and unit tests for
dgR
,dtR
anddsRMatrix
(#139)Add conversion and unit tests for
pMatrix
andddiMatrix
(#140)Rewrite conversion for
dgT
,dtT
anddsTMatrix
, and add file-based tests (#142)Add conversion and unit tests for
indMatrix
(#144)Rewrite conversion for
ddiMatrix
(#145)Add a warning message for matrices that cannot be converted (#147)
Add new vignette for sparse matrix support (#152; Dirk in #153)
Add support for sparse matrix conversion from Python SciPy (#158 addressing #141)
Optional return of row or column vectors in collapsed form if appropriate
#define
is set (Serguei Sokol in #151 and #154)Correct
speye()
for non-symmetric cases (Qiang Kou in #150 closing #149).Ensure tests using Scientific Python and reticulate are properly conditioned on the packages being present.
Added
.aspell/
directory with small local directory now supported by R-devel.
Courtesy of CRANberries, there is a diffstat report. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the nineth post in the recognisably rancid R randomness series, or R4 for short. Following on the heels of last week's post, we aim to look into the shared libraries created by R.
We love the R build process. It is robust, cross-platform, reliable and rather predicatable. It. Just. Works.
One minor issue, though, which has come up once or twice in the past is the (in)ability to fully control all compilation options. R will always recall CFLAGS
, CXXFLAGS
, ... etc as used when it was compiled. Which often entails the -g
flag for debugging which can seriously inflate the size of the generated object code. And once stored in ${RHOME}/etc/Makeconf
we cannot on the fly override these values.
But there is always a way. Sometimes even two.
The first is local and can be used via the (personal) ~/.R/Makevars
file (about which I will have to say more in another post). But something I have been using quite a bite lately uses the flags for the shared library linker. Given that we can have different code flavours and compilation choices---between C, Fortran and the different C++ standards---one can end up with a few lines. I currently use this which uses -Wl,
to pass an the -S
(or --strip-debug
) option to the linker (and also reiterates the desire for a shared library, presumably superfluous):
SHLIB_CXXLDFLAGS = -Wl,-S -shared
SHLIB_CXX11LDFLAGS = -Wl,-S -shared
SHLIB_CXX14LDFLAGS = -Wl,-S -shared
SHLIB_FCLDFLAGS = -Wl,-S -shared
SHLIB_LDFLAGS = -Wl,-S -shared
Let's consider an example: my most recently uploaded package RProtoBuf. Built under a standard 64-bit Linux setup (Ubuntu 17.04, g++ 6.3) and not using the above, we end up with library containing 12 megabytes (!!) of object code:
edd@brad:~/git/rprotobuf(feature/fewer_warnings)$ ls -lh src/RProtoBuf.so
-rwxr-xr-x 1 edd edd 12M Aug 14 20:22 src/RProtoBuf.so
edd@brad:~/git/rprotobuf(feature/fewer_warnings)$
However, if we use the flags shown above in .R/Makevars
, we end up with much less:
edd@brad:~/git/rprotobuf(feature/fewer_warnings)$ ls -lh src/RProtoBuf.so
-rwxr-xr-x 1 edd edd 626K Aug 14 20:29 src/RProtoBuf.so
edd@brad:~/git/rprotobuf(feature/fewer_warnings)$
So we reduced the size from 12mb to 0.6mb, an 18-fold decrease. And the file
tool still shows the file as 'not stripped' as it still contains the symbols. Only debugging information was removed.
What reduction in size can one expect, generally speaking? I have seen substantial reductions for C++ code, particularly when using tenmplated code. More old-fashioned C code will be less affected. It seems a little difficult to tell---but this method is my new build default as I continually find rather substantial reductions in size (as I tend to work mostly with C++-based packages).
The second option only occured to me this evening, and complements the first which is after all only applicable locally via the ~/.R/Makevars
file. What if we wanted it affect each installation of a package? The following addition to its src/Makevars
should do:
strippedLib: $(SHLIB)
if test -e "/usr/bin/strip"; then /usr/bin/strip --strip-debug $(SHLIB); fi
.phony: strippedLib
We declare a new Makefile
target strippedLib
. But making it dependent on $(SHLIB)
, we ensure the standard target of this Makefile
is built. And by making the target .phony
we ensure it will always be executed. And it simply tests for the strip
tool, and invokes it on the library after it has been built. Needless to say we get the same reduction is size. And this scheme may even pass muster with CRAN, but I have not yet tried.
Lastly, and acknowledgement. Everything in this post has benefited from discussion with my former colleague Dan Dillon who went as far as setting up tooling in his r-stripper repository. What we have here may be simpler, but it would not have happened with what Dan had put together earlier.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
RProtoBuf provides R bindings for the Google Protocol Buffers ("ProtoBuf") data encoding and serialization library used and released by Google, and deployed fairly widely in numerous projects as a language and operating-system agnostic protocol.
A new releases RProtoBuf 0.4.10 just appeared on CRAN. It is a maintenance releases replacing one leftover errorenous use of package=
in .Call
with the correct PACKAGE=
(as requsted by CRAN). It also integrates a small robustification in the deserializer when encountering invalide objects; this was both reported and fixed by Jeffrey Shen.
Changes in RProtoBuf version 0.4.10 (2017-08-13)
CRANberries also provides a diff to the previous release. The RProtoBuf page has an older package vignette, a 'quick' overview vignette, a unit test summary vignette, and the pre-print for the JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the eight post in the ramblingly random R rants series, or R4 for short. We took a short break over the last few weeks due to some conferencing followed by some vacationing and general chill.
But we're back now, and this post gets us back to initial spirit of (hopefully) quick and useful posts. Perusing yesterday's batch of CRANberries posts, I noticed a peculiar new directory shown the in the diffstat output we use to compare two subsequent source tarballs. It was entitled .aspell/
, in the top-level directory, and in two new packages by R Core member Kurt Hornik himself.
The context is, of course, the not infrequently-expressed desire to customize the spell checking done on CRAN incoming packages, see e.g. this r-package-devel thread.
And now we can as I verified with (the upcoming next release of) RcppArmadillo, along with a recent-enough (i.e. last few days) version of r-devel. Just copying what Kurt did, i.e. adding a file .aspell/defaults.R
, and in it pointing to rds file (named as the package) containing a character vector with words added to the spell checker's universe is all it takes. For my package, see here for the peculiars.
Or see here:
edd@bud:~/git/rcpparmadillo/.aspell(master)$ cat defaults.R
Rd_files <- vignettes <- R_files <- description <-
list(encoding = "UTF-8",
language = "en",
dictionaries = c("en_stats", "RcppArmadillo"))
edd@bud:~/git/rcpparmadillo/.aspell(master)$ r -p -e 'readRDS("RcppArmadillo.rds")'
[1] "MPL" "Sanderson" "Templated"
[4] "decompositions" "onwards" "templated"
edd@bud:~/git/rcpparmadillo/.aspell(master)$
And now R(-devel) CMD check --as-cran ...
is silent about spelling. Yay!
But take this with a grain of salt as this does not yet seem to be "announced" as e.g. yesterday's change in the CRAN Policy did not mention it. So things may well change -- but hey, it worked for me.
And this all is about aspell
, here is something topical about a spell to close the post:
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Just getting back from the most fun meetup I have been to in quite some time: episode 23 (by their count) of Open Source Open Mic hosted by Matt Godbolt and Joe Walnes here in Chicago. Nothing but a sequence of lightning talks. Plus beer and pizza. Sounds awesome? It was!
We had fantastic talks across at least half a dozen languages, covering both new-ish (Pony) and interesting ones such (Rust, Go, ...) plus of course some Javascript and some Python, no Java (yay!) and a few batshit crazy things like a self-hosting database in its own (shell) code, a terminal gif viewer (!!), and more. And it gave me an opportunity to quickly (one evening and morning commute) jam out a presentation about what is in the title: R for system administration.
And I am only half-joking. I had used R a couple of years ago when I needed to select, subset, modify, ... a large number of image files given some timestamp and filename patterns. And given how well R works in a vectorised manner with both regular expressions and timestamps, as well as on top of essentially all standard POSIX-style operating system / file-system functions, I picked up that thread again on the problem of ... cleaning up the file storage underlying CRANberries which by now has well over fifty-seven thousand (!!) tarballs of CRAN packages based on now ten years of CRANberries. So I showed how to prune this in essentially half a dozen lines of R (and data.table code), plus some motivation---all just right for a lightning talk. Seemingly the talk went well enough as quite a few folks gave a thumbs up and compliments over beers afterwards.
But see for yourself as the slides are now uploaded to my standard talks page.
My thanks to Matt and Joe for organizing the meetup. I think I will be back.
A good six years ago I blogged about plotOBOS()
which charts a moving average (from one of several available variants) along with shaded standard deviation bands. That post has a bit more background on the why/how and motivation, but as a teaser here is the resulting chart of the SP500 index (with ticker ^GSCP):
The code uses a few standard finance packages for R (with most of them maintained by Joshua Ulrich given that Jeff Ryan, who co-wrote chunks of these, is effectively retired from public life). Among these, xts had a recent release reflecting changes which occurred during the four (!!) years since the previous release, and covering at least two GSoC projects. With that came subtle API changes: something we all generally try to avoid but which is at times the only way forward. In this case, the shading code I used (via polygon()
from base R) no longer cooperated with the beefed-up functionality of plot.xts()
. Luckily, Ross Bennett incorporated that same functionality into a new function addPolygon
--- which even credits this same post of mine.
With that, the updated code becomes
## plotOBOS -- displaying overbough/oversold as eg in Bespoke's plots
##
## Copyright (C) 2010 - 2017 Dirk Eddelbuettel
##
## This is free software: you can redistribute it and/or modify it
## under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 2 of the License, or
## (at your option) any later version.
suppressMessages(library(quantmod)) # for getSymbols(), brings in xts too
suppressMessages(library(TTR)) # for various moving averages
plotOBOS <- function(symbol, n=50, type=c("sma", "ema", "zlema"),
years=1, blue=TRUE, current=TRUE, title=symbol,
ticks=TRUE, axes=TRUE) {
today <- Sys.Date()
if (class(symbol) == "character") {
X <- getSymbols(symbol, from=format(today-365*years-2*n), auto.assign=FALSE)
x <- X[,6] # use Adjusted
} else if (inherits(symbol, "zoo")) {
x <- X <- as.xts(symbol)
current <- FALSE # don't expand the supplied data
}
n <- min(nrow(x)/3, 50) # as we may not have 50 days
sub <- ""
if (current) {
xx <- getQuote(symbol)
xt <- xts(xx$Last, order.by=as.Date(xx$`Trade Time`))
colnames(xt) <- paste(symbol, "Adjusted", sep=".")
x <- rbind(x, xt)
sub <- paste("Last price: ", xx$Last, " at ",
format(as.POSIXct(xx$`Trade Time`), "%H:%M"), sep="")
}
type <- match.arg(type)
xd <- switch(type, # compute xd as the central location via selected MA smoother
sma = SMA(x,n),
ema = EMA(x,n),
zlema = ZLEMA(x,n))
xv <- runSD(x, n) # compute xv as the rolling volatility
strt <- paste(format(today-365*years), "::", sep="")
x <- x[strt] # subset plotting range using xts' nice functionality
xd <- xd[strt]
xv <- xv[strt]
xyd <- xy.coords(.index(xd),xd[,1]) # xy coordinates for direct plot commands
xyv <- xy.coords(.index(xv),xv[,1])
n <- length(xyd$x)
xx <- xyd$x[c(1,1:n,n:1)] # for polygon(): from first point to last and back
if (blue) {
blues5 <- c("#EFF3FF", "#BDD7E7", "#6BAED6", "#3182BD", "#08519C") # cf brewer.pal(5, "Blues")
fairlylight <<- rgb(189/255, 215/255, 231/255, alpha=0.625) # aka blues5[2]
verylight <<- rgb(239/255, 243/255, 255/255, alpha=0.625) # aka blues5[1]
dark <<- rgb(8/255, 81/255, 156/255, alpha=0.625) # aka blues5[5]
## buglet in xts 0.10-0 requires the <<- here
} else {
fairlylight <<- rgb(204/255, 204/255, 204/255, alpha=0.5) # two suitable grays, alpha-blending at 50%
verylight <<- rgb(242/255, 242/255, 242/255, alpha=0.5)
dark <<- 'black'
}
plot(x, ylim=range(range(x, xd+2*xv, xd-2*xv, na.rm=TRUE)), main=title, sub=sub,
major.ticks=ticks, minor.ticks=ticks, axes=axes) # basic xts plot setup
addPolygon(xts(cbind(xyd$y+xyv$y, xyd$y+2*xyv$y), order.by=index(x)), on=1, col=fairlylight) # upper
addPolygon(xts(cbind(xyd$y-xyv$y, xyd$y+1*xyv$y), order.by=index(x)), on=1, col=verylight) # center
addPolygon(xts(cbind(xyd$y-xyv$y, xyd$y-2*xyv$y), order.by=index(x)), on=1, col=fairlylight) # lower
lines(xd, lwd=2, col=fairlylight) # central smooted location
lines(x, lwd=3, col=dark) # actual price, thicker
}
and the main change are the three calls to addPolygon
. To illustrate, we call plotOBOS("SPY", years=2)
with an updated plot of the ETF representing the SP500 over the last two years:
Comments and further enhancements welcome!
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new version of RcppAPT -- our interface from R to the C++ library behind the awesome apt
, apt-get
, apt-cache
, ... commands and their cache powering Debian, Ubuntu and the like -- arrived on CRAN yesterday.
We added a few more functions in order to compute on the package graph. A concrete example is shown in this vignette which determines the (minimal) set of remaining Debian packages requiring a rebuild under R 3.4.* to update their .C()
and .Fortran()
registration code. It has been used for the binNMU request #868558.
As we also added a NEWS file, its (complete) content covering all releases follows below.
Changes in version 0.0.4 (2017-07-16)
New function
getDepends
New function
reverseDepends
Added package registration code
Added usage examples in
scripts
directoryAdded vignette, also in
docs
as rendered copyChanges in version 0.0.3 (2016-12-07)
- Added
dumpPackages
,showSrc
Changes in version 0.0.2 (2016-04-04)
- Added
reverseDepends
,dumpPackages
,showSrc
Changes in version 0.0.1 (2015-02-20)
- Initial version with
getPackages
andhasPackages
A bit more information about the package is available here as well as as the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The twelveth update in the 0.12.* series of Rcpp landed on CRAN this morning, following two days of testing at CRAN preceded by five full reverse-depends checks we did (and which are always logged in this GitHub repo). The Debian package has been built and uploaded; Windows and macOS binaries should follow at CRAN as usual. This 0.12.12 release follows the 0.12.0 release from late July, the 0.12.1 release in September, the 0.12.2 release in November, the 0.12.3 release in January, the 0.12.4 release in March, the 0.12.5 release in May, the 0.12.6 release in July, the 0.12.7 release in September, the 0.12.8 release in November, the 0.12.9 release in January, the 0.12.10.release in March, and the 0.12.11.release in May making it the sixteenth release at the steady and predictable bi-montly release frequency.
Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 1097 packages (and hence 71 more since the last release in May) on CRAN depend on Rcpp for making analytical code go faster and further, along with another 91 in BioConductor.
This releases contain a fairly large number of fairly small and focused pull requests most of which either correct some corner cases or improve other aspects. JJ tirelessly improved the package registration added in the previous release and following R 3.4.0. Kirill tidied up a number of small issues allowing us to run compilation in even more verbose modes---usually a good thing. Jeroen, Elias Pipping and Yo Gong all contributed as well, and we thank everybody for their contributions.
All changes are listed below in some detail.
Changes in Rcpp version 0.12.12 (2017-07-13)
Changes in Rcpp API:
The
tinyformat.h
header now ends in a newline (#701).Fixed rare protection error that occurred when fetching stack traces during the construction of an Rcpp exception (Kirill Müller in #706).
Compilation is now also possibly on Haiku-OS (Yo Gong in #708 addressing #707).
Dimension attributes are explicitly cast to
int
(Kirill Müller in #715).Unused arguments are no longer declared (Kirill Müller in #716).
Visibility of exported functions is now supported via the R macro
atttribute_visible
(Jeroen Ooms in #720).The
no_init()
constructor acceptsR_xlen_t
(Kirill Müller in #730).Loop unrolling used
R_xlen_t
(Kirill Müller in #731).Two unused-variables warnings are now avoided (Jeff Pollock in #732).
Changes in Rcpp Attributes:
Execute tools::package_native_routine_registration_skeleton within package rather than current working directory (JJ in #697).
The R portion no longer uses
dir.exists
to no require R 3.2.0 or newer (Elias Pipping in #698).Fix native registration for exports with name attribute (JJ in #703 addressing #702).
Automatically register init functions for Rcpp Modules (JJ in #705 addressing #704).
Add Shield around parameters in Rcpp::interfaces (JJ in #713 addressing #712).
Replace dot (".") with underscore ("_") in package names when generating native routine registrations (JJ in #722 addressing #721).
Generate C++ native routines with underscore ("_") prefix to avoid exporting when standard exportPattern is used in NAMESPACE (JJ in #725 addressing #723).
Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new version of the nanotime package for working with nanosecond timestamps just arrived on CRAN.
nanotime uses the RcppCCTZ package for (efficient) high(er) resolution time parsing and formatting up to nanosecond resolution, and the bit64 package for the actual integer64
arithmetic.
Thanks to a metric ton of work by Leonardo Silvestri, the package now uses S4 classes internally allowing for greater consistency of operations on nanotime objects.
Changes in version 0.2.0 (2017-06-22)
Rewritten in S4 to provide more robust operations (#17 by Leonardo)
Ensure
tz=""
is treated as unset (Leonardo in #20)Added
format
andtz
arguments tonanotime
,format
,Ensure printing respect
options()$max.print
, ensure names are kept with vector (#23 by Leonardo)Correct
summary()
by definingnames<-
(Leonardo in #25 fixing #24)Report error on operations that are meaningful for type; handled NA, NaN, Inf, -Inf correctly (Leonardo in #27 fixing #26)
We also have a diff to the previous version thanks to CRANberries. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new minor version 0.2.3 of RcppCCTZ is now on CRAN.
RcppCCTZ uses Rcpp to bring CCTZ to R. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. The RcppCCTZ page has a few usage examples and details.
This version ensures that we set the TZDIR
environment variable correctly on the old dreaded OS that does not come with proper timezone information---an issue which had come up while preparing the next (and awesome, trust me) release of nanotime. It also appears that I failed to blog about 0.2.2, another maintenance release, so changes for both are summarised next.
Changes in version 0.2.3 (2017-06-19)
On Windows, the
TZDIR
environment variable is now set in.onLoad()
Replaced
init.c
with registration code inside ofRcppExports.cpp
thanks to Rcpp 0.12.11.Changes in version 0.2.2 (2017-04-20)
Synchronized with upstream CCTZ
The
time_point
object is instantiated explicitly for nanosecond use which appears to be required on macOS
We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the seventh post in the rarely relevant R ramblings series, or R4 for short.
We took a short break as several conferences and other events interfered during the month of May, keeping us busy and away from this series. But we are back now with a short and useful hack I came up with this weekend.
The topic is C++14, i.e. the newest formally approved language standard for C++, and its support in R and on Travis CI. With release R 3.4.0 of a few weeks ago, R now formally supports C++14. Which is great.
But there be devils. A little known fact is that R hangs on to its configuration settings from its own compile time. That matters in cases such as the one we are looking at here: Travis CI. Travis is a tremendously useful and widely-deployed service, most commonly connected to GitHub driving "continuous integration" (the 'CI') testing after each commit. But Travis CI, for as useful as it is, is also maddingly conservative still forcing everybody to live and die by [Ubuntu 14.04]http://releases.ubuntu.com/14.04/). So while we all benefit from the fine work by Michael who faithfully provides Ubuntu binaries for distribution via CRAN (based on the Debian builds provided by yours truly), we are stuck with Ubuntu 14.04. Which means that while Michael can provide us with current R 3.4.0 it will be built on ancient Ubuntu 14.04.
Why does this matter, you ask? Well, if you just try to turn the very C++14 support added to R 3.4.0 on in the binary running on Travis, you get this error:
** libs
Error in .shlib_internal(args) :
C++14 standard requested but CXX14 is not defined
And you get it whether or not you define CXX14
in the session.
So R (in version 3.4.0) may want to use C++14 (because a package we submitted requests it), but having been built on the dreaded Ubuntu 14.04, it just can't oblige. Even when we supply a newer compiler. Because R hangs on to its compile-time settings rather than current environment variables. And that means no C++14 as its compile-time compiler was too ancient. Trust me, I tried: adding not only g++-6
(from a suitable repo) but also adding C++14
as the value for CXX_STD
. Alas, no mas.
The trick to overcome this is twofold, and fairly straightforward. First off, we just rely on the fact that g++
version 6 defaults to C++14. So by supplying g++-6
, we are in the green. We have C++14 by default without requiring extra options. Sweet.
The remainder is to tell R to not try to enable C++14 even though we are using it. How? By removing CXX_STD=C++14
on the fly and just for Travis. And this can be done easily with a small script configure
which conditions on being on Travis by checking two environment variables:
#!/bin/bash
## Travis can let us run R 3.4.0 (from CRAN and the PPAs) but this R version
## does not know about C++14. Even though we can select CXX_STD = C++14, R
## will fail as the version we use there was built in too old an environment,
## namely Ubuntu "trusty" 14.04.
##
## So we install g++-6 from another repo and rely on the fact that is
## defaults to C++14. Sadly, we need R to not fail and hence, just on
## Travis, remove the C++14 instruction
if [[ "${CI}" == "true" ]]; then
if [[ "${TRAVIS}" == "true" ]]; then
echo "** Overriding src/Makevars and removing C++14 on Travis only"
sed -i 's|CXX_STD = CXX14||' src/Makevars
fi
fi
I have deployed this now for two sets of builds in two distinct repositories for two "under-development" packages not yet on CRAN, and it just works. In case you turn on C++14 via SystemRequirements:
in the file DESCRIPTION
, you need to modify it here.
So to sum up, there it is: C++14 with R 3.4.0 on Travis. Only takes a quick Travis-only modification.
A new package! Or at least new on CRAN as the very initial version 0.1.0 had been available via the ghrr drat for over a year. But now we have version 0.1.1 to announce as a CRAN package.
RcppMspPack provides R with MessagePack header files for use via C++ (or C, if you must) packages such as RcppRedis.
MessagePack itself is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it is faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.
MessagePack is used by Redis and many other projects, and has bindings to just about any language.
To use this package, simply add it to the LinkingTo:
field in the DESCRIPTION
field of your R package---and the R package infrastructure tools will then know how to set include flags correctly on all architectures supported by R.
More information may be on the RcppMsgPack page. Issues and bugreports should go to the GitHub issue tracker.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new version of the anytime package is now on CRAN. It marks the eleventh release since the inaugural version late last summer.
anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, ... format to either POSIXct or Date objects -- and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples.
This release brings a little more consistency to how numeric or integer arguments are handled. Previously, we were overly eager in accepting something such as 20150605 (i.e. today) as a (numerical or integer) input to both anytime()
and anydate()
. That is well-intentioned, but ultimately foolish. We relied on heuristic cutoffs to determine whether input was "meant to be" a date or time offset. There lies madness. We now differentiate whether we were called via anytime()
(in which case numerical data is second offset to the epoch, just as.POSICct()
) or anytime()
(in which case it is days offset to the (date) epoch, just like as.Date()
). The previous behaviour can be restored via a options, both function-local as well as global are supported. And of course, there is no change for all other (and more common) input formats, notably character
or factor
. A full list of changes follows.
Changes in anytime version 0.3.0 (2017-06-05)
Numeric input is now always an offset to epoch, with
anytime()
using seconds, andanydate()
using dates. (#65 fixing #63).Old behaviour can be re-enabled with an option also supporting a global setting
getOption("anytimeOldHeuristic")
RStudio versions 1.1.129 or later can run all functions without fear of crashing due to a change in their use of Boost.
Replaced
init.c
with registration code inside ofRcppExports.cpp
thanks to Rcpp 0.12.11.
Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page.
For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The new RcppArmadillo release 0.7.900.2.0 is now on CRAN, and the Debian package was just updated as well.
Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language--and is widely used by (currently) 350 other packages on CRAN---an increase of 32 since the last CRAN release of 0.7.800.2.0 in April!
With the 7.900.* series of Armadillo, Conrad has started to more fully utilize OpenMP (also see Wikipedia on OpenMP) for operations that can be parallelized. To use this in your package you need to update its src/Makevars{,.win}
file similarly to what the skeleton default now uses
PKG_CXXFLAGS = $(SHLIB_OPENMP_CXXFLAGS)
PKG_LIBS = $(SHLIB_OPENMP_CFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS)
and you may want to enable C++11 while you are at it---though this may pose issues with older-than-ancient RHEL installations which are still (way too) pervasive so we do not do it by default (yet).
Here, we once again rely on the build infrastructure automagically provided by R itself: if and when OpenMP is available, R will use it via $(SHLIB_OPENMP_CXXFLAGS)
etc; see the fine WRE manual for details. That said, some operating systems make this harder than other, and macOS usually takes the crown. See for example this blog post by James for surviving in that environment. I am a little short of details because on Linux these things just work, and have for well over a decade. The rcpp-devel mailing list will be the best place for questions.
Changes in this release relative to the previous CRAN release are as follows:
Changes in RcppArmadillo version 0.7.900.2.0 (2017-06-02)
Upgraded to Armadillo release 7.900.2 (Evil Banana Republic)
Expanded
clamp()
to handle cubesComputationally expensive element-wise functions (such as
exp()
,log()
,cos()
, etc) can now be automatically sped up via OpenMP; this requires a C++11/C++14 compiler with OpenMP 3.0+ support for GCC and clang compilersOne caveat: when using GCC, use of
-march=native
in conjunction with-fopenmp
may lead to speed regressions on recent processorsAdded gcc 7 to support compiler check (James Balamuta in #128 addressing #126).
A unit test helper function for
rmultinom
was corrected (#133).OpenMP support was added to the skeleton helper in
inline.R
Courtesy of CRANberries, there is a diffstat report. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The elevent update in the 0.12.* series of Rcpp landed on CRAN yesterday following the initial upload on the weekend, and the Debian package and Windows binaries should follow as usual. The 0.12.11 release follows the 0.12.0 release from late July, the 0.12.1 release in September, the 0.12.2 release in November, the 0.12.3 release in January, the 0.12.4 release in March, the 0.12.5 release in May, the 0.12.6 release in July, the 0.12.7 release in September, the 0.12.8 release in November, the 0.12.9 release in January, and the 0.12.10.release in March --- making it the fifteenth release at the steady and predictable bi-montly release frequency.
Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 1026 packages on CRAN depend on Rcpp for making analytical code go faster and further, along with another 91 in BioConductor.
This releases follows on the heels of R's 3.4.0 release and addresses on or two issues from the transition, along with a literal boatload of other fixes and enhancements. James "coatless" Balamuta was once restless in making the documentation better, Kirill Mueller addressed a number of more obscure compiler warnings (triggered under under -Wextra
and the like), Jim Hester improved excecption handling, and much more mostly by the Rcpp Core team. All changes are listed below in some detail.
One big change that JJ made is that Rcpp Attributes also generate the now-almost-required package registration. (For background, I blogged about this one, two, three times.) We tested this, and do not expect it to throw curveballs. If you have an existing src/init.c
, or if you do not have registration set in your NAMESPACE
. It should cover most cases. But one never knows, and one first post-release buglet related to how devtools tests things has already been fixed in this PR by JJ.
Changes in Rcpp version 0.12.11 (2017-05-20)
Changes in Rcpp API:
Rcpp::exceptions can now be constructed without a call stack (Jim Hester in #663 addressing #664).
Somewhat spurious compiler messages under very verbose settings are now suppressed (Kirill Mueller in #670, #671, #672, #687, #688, #691).
Refreshed the included
tinyformat
template library (James Balamuta in #674 addressing #673).Added
printf
-like syntax support for exception classes and variadic templating forRcpp::stop
andRcpp::warning
(James Balamuta in #676).Exception messages have been rewritten to provide additional information. (James Balamuta in #676 and #677 addressing #184).
One more instance of
Rf_mkString
is protected from garbage collection (Dirk in #686 addressing #685).Two exception specification that are no longer tolerated by
g++-7.1
or later were removed (Dirk in #690 addressing #689)Changes in Rcpp Documentation:
Changes in Rcpp Sugar:
Changes in Rcpp Attributes:
Changes in Rcpp support functions:
- The
Rcpp.package.skeleton()
function now creates a package registration file provided R 3.4.0 or later is used (Dirk in #692)
Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Very excited about the next few weeks which will cover a number of R conferences, workshops or classes with talks, mostly around Rcpp and one notable exception:
May 19: Rcpp: From Simple Examples to Machine learning, pre-conference workshop at our R/Finance 2017 conference here in Chicago
May 26: Extending R with C++: Motivation and Examples, invited keynote at R à Québec 2017 at Université Laval in Quebec City, Canada
June 28-29: Higher-Performance R Programming with C++ Extensions, two-day course at the Zuerich R Courses @ U Zuerich in Zuerich, Switzerland
July 3: Rcpp at 1000+ reverse depends: Some Lessons Learned (working title), at DSC 2017 preceding useR! 2017 in Brussels, Belgium
July 4: Extending R with C++: Motivation, Introduction and Examples, tutorial preceding useR! 2017 in Brussels, Belgium
July 5, 6, or 7: Hosting Data Packages via drat
: A Case Study with Hurricane Exposure Data, accepted presentation, joint with Brooke Anderson
If you are near one those events, interested and able to register (for the events requiring registration), I would love to chat before or after.
A new release 0.2.14 of RInside is now on CRAN and in Debian.
RInside provides a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by Rcpp.
It has been nearly two years since the last release, and a number of nice extensions, build robustifications and fixes had been submitted over this period---see below for more. The only larger and visible extension is both a new example and some corresponding internal changes to allow a readline prompt in an RInside application, should you desire it.
RInside is stressing the CRAN system a little in that it triggers a number of NOTE and WARNING messages. Some of these are par for the course as we get close to R internals not all of which are "officially" in the API. This lead to the submission sitting a little longer than usual in incoming queue. Going forward we may need to find a way to either sanction these access point, whitelist them or, as a last resort, take the package off CRAN. Time will tell.
Changes since the last release were:
Changes in RInside version 0.2.14 (2017-04-28)
Interactive mode can use readline REPL (Łukasz Łaniewski-Wołłk in #25, and Dirk in #26)
Windows macros checks now uses
_WIN32
(Kevin Ushey in #22)The wt example now links with
libboost_system
The
Makevars
file is now more robist (Mattias Ellert in #21)A problem with empty environment variable definitions on Windows was addressed (Jeroen Ooms in #17 addressing #16)
HAVE_UINTPTR_T
is defined only if not already definedTravis CI is now driven via
run.sh
from our forked r-travis
CRANberries also provides a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page, or to issues tickets at the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The US Census Bureau released a new build 1.1.39 of their X-13ARIMA-SEATS program, released as binary and source. So Christoph and went to work and updated our x13binary package on CRAN.
The x13binary package takes the pain out of installing X-13ARIMA-SEATS by making it a fully resolved CRAN dependency. For example, if you install the excellent seasonal package by Christoph, then X-13ARIMA-SEATS will get pulled in via the x13binary package and things just work: Depend on x13binary and on all relevant OSs supported by R, you should have an X-13ARIMA-SEATS binary installed which will be called seamlessly by the higher-level packages such as seasonal or gunsales.
So now the full power of the what is likely the world's most sophisticated deseasonalization and forecasting package is now at your fingertips and the R prompt, just like any other of the 10,500+ CRAN packages.
Not many packaging changes in this release besides updating the underlying builds, but we switched our versioning scheme to reflect that our releases are driven by the US Census Bureau releases. But thanks to an initial contribution by David Schaub we now support the 'armhf' architecture common on Chromebooks running Linux.
Courtesy of CRANberries, there is also a diffstat
report for this release.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new RcppEigen release 0.3.3.3.0 was put into CRAN (and Debian) a few days ago. It brings Eigen 3.3.* to R.
Once again, Yixuan Qiu did most of the heavy lifting over a multi-month period as some adjustments needed to be made in the package itself, along with coordination downstream.
The complete NEWS
file entry follows.
Updated to version 3.3.3 of Eigen
Fixed incorrect function names in the examples, thanks to Ching-Chuan Chen
The class MappedSparseMatrix<T>
has been deprecated since Eigen 3.3.0. The new structure Map<SparseMatrix<T> >
should be used instead
Exporters for the new type Map<SparseMatrix<T> >
were added
Travis CI is now driven via run.sh
from our forked r-travis
Courtesy of CRANberries, there is also a diffstat report for the most recent release.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the sixth post in the really random R riffs series, or R4 for short.
Posts #1 and #2 discussed how to get the now de rigeur package registration information computed. In essence, we pointed to something which R 3.4.0 would have, and provided tricks for accessing it while R 3.3.3 was still R-released.
But now R 3.4.0 is out, and life is good! Or at least this is easier. For example, a few days ago I committed this short helper script pnrrs.r
to littler:
#!/usr/bin/r
if (getRversion() < "3.4.0") stop("Not available for R (< 3.4.0). Please upgrade.", call.=FALSE)
tools::package_native_routine_registration_skeleton(".")
So with this example script pnrrs.r
soft-linked to /usr/local/bin
(or ~/bin
) as I commonly do with littler helpers, all it takes is
cd some/R/package/source
pnrrs.r
and the desired file usable as src/init.c
is on stdout
. Editing NAMESPACE
is quick too, and we're all done. See the other two posts for additional context. If you don't have littler, the above also works with Rscript
.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A bug fix release of RcppTOML arrived on CRAN today. Table arrays were (wrongly) not allowing for nesting; a simply recursion fix addresses this.
RcppTOML brings TOML to R. TOML is a file format that is most suitable for configurations, as it is meant to be edited by humans but read by computers. It emphasizes strong readability for humans while at the same time supporting strong typing as well as immediate and clear error reports. On small typos you get parse errors, rather than silently corrupted garbage. Much preferable to any and all of XML, JSON or YAML -- though sadly these may be too ubiquitous now. TOML is making good inroads with newer and more flexible projects such as the Hugo static blog compiler, or the Cargo system of Crates (aka "packages") for the Rust language.
Changes in version 0.1.3 (2017-04-25)
- Nested TableArray types are now correctly handled (#16)
Courtesy of CRANberries, there is a diffstat report for this release.
More information is on the RcppTOML page page. Issues and bugreports should go to the GitHub issue tracker.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Time for a new release of Rblpapi -- version 0.3.6 is now on CRAN. Rblpapi provides a direct interface between R and the Bloomberg Terminal via the C++ API provided by Bloomberg Labs (but note that a valid Bloomberg license and installation is required).
This is the seventh release since the package first appeared on CRAN last year. This release brings a very nice new function lookupSecurity()
contributed by Kevin Jin as well as a number of small fixes and enhancements. Details below:
Changes in Rblpapi version 0.3.6 (2017-04-20)
bdh
can now store indouble
preventing overflow (Whit and John in #205 closing #163)
bdp
documentation has another ovveride exampleA new function
lookupSecurity
can search for securities, optionally filtered by yellow key (Kevin Jin and Dirk in #216 and #217 closing #215)Added file
init.c
with calls toR_registerRoutines()
andR_useDynamicSymbols()
; also use.registration=TRUE
inuseDynLib
inNAMESPACE
(Dirk in #220)
getBars
andgetTicks
can now returndata.table
objects (Dirk in #221)
bds
has improved internal protect logic viaRcpp::Shield
(Dirk in #222)
Courtesy of CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the Rblpapi page. Questions, comments etc should go to the issue tickets system at the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
New package! And, as it happens, a effectively a subset or variant of one my oldest packages, RQuantLib.
Fairly recently, Peter Caspers started to put together a header-only subset of QuantLib. He called this Quantuccia, and, upon me asking, said that it stands for "little sister" of QuantLib. Very nice.
One design goal is to keep Quantuccia header-only. This makes distribution and deployment much easier. In the fifteen years that we have worked with QuantLib by providing the R bindings via RQuantLib, it has always been a concern to provide current QuantLib libraries on all required operating systems. Many people helped over the years but it is still an issue, and e.g. right now we have no Windows package as there is no library build it against.
Enter RcppQuantuccia. It only depends on R, Rcpp (for seamless R and C++ integrations) and BH bringing Boost headers. This will make it much easier to have Windows and macOS binaries.
So what can it do right now? We started with calendaring, and you can compute date pertaining to different (ISDA and other) business day conventions, and compute holiday schedules. Here is one example computing inter alia under the NYSE holiday schedule common for US equity and futures markets:
R> library(RcppQuantuccia)
R> fromD <- as.Date("2017-01-01")
R> toD <- as.Date("2017-12-31")
R> getHolidays(fromD, toD) # default calender ie TARGET
[1] "2017-04-14" "2017-04-17" "2017-05-01" "2017-12-25" "2017-12-26"
R> setCalendar("UnitedStates")
R> getHolidays(fromD, toD) # US aka US::Settlement
[1] "2017-01-02" "2017-01-16" "2017-02-20" "2017-05-29" "2017-07-04" "2017-09-04"
[7] "2017-10-09" "2017-11-10" "2017-11-23" "2017-12-25"
R> setCalendar("UnitedStates::NYSE")
R> getHolidays(fromD, toD) # US New York Stock Exchange
[1] "2017-01-02" "2017-01-16" "2017-02-20" "2017-04-14" "2017-05-29" "2017-07-04"
[7] "2017-09-04" "2017-11-23" "2017-12-25"
R>
The GitHub repo already has a few more calendars, and more are expected. Help is of course welcome for both this, and for porting over actual quantitative finance calculations.
More information is on the RcppQuantuccia page. Issues and bugreports should go to the GitHub issue tracker.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Moments ago Rcpp passed a big milestone as there are now 1000 packages on CRAN depending on it (as measured by Depends, Imports and LinkingTo, but excluding Suggests). The graph is on the left depicts the growth of Rcpp usage over time.
One easy way to compute such reverse dependency counts is the tools::dependsOnPkgs()
function that was just mentioned in yesterday's R^4 blog post. Another way is to use the reverse_dependencies_with_maintainers()
function from this helper scripts file on CRAN. Lastly, devtools has a function revdep()
but it has the wrong default parameters as it includes Suggests:
which you'd have to override to get the count I use here (it currently gets 1012 in this wider measure).
Rcpp cleared 300 packages in November 2014. It passed 400 packages in June 2015 (when I only tweeted about it), 500 packages in late October 2015, 600 packages last March, 700 packages last July, 800 packages last October and 900 packages early January. The chart extends to the very beginning via manually compiled data from CRANberries and checked with crandb. The next part uses manually saved entries. The core (and by far largest) part of the data set was generated semi-automatically via a short script appending updates to a small file-based backend. A list of packages using Rcpp is kept on this page.
Also displayed in the graph is the relative proportion of CRAN packages using Rcpp. The four per-cent hurdle was cleared just before useR! 2014 where I showed a similar graph (as two distinct graphs) in my invited talk. We passed five percent in December of 2014, six percent July of 2015, seven percent just before Christmas 2015, eight percent last summer, and nine percent mid-December 2016. Ten percent is next; we may get there during the summer.
1000 user packages is a really large number. This puts a whole lot of responsibility on us in the Rcpp team as we continue to keep Rcpp as performant and reliable as it has been.
And with that a very big Thank You! to all users and contributors of Rcpp for help, suggestions, bug reports, documentation or, of course, code.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the fifth post in the recklessly rambling R rants series, or R4 for short.
The third post showed an easy way to follow R development by monitoring (curated) changes on the NEWS file for the development version r-devel. As a concrete example, I mentioned that it has shown a nice new function (tools::CRAN_package_db()
) coming up in R 3.4.0. Today we will build on that.
Consider the following short snippet:
library(data.table)
getPkgInfo <- function() {
if (exists("tools::CRAN_package_db")) {
dat <- tools::CRAN_package_db()
} else {
tf <- tempfile()
download.file("https://cloud.r-project.org/src/contrib/PACKAGES.rds", tf, quiet=TRUE)
dat <- readRDS(tf) # r-devel can now readRDS off a URL too
}
dat <- as.data.frame(dat)
setDT(dat)
dat
}
It defines a simple function getPkgInfo()
as a wrapper around said new function from R 3.4.0, ie tools::CRAN_package_db()
, and a fallback alternative using a tempfile (in the automagically cleaned R temp directory) and an explicit download and read of the underlying RDS file. As an aside, just this week the r-devel NEWS told us that such readRDS()
operations can now read directly from URL connection. Very nice---as RDS is a fantastic file format when you are working in R.
Anyway, back to the RDS file! The snippet above returns a data.table object with as many rows as there are packages on CRAN, and basically all their (parsed !!) DESCRIPTION info and then some. A gold mine!
Consider this to see how many package have a dependency (in the sense of Depends, Imports or LinkingTo, but not Suggests because Suggests != Depends) on Rcpp:
R> dat <- getPkgInfo()
R> rcppRevDepInd <- as.integer(tools::dependsOnPkgs("Rcpp", recursive=FALSE, installed=dat))
R> length(rcppRevDepInd)
[1] 998
R>
So exciting---we will hit 1000 within days! But let's do some more analysis:
R> dat[ rcppRevDepInd, RcppRevDep := TRUE] # set to TRUE for given set
R> dat[ RcppRevDep==TRUE, 1:2]
Package Version
1: ABCoptim 0.14.0
2: AbsFilterGSEA 1.5
3: acc 1.3.3
4: accelerometry 2.2.5
5: acebayes 1.3.4
---
994: yakmoR 0.1.1
995: yCrypticRNAs 0.99.2
996: yuima 1.5.9
997: zic 0.9
998: ziphsmm 1.0.4
R>
Here we index the reverse dependency using the vector we had just computed, and then that new variable to subset the data.table
object. Given the aforementioned parsed information from all the DESCRIPTION files, we can learn more:
R> ## likely false entries
R> dat[ RcppRevDep==TRUE, ][NeedsCompilation!="yes", c(1:2,4)]
Package Version Depends
1: baitmet 1.0.0 Rcpp, erah (>= 1.0.5)
2: bea.R 1.0.1 R (>= 3.2.1), data.table
3: brms 1.6.0 R (>= 3.2.0), Rcpp (>= 0.12.0), ggplot2 (>= 2.0.0), methods
4: classifierplots 1.3.3 R (>= 3.1), ggplot2 (>= 2.2), data.table (>= 1.10),
5: ctsem 2.3.1 R (>= 3.2.0), OpenMx (>= 2.3.0), Rcpp
6: DeLorean 1.2.4 R (>= 3.0.2), Rcpp (>= 0.12.0)
7: erah 1.0.5 R (>= 2.10), Rcpp
8: GxM 1.1 NA
9: hmi 0.6.3 R (>= 3.0.0)
10: humarray 1.1 R (>= 3.2), NCmisc (>= 1.1.4), IRanges (>= 1.22.10),\nGenomicRanges (>= 1.16.4)
11: iNextPD 0.3.2 R (>= 3.1.2)
12: joinXL 1.0.1 R (>= 3.3.1)
13: mafs 0.0.2 NA
14: mlxR 3.1.0 R (>= 3.0.1), ggplot2
15: RmixmodCombi 1.0 R(>= 3.0.2), Rmixmod(>= 2.0.1), Rcpp(>= 0.8.0), methods,\ngraphics
16: rrr 1.0.0 R (>= 3.2.0)
17: UncerIn2 2.0 R (>= 3.0.0), sp, RandomFields, automap, fields, gstat
R>
There are a full seventeen packages which claim to depend on Rcpp while not having any compiled code of their own. That is likely false---but I keep them in my counts, however relunctantly. A CRAN-declared Depends: is a Depends:, after all.
Another nice thing to look at is the total number of package that declare that they need compilation:
R> ## number of packages with compiled code
R> dat[ , .(N=.N), by=NeedsCompilation]
NeedsCompilation N
1: no 7625
2: yes 2832
3: No 1
R>
Isn't that awesome? It is 2832 out of (currently) 10458, or about 27.1%. Just over one in four. Now the 998 for Rcpp look even better as they are about 35% of all such packages. In order words, a little over one third of all packages with compiled code (which may be legacy C, Fortran or C++) use Rcpp. Wow.
Before closing, one shoutout to Dirk Schumacher whose thankr which I made the center of the last post is now on CRAN. As a mighty fine and slim micropackage without external dependencies. Neat.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new RcppArmadillo version 0.7.800.2.0 is now on CRAN.
Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language--and is widely used by (currently) 318 other packages on CRAN -- an increase of 20 just since the last CRAN release of 0.7.600.1.0 in December!
Changes in this release relative to the previous CRAN release are as follows:
Changes in RcppArmadillo version 0.7.800.2.0 (2017-04-12)
Upgraded to Armadillo release 7.800.2 (Rogue State Redux)
- The Armadillo license changed to Apache License 2.0
The
DESCRIPTION
file now mentions the Apache License 2.0, as well as the former MPL2 license used for earlier releases.A new file
init.c
was added with calls toR_registerRoutines()
andR_useDynamicSymbols()
Symbol registration is enabled in
useDynLib
The
fastLm
example was updated
Courtesy of CRANberries, there is a diffstat report. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the fourth post in the repulsively random R ramblings series, or R4 for short.
My twitter feed was buzzing about a nice (and as yet unpublished, ie not-on-CRAN) package thankr by Dirk Schumacher which compiles a list of packages (ordered by maintainer count) for your current session (or installation or ...) with a view towards saying thank you to those whose packages we rely upon. Very nice indeed.
I had a quick look and run it twice ... and had a reaction of ewwww, really? as running it twice gave different results as on the second instance a boatload of tibblyverse packages appeared. Because apparently kids these day can only slice data that has been tidied or something.
So I had another quick look ... and put together an alternative version using just base R (as there was only one subfunction that needed reworking):
source(file="https://raw.githubusercontent.com/dirkschumacher/thankr/master/R/shoulders.R")
format_pkg_df <- function(df) { # non-tibblyverse variant
tb <- table(df[,2])
od <- order(tb, decreasing=TRUE)
ndf <- data.frame(maint=names(tb)[od], npkgs=as.integer(tb[od]))
colpkgs <- function(m, df) { paste(df[ df$maintainer == m, "pkg_name"], collapse=",") }
ndf[, "pkg"] <- sapply(ndf$maint, colpkgs, df)
ndf
}
A nice side benefit is that the function is now free of external dependencies (besides, of course, base R). Running this in the ESS session I had open gives:
R> shoulders() ## by Dirk Schumacher, with small modifications
maint npkgs pkg
1 R Core Team <R-core@r-project.org> 9 compiler,graphics,tools,utils,grDevices,stats,datasets,methods,base
2 Dirk Eddelbuettel <edd@debian.org> 4 RcppTOML,Rcpp,RApiDatetime,anytime
3 Matt Dowle <mattjdowle@gmail.com> 1 data.table
R>
and for good measure a screenshot is below:
I think we need a catchy moniker for R work using good old base R. SoberVerse? GrumbyOldFolksR? PlainOldR? Better suggestions welcome.
Edit on 2017-04-09: And by now Dirk Schumacher fixed that little bug in thankr which was at the start of this. His shoulders()
function is now free of side effects, and thankr is now a clean micropackage free of external depends from any verse, be it tiddly or grumpy. I look forward to seeing it on CRAN soon!
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the third post in the rarely relevant R recommendation series, or R4 for short.
Today will be brief, but of some importance. In order to know where R is going next, few places provide a better vantage point than the actual ongoing development.
A few years ago, I mentioned to Duncan Murdoch how straightforward the setup of my CRANberries feed (and site) was. After all, static blog compilers converting textual input to html, rss feed and whatnot have been around for fifteen years (though they keep getting reinvented). He took this to heart and built the (not too pretty) R-devel daily site (which also uses a fancy diff tool as it shows changes in NEWS
) as well as a more general description of all available sub-feeds. I follow this mostly through blog aggregations -- Google Reader in its day, now Feedly. A screenshot is below just to show that it doesn't have to be ugly just because it is on them intertubes:
This shows a particularly useful day when R-devel folded into the new branch for what will be the R 3.4.0 release come April 21. The list of upcoming changes is truly impressive and quite comprehensive -- and the package registration helper, focus of posts #1 and #2 here, is but one of these many changes.
One function I learned about that day is tools::CRAN_package_db()
, a helper to get a single (large) data.frame
with all package DESCRIPTION information. Very handy. Others may have noticed that CRAN repos now have a new top-level file PACKAGES.rds
and this function does indeed just fetch it--which you could do with a similar one-liner in R-release as well. Still very handy.
But do read about changes in R-devel and hence upcoming changes in R 3.4.0. Lots of good things coming our way.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A brown bag bug fix release 0.0.3 of RApiDatetime is now on CRAN.
RApiDatetime provides six entry points for C-level functions of the R API for Date
and Datetime
calculations. The functions asPOSIXlt
and asPOSIXct
convert between long and compact datetime representation, formatPOSIXlt
and Rstrptime
convert to and from character strings, and POSIXlt2D
and D2POSIXlt
convert between Date
and POSIXlt
datetime. These six functions are all fairly essential and useful, but not one of them was previously exported by R.
I left two undefined variables in calls in the exported header file; this only become an issue once I actually tried accessing the API from another package as I am now doing in anytime.
Changes in RApiDatetime version 0.0.3 (2017-04-02)
Correct two simple copy-and-paste errors in
RApiDatetime.h
Also enable registration in
useDynLib
, and explicitly export known and documented R access functions provided for testing
Courtesy of CRANberries, there is a comparison to the previous release. More information is on the rapidatetime page.
For questions or comments please use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the second post in rambling random R recommendation series, or R4 for short.
Two days ago I posted the initial (actual) post. It provided context for why we need package registration entries (tl;dr: because R CMD check
now tests for it, and because it The Right Thing to do, see documentation in the posts). I also showed how generating such a file src/init.c
was essentially free as all it took was single call to a new helper function added to R-devel by Brian Ripley and Kurt Hornik.
Now, to actually use R-devel you obviously need to have it accessible. There are a myriad of ways to achieve that: just compile it locally as I have done for years, use a Docker image as I showed in the post -- or be creative with eg Travis or win-builder both of which give you access to R-devel if you're clever about it.
But as no good deed goes unpunished, I was of course told off today for showing a Docker example as Docker was not "Easy". I think the formal answer to that is baloney. But we leave that aside, and promise to discuss setting up Docker at another time.
R is after all ... just R. So below please find a script you can save as, say, ~/bin/pnrrs.r
. And calling it---even with R-release---will generate the same code snippet as I showed via Docker. Call it a one-off backport of the new helper function -- with a half-life of a few weeks at best as we will have R 3.4.0 as default in just a few weeks. The script will then reduce to just the final line as the code will be present with R 3.4.0.
#!/usr/bin/r
library(tools)
.find_calls_in_package_code <- tools:::.find_calls_in_package_code
.read_description <- tools:::.read_description
## all what follows is from R-devel aka R 3.4.0 to be
package_ff_call_db <- function(dir) {
## A few packages such as CDM use base::.Call
ff_call_names <- c(".C", ".Call", ".Fortran", ".External",
"base::.C", "base::.Call",
"base::.Fortran", "base::.External")
predicate <- function(e) {
(length(e) > 1L) &&
!is.na(match(deparse(e[[1L]]), ff_call_names))
}
calls <- .find_calls_in_package_code(dir,
predicate = predicate,
recursive = TRUE)
calls <- unlist(Filter(length, calls))
if(!length(calls)) return(NULL)
attr(calls, "dir") <- dir
calls
}
native_routine_registration_db_from_ff_call_db <- function(calls, dir = NULL, character_only = TRUE) {
if(!length(calls)) return(NULL)
ff_call_names <- c(".C", ".Call", ".Fortran", ".External")
ff_call_args <- lapply(ff_call_names,
function(e) args(get(e, baseenv())))
names(ff_call_args) <- ff_call_names
ff_call_args_names <-
lapply(lapply(ff_call_args,
function(e) names(formals(e))), setdiff,
"...")
if(is.null(dir))
dir <- attr(calls, "dir")
package <- # drop name
as.vector(.read_description(file.path(dir, "DESCRIPTION"))["Package"])
symbols <- character()
nrdb <-
lapply(calls,
function(e) {
if (startsWith(deparse(e[[1L]]), "base::"))
e[[1L]] <- e[[1L]][3L]
## First figure out whether ff calls had '...'.
pos <- which(unlist(Map(identical,
lapply(e, as.character),
"...")))
## Then match the call with '...' dropped.
## Note that only .NAME could be given by name or
## positionally (the other ff interface named
## arguments come after '...').
if(length(pos)) e <- e[-pos]
## drop calls with only ...
if(length(e) < 2L) return(NULL)
cname <- as.character(e[[1L]])
## The help says
##
## '.NAME' is always matched to the first argument
## supplied (which should not be named).
##
## But some people do (Geneland ...).
nm <- names(e); nm[2L] <- ""; names(e) <- nm
e <- match.call(ff_call_args[[cname]], e)
## Only keep ff calls where .NAME is character
## or (optionally) a name.
s <- e[[".NAME"]]
if(is.name(s)) {
s <- deparse(s)[1L]
if(character_only) {
symbols <<- c(symbols, s)
return(NULL)
}
} else if(is.character(s)) {
s <- s[1L]
} else { ## expressions
symbols <<- c(symbols, deparse(s))
return(NULL)
}
## Drop the ones where PACKAGE gives a different
## package. Ignore those which are not char strings.
if(!is.null(p <- e[["PACKAGE"]]) &&
is.character(p) && !identical(p, package))
return(NULL)
n <- if(length(pos)) {
## Cannot determine the number of args: use
## -1 which might be ok for .External().
-1L
} else {
sum(is.na(match(names(e),
ff_call_args_names[[cname]]))) - 1L
}
## Could perhaps also record whether 's' was a symbol
## or a character string ...
cbind(cname, s, n)
})
nrdb <- do.call(rbind, nrdb)
nrdb <- as.data.frame(unique(nrdb), stringsAsFactors = FALSE)
if(NROW(nrdb) == 0L || length(nrdb) != 3L)
stop("no native symbols were extracted")
nrdb[, 3L] <- as.numeric(nrdb[, 3L])
nrdb <- nrdb[order(nrdb[, 1L], nrdb[, 2L], nrdb[, 3L]), ]
nms <- nrdb[, "s"]
dups <- unique(nms[duplicated(nms)])
## Now get the namespace info for the package.
info <- parseNamespaceFile(basename(dir), dirname(dir))
## Could have ff calls with symbols imported from other packages:
## try dropping these eventually.
imports <- info$imports
imports <- imports[lengths(imports) == 2L]
imports <- unlist(lapply(imports, `[[`, 2L))
info <- info$nativeRoutines[[package]]
## Adjust native routine names for explicit remapping or
## namespace .fixes.
if(length(symnames <- info$symbolNames)) {
ind <- match(nrdb[, 2L], names(symnames), nomatch = 0L)
nrdb[ind > 0L, 2L] <- symnames[ind]
} else if(!character_only &&
any((fixes <- info$registrationFixes) != "")) {
## There are packages which have not used the fixes, e.g. utf8latex
## fixes[1L] is a prefix, fixes[2L] is an undocumented suffix
nrdb[, 2L] <- sub(paste0("^", fixes[1L]), "", nrdb[, 2L])
if(nzchar(fixes[2L]))
nrdb[, 2L] <- sub(paste0(fixes[2L]), "$", "", nrdb[, 2L])
}
## See above.
if(any(ind <- !is.na(match(nrdb[, 2L], imports))))
nrdb <- nrdb[!ind, , drop = FALSE]
## Fortran entry points are mapped to l/case
dotF <- nrdb$cname == ".Fortran"
nrdb[dotF, "s"] <- tolower(nrdb[dotF, "s"])
attr(nrdb, "package") <- package
attr(nrdb, "duplicates") <- dups
attr(nrdb, "symbols") <- unique(symbols)
nrdb
}
format_native_routine_registration_db_for_skeleton <- function(nrdb, align = TRUE, include_declarations = FALSE) {
if(!length(nrdb))
return(character())
fmt1 <- function(x, n) {
c(if(align) {
paste(format(sprintf(" {\"%s\",", x[, 1L])),
format(sprintf(if(n == "Fortran")
"(DL_FUNC) &F77_NAME(%s),"
else
"(DL_FUNC) &%s,",
x[, 1L])),
format(sprintf("%d},", x[, 2L]),
justify = "right"))
} else {
sprintf(if(n == "Fortran")
" {\"%s\", (DL_FUNC) &F77_NAME(%s), %d},"
else
" {\"%s\", (DL_FUNC) &%s, %d},",
x[, 1L],
x[, 1L],
x[, 2L])
},
" {NULL, NULL, 0}")
}
package <- attr(nrdb, "package")
dups <- attr(nrdb, "duplicates")
symbols <- attr(nrdb, "symbols")
nrdb <- split(nrdb[, -1L, drop = FALSE],
factor(nrdb[, 1L],
levels =
c(".C", ".Call", ".Fortran", ".External")))
has <- vapply(nrdb, NROW, 0L) > 0L
nms <- names(nrdb)
entries <- substring(nms, 2L)
blocks <- Map(function(x, n) {
c(sprintf("static const R_%sMethodDef %sEntries[] = {",
n, n),
fmt1(x, n),
"};",
"")
},
nrdb[has],
entries[has])
decls <- c(
"/* FIXME: ",
" Add declarations for the native routines registered below.",
"*/")
if(include_declarations) {
decls <- c(
"/* FIXME: ",
" Check these declarations against the C/Fortran source code.",
"*/",
if(NROW(y <- nrdb$.C)) {
args <- sapply(y$n, function(n) if(n >= 0)
paste(rep("void *", n), collapse=", ")
else "/* FIXME */")
c("", "/* .C calls */",
paste0("extern void ", y$s, "(", args, ");"))
},
if(NROW(y <- nrdb$.Call)) {
args <- sapply(y$n, function(n) if(n >= 0)
paste(rep("SEXP", n), collapse=", ")
else "/* FIXME */")
c("", "/* .Call calls */",
paste0("extern SEXP ", y$s, "(", args, ");"))
},
if(NROW(y <- nrdb$.Fortran)) {
args <- sapply(y$n, function(n) if(n >= 0)
paste(rep("void *", n), collapse=", ")
else "/* FIXME */")
c("", "/* .Fortran calls */",
paste0("extern void F77_NAME(", y$s, ")(", args, ");"))
},
if(NROW(y <- nrdb$.External))
c("", "/* .External calls */",
paste0("extern SEXP ", y$s, "(SEXP);"))
)
}
headers <- if(NROW(nrdb$.Call) || NROW(nrdb$.External))
c("#include <R.h>", "#include <Rinternals.h>")
else if(NROW(nrdb$.Fortran)) "#include <R_ext/RS.h>"
else character()
c(headers,
"#include <stdlib.h> // for NULL",
"#include <R_ext/Rdynload.h>",
"",
if(length(symbols)) {
c("/*",
" The following symbols/expresssions for .NAME have been omitted",
"", strwrap(symbols, indent = 4, exdent = 4), "",
" Most likely possible values need to be added below.",
"*/", "")
},
if(length(dups)) {
c("/*",
" The following name(s) appear with different usages",
" e.g., with different numbers of arguments:",
"", strwrap(dups, indent = 4, exdent = 4), "",
" This needs to be resolved in the tables and any declarations.",
"*/", "")
},
decls,
"",
unlist(blocks, use.names = FALSE),
## We cannot use names with '.' in: WRE mentions replacing with "_"
sprintf("void R_init_%s(DllInfo *dll)",
gsub(".", "_", package, fixed = TRUE)),
"{",
sprintf(" R_registerRoutines(dll, %s);",
paste0(ifelse(has,
paste0(entries, "Entries"),
"NULL"),
collapse = ", ")),
" R_useDynamicSymbols(dll, FALSE);",
"}")
}
package_native_routine_registration_db <- function(dir, character_only = TRUE) {
calls <- package_ff_call_db(dir)
native_routine_registration_db_from_ff_call_db(calls, dir, character_only)
}
package_native_routine_registration_db <- function(dir, character_only = TRUE) {
calls <- package_ff_call_db(dir)
native_routine_registration_db_from_ff_call_db(calls, dir, character_only)
}
package_native_routine_registration_skeleton <- function(dir, con = stdout(), align = TRUE,
character_only = TRUE, include_declarations = TRUE) {
nrdb <- package_native_routine_registration_db(dir, character_only)
writeLines(format_native_routine_registration_db_for_skeleton(nrdb,
align, include_declarations),
con)
}
package_native_routine_registration_skeleton(".") ## when R 3.4.0 is out you only need this line
Here I use /usr/bin/r
as I happen to like littler a lot, but you can use Rscript
the same way.
Easy enough now?
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Welcome to the first actual post in the R4 series, following the short announcement earlier this week.
Last month, Brian Ripley announced on r-devel that registration of routines would now be tested for by R CMD check
in r-devel (which by next month will become R 3.4.0). A NOTE will be issued now, this will presumably turn into a WARNING at some point. Writing R Extensions has an updated introduction) of the topic.
Package registration has long been available, and applies to all native (i.e. "compiled") function via the .C()
, .Call()
, .Fortran()
or .External()
interfaces. If you use any of those -- and .Call()
may be the only truly relevant one here -- then is of interest to you.
Brian Ripley and Kurt Hornik also added a new helper function: tools::package_native_routine_registration_skeleton()
. It parses the R code of your package and collects all native function entrypoints in order to autogenerate the registration. It is available in R-devel now, will be in R 3.4.0 and makes adding such registration truly trivial.
But as of today, it requires that you have R-devel. Once R 3.4.0 is out, you can call the helper directly.
As for R-devel, there have always been at least two ways to use it: build it locally (which we may cover in another R4 installment), or using Docker. Here will focus on the latter by relying on the Rocker project by Carl and myself.
We assume you can run Docker on your system. How to add it on Windows, macOS or Linux is beyond our scope here today, but also covered extensively elsewhere. So we assume you can execute docker
and e.g. bring in the 'daily r-devel' image drd from our Rocker project via
~$ docker pull rocker/drd
With that, we can use R-devel to create the registration file very easily in a single call (which is a long command-line we have broken up with one line-break for the display below).
The following is real-life example when I needed to add registration to the RcppTOML package for this week's update:
~/git/rcpptoml(master)$ docker run --rm -ti -v $(pwd):/mnt rocker/drd \ ## line break
RD --slave -e 'tools::package_native_routine_registration_skeleton("/mnt")'
#include <R.h>
#include <Rinternals.h>
#include <stdlib.h> // for NULL
#include <R_ext/Rdynload.h>
/* FIXME:
Check these declarations against the C/Fortran source code.
*/
/* .Call calls */
extern SEXP RcppTOML_tomlparseImpl(SEXP, SEXP, SEXP);
static const R_CallMethodDef CallEntries[] = {
{"RcppTOML_tomlparseImpl", (DL_FUNC) &RcppTOML_tomlparseImpl, 3},
{NULL, NULL, 0}
};
void R_init_RcppTOML(DllInfo *dll)
{
R_registerRoutines(dll, NULL, CallEntries, NULL, NULL);
R_useDynamicSymbols(dll, FALSE);
}
edd@max:~/git/rcpptoml(master)$
We can understand the docker
command invocation above through its components:
docker run
is the basic call to a container--rm -ti
does subsequent cleanup (--rm
) and gives a terminal (-t
) that is interactive (-i
)-v $(pwd):/mnt
uses the -v a:b
options to make local directory a
available as b
in the container; here $(pwd)
calls print working directory to get the local directory which is now mapped to /mnt
in the containerrocker/drd
invokes the 'drd' container of the Rocker projectRD
is a shorthand to the R-devel binary inside the container, and the main reason we use this container-e 'tools::package_native_routine_registration_skeleton("/mnt")
calls the helper function of R (currently in R-devel only, so we use RD) to compute a possible init.c
file based on the current directory -- which is /mnt
inside the containerThat it. We get a call to the R function executed inside the Docker container, examining the package in the working directory and creating a registration file for it which is display to the console.
src/init.c
We simply copy the output to a file src/init.c; I often fold one opening brace one line up.
We also change one line in NAMESPACE from (for this package) useDynLib("RcppTOML")
to useDynLib("RcppTOML", .registration=TRUE)
. Adjust accordingly for other package names.
And with that we a have a package which no longer provokes the NOTE as seen by the checks page. Calls to native routines are now safer (less of a chance for name clashing), get called quicker as we skip the symbol search (see the WRE discussion) and best of all this applies to all native routines whether written by hand or written via a generator such as Rcpp Attributes.
So give this a try to get your package up-to-date.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new minor version of the nanotime package for working with nanosecond timestamps arrived yesterday on CRAN.
nanotime uses the RcppCCTZ package for (efficient) high(er) resolution time parsing and formatting up to nanosecond resolution, and the bit64 package for the actual integer64
arithmetic.
This release just arranges things neatly before Leonardo Silvestri and I may shake things up with a possible shift to doing it all in S4 as we may need the added rigour for nanotime
object operations for use in his ztsdb project.
Changes in version 0.1.2 (2017-03-27)
- The
as.integer64
function is now exported as well.
We also have a diff to the previous version thanks to CRANberries. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
So I had been toying with the idea of getting back to the blog and more regularly writing / posting little tips and tricks. I even started taking some notes but because perfect is always the enemy of the good it never quite materialized.
But the relatively broad discussion spawned by last week's short rant on Suggests != Depends made a few things clear. There appears to be an audience. It doesn't have to be long. And it doesn't have to be too polished.
So with that, let's get the blogging back from micro-blogging.
This note forms post zero of what will a new segment I call R4 which is shorthand for relatively random R rambling.
Stay tuned.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new release of RcppTOML is now on CRAN. This release fixes a few parsing issues for less frequently-used inputs: vectors of boolean or date(time) types, as well as table array input.
RcppTOML brings TOML to R. TOML is a file format that is most suitable for configurations, as it is meant to be edited by humans but read by computers. It emphasizes strong readability for humans while at the same time supporting strong typing as well as immediate and clear error reports. On small typos you get parse errors, rather than silently corrupted garbage. Much preferable to any and all of XML, JSON or YAML -- though sadly these may be too ubiquitous now. TOML is making good inroads with newer and more flexible projects such as the Hugo static blog compiler, or the Cargo system of Crates (aka "packages") for the Rust language.
Changes in version 0.1.2 (2017-03-26)
Dates and Datetimes in arrays in the input now preserve their types instead of converting to numeric vectors (#13)
Boolean vectors are also correctly handled (#14)
TableArray types are now stored as lists in a single named list (#15)
The README.md file was expanded with an example and screenshot.
Added file
init.c
with calls toR_registerRoutines()
and R_useDynamicSymbols()
; also use.registration=TRUE
inuseDynLib
inNAMESPACE
Two example files were updated.
Courtesy of CRANberries, there is a diffstat report for this release.
More information is on the RcppTOML page page. Issues and bugreports should go to the GitHub issue tracker.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Two days after the initial 0.0.1 release, a new version of RApiDatetime has just arrived on CRAN.
RApiDatetime provides six entry points for C-level functions of the R API for Date
and Datetime
calculations. The functions asPOSIXlt
and asPOSIXct
convert between long and compact datetime representation, formatPOSIXlt
and Rstrptime
convert to and from character strings, and POSIXlt2D
and D2POSIXlt
convert between Date
and POSIXlt
datetime. These six functions are all fairly essential and useful, but not one of them was previously exported by R.
Josh Ulrich took one hard look at the package -- and added the one line we needed to enable the Windows support that was missing in the initial release. We now build on all platforms supported by R and CRAN. Otherwise, I just added a NEWS file and called it a bugfix release.
Changes in RApiDatetime version 0.0.2 (2017-03-25)
- Windows support has added (Josh Ulrich in #1)
Changes in RApiDatetime version 0.0.1 (2017-03-23)
- Initial release with six accessible functions
Courtesy of CRANberries, there is a comparison to the previous release. More information is on the rapidatetime page.
For questions or comments please use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Very happy to announce a new package of mine is now up on the CRAN repository network: RApiDatetime.
It provides six entry points for C-level functions of the R API for Date
and Datetime
calculations: asPOSIXlt
and asPOSIXct
convert between long and compact datetime representation, formatPOSIXlt
and Rstrptime
convert to and from character strings, and POSIXlt2D
and D2POSIXlt
convert between Date
and POSIXlt
datetime. These six functions are all fairly essential and useful, but not one of them was previously exported by R. Hence the need to put them together in the this package to complete the accessible API somewhat.
These should be helpful for fellow package authors as many of us have either our own partial copies of some of this code, or rather farm back out into R to get this done.
As a simple (yet real!) illustration, here is an actual Rcpp function which we could now cover at the C level rather than having to go back up to R (via Rcpp::Function()
):
inline Datetime::Datetime(const std::string &s, const std::string &fmt) {
Rcpp::Function strptime("strptime"); // we cheat and call strptime() from R
Rcpp::Function asPOSIXct("as.POSIXct"); // and we need to convert to POSIXct
m_dt = Rcpp::as<double>(asPOSIXct(strptime(s, fmt)));
update_tm();
}
I had taken a first brief stab at this about two years ago, but never finished. With the recent emphasis on C-level function registration, coupled with a possible use case from anytime I more or less put this together last weekend.
It currently builds and tests fine on POSIX-alike operating systems. If someone with some skill and patience in working on Windows would like to help complete the Windows side of things then I would certainly welcome help and pull requests.
For questions or comments please use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A number of packages on CRAN use Suggests: casually.
They list other packages as "not required" in Suggests: -- as opposed to absolutely required via Imports: or the older Depends: -- yet do not test for their use in either examples or, more commonly, unit tests.
So e.g. the unit tests are bound to fail because, well, Suggests != Depends.
This has been accomodated for many years by all parties involved by treating Suggests as a Depends and installing unconditionally. As I understand it, CRAN appears to flip a switch to automatically install all Suggests from major repositories glossing over what I consider to be a packaging shortcoming. (As an aside, treatment of Additonal_repositories: is indeed optional; Brooke Anderson and I have a fine paper under review on this)
I spend a fair amount of time with reverse dependency ("revdep") checks of packages I maintain, and I will no longer accomodate these packages.
These revdep checks take long enough as it is, so I will now blacklist these packages that are guaranteed to fail when their "optional" dependencies are not present.
Writing R Extensions says in Section 1.1.3
All packages that are needed10 to successfully run R CMD check on the package must be listed in one of ‘Depends’ or ‘Suggests’ or ‘Imports’. Packages used to run examples or tests conditionally (e.g. via
if(require(pkgname)))
should be listed in ‘Suggests’ or ‘Enhances’. (This allows checkers to ensure that all the packages needed for a complete check are installed.)In particular, packages providing “only” data for examples or vignettes should be listed in ‘Suggests’ rather than ‘Depends’ in order to make lean installations possible.
[...]
It used to be common practice to use require calls for packages listed in ‘Suggests’ in functions which used their functionality, but nowadays it is better to access such functionality via :: calls.
and continues in Section 1.1.3.1
Note that someone wanting to run the examples/tests/vignettes may not have a suggested package available (and it may not even be possible to install it for that platform). The recommendation used to be to make their use conditional via
if(require("pkgname")))
: this is fine if that conditioning is done in examples/tests/vignettes.
I will now exercise my option to use 'lean installations' as discussed here. If you want your package included in tests I run, please make sure it tests successfully when only its required packages are present.
A bugfix release of the anytime package arrived at CRAN earlier today. This is tenth release since the inaugural version late last summer, and the second (bugfix / feature) release this year.
anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, ... format to either POSIXct or Date objects -- and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples.
This releases addresses an annoying bug related to British TZ settings and the particular impact of a change in 1971, and generalizes input formats to accept integer or numeric format in two specific ranges. Details follow below:
Changes in anytime version 0.2.2 (2017-03-21)
Address corner case of integer-typed (large) values corresponding to
POSIXct
time (PR #57 closing ##56)Add special case for ‘Europe/London’ and 31 Oct 1971 BST change to avoid a one-hour offset error (#58 fixing #36 and #51)
Address another corner case of numeric values corresponding to
Date
types which are now returned asDate
Added file
init.c
with calls toR_registerRoutines()
and R_useDynamicSymbols()
; already used.registration=TRUE
inuseDynLib
inNAMESPACE
Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page.
For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The tenth update in the 0.12.* series of Rcpp just made it to the main CRAN repository providing GNU R with by now over 10,000 packages. Windows binaries for Rcpp, as well as updated Debian packages will follow in due course. This 0.12.10 release follows the 0.12.0 release from late July, the 0.12.1 release in September, the 0.12.2 release in November, the 0.12.3 release in January, the 0.12.4 release in March, the 0.12.5 release in May, the 0.12.6 release in July, the 0.12.7 release in September, the 0.12.8 release in November, and the 0.12.9 release in January --- making it the fourteenth release at the steady and predictable bi-montly release frequency.
Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 975 packages on CRAN depend on Rcpp for making analytical code go faster and further. That is up by sixtynine packages over the two months since the last release -- or just over a package a day!
The changes in this release are almost exclusively minor bugfixes and enhancements to documentation and features: James "coatless" Balamuta rounded out the API, Iñaki Ucar fixed a bug concerning one-character output, Jeroen Ooms allowed for finalizers on XPtr
objects, Nathan Russell corrected handling of lower (upper) triangular matrices, Dan Dillon and I dealt with Intel compiler quirks for his algorithm.h
header, and I added a C++17 plugin along with some (overdue!) documentation regarding the various C++ standards that are supported by Rcpp (which is in essence whatever your compiler supports, i.e., C++98, C++11, C++14 all the way to C++17 but always keep in mind what CRAN and different users may deploy).
Changes in Rcpp version 0.12.10 (2017-03-17)
Changes in Rcpp API:
Added new size attribute aliases for number of rows and columns in DataFrame (James Balamuta in #638 addressing #630).
Fixed single-character handling in
Rstreambuf
(Iñaki Ucar in #649 addressing #647).XPtr gains a parameter
finalizeOnExit
to enable running the finalizer when R quits (Jeroen Ooms in #656 addressing #655).Changes in Rcpp Sugar:
Changes in Rcpp Attributes
- The C++17 standard is supported with a new plugin (used eg for
g++-6.2
).Changes in Rcpp Documentation:
- An overdue explanation of how C++11, C++14, and C++17 can be used was added to the Rcpp FAQ.
Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new maintenance release 0.3.2.9.1 of RcppEigen, still based on Eigen 3.2.9 is now on CRAN and is now going into Debian soon.
This update ensures that RcppEigen and the Matrix package agree on their #define
statements for the CholMod / SuiteSparse library. Thanks to Martin Maechler for the pull request. I also added a file src/init.c
as now suggested (soon: requested) by the R CMD check
package validation.
The complete NEWS
file entry follows.
Changes in RcppEigen version 0.3.2.9.1 (2017-03-14)
Synchronize CholMod header file with Matrix package to ensure binary compatibility on all platforms (Martin Maechler in #42)
Added file
init.c
with calls toR_registerRoutines()
and R_useDynamicSymbols()
; also use.registration=TRUE
inuseDynLib
inNAMESPACE
Courtesy of CRANberries, there is also a diffstat report for the most recent release.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
RProtoBuf provides R bindings for the Google Protocol Buffers ("Protobuf") data encoding and serialization library used and released by Google, and deployed as a language and operating-system agnostic protocol by numerous projects.
The RProtoBuf 0.4.9 release is the fourth and final update this weekend following the request by CRAN to not use package=
in .Call()
when PACKAGE=
is really called for.
Some of the code in RProtoBuf 0.4.9 had this bug; some other entry points had neither (!!). With the ongoing drive to establish proper registration of entry points, a few more issues were coming up, all of which are now addressed. And we had some other unreleased minor cleanup, so this made for a somewhat longer (compared to the other updates this weekend) NEWS list:
Changes in RProtoBuf version 0.4.9 (2017-03-06)
A new file
init.c
was added with calls toR_registerRoutines()
andR_useDynamicSymbols()
Symbol registration is enabled in
useDynLib
Several missing
PACKAGE=
arguments were added to the corresponding.Call
invocationsTwo (internal) C++ functions were renamed with suffix
_cpp
to disambiguate them from R functions with the same nameAll of above were part of #26
Some editing corrections were made to the introductory vignette (David Kretch in #25)
The 'configure.ac' file was updated, and renamed from the older converntion 'configure.in', along with 'src/Makevars'. (PR #24 fixing #23)
CRANberries also provides a diff to the previous release. The RProtoBuf page has an older package vignette, a 'quick' overview vignette, a unit test summary vignette, and the pre-print for the JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The RVowpalWabbit package update is the third of four upgrades requested by CRAN, following RcppSMC 0.1.5 and RcppGSL 0.3.2.
This package being somewhat raw, the change was simple and just meant converting the single entry point to using Rcpp Attributes -- which addressed the original issue in passing.
No new code or features were added.
We should mention that is parallel work ongoing in a higher-level package interfacing the vw
binary -- rvw -- as well as plan to redo this package via the external libraries. In that sounds interesting to you, please get in touch.
More information is on the RVowpalWabbit page. Issues and bugreports should go to the GitHub issue tracker.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The RcppGSL package provides an interface from R to the GNU GSL using the Rcpp package.
RcppGSL release 0.3.2 is one of several maintenance releases this weekend to fix an issue flagged by CRAN: calls to .Call()
sometimes used package=
where PACKAGE=
was meant. This came up now while the registration mechanism is being reworked.
So RcppGSL was updated too, and we took the opportunity to bring several packaging aspects up to the newest standards, including support for the soon-to-be required registration of routines.
No new code or features were added. The NEWS file entries follow below:
Changes in version 0.3.2 (2017-03-04)
In the
fastLm
function,.Call
now uses the correctPACKAGE=
argumentAdded file
init.c
with calls toR_registerRoutines()
and R_useDynamicSymbols()
; also use.registration=TRUE
inuseDynLib
inNAMESPACE
The skeleton configuration for created packages was updated.
Courtesy of CRANberries, a summary of changes to the most recent release is available.
More information is on the RcppGSL page. Questions, comments etc should go to the issue tickets at the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
RcppSMC provides Rcpp-based bindings to R for them Sequential Monte Carlo Template Classes (SMCTC) by Adam Johansen described in his JSS article.
RcppSMC release 0.1.5 is one of several maintenance releases this weekend to fix an issue flagged by CRAN: calls to .Call()
sometimes used package=
where PACKAGE=
was meant. This came up now while the registration mechanism is being reworked.
Hence RcppSMC was updated, and we took the opportunity to bring several packaging aspects up to the newest standards, including support for the soon-to-be required registration of routines.
No new code or features were added. The NEWS file entries follow below:
Changes in RcppSMC version 0.1.5 (2017-03-03)
Correct
.Call
to usePACKAGE=
argument
DESCRIPTION
,NAMESPACE
,README.md
changes to comply with currentR CMD check
levelsAdded file
init.c
with calls toR_registerRoutines()
and R_useDynamicSymbols()
Updated
.travis.yml
file for continuous integration
Courtesy of CRANberries, there is a diffstat report for this release.
More information is on the RcppSMC page. Issues and bugreports should go to the GitHub issue tracker.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new release 0.3.1 of the RPushbullet package, following the recent 0.3.0 release is now on CRAN. RPushbullet is interfacing the neat Pushbullet service for inter-device messaging, communication, and more. It lets you easily send alerts like the one to the to your browser, phone, tablet, ... -- or all at once.
This release owes once again a lot to Seth Wenchel who helped to update and extend a number of features. We fixed one more small bug stemming from the RJSONIO to jsonlite transition, and added a few more helpers. We also enabled Travis testing and with it covr-based coverage analysis using pretty much the same setup I described in this recent blog post.
Changes in version 0.3.1 (2017-02-17)
The target device designation was corrected (#39).
Three new (unexported) helper functions test the validity of the api key, device and channel (Seth in #41).
The
summary
method for thepbDevices
class was corrected (Seth in #43).New helper functions
pbValidateConf
,pbGetUser
,pbGetChannelInfo
were added (Seth in #44 closing #40).New classes
pbUser
andpbChannelInfo
were added (Seth in #44).Travis CI tests (and covr coverage analysis) are now enabled via an encrypted config file (#45).
Courtesy of CRANberries, there is also a diffstat report for this release.
More details about the package are at the RPushbullet webpage and the RPushbullet GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The third release of littler as a CRAN package is now available, following in the now more than ten-year history as a package started by Jeff in the summer of 2006, and joined by me a few weeks later.
littler is the first command-line interface for R and predates Rscript
. It is still faster, and in my very biased eyes better as it allows for piping as well shebang scripting via #!
, uses command-line arguments more consistently and still starts faster. It prefers to live on Linux and Unix, has its difficulties on OS X due to yet-another-braindeadedness there (who ever thought case-insensitive filesystems where a good idea?) and simply does not exist on Windows (yet -- the build system could be extended -- see RInside for an existence proof, and volunteers welcome!).
This release brings several new examples script to run package checks, use the extraordinary R Hub, download RStudio daily builds, and more -- see below for details. No internals were changed.
The NEWS
file entry is below.
Changes in littler version 0.3.2 (2017-02-14)
Changes in examples
New scripts
getRStudioServer.r
andgetRStudioDesktop.r
to download daily packages, currently defaults to Ubuntu amd64New script
c4c.r
callingrhub::check_for_cran()
New script
rd2md.r
to convert Rd to markdown.New script
build.r
to create a source tarball.The
installGitHub.r
script now use package remotes (PR #44, #46)
Courtesy of CRANberries, there is a comparison to the previous release. Full details for the littler release are provided as usual at the ChangeLog page. The code is available via the GitHub repo, from tarballs off my littler page and the local directory here -- and now of course all from its CRAN page and via install.packages("littler")
. Binary packages are available directly in Debian as well as soon via Ubuntu binaries at CRAN thanks to the tireless Michael Rutter.
Comments and suggestions are welcome at the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Following up on the somewhat important RcppTOML 0.1.0 releaseas which brought RcppTOML to Windows, we have a first minor update 0.1.1. Two things changed: once again updated upstream code from Chase Geigle's cpptoml which now supports Date types too, and we added the ability to parse TOML from strings as opposed to only from files.
TOML is a file format that is most suitable for configurations, as it is meant to be edited by humans but read by computers. It emphasizes strong readability for humans while at the same time supporting strong typing as well as immediate and clear error reports. On small typos you get parse errors, rather than silently corrupted garbage. Much preferable to any and all of XML, JSON or YAML -- though sadly these may be too ubiquitous now.
TOML is making good inroads with newer and more flexible projects such as the Hugo static blog compiler, or the Cargo system of Crates (aka "packages") for the Rust language.
Changes in version 0.1.1 (2017-xx-yy)
Synchronized multiple times with ccptoml upstream adding support for local datetime and local date and more (PR #9, #10, PR #11)
Dates are now first class types, some support for local versus UTC times was added (though it may be adviseable to stick with UTC)
Parsing from (R) character variables is now supported as well
Output from
print.toml
no longer prints extra newlines
Courtesy of CRANberries, there is a diffstat report for this release.
More information and examples are on the RcppTOML page. Issues and bugreports should go to the GitHub issue tracker.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
More and more packages, be it for R or another language, are now interfacing different application programming interfaces (API) which are exposed to the web. And many of these may require an API key, or token, or account and password.
Which traditionally poses a problem in automated tests such as those running on the popular Travis CI service which integrates so well with GitHub. A case in point is the RPushbullet package where Seth Wenchel and I have been making a few recent changes and additions.
And yesterday morning, I finally looked more closely into providing Travis CI with the required API key so that we could in fact run continuous integration with unit tests following each commit. And it turns that it is both easy and quick to do, and yet another great showcase for ad-hoc Docker use.
The rest of this post will give a quick minimal run-down, this time using the gtrendsR package by Philippe Massicotte and myself. Start by glancing at the 'encrypting files' HOWTO from Travis itself.
We assume you have Docker installed, and a suitable base package. We will need Ruby, so any base Linux image will do. In what follows, I use Ubuntu 14.04 but many other Debian, Ubunti, Fedora, ... flavours could be used provided you know how to pick the relevant packages. What is shown here should work on any recent Debian or Ubuntu flavour 'as is'.
We start by firing off the Docker engine in the repo directory for which we want to create an encrypted file. The -v $(pwd):/mnt
switch mounts the current directory as /mnt
in the Docker instance:
edd@max:~/git/gtrendsr(master)$ docker run --rm -ti -v $(pwd):/mnt ubuntu:trusty
root@38b478356439:/# apt-get update ## this takes a minute or two
Ign http://archive.ubuntu.com trusty InRelease
Get:1 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Get:2 http://archive.ubuntu.com trusty-security InRelease [65.9 kB]
# ... a dozen+ lines omitted ...
Get:21 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]
Get:22 http://archive.ubuntu.com trusty/universe amd64 Packages [7589 kB]
Fetched 22.4 MB in 6min 40s (55.8 kB/s)
Reading package lists... Done
root@38b478356439:/#
We then install what is needed to actually install the travis
(Ruby) gem, as well as git
which is used by it:
root@38b478356439:/# apt-get install -y ruby ruby-dev gem build-essential git
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
# ... lot of output ommitted ...
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
root@38b478356439:/#
This too may take a few minutes, depending on the networking bandwidth and other factors, and should in general succeed without the need for any intervention. Once it has concluded, we can use the now-complete infrastructure to install the travis
command-line client:
root@38b478356439:/# gem install travis
Fetching: multipart-post-2.0.0.gem (100%)
Fetching: faraday-0.11.0.gem (100%)
Fetching: faraday_middleware-0.11.0.1.gem (100%)
Fetching: highline-1.7.8.gem (100%)
Fetching: backports-3.6.8.gem (100%)
Fetching: multi_json-1.12.1.gem (100%
# ... many lines omitted ...
Installing RDoc documentation for websocket-1.2.4...
Installing RDoc documentation for json-2.0.3...
Installing RDoc documentation for pusher-client-0.6.2...
Installing RDoc documentation for travis-1.8.6...
root@38b478356439:/#
This in turn will take a moment.
Once done, we can use the travis
client to login into GitHub. In my base this requires a password and a two-factor authentication code. Also note that we switch directories first to be in the actual repo we had mounted when launching docker
.
root@38b478356439:/# cd /mnt/ ## change to repo directory
root@38b478356439:/mnt# travis --login
Shell completion not installed. Would you like to install it now? |y| y
We need your GitHub login to identify you.
This information will not be sent to Travis CI, only to api.github.com.
The password will not be displayed.
Try running with --github-token or --auto if you don't want to enter your password anyway.
Username: eddelbuettel
Password for eddelbuettel: ****************
Two-factor authentication code for eddelbuettel: xxxxxx
Successfully logged in as eddelbuettel!
root@38b478356439:/mnt#
Now the actual work of encrypting. For this particular package, we need a file .Rprofile
containing a short option()
segment setting a user-id and password:
root@38b478356439:/mnt# travis encrypt-file .Rprofile
Detected repository as PMassicotte/gtrendsR, is this correct? |yes|
encrypting .Rprofile for PMassicotte/gtrendsR
storing result as .Rprofile.enc
storing secure env variables for decryption
Please add the following to your build script (before_install stage in your .travis.yml, for instance):
openssl aes-256-cbc -K $encrypted_988d19a907a0_key -iv $encrypted_988d19a907a0_iv -in .Rprofile.enc -out .Rprofile -d
Pro Tip: You can add it automatically by running with --add.
Make sure to add .Rprofile.enc to the git repository.
Make sure not to add .Rprofile to the git repository.
Commit all changes to your .travis.yml.
root@38b478356439:/mnt#
That's it. Now we just need to follow-through as indicated, committing the .Rprofile.enc
file, making sure to not commit its input file .Rprofile
, and adding the proper openssl
invocation with the keys known only to Travis to the file .travis.yml
.
An updated anytime package arrived at CRAN yesterday. This is release number nine, and the first with a little gap to the prior release on Christmas Eve as the features are stabilizing, as is the implementation.
anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, ... format to either POSIXct or Date objects -- and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples.
This releases addresses two small things related to the anydate()
and utcdate()
conversion (see below) and adds one nice new format, besides some internal changes detailed below:
R> library(anytime)
R> anytime("Thu Sep 01 10:11:12 CDT 2016")
[1] "2016-09-01 10:11:12 CDT"
R> anytime("Thu Sep 01 10:11:12.123456 CDT 2016") # with frac. seconds
[1] "2016-09-01 10:11:12.123456 CDT"
R>
Of course, all commands are also fully vectorised. See the anytime page, or the GitHub README.md for more examples.
Changes in anytime version 0.2.1 (2017-02-09)
The new
DatetimeVector
class from Rcpp is now used, and proper versioned Depends: have been added (#43)The
anydate
andutcdate
functions convert again from factor and ordered (#46 closing #44)A format similar to RFC 28122 but with additonal timezone text can now be parsed (#48 closing #47)
Conversion from
POSIXt
toDate
now also respect the timezone (#50 closing #49)The internal
.onLoad
functions was updatedThe Travis setup uses https to fetch the run script
Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page.
For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Time for another update of RcppArmadillo with a new release 0.7.700.0.0 based on a fresh Armadillo 7.700.0. Following my full reverse-dependency check of 318 package (commit of log here), CRAN took another day to check again.
Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language--and is widely used by (currently) 318 other packages on CRAN -- an increase of 20 just since the last CRAN release of 0.7.600.1.0 in December!
Changes in this release relative to the previous CRAN release are as follows:
Changes in RcppArmadillo version 0.7.700.0.0 (2017-02-07)
Upgraded to Armadillo release 7.700.0 ("Rogue State")
added
polyfit()
andpolyval()
added second form of
log_det()
to directly return the result as a complex numberadded
range()
to statistics functionsexpanded
trimatu()
/trimatl()
andsymmatu()
/symmatl()
to handle sparse matriceChanges in RcppArmadillo version 0.7.600.2.0 (2017-01-05)
Upgraded to Armadillo release 7.600.2 (Coup d'Etat Deluxe)
- Bug fix to memory allocation for
fields
Courtesy of CRANberries, there is a diffstat report. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A pure maintenance release of the random package for truly (hardware-based) random numbers as provided by random.org is now on CRAN. As requested by CRAN, we made running tests optional. Not running tests is clearly one way of not getting (spurious, networking-related) failures ...
Courtesy of CRANberries comes a diffstat report for this release. Current and previous releases are available here as well as on CRAN.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new minor version 0.2.1, of RcppCCTZ is now on CRAN. It corrects a possible shortcoming and rounding in the conversion from internal representation (in C++11 using int64_t
) to the two double
values for seconds and nanoseconds handed to R. Two other minor changes are also summarized below.
RcppCCTZ uses Rcpp to bring CCTZ to R. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. The RcppCCTZ page has a few usage examples and details.
The changes in this version are summarized here:
Changes in version 0.2.1 (2017-02-04)
We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new version of the nanotime package for working with nanosecond timestamps is now on CRAN.
nanotime uses the RcppCCTZ package for (efficient) high(er) resolution time parsing and formatting, and the bit64 package for the actual integer64
arithmetic.
This release adds an improved default display format always showing nine digits of fractional seconds. It also changes the print()
method to call format()
first, and we started to provide some better default Ops
methods. These fixes were suggested by Matt Dowle. We also corrected a small issue which could lead to precision loss in formatting as pointed out by Leonardo Silvestri.
Changes in version 0.1.1 (2017-02-04)
The default display format now always shows nine digits (#10 closing #9)
The default print method was updated to use formated output, and a new new converter
as.integer64
was addedSeveral 'Ops' method are now explicitly defined allowing casting of results (rather than falling back on bit64 behaviour)
The format routine is now more careful about not loosing precision (#13 closing #12)
We also have a diff to the previous version thanks to CRANberries. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A major new update of the RPushbullet package is now on CRAN. RPushbullet interfacing the neat Pushbullet service for inter-device messaging, communication, and more. It lets you easily send alerts like the one to the to your browser, phone, tablet, ... -- or all at once.
This release owes a lot to Seth Wenchel who was instrumental in driving several key refactorings. We now use the curl package instead of relying on system()
calls to the binary. We also switched from RJSONIO to jsonlite. A new helper function to create the required resourcefile was added, and several other changes were made as detailed below in the extract from the NEWS.Rd
file.
Changes in version 0.3.0 (2017-02-03)
The
curl
binary use was replaced by use of the curl package; several new helper functions added (PRs #30, #36 by Seth closing #29)Use of RJSONIO was replaced by use of jsonlite (PR #32 by Seth closing #31)
A new function
pbSetup
was added to aid creating the resource file (PRs #34, #37 by Seth and Dirk)The package intialization was refactored so that non-loading calls such as
RPushbullet::pbPost(...)
now work (#33 closing #26)The test suite was updated and extended
The Travis script was updated use run.sh
DESCRIPTION, README.md and other files were updated for current
R CMD check
standardsDeprecated parts such as 'type=address' were removed, and the documentation was updated accordingly.
Coverage support was added (in a 'on-demand' setting as automated runs would need a Pushbullet API token)
Courtesy of CRANberries, there is also a diffstat report for this release.
More details about the package are at the RPushbullet webpage and the RPushbullet GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
An update to the gunsales package is now on the CRAN network. As in the last update, some changes are mostly internal. We removed the need to import two extra packages with were used in one line each -- easy enough to be replaced with base R. We also update the included data sets, and update a few things for current R packaging standards.
Courtesy of CRANberries, there is also a diffstat report for the most recent release.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new release, now at version 0.6.12, of the digest package is now on CRAN and in Debian.
The digest creates hash digests of arbitrary R objects (using the 'md5', 'sha-1', 'sha-256', 'crc32', 'xxhash' and 'murmurhash' algorithms) permitting easy comparison of R language objects.
This release extends sha1 digest methods to even more types, thanks to another contribution by Thierry Onkelinx.
CRANberries provides the usual summary of changes to the previous version.
For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
During useR! 2016, Nick Tierney had asked on Twitter about rmarkdown and metropolis and whether folks had used RMarkdown-driven LaTeX Beamer presentations. My firm hell yeah answer, based on having used mtheme outright or in local mods for quite some time (see my talks page), lead to this blog post of mine describing this GitHub repo I had quickly set up during breaks at useR! 2016. The corresponding blog post and the repo have some more details on how I do this, in particular about local packages (also with sources on GitHub) for the non-standard fonts I use.
This week I got around to updating the repo / example a little by making the default colours (in my example) a little less awful, and a page on blocks and, most importantly, turning the example into the animated gif below:
And thanks to the beautiful tint package -- see its repo and CRAN package --- I now know how to create a template package. So if there is interest (and spare time), we could build a template package for RStudio too.
With that, may I ask a personal favour of anybody still reading the post? Please do not hit my twitter handle with questions for support. All my code is an GitHub, and issue tickets there are much preferred. Larger projects like Rcpp also have their own mailing lists, and it is much better to use those. And if you like neither, maybe ask on StackOverflow. But please don't spam my Twitter handle. Thank you.
Issue ticket #20 demonstrated that we had not yet set up Windows for version 3 of Google Protocol Buffers ("Protobuf") -- while the other platforms support it. So I made the change, and there is release 0.4.8.
RProtoBuf provides R bindings for the Google Protocol Buffers ("Protobuf") data encoding and serialization library used and released by Google, and deployed as a language and operating-system agnostic protocol by numerous projects.
The NEWS
file summarises the release as follows:
Changes in RProtoBuf version 0.4.8 (2017-01-17)
CRANberries also provides a diff to the previous release. The RProtoBuf page has an older package vignette, a 'quick' overview vignette, a unit test summary vignette, and the pre-print for the JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Yesterday afternoon, the nineth update in the 0.12.* series of Rcpp made it to the CRAN network for GNU R. Windows binaries have by now been generated; and the package was updated in Debian too. This 0.12.9 release follows the 0.12.0 release from late July, the 0.12.1 release in September, the 0.12.2 release in November, the 0.12.3 release in January, the 0.12.4 release in March, the 0.12.5 release in May, the 0.12.6 release in July, the 0.12.7 release in September, and the 0.12.8 release in November --- making it the thirteenth release at the steady bi-montly release frequency.
Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 906 packages on CRAN depend on Rcpp for making analytical code go faster and further. That is up by sixthythree packages over the two months since the last release -- or about a package a day!
Some of the changes in this release are smaller and detail-oriented. We did squash one annoying bug (stemming from the improved exception handling) in Rcpp::stop()
that hit a few people. Nathan Russell added a sample()
function (similar to the optional one in RcppArmadillo; this required a minor cleanup by for small number of other packages which used both namespaces 'opened'. Date
and Datetime
objects now have format()
methods and <<
output support. We now have coverage reports via covr as well. Last but not least James "coatless" Balamuta was once more tireless on documentation and API consistency --- see below for more details.
Changes in Rcpp version 0.12.9 (2017-01-14)
Changes in Rcpp API:
The exception stack message is now correctly demangled on all compiler versions (Jim Hester in #598)
Date and Datetime object and vector now have format methods and
operator<<
support (#599).The
size
operator inMatrix
is explicitly referenced avoiding a g++-6 issues (#607 fixing #605).The underlying date calculation code was updated (#621, #623).
Addressed improper diagonal fill for non-symmetric matrices (James Balamuta in #622 addressing #619)
Changes in Rcpp Sugar:
Changes in Rcpp unit tests
Changes in Rcpp Documentation:
Changes in Rcpp build system
Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Last week, Josh sent the call for papers to the R-SIG-Finance list making everyone aware that we will have our nineth annual R/Finance conference in Chicago in May. Please see the call for paper (at the link, below, or at the website) and consider submitting a paper.
We are once again very excited about our conference, thrilled about upcoming keynotes and hope that many R / Finance users will not only join us in Chicago in May 2017 -- but also submit an exciting proposal.
We also overhauled the website, so please see R/Finance. It should render well and fast on devices of all sizes: phones, tablets, desktops with browsers in different resolutions. The program and registration details still correspond to last year's conference and will be updated in due course.
So read on below, and see you in Chicago in May!
R/Finance 2017: Applied Finance with R
May 19 and 20, 2017
University of Illinois at Chicago, IL, USA
The ninth annual R/Finance conference for applied finance using R will be held on May 19 and 20, 2017 in Chicago, IL, USA at the University of Illinois at Chicago. The conference will cover topics including portfolio management, time series analysis, advanced risk tools, high-performance computing, market microstructure, and econometrics. All will be discussed within the context of using R as a primary tool for financial risk management, portfolio construction, and trading.
Over the past eight years, R/Finance has included attendees from around the world. It has featured presentations from prominent academics and practitioners, and we anticipate another exciting line-up for 2017.
We invite you to submit complete papers in pdf format for consideration. We will also consider one-page abstracts (in txt or pdf format) although more complete papers are preferred. We welcome submissions for both full talks and abbreviated "lightning talks." Both academic and practitioner proposals related to R are encouraged.
All slides will be made publicly available at conference time. Presenters are strongly encouraged to provide working R code to accompany the slides. Data sets should also be made public for the purposes of reproducibility (though we realize this may be limited due to contracts with data vendors). Preference may be given to presenters who have released R packages.
Financial assistance for travel and accommodation may be available to presenters, however requests must be made at the time of submission. Assistance will be granted at the discretion of the conference committee.
Please submit proposals online at http://go.uic.edu/rfinsubmit.
Submissions will be reviewed and accepted on a rolling basis with a final deadline of February 28, 2017. Submitters will be notified via email by March 31, 2017 of acceptance, presentation length, and financial assistance (if requested).
Additional details will be announced via the conference website as they become available. Information on previous years' presenters and their presentations are also at the conference website. We will make a separate announcement when registration opens.
For the program committee:
Gib Bassett, Peter Carl, Dirk Eddelbuettel, Brian Peterson,
Dale Rosenthal, Jeffrey Ryan, Joshua Ulrich
Last month, we released nanotime, a package to work with nanosecond timestamps. See the initial release announcement for some background material and a few first examples.
nanotime relies on the RcppCCTZ package for high(er) resolution time parsing and formatting: R itself stops a little short of a microsecond. And it uses the bit64 package for the actual arithmetic: time at this granularity is commonly represented at (integer) increments (at nanosecond resolution) relative to an offset, for which the standard epoch of Januar 1, 1970 is used. int64
types are a perfect match here, and bit64 gives us an integer64
. Naysayers will point out some technical limitations with R's S3 classes, but it works pretty much as needed here.
The one thing we did not have was Windows support. RcppCCTZ and the CCTZ library it uses need real C++11 support, and the g++-4.9
compiler used on Windows falls a little short lacking inter alia a suitable std::get_time()
implementation. Enter Dan Dillon who ported this from LLVM's libc++ which lead to Sunday's RcppCCTZ 0.2.0 release.
And now we have all our ducks in a row: everything works on Windows too. The next paragraph summarizes the changes for both this release as well as the initial one last month:
Changes in version 0.1.0 (2017-01-10)
Added Windows support thanks to expanded RcppCCTZ (closes #6)
Added "mocked up" demo with nanosecond delay networking analysis
Added 'fmt' and 'tz' options to output functions, expanded
format.nanotime
(closing #2 and #3)Added data.frame support
Expanded tests
Changes in version 0.0.1 (2016-12-15)
Initial CRAN upload.
Package is functional and provides examples.
We also have a diff to the previous version thanks to CRANberries. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new version, now at 0.2.0, of RcppCCTZ is now on CRAN. And it brings a significant change: windows builds! Thanks to Dan Dillon who dug deep enough into the libc++ sources from LLVM to port the std::get_time()
function that is missing from the 4.* series of g++. And with Rtools being fixed at g++-4.9.3
this was missing for us here. Now we can parse dates for use by RcppCCTZ on Windows as well. That is important not only for RcppCCTZ but also particularly for the one package (so far) depending on it: nanotime.
CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. It requires only a proper C++11 compiler and the standard IANA time zone data base which standard Unix, Linux, OS X, ... computers tend to have in /usr/share/zoneinfo
-- and for which R on Windows ships its own copy we can use. RcppCCTZ connects this library to R by relying on Rcpp.
The RcppCCTZ page has a few usage examples, as does the post announcing the previous release.
The changes in this version are summarized here:
Changes in version 0.2.0 (2017-01-08)
Windows compilation was enabled by defining
OFFSET()
andABBR()
for MinGW (#10 partially addressing #9)Windows use completed with backport of
std::get_time
from LLVM's libc++ to enablestrptime
semantics (Dan Dillon in #11 completing #9)Timezone information on Windows is supplied via R's own copy of zoneinfo with
TZDIR
set (also #10)The interface to
formatDouble
was cleaned up
We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Today, Rcpp passed another milestone as 900 packages on CRAN now depend on it (as measured by Depends, Imports and LinkingTo declarations). The graph is on the left depicts the growth of Rcpp usage over time.
The easiest way to compute this is to use the reverse_dependencies_with_maintainers()
function from a helper scripts file on CRAN. This still gets one or two false positives of packages declaring a dependency but not actually containing C++ code and the like. There is also a helper function revdep()
in the devtools package but it includes Suggests:
which does not firmly imply usage, and hence inflates the count. I have always opted for a tighter count with corrections.
Rcpp cleared 300 packages in November 2014. It passed 400 packages in June 2015 (when I only tweeted about it), 500 packages in late October 2015, 600 packages last March, 700 packages last July and 800 packages last October. The chart extends to the very beginning via manually compiled data from CRANberries and checked with crandb. The next part uses manually saved entries. The core (and by far largest) part of the data set was generated semi-automatically via a short script appending updates to a small file-based backend. A list of packages using Rcpp is kept on this page.
Also displayed in the graph is the relative proportion of CRAN packages using Rcpp. The four per-cent hurdle was cleared just before useR! 2014 where I showed a similar graph (as two distinct graphs) in my invited talk. We passed five percent in December of 2014, six percent July of last year, seven percent just before Christmas eight percent this summer, and nine percent mid-December.
900 user packages is a really large number. This puts more than some responsibility on us in the Rcpp team as we continue to keep Rcpp as performant and reliable as it has been.
At the rate things are going, the big 1000 may be hit some time in April.
And with that a very big Thank You! to all users and contributors of Rcpp for help, suggestions, bug reports, documentation or, of course, code.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Big news: RcppTOML now works on Windows too!
This package had an uneventful 2016 without a single update. Release 0.0.5 had come out in late 2015 and we had no bugs or issues to fix. We use the package daily in production: a key part of our parameterisation is in TOML files
In the summer, I took one brief stab at building on Windows now that R sports itself a proper C++11 compiler on Windows too. I got stuck over the not-uncommon problem of incomplete POSIX and/or C++11 support with MinGW and g++-4.9. And sadly ... I appears I wasn't quite awake enough to realize that the missing functionality was right there exposed by Rcpp! Having updated that date / datetime functionality very recently, I was in a better position to realize this when Devin Pastoor asked two days ago. I was able to make a quick suggestion which he tested, which I then refined ... here we are: RcppTOML on Windows too! (For the impatient: CRAN has reported that it has built the Windows binaries, they should hit mirrors such as this CRAN package for RcppTOML shortly.)
So what is this TOML thing, you ask? A file format, very suitable for configurations, meant to be edited by humans but read by computers. It emphasizes strong readability for humans while at the same time supporting strong typing as well as immediate and clear error reports. On small typos you get parse errors, rather than silently corrupted garbage. Much preferable to any and all of XML, JSON or YAML -- though sadly these may be too ubiquitous now. But TOML is making good inroads with newer and more flexible projects. The Hugo static blog compiler is one example; the Cargo system of Crates (aka "packages") for the Rust language is another example.
The new release updates the included cpptoml template header by Chase Geigle, brings the aforementioned Windows support and updates the Travis configuration. We also added a NEWS file for the first time so here are all changes so far:
Changes in version 0.1.0 (2017-01-05)
Added Windows support by relying on
Rcpp::mktime00()
(#6 and #8 closing #5 and #3)Synchronized with
cpptoml
upstream (#9)Updated Travis CI support via newer
run.sh
Changes in version 0.0.5 (2015-12-19)
Synchronized with
cpptoml
upstream (#4)Improved and extended examples
Changes in version 0.0.4 (2015-07-16)
Minor update of upstream
cpptoml.h
More explicit call of
utils::str()
Properly cope with empty lists (#2)
Changes in version 0.0.3 (2015-04-27)
- First CRAN release after four weeks of initial development
Courtesy of CRANberries, there is a diffstat report for this release.
More information and examples are on the RcppTOML page. Issues and bugreports should go to the GitHub issue tracker.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new minor release with version number 0.6.11 of the digest package is now on CRAN and in Debian.
It is mostly a maintenance release. Sometime last spring we were asked to consider changing the license GPL-2 to GPL (>= 2). Having gotten agreement of all copyright holders, this could finally happen. But it so happens that the last yay just after the last release, so it took another cycle. In other changes, I also made the makeRaw
function fully generic and added documentation. The pull request by Jim Hester to add covr support was finally folded in, leading as always to some gaming and improvement of the coverage metrics. A polite request by Radford Neal to support a nosharing
option in base::serialize
was also honoured; this should help for use with his interesting pqR variant of R.
CRANberries provides the usual summary of changes to the previous version.
For questions or comments use the issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.