prrd 0.1.0 at CRAN: Now With Real Concurrency
A new release of prrd arrived at CRAN today, and has already been built for r2u and on r-universe. The prrd package facilitates the parallel running [of] reverse dependency [checks] when preparing R packages. It is used extensively for releases I make of Rcpp, RcppArmadillo, RcppEigen, BH, and others.
The key idea of prrd is simple, and described in some more detail on its webpage and its GitHub repo. Reverse dependency checks are an important part of package development that is easily done in a (serial) loop. But these checks are also generally embarassingly parallel as there is no or little interdependency between them (besides maybe shared build depedencies). See the (somewhat dated) screenshot (running six parallel workers, arranged in a split byobu session).
This release updates internals to allow for greater concurrency. SQLite is used as the backend, both
for the job queue (managed via the liteq package) as
well as for storing actual reverse-dependency test results. It is
well-known that SQLite copes quite
well with multiple readers. Multiple concurrent
writers are more challenging. We tended to get by via
BEGIN EXCLUSIVE pragma use, but still get occassional issue
even at a somewhat moderate concurrency levels. After some back and
forth and trying a few approaches, more recently we established one
first solution relying on very explicity (OS-level, filesystem-based)
locking via package filelock. That
worked.
We mentioned this in an
old liteq issue that had circled around the same
question with a number of posts over the years, but no resolution. This
comment seemingly awoke Gábor, and he nailed this
years-open issue with a fine pull request deploying
BEGIN IMMEDIATE and well as a loop around
tryCatch to not immediately fail. I borrowed/ported that
approach to prrd and we now
have a second solution (and no longer need filelock). On a
somewhat better server with enough cores and memory, I am now running
eighteen concurrent workers—watched via three multi-pane byobu windows. Works like a charm without
issues. At last! A big thanks to Gábor for finding another
solution, and especially that only relies on SQLite.
The release is summarised in the NEWS entry:
Changes in prrd version 0.1.0 (2026-11-07)
Concurrent operations are now supported via updated sqlite semantics borrowed with appreciation from liteq
This follows an interim solution relying on filelock use; however this later package is not needed so no increase in dependencies
Courtesy of my CRANberries, there is also a diffstat report for this release.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.
