||Thinking inside the box|
Sun, 27 Nov 2011
A Story of Life and Death. On CRAN. With Packages.
The success of CRAN is due to a lot of hard work by the CRAN maintainers, lead for many years and still today by Kurt Hornik whose dedication is unparalleled. Even at the current growth rate of several packages a day, all submissions are still rigorously quality-controlled using strong testing features available in the R system.
And for all its successes, and without trying to sound ungrateful, there have always been some things missing at CRAN. It has always been difficult to keep a handle on the rapidly growing archive. Task Views for particular fields, edited by volunteers with specific domain knowledge (including yours truly) help somewhat, but still cannot keep up with the flow. What is missing are regular updates on packages. What is also missing is a better review and voting system (and while Hadley Wickham mentored a Google Summer of Code student to write CRANtastic, it seems fair to say that this subproject didn't exactly take off either).
Following useR! 2007 in Ames, I decided to do something and noodled over a first design on the drive back to Chicago. A weekend of hacking
lead to CRANberries. CRANberries uses existing R functions to learn which packages
are available right now, and compares that to data stored in a local SQLite database. This is enough to learn two things: First, which
new packages were added since the last run. That is very useful information, and it feeds a website with blog
subscriptions (for the technically minded: an RSS feed, at this URL).
Second, it can also compare current versions numbers with the most recent stored
version number, and thereby learns about updated packages. This too is useful, and also feeds a website and RSS
stream (at this URL; there is also a combined one for new and updated packages.) CRANberries
writes out little summaries for both new packages (essentially copying what the DESCRIPTION file contains), and a quick
All this has been operating since 2007. Google Reader tells me the the RSS feed averages around 137 posts per week, and has about 160 subscribers. It does feed to Planet R which itself redistributes so it is hard to estimate the absolute number of readers. My weblogs also indicate a steady number of visits to the html versions.
The most recent innovation was to add tweeting earlier in 2011 under the
Which leads us to today's innovation. One feature which has truly been missing from CRAN was updates about withdrawn packages. Packages can be withdrawn for a number of reasons. Back in the day, CRAN carried so-called bundles carrying packages inside. Examples were VR and gregmisc. Both had long been split into their component packages, making VR and gregmisc part of the set of packages no longer on the top page of CRAN, but only its archive section. Other examples are packages such as Design, which its author Frank Harrell renamed to rms to match to title of the book covering its methodology. And then there are of course package for which the maintainer disappeared, or lost interest, or was unable to keep up with quality requirements imposed by CRAN. All these packages are of course still in the Archive section of CRAN.
But how many packages did disappear? Well, compared to the information accumulated by CRANberries over the years, as of today a staggering 282 packages have been withdrawn for various reasons. And at least I would like to know more regularly when this happens, if only so I have a chance to see if the retired package is one the 120+ packages I still look after for Debian (as happened recently with two Rmetrics packages).
So starting with the next scheduled run, CRANberries
will also report removed packages, in its own subtree of the website and its own RSS feed (which should appear at
this URL). I made the required code changes (all of about two
dozen lines), and did some light testing. To not overwhelm us all with line noise while we catch up to the current steady state of packages, I
have (temporarily) lowered the frequency with which CRANberries is called by
I hope this is of interest and use to others. If so, please use the RSS feeds in your RSS readers, and subscribe to the