R/Finance conference in Chicago in April: Call for Papers
The following went out to the R-announce and R-SIG-Finance mailing lists a few days ago. The conference already has a very strong lineup of invited speakers, and we are now asking R / Finance users from both academia and industry to submit suitable one-page abstracts:

Call for Papers

The Finance Department of the University of Illinois at Chicago (UIC),
the International Center for Futures and Derivatives at UIC, and
members of the R finance community are pleased to announce

R/Finance 2009: Applied Finance with R

on April 24 and 25, 2009, in Chicago, IL, USA

Confirmed keynote speakers include:

Patrick Burns (Burns Statistics)
David Kane (Kane Capital)
Roger Koenker (U of Illinois at Urbana/Champaign)
David Ruppert (Cornell)
Diethelm Wuertz (ETH Zuerich)
Eric Zivot (U of Washington)

We invite all users of R in Finance to submit one-page abstracts or
complete papers (in txt/pdf/doc format). We encourage papers both on
academic research topics and related to use of R by Finance practitioners.

Presenters are strongly encouraged to provide working R code to accompany
the presentation/paper. Datasets need not be made public.

Please send submissions to committee@RinFinance.com.
The submission deadline is January 31st, 2009.
Submissions will be evaluated and submitters notified via email on a rolling basis.

Additional details about the conference will be announced as available.

For the program committee:

Gib Bassett, Peter Carl, Dirk Eddelbuettel, John Miller,
Brian Peterson, Dale Rosenthal, Jeffrey Ryan

/computers/R | permanent link

Updated 'Introduction to High-Performance Computing with R'
Fellow R user Paul Gilbert had invited me to come to Ottawa and the Bank of Canada to give a presentation/workshop on 'high-performance computing with R' similar to the UseR 2008 tutorial and talk.

I just posted the updated slides from this talk, and there is also an updated live cdrom on the Alioth server. Also, it looks like the tutorial will be held again at UseR 2009 in Rennes, see here for a brief synopsis.

It was nice to get back to Canada, even if it was a 24 hour whirlwind trip. Ottaws looked quite pretty in all the snow. And it seems that I got rather lucky with the travel dates as both the days before and after my trip had a large number of flight cancellations and delays due to snow storms.

/computers/R | permanent link

CRANberries prettified
Judging from my html logs, a fair number of folks go to the html version of my CRANberries feed (which was originally announced here) of new or updated packages for R,

So I quickly put together some simple css formatting to make it look a little better than the default blosxom theme it sported previously. That said, you probably should read the rss version (more about rss here) anyway!

Update: Oops. And it even works with a correct path to the css file. Now fixed.

/computers/R | permanent link

UseR! 2008 talk
Besides the slides from the tutorial at UseR! 2008 that were mentioned here previously, I also gave a short talk on scripting with R in high-performance computing using our littler frontend to R.

The talk introduces and extends an example related to some of the material from the tutorial itself. The slides from the talk are a little rough as the talk was somewhat ad-hoc: As session chair, I was confronted with a fairly last-minute cancellation and a 15 minute hole, and thought this would make a good little talk. It does show a nice trick for using littler with Open MPI (via snow) under the powerful slurm resource manager and batch/queue engine.

/computers/R | permanent link

UseR! 2008 tutorial
Earlier today, I presented a 3 1/2 hour tutorial Introduction to high-performance R (here is a brief description of the talk) at the UseR! 2008 conference at the TU Dortmund.

In a nutshell, the tutorial covered how to measure / profile R performance for speed and memory use, how to accelerate R using vectorised expression and tools like Ra / jit, how to add compiled code to R using either the .C or .Call interface and using the inline and RCpp packages, how to use R code in parallel (explicitly using NWS, Rmpi or snow as well as implicitly using pnmath / OpenMP), and how to script / automate R using littler, Rscript or RPy.

The final version of the slides is now available via my presentations page, and the live cdrom with software support for all the software used is at Alioth.

Update: Corrected link to presentations page thanks to heads-up by Charles. Thanks!

/computers/R | permanent link

CRANberries updated
The good folks at CRAN, the R package network which by now contains over thirteen hundred packages, have reorganised the website slightly. As far as I can tell, the changes are all for the better: nicer URLs, lots more per-package information, and other changes such as an updated 'task view' format (which is the part I knew via my maintenance of the Finance task view).

But these changes also affected my my CRANberries (see the html or better yet rss view) summaries of new packages as some of the source information moved. So I just updated the (surprisingly short at 189 lines including plenty of whitespace and comments) script, and things should work now come the next update.

While updating the 'more info' link for new and updates posts to point to the new-style entry at CRAN, I also took the opportunity to update the format of the `blog' entry for updates where we now show title and description along with the diffstat output,

I also manually copied in two of the recent entries: the new package emu where CRANberries had fallen over as we could not find the package description (in the new spot), and the existing package GEOmap where diffstat failed as we somehow didn't have a proper tarnall of the previous sources.

/computers/R | permanent link

The amazing Prof. Ripley (cont'ed)
A little mini-meme got started on August 1 when Ben Bolker posted the following code to the r-devel list (and here I substituted the more standard '<-' assignment operator for the less standard though-now permitted '='):

x <- readLines("http://developer.r-project.org/R.svnlog.2007")
rx <- x[grep("^r",x)]
who <- gsub(" ","",sapply(strsplit(rx,"\\|"),"[",2))
twho <- table(who)
twho["ripley"]/sum(twho)

[ Oh, and for those who don't know him, he's also got a day job which presumably entails looking after his graduate students at Oxford. Who knows, he may even teach. Kidding aside, he's actually one of the nicest persons you'll ever meet in real life. ]

Now yesterday, Simon Jackman who had at first simply repeated Ben's analysis on his own blog followed up with a nice analysis (albeit typeset in a way that rendered the code inoperational, which has now been fixes) that creates both a histogram and a dotplot of commits per hour of the day. Omitting Ben's code which Simon reuses, we have the following for histogram and dotchart:

tod <- unlist(sapply(rx,function(x)strsplit(x,split=" ")[[1]][6]))
tod <- tod[who=="ripley"]

tz <- sub(pattern=".*(-[0-9]{4}).*",replacement="\\1",x=rx)
tz <- tz[who=="ripley"]
tz <- as.numeric(tz)/100
offset <- 3600*tz

z <- strptime(tod,format="%H:%M:%S")
hist(z,"hours",main="Ripley Commit Times in SVN TZ")

h <- z - offset
h <- format(h,format="%H")
h <- factor(as.numeric(h), levels=0:23)
dotchart(table(h), main="Ripley Commit Times, By Hour in GMT",
         labels=paste(0:23,1:24,sep=":"))

Now, nobody has looked at the time series. So we correct this and add the following:

## rather extract both  date and time
dat <- unlist(sapply(rx, function(x) {
  txt <- strsplit(x,split=" ")[[1]]
  paste(txt[5], txt[6])
}))
## subset on Prof Ripley
dat <- dat[who == "ripley"]
## and convert to POSIXct, correcting by tz as well
datpt <- as.POSIXct(strptime(dat,format="%Y-%m-%d %H:%M:%S")) - offset

## turn into zoo -- we use a constant series of ones as each
## committ is taken as a timestamped event
datzoo <- zoo(1, order.by=datpt)
## and use zoo to aggregate into commits per date
daily <- aggregate(datzoo, as.Date(index(datzoo)), sum)

## now plot as grey bars
plot(daily, col='darkgrey', type='h', lwd=2,
     ylab="Nb of SVN commits, three-week median",
     xlab="R release dates 2.5.0 and 2.5.1 shown in orange",
     main="The amazing Prof. Ripley")
## mark the two R releases of 2007
abline(v=c(as.Date("2007-04-24"),as.Date("2007-06-28")),col='orange',lwd=1.5)
## and do a quick centered rolling median
lines(rollmedian(daily, 21, align="center"), lwd=3)

This shows that Prof Ripley averaged about ten commits a day before and after the release of R 2.5.0, and that he has slowed down ever so slightly since then to end up at around a mere seven commits a day. Every day. For the seven-plus months we looked at.

So, anyone for analysing his r-help posting frequencies ?

/computers/R | permanent link

Announcing CRANberries
Earlier today I sent an announcement to the r-packages list. It describes CRANberries, two simple RSS feeds that summarize both 'new' and 'updated' packages at CRAN, the archive network for R. I cooked this up rather quickly using a few lines of R, a small SQLite db backend and the old Blosxom blog engine. A tip of the hat to Barry Rowlingson who almost immediately suggested to use the lol format instead.

The hope is that this proves helpful for keeping tabs on the amazing growth of CRAN (which is now at over one thousand packages) as well as the number of updates to existing packages. The feed(s) can be consumed standalone, or via the brand new Planet R aggregator that Elijah announced today too.

/computers/R | permanent link

More on 'nicer charts'
Via the Planet Debian aggregator and his blog, Sven followed up on my post regarding Lucas' plot of the package age distribution.

As some of my points didn't seem to make it across, I will reiterate them more plainly:

• GNUplot, while easy to use, creates charts that aren't terribly pretty;
• Lucas' original chart had, to paraphrase an expression by Tufte, a poor 'ink to paper ratio': the data is too concentrated in the last quartile;
• for that very reason, taking logs is a good thing here

Sven also addresses the fact that what we really want is to see the quantiles of the data set. Quite right, and taking logs makes that easier. Consider the two charts below which plot the 'package age in days' as an empirical cumulative distribution function using built-in R functions ecdf and plot.stepfun (rather than redoing it ad-hoc as I had done), and also add explicitly quantiles. The two charts use the exact same instructions; however the second chart transforms the x-axis to a logarithmic scale.

While it is close to impossible to find the 25 or 50 percentile on the first chart, it becomes a lot easier on the second chart because the x-axis is 'stretched' using the log transform. About one quarters of the distribution appears to be rebuild within 1.5 months old, and about half is younger than four months (as a quick call to summary(pkgAge) confirms). Reading these proprtions off the original chart, or the non-log chart, is much more difficult.

/computers/R | permanent link

Improving simple charts
Earlier today and via Planet Debian, Lucas blogged about the 'age distribution' of Debian packages, defined as the time since the last (re-)compilation. He illustrated his findings with an, umm, rather ugly chart. Having climbed onto the soap box once before, I would like to point out how easy it can be to create simple, informative, and, at to least to me, prettier charts using R.

Lucas included a URL to the data. The first nice thing to note that we can read the data directly from the URL -- no need to copy the file:

pkgAge <- read.table(file="http://people.debian.org/~lucas/arch-age/arch-age.log", col.names=c("pkg","yyyymmdd"))

pkgAge[,"date"] <- as.Date(as.character(pkgAge[,"yyyymmdd"]), "%Y%m%d")
pkgAge[,"age"] <- as.numeric(difftime(Sys.Date(), pkgAge[,"date"], units="day"))
pkgAge[,"prop"] <- (1:nrow(pkgAge)) / nrow(pkgAge) * 100

We can then view the date. Before we plot,

png("packageAges.png", quality=100, width=640, height=480, pointsize=10)
oldpar <- par(mfrow=c(2,2), mar=c(2.5,2.5,3,1))

The first chart shows again proportion over date:

with(pkgAge, plot(date, prop, type='l', main="Standard Plot"))

As it clear that the data has a fairly long tail in the older dates, we can also try to plot the plot over logarithmic time differences. This doesn't work for dates, but it works for our (positive-valued) age variable:

with(pkgAge, plot(age, prop, type='l', log="x", main="More linear as log(age in days)"))

The very far left tail below 0.5 percent is interesting as the one very old package is clearly an outlier within an outlier region. We use the subset function to take just one portion of the data, use logs, and explicit plotting symbols '+' in a points-and-lines plot:

with(subset(pkgAge, prop<0.5), plot(date, prop, type='b', log="y", pch="+", main="Detail in left tail, up to 0.5%"))

Lastly, the upper quartile is fairly linear.

with(subset(pkgAge, prop>75), plot(date, prop, type='l', pch=".", main="Yet fairly linear in top 25%"))

At the end

oldpar <- par(mfrow=c(2,3))
dev.off()

Updated to correctly display the assignment operator <-

/computers/R | permanent link

RGtk2 packages in Debian
As previously mentioned, I had built local packages of Michael Lawrence's RGtk2 port of the Gtk v2 widgets / toolkit to the R language and environment. This package, along with the related cairoDevice package, is now in Debian's main archive. RGtk2 extends and replaces the older r-omegahat-rgtk package.

For an two interesting applications using RGtk2, look at Graham William's rattle, a glade-based data-mining user-interface to several R functions, and at John Verzani's PMG (aka Poor Man's GUI) generic GUI for R.

/computers/R | permanent link

RGtk2 packages available
Michael Lawrence announced the release of RGtk2 yesterday on both r-packages and r-sig-gui. This follows his impressive presentation at DSC 2005 and finally makes the code available.

RGtk2 is, as the name suggests, an update of the older RGtk package (that I'd been maintaining in Debian as r-omegahat-rgtk) to the 'newer' version 2 of Gtk (aka The GIMP Toolkit). It provides Gtk goodies for the amazing R language and environment many of us dig for its use in statistical computing, visualization, data analysis, estimation and more.

RGtk2 is quite an achievement. The source package is huge at 1.9mb, the resulting Debian package even huger at 5.3mb, and it all seems to work fine based on some initial tests of running the demos. Based on the two source packages, I created two packages of RGtk2 (as r-omegahat-rgtk2) and the Cairo device (as r-omegahat-cairodevice) which you can fetch from here.

Not sure if I feel the urge to maintain them, but if someone else wants to step forward, let me know. Sources and diffs are in the same directory. Feedback welcome.

Update: By the way, what do we need to build with gtkmozembed? I didn't find a proper Build-Depends: for this.

/computers/R | permanent link

First `r-devel' builds leading up to R 2.1.0
As usual, the next GNU R release (expected in April) will bring yet another set of changes. Among the very user-visible changes are initial localisation support, and a disappearance of the (never completed) Gnome front-end.

I have built an initial set of packages, mostly to convince myself that nothing too drastic was needed inside the debian/ directory. These packages are for now here on my box but I guess I could upload them to Debian's experimental distribution for which the changelog entry is tagged.

I tend not to install any locales packages on my machines, so I can't really test if the localisation support works -- configure and friends certainly suggest it based on the compile-time messages. So my dear reader, if you are into non-default locales and R, and have a minute, grab the package and try something like

$ LANGUAGE=de LC_all=de LC_MESSAGES=de LANG=de LOCALE=de R

R : Copyright 2005, The R Foundation for Statistical Computing
Version 2.1.0 Under development (unstable) (2005-02-27), ISBN 3-900051-07-0

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for a HTML browser interface to help.
Type 'q()' to quit R.

> Sys.getlocale()
[1] "C"
> q("no")

/computers/R | permanent link

Nicer charts
Wouter had blogged about his charts based on Takuo's per-maintainer statistics from the Debian BTS.

In an effort of true R evangelism, I bugged Wouter about the gnuplot ugliness in those charts and offered an R script as an alternate. Per his reply, he seemed please with the output [1] -- click on the png for a nicer pdf:

For anybody interested, the R code is available. It provides two simple functions. The first actually creates the 4x1 chart. The second loops over all datafiles ending in .csv in a given directory, assuming the filename before the .csv ending provides the unique identifier (here the maintainer name and email as per Takuo's and Wouter's setup). On a per file basis, data is loaded, and a pdf and png are produced. This can be called as in

R CMD BATCH debian_bts_chart.R

Lastly, I should note that I do think that the underlying data is wrongly classified. Counting a bug as 'active' even when it has been closed, but is not yet archived merely because the 28 days period hasn't passed is plain wrong. In my book, a closed bug cannot count as an active one.

[1] This is my own data, and it shows the drop-off a few weeks ago when I passed maintainership of a good dozen packages on to others.

/computers/R | permanent link

Truly random numbers
The neat random.org service started by Mads Haahr a few years ago supplies truly random numbers. This is not the place to get into details about why pseudo and quasi (aka low-discrepancy sequences) random numbers are only 'random'.

Mads' service samples atmospheric noise -- see his background essay for more details -- which gets aggregated and can then be had via Corba, HTTP or SOAP. Given how R has such a wonderful (and probably little know) url() function to aquire data over the web, I figured it might be worthwhile to show how R can acquire truly random numbers. This snipped downloads a 10,000 x 2 vector, and plots it:

> X <- read.table(url(paste("http://www.random.org/cgi-bin/randnum",
                            "?num=10000&min=-1000000000&max=1000000000&col=2",
                            sep="")),
                      header=FALSE)
> plot(X, pch=".")

As repeated simulations are often rather time-intensive, downloading random sequences may not be the fasted way to go about things. However, this method would provide a portable way to seed a pseudo random number generator in a portable fashion for platforms that do not have an entropy provider under /dev.

/computers/R | permanent link

Three-character patch for Rpy to build under R 2.0.0
As we mentioned here before, R 2.0.0 is out and in Debian. Graham reminded us about the need for a fresh version of Rpy. It turns out that a minor patch is needed to adjust for the new location of libR.so:

--- rpy-0.3.5.orig/setup.py
+++ rpy-0.3.5/setup.py
@@ -54,7 +54,7 @@
 RHOME = get_R_HOME()
 DEFINE.append(('R_HOME', '"%s"' %RHOME))

-r_libs = os.path.join(RHOME, 'bin')
+r_libs = os.path.join(RHOME, 'lib') # edd 11 Oct 2004: changed to 'lib' for 2.0.0
 source_files = ["src/rpymodule.c", "src/R_eval.c",
                 "src/io.c"]
 if sys.platform=='win32':

Update: Greg just told me by email that the fix is in CVS too, so a new RPy will have it.

/computers/R | permanent link

R 2.0.0 now in Debian
It's this time of year again, and the bi-annual GNU R release brings in a wider than usual list of changes as indicated by the new major release version 2.0.0. One change in particular, lazy loading, promises to be most useful, if only for faster startup times.. Unfortunately, the internal code changes also triggered two ugly bugs that seemed to appear only in the pbuilder chroot builds I made. After a couple of frustrating days spent trying different things, Brian Ripley kindly supplied a patch earlier today. This allows us build R, and the core 'recommended' packages, in the chroot environment used by the Debian autobuilders.

So after some further testing, packages of R 2.0.0 are now on the Debian servers and should be in unstable tomorrow. Because of the new internal interface to R packages, older packages will not load. Users can either call update.packages() directly, or wait a few days until we (as in Chris, Doug, Rafael, Steffen and myself) get the various r-(cran|bioc|omegahat|other|noncran)-* packages rebuilt under 2.0.0. We will try our best to have them all updated by Sunday.

/computers/R | permanent link

New 'R in Finance' list up and running
Yesterday, I sent this note to the r-help list:

Thanks again to everybody who participated in the finance sessions at the recent useR! 2004 conference. During the discussions, the idea of a mailing list for R and Finance came up. Thanks to Martin, such a list has now been created and can be accessed via the page

https://www.stat.math.ethz.ch/mailman/listinfo/r-sig-finance

from which subscription requests can be made using the usual confirmation system employed by the mailman software. Everybody interested in 'Finance' (we will try not to be too picky regarding definitions) and R us cordially invited to subscribe. Also feel free to forward this message to interested colleagues.

/computers/R | permanent link

Slides for my useR! 2004 presentations are up
Now back from the excellent useR! 2004, the first international R User Conference in Vienna, I have put the slides for my talk Programming with financial data: Connecting R to Lim and Bloomberg, as well the two shorter ones Quantian: A single-system image scientifc cluster programming environment, and R on Debian: Past, Present, Future (joint with Doug Bates and Albrecht Gebhardt) up on my website.

(And no, I cannot distribute the two packages described in the main talk as they were done at work where open source releases are not yet an accepted methodology. Hopefully one day.)

/computers/R | permanent link

Debian R Policy draft released
We just posted a draft with requests for comments to the Debian and R developer lists. Comments welcome!

/computers/R | permanent link

Accelerating plotOHLC by a few orders of magnitude
The plotOHLC function in the tseries package is useful to plot timeseries of various financial assets with open/high/low/close data. I had often wondered if it could be made to run a little faster. It turns out that the following patch does

--- plotOHLC.R.orig     2003-12-14 12:02:20.000000000 -0600
+++ plotOHLC.R  2003-12-14 12:03:42.000000000 -0600
@@ -21,14 +21,9 @@
         ylim <- range(x[is.finite(x)])
     plot.new()
     plot.window(xlim, ylim, ...)
-    for (i in 1:NROW(x)) {
-        segments(time.x[i], x[i, "High"], time.x[i], x[i, "Low"],
-            col = col[1], bg = bg)
-        segments(time.x[i] - dt, x[i, "Open"], time.x[i], x[i,
-            "Open"], col = col[1], bg = bg)
-        segments(time.x[i], x[i, "Close"], time.x[i] + dt, x[i,
-            "Close"], col = col[1], bg = bg)
-    }
+    segments(time.x, x[, "High"], time.x, x[, "Low"], col = col[1], bg = bg)
+    segments(time.x - dt, x[, "Open"], time.x, x[, "Open"], col = col[1], bg =$
+    segments(time.x, x[, "Close"], time.x + dt, x[, "Close"], col = col[1], bg$
     if (ann)
         title(main = main, xlab = xlab, ylab = ylab, ...)
     if (axes) {

> IBM<-get.hist.quote("IBM", "2001-12-14")
trying URL
http://chart.yahoo.com/table.csv?s=IBM&a=11&b=13&c=2001&d=11&e=12&f=2003&g=d&q=$
Content type application/octet-stream' length unknown
opened URL
.......... .......... ...
downloaded 23Kb

time series starts 2001-12-12
time series ends   2003-12-11
> system.time(plotOHLC(IBM))                    # original
[1] 1.56 0.26 5.11 0.00 0.00
> system.time(fastplotOHLC(IBM))                # patched
[1] 0.02 0.00 0.05 0.00 0.00

/computers/R | permanent link

Small patch for Rpy and recent R versions
After I asked on R-devel about the failure of Rpy to built with more recent versions of GNU R, I every now and then get a request for the patch mentioned in my follow-up on on R-devel.

To make life easier for everybody, and as the patch is so simple, here it comes:

--- rpy-0.3.1.orig/src/RPy.h
+++ rpy-0.3.1/src/RPy.h
@@ -90,7 +90,8 @@
 PyOS_sighandler_t python_sigint;
 
 /* R function for jumping to toplevel context */
-extern void jump_now(void);
+/* extern void jump_now(void); */
+extern void Rf_onintr(void); 
 
 /* Global interpreter */
 PyInterpreterState *my_interp;
--- rpy-0.3.1.orig/src/R_eval.c
+++ rpy-0.3.1/src/R_eval.c
@@ -65,7 +65,8 @@
 void interrupt_R(int signum)
 {
   interrupted = 1;
-  jump_now();
+  /* jump_now(); */
+  Rf_onintr();
 }
 

Thanks to Luke Tierney for the hint regarding Rf_onintr(). If you're into Debian, I also have a local package I could sent you. I'd upload it, but I don't really want to be stuck maintaining it when Rpy looks as if it has been orphaned by its original author.

/computers/R | permanent link

Announcement for useR! 2004 is out
Torsten sent out the announcement for the first international R User Conference to be held in Vienna next May. See you there!

/computers/R | permanent link