RcppSimdJSON: Rcpp Bindings for the simdjson Header Library
Motivation
simdjson by Daniel Lemire (with contributions by Geoff Langdale, John Keiser and many others) is an engineering marvel. Through very clever use of SIMD instructions, it manages to parse JSON files faster than disc access. Wut? Yes you read that right: parallel processing with so little overhead that the net throughput is limited only by disk speed.
Moreover, it is implemented in neat modern C++ and can be accessed as a header-only library. (Well, one library in two files, really.) Which makes R packaging easy and convenient and compelling. So here we are.
For further introduction, see the arXiv paper by Langdale and Lemire (out/to appear in VLDB Journal 28(6) as well) and/or the video of the recent talk by Daniel Lemire at QCon (voted best talk).
Example
jsonfile <- system.file("jsonexamples", "twitter.json", package="RcppSimdJson")
library(RcppSimdJson)
validateJSON(jsonfile) # validate a JSON file
res <- fload(jsonfile) # parse a JSON fileComparison
A simple file-oriented parsing benchmark against the other R-accessible JSON parsers:
> print(res)
Unit: microseconds
expr min lq mean median uq max neval cld
simdjson 101.991 114.764 127.742 123.070 133.514 202.078 100 a
yyjsonr 117.089 140.452 157.350 156.107 169.136 281.537 100 a
jsonify 1092.174 1112.041 1228.807 1137.108 1367.544 1778.297 100 b
jsonlite 1941.132 1972.886 2022.138 1989.743 2016.072 2766.606 100 c
RJSONIO 4459.464 4524.275 4633.694 4552.539 4610.932 6516.523 100 d
ndjson 35795.875 36269.607 36495.140 36378.285 36636.018 44191.303 100 e
>
> print(res, unit="relative")
Unit: relative
expr min lq mean median uq max neval cld
simdjson 1.00000 1.00000 1.00000 1.00000 1.0000 1.00000 100 a
yyjsonr 1.14803 1.22383 1.23178 1.26844 1.2668 1.39321 100 a
jsonify 10.70853 9.68976 9.61944 9.23952 10.2427 8.80005 100 b
jsonlite 19.03239 17.19073 15.82986 16.16757 15.1000 13.69078 100 c
RJSONIO 43.72409 39.42226 36.27384 36.99146 34.5351 32.24756 100 d
ndjson 350.97092 316.03507 285.69405 295.59019 274.3973 218.68438 100 e
>Or in chart form, also including the second benchmark parsing strings:
Status
All three major OSs are supported, and JSON can be parsed from file and string under a variety of settings. A C++17 compiler is required for ease of setup (though the upstream can fall back to older compiler; one can edit src/Makevars accordingly if need be).
Contributing
Any problems, bug reports, or features requests for the package can be submitted and handled most conveniently as Github issues in the repository.
Before submitting pull requests, it is frequently preferable to first discuss need and scope in such an issue ticket. See the file Contributing.md (in the Rcpp repo) for a brief discussion.
See Also
For standard JSON work on R, as well as for other nicely done C++ libraries, consider these:
- jsonlite by Jeroen Ooms is excellent, very versatile, and probably most-widely used;
- rapidjsonr and jsonify by David Cooley bringing RapidJSON to R;
- ndjson by Bob Rudis builds on the JSON for Modern C++ library by Niels Lohmann;
- RJSONIO by Duncan Temple Lang started all this but could use a little love;
- yyjsonr by Mike Cheng is a more recent and performant addition based on yyjson.
Author
For the R package, Dirk Eddelbuettel and Brendan Knapp.
For everything pertaining to simdjson, Daniel Lemire (and many contributors).
License
GPL (>= 2)