Thu, 11 Dec 2014

RProtoBuf 0.4.2

A new release 0.4.2 of RProtoBuf is now on CRAN. RProtoBuf provides R bindings for the Google Protocol Buffers ("Protobuf") data encoding library used and released by Google, and deployed as a language and operating-system agnostic protocol by numerous projects.

Murray and Jeroen did almost all of the heavy lifting. Many changes were triggered by two helpful referee reports, and we are slowly getting to the point where we will resubmit a much improved paper. Full details are below.

Changes in RProtoBuf version 0.4.2 (2014-12-10)

  • Address changes suggested by anonymous reviewers for our Journal of Statistical Software submission.

  • Make Descriptor and EnumDescriptor objects subsettable with "[[".

  • Add length() method for Descriptor objects.

  • Add names() method for Message, Descriptor, and EnumDescriptor objects.

  • Clarify order of returned list for descriptor objects in as.list documentation.

  • Correct the definition of as.list for EnumDescriptors to return a proper list instead of a named vector.

  • Update the default print methods to use cat() with fill=TRUE instead of show() to eliminate the confusing [1] since the classes in RProtoBuf are not vectorized.

  • Add support for serializing function, language, and environment objects by falling back to R's native serialization with serialize_pb and unserialize_pb to make it easy to serialize into a Protocol Buffer all of the more than 100 datasets which come with R.

  • Use normalizePath instead of creating a temporary file with file.create when getting absolute path names.

  • Add unit tests for all of the above.

CRANberries also provides a diff to the previous release. RProtoBuf page which has a draft package vignette, a a 'quick' overview vignette, and a unit test summary vignette. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

/code/rprotobuf | permanent link

Tue, 25 Mar 2014

RProtoBuf 0.4.1

A new bug-fix release release 0.4.1 of RProtoBuf, is now on CRAN. RProtoBuf provides GNU R bindings for the Google Protocol Buffers ("Protobuf") data encoding library used and released by Google, and deployed as a language and operating-system agnostic protocol by numerous projects.

Murray once more shouldered most of the actual burden and fixed a number of issues detailed below.

Changes in RProtoBuf version 0.4.1 (2014-03-25)

  • Document and add a test for the deprecated group functionality.

  • Add a CITATION file pointing to our arXiv.org preprint.

  • Fix a bug in the show method for EnumDescriptor types.

  • Import all top-level enums from imported .proto files.

  • Removed duplicate enum value type from the unit tests that caused problems with the most recent libprotobuf-2.5. (without option allow_alias).

CRANberries also provides a diff to the previous release. More information is at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

/code/rprotobuf | permanent link

Mon, 20 Jan 2014

RProtoBuf 0.4.0: A whole lot of goodies and Windoze support

A new major release 0.4.0 of RProtoBuf, is now on CRAN. RProtoBuf provides GNU R bindings for the Google Protocol Buffers ("Protobuf") data encoding library used and released by Google, and deployed as a language and operating-system agnostic protocol by numerous projects.

With this release, we are welcoming Jeroen Ooms to the team. Jeroen had already worked on some RProtoBuf extensions in the context of his OpenCPU project; we have now integrated those Protocol Buffers functions. And Jeroen pushed all the right buttons to finally get RProtoBuf built on everybody's least-favourite operating system by provding a static library for use by win-builder and CRAN. Murray once again did a lot of work on internals. His use of the LLVM tool llvm-format was particular helpful to make our coding style a little more consistent.

The complete NEWS file entry for this release follows:

Changes in RProtoBuf version 0.4.0 (2014-01-14)

  • Changes to support CRAN builds for MS Windows.

  • Added functions serialize_pb, unserialize_pb, and can_serialize_pb plus documentation from Jeroen Ooms RProtoBufUtils package.

  • New dir inst/python with some Python examples.

  • Added Jeroen Ooms as author.

  • Vignettes have been converted to the R 3.0.0 or later use of external vignette builders, no longer need a Makefile

  • Added missing methods to dollar completion list for Message, Descriptor, EnumValueDescriptor, and FileDescriptor classes.

  • Add missing export for .DollarNames EnumValueDescriptor to allow completion on that class.

  • Add more than 15 additional pages to the main Intro vignette documenting better all of the S4 classes implemented by RProtoBuf, updating the type mapping tables to take into account 64-bit support, and documenting advanced features such as Extensions.

  • Added better error checking in EnumDescriptors to catch the case when wrong types are provided.

  • Updated the FileDescriptor name() method to accept a boolean for full paths just like the generic name() method.

  • Correct a bug that incorrectly dispatched as.character() when as.list() was called on Descriptor objects.

  • Update FileDescriptor $ dispatch to work properly for the names of fields defined in the FileDescriptor, instead of just returning NULL even for types returned by $ completion.

  • Added a reservation for extension fields in the example tutorial.Person schema.

  • Support setting int32 fields with character representations and raise an R-level stop() error if the provided string can not be parsed as a 32-bit integer, rather than crashing the R instance.

  • Update the project TODO file.

  • Add better documentation and tests for all of the above.

  • Corrected the handling of uint32 and fixed32 types in protocol buffers to ensure that they work with numbers as large as 2^32 - 1, which is larger than an integer can hold in R since R does not have an unsigned integer class. These values are stored as doubles internally now to avoid losing precision.

  • Added unit tests to verify behavior of RProtoBuf with extreme values for uint32 types.

  • Removed old exception handling code and instead rely on the more modern Rcpp::stop method maintained in Rcpp.

  • Add better error messages when setting a repeated field of messages to inform the user which element index was of the wrong type and what the expected type was.

  • Add an optional 'partial' argument to readASCII allowing uninitialized message fragments to be read in.

  • (internal) Added const qualifiers in more places throughout the C++ code for type safety.

  • (internal) Standardize coding conventions of the C++ files and run them through clang-format for consistency. A STYLE file has been submitted to R-Forge with details about the coding standards and how they are enforced with Emacs and clang-format.

  • Applied changes suggested by Kevin Ushey to the S4 class handling to support both the currently released Rcpp as well as the currently pending next version.

CRANberries also provides a diff to the previous release 0.3.2. More information is at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

/code/rprotobuf | permanent link

Mon, 16 Dec 2013

RProtoBuf 0.3.2

A new version 0.3.2 of RProtoBuf, is now on CRAN. RProtoBuf provides GNU R bindings for the Google Protobuf data encoding library used and released by Google.

As for the last few releases, Murray took charge of most changes. The NEWS file entry follows:

Changes in RProtoBuf version 0.3.2 (2013-12-15)

  • Fixed a bug that erroneously prevented users from setting raw byte fields in protocol buffers under certain circumstances.

  • Give a user friendly error message when seting an extension to a message of the wrong type instead of causing a C++ check failure that terminates the Rsession.

  • Change object table lookup slightly to allow users to use the <<- operator in code using RProtoBuf without hitting a stop() error in the lookup routine.

  • Add missing enum_type method and improve show method for EnumValueDescriptors.

  • Improve documentation and tests for all of the above.

  • Rewrote tests/ script calling RUnit tests

CRANberries also provides a diff to the previous release 0.3.1. More information is at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

/code/rprotobuf | permanent link

Sun, 15 Sep 2013

RProtoBuf 0.3.1

Two months to the day after the previous 0.3.0 release of RProtoBuf, we are happy to announce a maintenance release 0.3.1 which has now arrived on CRAN. RProtoBuf provides GNU R bindings for the Google Protobuf data encoding library used and released by Google.

Once again, Murray lead this release and added a number improvements. The NEWS file entry follows:

Changes in RProtoBuf version 0.3.1 (2013-09-13)

  • Added support for setting and getting 64-bit integer types as character strings of decimal integers to work around R's lack of native 64-bit integer types.

  • Added better error handling, documentation, and tests to the extensions support (getExtension and setExtension).

  • Add support to P for returning extension descriptors.

  • Improved error messages to include field names when invalid fields are specified to protocol buffer messages with new, update, etc.

  • Improved configure to detect and pass -std=c++0x if it is available to enable long long 64-bit integer support in Rcpp.

  • Improved configure in the case when pkg-config is not available or does not know about the google protocol buffer includes.

  • Replaced newly deprecated Rcpp macros with a simplified macro functionality to avoid warnings on the latest development version of Rcpp.

CRANberries also provides a diff to the previous release 0.3.0. More information is at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

/code/rprotobuf | permanent link

Mon, 15 Jul 2013

RProtoBuf 0.3.0

A new release of RProtoBuf -- now at version 0.3.0 -- arrived earlier on CRAN. RProtoBuf provides GNU R bindings for the Google Protobuf data encoding library used and released by Google.

Murray added a lot of new features which merited the increase in the minor number. The NEWS file entry follows below:

Changes in RProtoBuf version 0.3 (2013-07-13)

  • Added support for Protocol Buffer extensions through the new getExtension and setExtension methods for Message objects, and added top-level extensions to the descriptor pool.

  • Added more user-friendly show() methods for Messages, Descriptors, and other RProtoBuf objects. A common source of confusion on the mailing lists is to confuse Messages and Descriptors. The new show() methods succinctly display more information about e.g. the number of set fields in a message to make it a little clearer to users what type of object they are working with.

  • Added has method for EnumDescriptor objects to test existance of a named constant in an enumeration, and fix a bug in the length method for these objects.

  • Added a number method for EnumValueDescriptor objects to return the number of a named enumeration constant directly.

  • Improved documentation and unit tests.

CRANberries also provides a diff to the previous release 0.2.6. More information is at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

/code/rprotobuf | permanent link

Thu, 04 Oct 2012

RProtoBuf 0.2.6

Release 0.2.6 of RProtoBuf arrived on CRAN earlier this morning. RProtoBuf provides GNU R bindings for the Google Protobuf data encoding library used and released by Google.

This release was once more driven largely by Murray whom we have now added among the list of authors of the package too. The NEWS file entry follows below:

Changes in version 0.2.6 (2012-10-04)

  • Applied several more patches by Murray to

    • correct '_' and '__' mismatches in wrapper calls

    • update a few manual pages for style, and add examples

    • fix a bug where NAs were silently treated as TRUE for logical/bool types

    • fix a bug that caused crashes when adding vectors to optional fields

    • fix bugs in readASCII that returned empty protocol buffers when the file or connection could not be opened

    • distinguish between non-existant and not-set fieldswith has() by returning NULL in the former case.

    • fix a bug that caused non-deterministic behavior when setting a repeated message field in a protobuf to a single Message.

    • add unit tests for all of the above.

  • Added Murray to Authors: field in DESCRIPTION

  • Removed old and unconvincing example on RProtoBuf for storage and serialization in an imagined HighFrequencyFinance context

CRANberries also provides a diff to the previous release 0.2.5. More information is at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

/code/rprotobuf | permanent link

Thu, 09 Aug 2012

RProtoBuf 0.2.5

A new release 0.2.5 of RProtoBuf is now on CRAN. RProtoBuf provides GNU R bindings for the Google Protobuf data encoding library used and released by Google.

This release once again contains a number of patches kindly contributed by Murray Stokely, as well as some updates to conform to CRAN Policy changes.

The NEWS file entry follows below:

Changes in version 0.2.5 (2012-08-08)

  • Applied patches by Murray to

    • correctly deal with nested Protocol Buffer definitions, and also add new unit test for this

    • test a a protocol buffer for missing required fields before serializing it, also add a unit test

    • add a small stylistic fix and examples to the 'add.Rd' manual page

  • Moved inst/doc/ to vignettes/ per newer CRAN Policy

CRANberries also provides a diff to the previous release 0.243. More information is at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

/code/rprotobuf | permanent link

Wed, 16 May 2012

RProtoBuf 0.2.4

A new release 0.2.4 of RProtoBuf is now on CRAN. RProtoBuf provides GNU R bindings for the Google Protobuf data encoding library used and released by Google.

This release once again contains a number of patches kindly contributed by Murray Stokely, as well as an added header file needed to build with the g++ 4.7 version which has become the build standard on CRAN.

The NEWS file entry follows below:

0.2.4   2012-05-15

    o   Applied several patches kindly supplied by Murray Stokely to
         - properly work with repeated strings 
         - correct C++ function naming in a few instances
         - add an example of ascii export/import of messages

    o   Suppport g++-4.7 and stricter #include file checking by adding unistd

    o   Made small improvements to the startup code

CRANberries also provides a diff to the previous release 0.2.3. More information is at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge. Updated to show NEWS rather than ChangeLog

/code/rprotobuf | permanent link

Wed, 13 Apr 2011

RProtoBuf 0.2.3

A maintenance release 0.2.3 of RProtoBuf is now on CRAN. RProtoBuf provides GNU R bindings for the Google Protobuf data encoding library used and released by Google.

The NEWS file entry follows below:

0.2.3   2011-04-12

   o    Protect UINT64 and INT64 with '#ifdef RCPP_HAS_LONG_LONG' which
        itself is valid with either g++ <= 4.4, or newer versions if the 
        -std=c++0x is used

    o   The documentation Makefile now uses the $R_HOME environment variable

    o   The documentation Makefile no longer calls clean in the all target

And courtesy of CRANberries, here is the diff to the previous release 0.2.2:
Diff between RProtoBuf versions 0.2.2 dated 2011-01-12 and 0.2.3 dated 2011-04-13

 RProtoBuf-0.2.2/RProtoBuf/inst/doc/RProtoBuf-quickref.Rnw                     |only
 RProtoBuf-0.2.3/RProtoBuf/ChangeLog                                           |   23 
 RProtoBuf-0.2.3/RProtoBuf/DESCRIPTION                                         |    8 
 RProtoBuf-0.2.3/RProtoBuf/R/with.R                                            |   12 
 RProtoBuf-0.2.3/RProtoBuf/cleanup                                             |    3 
 RProtoBuf-0.2.3/RProtoBuf/inst/NEWS                                           |   10 
 RProtoBuf-0.2.3/RProtoBuf/inst/doc/Makefile                                   |   42 
 RProtoBuf-0.2.3/RProtoBuf/inst/doc/RProtoBuf-quickref.pdf                     |binary
 RProtoBuf-0.2.3/RProtoBuf/inst/doc/RProtoBuf-unitTests.pdf                    |binary
 RProtoBuf-0.2.3/RProtoBuf/inst/doc/RProtoBuf.pdf                              | 1267 +++++-----
 RProtoBuf-0.2.3/RProtoBuf/inst/doc/unitTests-results/RProtoBuf-unitTests.html |   20 
 RProtoBuf-0.2.3/RProtoBuf/inst/doc/unitTests-results/RProtoBuf-unitTests.txt  |   14 
 RProtoBuf-0.2.3/RProtoBuf/src/extractors.cpp                                  |   44 
 RProtoBuf-0.2.3/RProtoBuf/src/mutators.cpp                                    |   55 
 RProtoBuf-0.2.3/RProtoBuf/src/wrapper_FieldDescriptor.cpp                     |   24 
 15 files changed, 819 insertions(+), 703 deletions(-)

As always, there is more information at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

/code/rprotobuf | permanent link

Wed, 12 Jan 2011

RProtoBuf 0.2.2

Thanks to two patches by Murray Stokely, we have a nice new minor release 0.2.2 of RProtoBuf out on CRAN. RProtoBuf provides GNU R bindings for the Google Protobuf data encoding library used and released by Google.

The NEWS file entry follows below:

0.2.2   2011-01-11

   o    Applied two patches by Murray Stokely which 
         - correct a typo preventing some functions from being called, 
         - add bounds checking in call to google protobuf library, 
         - add a new name method 
         - add a tiny unit test for the above
         - add more graceful error handling
         - add support for import patterns other than .proto$, 
         - add simple reading file that does not exist (exception) test
         - add simple repeated field unit test

And courtesy of CRANberries, here is the diff to the previous release 0.2.1:
 ChangeLog                                           |   35 
 DESCRIPTION                                         |    8 
 R/00classes.R                                       |   11 
 R/internals.R                                       |    7 
 R/wrapper_EnumDescriptor.R                          |    6 
 inst/NEWS                                           |   12 
 inst/THANKS                                         |only
 inst/doc/Makefile                                   |    4 
 inst/doc/RProtoBuf-quickref.pdf                     |binary
 inst/doc/RProtoBuf-unitTests.pdf                    |binary
 inst/doc/RProtoBuf.pdf                              | 2362 +++++++++-----------
 inst/doc/unitTests-results/RProtoBuf-unitTests.html |   20 
 inst/doc/unitTests-results/RProtoBuf-unitTests.txt  |   23 
 inst/unitTests/runit.addressbook.R                  |   11 
 inst/unitTests/runit.enums.R                        |only
 inst/unitTests/runit.golden.message.R               |    7 
 inst/unitTests/runit.import.R                       |only
 man/EnumDescriptor-class.Rd                         |   42 
 man/EnumValueDescriptor-class.Rd                    |   23 
 man/readProtoFiles.Rd                               |    3 
 src/DescriptorPoolLookup.cpp                        |   26 
 src/S4_classes.h                                    |    6 
 src/wrapper_EnumDescriptor.cpp                      |   25 
 src/wrapper_EnumValueDescriptor.cpp                 |   25 
 24 files changed, 1357 insertions(+), 1299 deletions(-)

As always, there is more information at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

/code/rprotobuf | permanent link

Mon, 18 Oct 2010

RProtoBuf 0.2.1

A fresh minor release of RProtoBuf, now at version 0.2.1, has appeared earlier today on CRAN. RProtoBuf provides GNU R bindings for the Google Protobuf data encoding library used and released by Google.

This releases extends the recent 0.2.0 release of RProtoBuf with a patch for raw bytes serialization which Koert Kuipers kindly contributed. This helps RProtoBuf for RPC communication where raw bytes are often a preferred form.

As always, there is more information at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

/code/rprotobuf | permanent link

Sat, 11 Sep 2010

RProtoBuf 0.2,0

A brand new and shiny release of RProtoBuf, now at version 0.2.0, arrived on CRAN earlier today. RProtoBuf provides GNU R bindings for the Google Protobuf data encoding library used and release by Google and others.

This is only the second release after 0.1-0 more than six months ago. Given that Rcpp is such a key ingedrient for RProtoBuf, and that Rcpp underwent so many exciting changes itself, Romain and I never got around to releasing new versions of RProtoBuf. This version is now much closer to the actual C++ API and fairly feature rich. We summarised a few of these new things in the presentation at useR! 2010.

There is more information at the RProtoBuf page; there is a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

/code/rprotobuf | permanent link

Wed, 03 Feb 2010

RProtoBuf 0.1-0

Romain uploaded our first release of RProtoBuf to CRAN yesterday. RProtoBuf provides bindings for GNU R to the Google Protobuf implementation. Google Protobuf is (and I quote) a way of encoding structured data in an efficient yet extensible format that is used for almost all internal RPC protocols and file formats at Google.

RProtoBuf had a funny start. I had blogged about the 12 hour passage from proof of concept to R-Forge project following the ORD session hackfest in October. What happened next was as good. Romain emailed within hours of the blog post and reminded me of a similar project that is part of Saptarshi Guha's RHIPE R/Hadoop implementation. So the three of us--Romain, Saptarshi and I---started emailing and before long it becomes clear that Romain is both rather intrigued by this (whereas Saptarshi has slightly different needs for the inner workings of his Hadoop bindings) and was able to devote some time to it. So the code kept growing and growing at a fairly rapid clip. Til that stopped as we switched to working feverishly on Rcpp to both support the needs of this project, and to implement ideas we had while working on this. That now lead to the point where Rcpp is maturing in terms of features, so we will probably have time come back to more work on RProtoBuf to take advantage of the nice templated autoconversions we now have in Rcpp. Oddly enough, the initial blog post seemed to anticipate changes in Rcpp.

Anyway -- RProtoBuf is finally here and it already does a fair amount of magic based of code reflection using the proto files. The Google documentation has a simple example of a 'person' entry in an 'addressbook' which, when translated to R, goes like this:

R> library( RProtoBuf )                      ## load the package
R> readProtoFiles( "addressbook.proto" )     ## acquire protobuf information
R> bob <- new( tutorial.Person,              ## create new object
+   email = "bob@example.com",
+   name = "Bob",
+   id = 123 )
R> writeLines( bob$toString() )              ## serialize to stdout
name: "Bob"
id: 123
email: "bob@example.com"

R> bob$email                                 ## access and/or override
[1] "bob@example.com"
R> bob$id <- 5
R> bob$id
[1] 5

R> serialize( bob, "person.pb" )             ## serialize to compact binary format

There is more information at the RProtoBuf page, and we already have a draft package vignette, a 'quick' overview vignette and a unit test summary vignette.

More changes should be forthcoming as Romain and I find time to code them up. Feedback is as always welcome.

/code/rprotobuf | permanent link