Welcome to post 59 in the R4 series.
How did we get here: A initial words about GitHub.
GitHub Actions
provides (essentially unlimited) compute time. This further boosts a
service already in a market-dominating position: GitHub as a code repository.
Those of us old enough to remember the start of git (the
program and protocol) may remember the extremely bare-bones
initial hosting site repo.or.cz (launched in 2006). GitHub
came two years later, and put an enormous amount of focus into design
and user interfaces. To cut a long story short, GitHub won the services
war. And with it git won the platform war. To a first
approximation, everybody and everything is on GitHub. So
the repository is already dominant. And then free compute
was added.
So given its scale and positioning, and its essentially free
provisioning of free multi-core compute setups with generally decent
connectivity, widespread adoption happened. And as is goes, some
mischief is bound to happen. And it did. More on that below.
A few words about r2u: r2u makes all packages on
CRAN, i.e. the code repository
network for R, install fast, reliably and
easy on Ubuntu by making them available to apt,
the native package manager. It is to our knowledge also the first and
only time an entire open source programming repository is available in
binary form with all dependencies resolved. It is going strongly: the
last monthly use topped
five million packages. See the r2u website for more.
r2u and GitHub: For the first few years,
builds for r2u
were done locally on my machine, and then uploaded to the primary
repositry r2u.stat.illinois.edu. I do not recall systemic outages or
connection issues though occassional network timeouts were seen. Once we
started to support arm64 (in addition to the default amd64) binaries,
building those switched to GitHub Actions simply because … they had
runners for arm64 while I had no arm64 hardware. The experience of
building packages (in bulk) was rather positive. So we investigated
builds for amd64 too. If memory serves we first did this for either one
of the semi-annual BioConductor updates. Before long, builds for amd64
followed meaning all of r2u was being built in GitHub Actions.
During these builds, I would regularly encounter builds failures:
“cannot connect to r2u.stat.illinois.edu”. I misdiagnosed this as a
resource issue on the GitHub side, and consequently made (several)
attempts at robustifying the builds via for example longer (download)
timeout limits as well as checks for build failures and conditional
rebuilds. Needless to say, and given what we know now (more on that
below), this did not work. But it went on for a few months this spring
and summer. What did work was to simply relaunch under ‘re-run failed
jobs’. Given the distributed nature of GitHub Action this generally
allocates to a different machine and address and succeeds. In the grand
scheme of things a nuisance as we a need second run, but given the
fourty (!!) concurrent jobs this tends to be quick. So a minor
nuisance.
This discribed the production side. On the
consumption side, one prominent user of r2u, especially at GitHub,
is our r-ci setup for
continuous integration. It too could fail at times, and a simple re-run
would fix it. Annoying, if addressable manually. Usage by others I
cannot monitor so I can only assume that the random failure nature must
have frustrated them too. Potentially a much bigger nuisance.
As users were getting annoyed, some took action. Jeffrey Girard opened discussion
topic #159 which contained a thorough investigation of his
confirming that only amd64 nodes were affected. This had not been
noticed before. Troy
Hernandez set up a full harness with tests in an ad-hoc repo
designed for repeated remote triggering. This also logged the IP
addresses for success or failure. Through both these approaches it
became (eventually) clear that the failures were limited to either
certain (individual) IP addresses, or IP subnets.
When taking the conversation back to network service at U of
Illinois, we realized that the issue was in fact caused by a network
policy at the university. And specific to GitHub.
In fact, what happened initially were waves of port scanning attacks
originating from GitHub IP addresses. As (essentially) “anybody” can run
code there, bad actors can too. The response from the university side
was reasonable and swift: Identified IP addresses were added to a
‘null-router’ that (essentially) swallows traffic. And that was the
cause of the perceived-as-random outages: Jobs that ended up failing at
GitHub Actions were the ones assigned to addresses that have previously
been seen as port scanning.
Shifting production: Once this was confirmed, I
investiaged alternatives. On the production side using
different machines would help. So I tried blacksmith.sh, a competing
alternate service offering faster runners as ‘drop-in replacements’ for
the GitHub Actions runners. This worked great, until I ran up
against my ‘free cpu minutes quota’. In a mere two days (that were
arguably overly busy as it was shortly after CRAN reopened after the summer
break). Given that the service would not sponsor us a supported open
source software project with sufficient quota, we moved off blacksmith.sh after two days.
A first programmatic response: consumption-side: For
the r-ci client side,
it was straightforward to setup a check and subsequent workaround. When
curl fails with a silent HEAD attempt at the primary
repository failed, we take this to be caused by presence of a
null-router entry for the IP we are on, and switch the apt
setup to the secondary repository. Which may be slower, or at rare times
unreachable itself – but still provides a fine fallback when a node is
‘prohibited’ from talking to U of Illinois resources such as
r2u.stat.illinois.edu. Having used this for a few days in r-ci it seems to work.
A second programmatic response: production-side: For
the r2u builds, and
given that blacksmith.sh would
not grant ‘most-favored status’ with sufficient free minutes, we
switched our Docker-based setup to switch to the secondary when an
initial probe fails. That was added last weekend, and appears to work
just swimmingly. Another application to the fundamental
theorem of software engineering: another layer of indirection can
solve just about any problem.
For completeness, the corresponding code is
webstatus=$(curl --head --silent --no-fail --output /dev/null \
--write-out "%{http_code}" https://r2u.stat.illinois.edu || true)
if test "${webstatus}" = "200"; then
echo "The r2u repository is reachable."
else
extip=$(curl --silent https://ipinfo.io/ip)
echo "::notice::The primary r2u repository is **not reachable** from ${extip}."
fi
We run an initial curl test (without failing) and have
it report the HTTP return code. 200 means no issue, all others are
suspect here—so we run a second curl query to obtain our
external IP and log it. We use the same logic in another spot from
inside the build container and use the else branch to switch
apt to the secondary repository via sed call
on the .sources file.
Logging of ‘bad’ IPs: On both our sides,
i.e. production as well as consumption, we now also log the IP addresses
of the failing nodes and will ask network security to remove these from
the null router. If our jobs can be assigned to them it clearly shows
the machines are part of the normal compute pool and are not
doing anything nefarious at the moment. So they should be removed from
the null-router list. We will see how that fares.
Putting it all together: Providing a free resources
can, sadly, lead to an a decline the service experience just as the tragedy of
the commons analysis would predict. Restricting, or ‘pricing’ use
may be a stock answer but I for one am glad GitHub Actions is still
free. But we need to do our bit of upkeep. Just as network security logs
bad actors (taking advantage of the free resource) we should make an
effort to unlist nodes no longer part of any portscan (or alike)
swarm.
For r-ci users, there is hopefully little to do (if
you rely on the standard
action). We do now catch a node that was assigned a continuous
integration job cannot connect to r2u as we can test this
easily (and cheaply). Pivoting to the secondary repository is a valid,
and working, answer. Hopefully over time we can also work towards
restricting the null-router list down to recent entries and fewer
overall, thereby lowering the chance of gitting a bad IP. Eventually, we
could also overly a CDN proxy to avoid the ‘bad IP’ problem. It is
something to consider.
Summing up: We are still chuffed at how successful
r2u has become, and how
much can be done with GitHub Actions. Sadly, as
we found out, there can also be a ‘tax’ on letting compute happen there
but as discussed in this note, there are ways to avoid it by pivoting to
alternate repository source.
This post by Dirk
Eddelbuettel originated on his Thinking inside the box
blog. If you like this or other open-source work I do, you can now sponsor me at
GitHub.
/code/r4 |
permanent link
