We are thrilled to announce rig 0.11.0 (and 0.10.0), with lots of new functionalities: user mode R installations, R package and project management and much more. This is the third of three blog posts about the new rig release. This post is about managing R projects.

The first post covered user mode R installations, the second focused on R packages.

Full changelog on the rig web site.

Introduction

The new rig proj command lets you manage R projects and virtual environments. An R project lets you share and run R code in a standardized way. rig’s R projects use an rproj.toml manifest, which is still under development.

rig’s R project implementation largely works the same way as other project management tools for Python, Rust, and other languages.

The project lifecycle

A project lifecycle typically involves

1. rig proj init (or rig proj import) to create a new project.
2. rig proj add (and rig proj remove) to manage project dependencies.
3. rig proj lock to resolve how the project is installed over a range of R versions in a range of platforms.
4. rig proj sync to install a project according to the lock file.
5. rig run to execute R code within the context of the project.

Creating projects

There are three ways to create an R project. One is to create a brand new project with rig proj init. This creates a minimal rproj.toml manifest file that you can either edit manually or via rig proj add:

❯ rig proj init
✓ Created rproj.toml
✓ Created .Renviron
✓ Created .gitignore
✓ Created .rvenvlib/rvenv
ℹ Project set up for R 4.6.1. Next: add dependencies to rproj.toml, then
  run `rig proj lock` and `rig proj sync`.

❯ cat rproj.toml
[project]
name = "x"
version = "0.1.0"
type = "project"

[dependencies]
R = ">= 4.6"

Another way is to convert an existing project to an rproj.toml manifest. Start with an existing R project with a DESCRIPTION file and run rig proj import to create an rproj.toml manifest from it. Or start with an renv lock file and call rig proj renv import. rig proj import keeps all fields of DESCRIPTION, including various types of dependencies. For R packages you can use rproj.toml as the source of truth for package metadata and dependencies and (re)generate DESCRIPTION from it as needed, e.g. for CRAN submission.

A third way to create a project is to write an rproj.toml manifest manually.

Project types

There are two types of R manifests: projects and packages. In general we use project as a generic name for both types and consider a package as a specialized project.

For a package, create the R/, src/, etc. directories as usual. See below for a possible way to use the devtools based development tools for a rig project.

For projects rig does not have a suggested way for organizing R and other code, documentation and tests. If in doubt, follow the conventions of R packages. Use [[bin]] to specify executable scripts within the project, relative to the project root. You’ll be able to run these scripts with rig run.

Locking a project

rig proj lock reads rproj.toml and works out which exact packages should be downloaded and installed to satisfy all requirements from rproj.toml. All requirements always include the optional and soft dependencies. The result is a lock file called rproj.lock. This is also a TOML file, typically much bigger than the original rproj.toml project manifest.

Universal lock files

The lock can and usually does include packages for multiple platforms. The exact set of platforms and R versions to include in the lock file can be specified in rproj.toml or at the command line. By default rig includes the current R release and five platforms: the platform of the current machine, macOS arm64, Windows x86_64, generic glibc Linux and source (four platforms if the current platform coincides with one of the others).

A multi-platform lock file means that it is much easier to plan the cross-platform deployment of a project: a team can include all relevant platforms and R versions in the lock file and commit a lock file to version control.

rig also records system requirements in the lock file, for platforms that need them.

Stable URLs

Universal lock files only really work well if they contain stable URLs for binary packages for all platforms and R versions. This is why it is important to use P3M (or another Posit Package Manager instance) with extended metadata about all older package versions and all their corresponding binary builds. With P3M rig is typically able to generate lock files with binary packages for the last 3-4 R versions on all common platforms.

Sticky lock files

Adding new dependencies to a project runs rig proj lock, but it keeps the package versions already in rproj.lock, unless they conflict with the new requirements. This is important to be able to maintain a stable set of package versions while still adding new dependencies. To update an already existing dependency to a newer version, you need to run rig proj lock --upgrade-package. Or run rig proj lock --upgrade to upgrade all packages.

Preferring binary packages

By default rig does not choose an older binary package over a newer source package. You can change that with the prefer-binary setting in rproj.toml. E.g. prefer-binary = true means that rig will choose an older binary package over a newer source package, if the older package also satisfies the dependency requirements.

Excluding newer package versions

You can prevent rig from automatically upgrading to newer package versions by using the exclude-newer setting in rproj.toml. For example, exclude-newer = "2026-09-30" will exclude any package versions that were published after the specified date. This works for P3M’s CRAN and Bioconductor repositories, but not for generic CRAN-like repositories which typically do not have release dates.

Repositories

Projects ignore the global repository settings configured via rig repos. By default projects use P3M’s CRAN and Bioconductor repositories. This can be configured with [[repository]] entries in rproj.toml. You can turn off P3M’s CRAN and Bioconductor repositories and you can also add your own custom repositories.

rig’s solver and universal lock files work best with P3M’s CRAN and Bioconductor metadata, so we strongly recommend using them whenever possible.

Syncing a project

rig proj sync installs the packages listed in rproj.lock into the project library in .rvenv/lib, inside the project’s virtual environment. It locks first if the project manifest has changed since creating the lock file.

If the current project is a package (type = "package"), then rig proj sync will also install the package itself into the project library.

rig proj sync also installs the R version requested in the lock file, if it is not already installed. This makes it very easy to deploy a project in an empty container or virtual machine: only rig and the lock file are needed and rig installs everything required for the project: R, system requirements and R packages.

rig proj sync installs the hard requirements and the dev dependency group by default. For a production deployment you can use the --no-dev option to skip installing development dependencies. Use --group and --extra to install optional dependency groups and extra dependencies.

Fast and lean libraries

rig clones the package files when installing packages into the project library, whenever possible. This means that installation is very fast and space-efficient. Having the same package version installed for ten projects on the same disk only takes up the space of a single copy.

The virtual environment

A rig project has two kinds of files. rig proj init creates the ones that belong in version control: rproj.toml, .Renviron, .rvenvlib/rvenv and .gitignore. Commit these, together with rproj.lock, and a fresh clone of the project works after a rig proj sync, or right away with rig run.

rig proj sync creates the rest in the .rvenv directory: the project library in .rvenv/lib, R and Rscript wrapper scripts in .rvenv/bin, and some configuration files. .rvenv is specific to the machine, it is in .gitignore, and you can delete it any time, rig proj sync creates it again.

The .Renviron file and the small rvenv package in .rvenvlib make sure that R uses the project library whenever it starts in the project directory: from a terminal, or from RStudio, Positron or VS Code. You do not need to activate the project explicitly, although .rvenv/bin has activation scripts for the common shells if you prefer that.

Inside the project R only sees the project library, the tools library (see below) and the packages that come with R. Packages from your user or site library do not leak into the project.

If the project library is out of date compared to rproj.lock, e.g. after a git pull, R prints a warning at startup, and rig proj status tells you what changed. If you start R with rig run in the project directory, then it automatically runs rig proj sync as needed. If you use Positron, VS Code, RStudio or another IDE, then R will warn at startup if the virtual environment is out of date and suggest that you run rig proj sync and restart R.

Managing dependencies

R package dependencies are specified in the rproj.toml manifest under three types of tables. The [dependencies] table includes packages that are always installed. These are considered as R packages required for the project. Packages in [dependency-groups.<name>] are groups of development dependencies and they are only installed if requested, except for the dev group which is installed by default. For example, the packages that are needed for building the pkgdown website can go into a [dependency-groups.website] table. dependency-groups is similar to Config/Needs/<name> in DESCRIPTION files of R packages. Dependencies of optional features go into [optional-dependencies.<name>] tables and are only installed if requested. For example, if your package has functions for visualization, but these are not required for the core functionality, you may put them in an [optional-dependencies.viz] table and other packages using yours may opt in to the viz feature. (The infrastructure for depending on optional features in packages is not built yet, and package repositories need to support it as well.)

You can edit the rproj.toml manifest manually to add dependencies or use rig proj commands to do it. rig proj add adds packages to rproj.toml, then updates rproj.lock and installs the new packages into the project library:

❯ rig proj add dplyr cli

❯ cat rproj.toml
[project]
name = "x"
version = "0.1.0"
type = "project"

[dependencies]
R = ">= 4.6"
cli = "*"
dplyr = "*"

Without a version requirement rig adds a package as "*", any version. The exact version is recorded in rproj.lock, so you only need a version requirement if the project really needs a certain version. Version requirements use the same syntax as for rig pkg install, e.g. 'cli@>= 3.6' or 'dplyr@^1.1.0'. rig proj remove removes packages.

Use --dev to add a development dependency, e.g. testthat. These go into the [dependency-groups.dev] table.

Packages from GitHub, GitLab, git repositories, URLs and local paths use the same syntax as rig pkg install, see the second post. Bioconductor or other packages coming from the configured repositories work the same way as CRAN packages. To pin a package to a certain repository, e.g. an R universe, use --with-repos:

❯ rig proj add --with-repos rlib=https://r-lib.r-universe.dev cli

This adds the repository to rproj.toml and pins cli to it:

[dependencies]
cli = { version = "*", repository = "rlib" }

[[repository]]
name = "rlib"
url = "https://r-lib.r-universe.dev"

If rig cannot resolve the new dependencies, e.g. because of a typo in a package name, it restores rproj.toml.

Running a project

rig run starts R in the project’s environment, from anywhere inside the project. It locks and syncs the project first, if needed, so day to day you mostly need rig proj add, rig proj remove and rig run.

rig run script.R runs a script in the project’s environment. Arguments after the script name go to the script, and commandArgs(TRUE) picks them up. Use --no-project to run the script without the project.

A project can give names to its scripts, in [[bin]] tables of rproj.toml:

[[bin]]
name = "report"
path = "scripts/report.R"
description = "Build the report"

Then rig run report --format pdf runs scripts/report.R with the --format pdf arguments, and rig run --list lists all named scripts of the project.

You can also run the .rvenv/bin/R and .rvenv/bin/Rscript wrappers directly, or start R from your IDE in the project directory, see the virtual environment above.

Package development with devtools

To develop an R package as a rig project, use type = "package" in rproj.toml, or run rig proj import on an existing package. rig proj sync then installs the package itself into the project library, and creates DESCRIPTION from rproj.toml (if missing), so R CMD build, R CMD check and other tools keep working.

Development tools like devtools, usethis and roxygen2 are not dependencies of the package, so they should not be in rproj.toml. rig has a special global package library for them, called tools, and adds it to the library path of every project. Install your tools there once:

❯ rig pkg install -l tools devtools usethis roxygen2

and then they are available in all projects:

❯ rig run
> devtools::load_all()
> devtools::document()
> devtools::test()

Packages that the package’s tests and documentation need, e.g. testthat or knitr, are different. Add these with rig proj add --dev, so they are locked together with the other dependencies.

Inline scripts

Sometimes a project is just a single R script. For these rig supports inline dependencies: the script declares the R version and the packages it needs in a special comment block with a # /// script header and a # /// footer:

# /// script
# [dependencies]
# R = ">= 4.4"
# cli = "*"
# dplyr = ">= 1.1"
# ///

library(dplyr)
cli::cli_text("Hello from {.pkg cli}!")

Without the leading # characters, the block is the same TOML as in rproj.toml.

rig run script.R installs the right R version if needed, installs the packages into a separate environment in rig’s cache, and then runs the script. Your own package libraries do not change. The next run reuses the environment, so it starts right away. You can share a single .R file, and it runs the same way on every machine that has rig.

You can write the block by hand, or let rig manage it:

❯ rig proj init --script script.R
❯ rig proj add --script script.R cli dplyr
❯ rig proj remove --script script.R dplyr

To pin the exact R and package versions, lock the script:

❯ rig proj lock --script script.R

This writes script.R.lock next to the script. Keep the two files together, and rig run will use the versions from the lock file.

On macOS and Linux, start the script with #!/usr/bin/env -S rig run and make it executable, to run it like any other command. On Windows, rig system script-assoc lets you run .R files from cmd and PowerShell. See the scripts guide for more.

Caches

rig caches the files it downloads: repository metadata, package files, packages it built from source, git repositories of git and GitHub dependencies, and the environments of inline scripts. rig pkg, rig proj and rig run share the same cache, and so do all projects, so rig downloads or builds a package only once. rig refreshes the repository metadata once a day by default.

rig cache info shows how much space the cache takes, by category, and rig cache clean deletes it, or only one category of it, with --category:

❯ rig cache info
❯ rig cache clean --category scripts

Use --no-cache to skip the cache for a single command, and rig system dirs --cache to see where the cache is.

Your feedback is welcome!

We would love to hear your thoughts, suggestions, and any issues you encounter while using rig. Please use the issue tracker for bug reports and feature requests, and the discussion forum or Posit community for general questions and discussions.

We are thrilled to announce rig 0.11.0 (and 0.10.0), with lots of new functionalities: user mode R installations, R package and project management and much more. This is the second of three blog posts about the new rig release. This post is about managing R packages.

The first post covered user mode R installations. The third post focuses on project management.

Full changelog on the rig web site.

Introduction

The new rig pkg command lets you manage R packages: installing, removing, and querying package dependencies. It supports CRAN-like repositories, packages in git repositories (including on GitHub), URLs pointing to package files and packages in local files and directories.

rig pkg is aimed at R users and developers who use a single (or a few) package libraries. If you prefer to use project-specific libraries, you probably want to use rig proj instead.

Managing installed packages

rig pkg list lists installed packages in a library. rig pkg remove removes a package from a library.

❯ rig pkg list
559 packages (R 4.6.1, main: /Users/gaborcsardi/Library/R/arm64/4.6/library)

Package              Version      Built   Platform                   Source
----------------------------------------------------------------------------------------------------------------
abind                1.4-8        4.6.0   -                          RSPM
ade4                 1.7-24       4.6.0   aarch64-apple-darwin23     CRAN
adegenet             2.1.11       4.6.0   aarch64-apple-darwin23     CRAN
AdmixPoly            1.0.1        4.6.0   aarch64-apple-darwin20     RSPM
affy                 1.90.0       4.6.1   aarch64-apple-darwin23     RSPM
affyio               1.82.0       4.6.0   aarch64-apple-darwin20     RSPM
[...]

rig pkg commands use the current default R version (as in rig default), unless the -r or --r-version option specifies otherwise. They use the default library (as in rig library default), unless the -l or --library option specifies otherwise.

rig pkg doctor checks the health of your package library, identifying any issues with installed packages or their dependencies: missing dependencies, dependency conflicts, ABI conflicts, etc. It also tells you how to fix these issues.

❯ rig pkg doctor
3 problems (1 error, 2 warnings) in 559 packages (R 4.6.1, main: /Users/gaborcsardi/Library/R/arm64/4.6/library)

Package              Version   Severity   Problem   Details
-------------------------------------------------------------------------------------------------------------
00LOCK-dress.graph   -         warning    lock      lock directory left behind by an interrupted installation
dress.graph          -         warning    broken    no DESCRIPTION file
missoNet             1.5.1     error      missing   Imports ComplexHeatmap, not installed

To fix the errors, run:
  rig pkg install ComplexHeatmap

To remove the leftover lock directories, if no installation is running, run:
  rm -rf /Users/gaborcsardi/Library/R/arm64/4.6/library/00LOCK-dress.graph

To remove the broken packages, run:
  rm -rf /Users/gaborcsardi/Library/R/arm64/4.6/library/dress.graph

✗ Found 1 error in /Users/gaborcsardi/Library/R/arm64/4.6/library

Installing packages

rig pkg install can install R packages from CRAN-like repositories, git repositories, GitHub, URLs, and local paths. It uses the default R version and the default package library (see below about libraries), unless otherwise specified. It caches metadata and package files heavily, avoids redundant downloads and installations, and performs downloads and package builds in parallel. It uses the host OS’s file cloning capabilities to efficiently copy files when installing packages.

If you used the pak R package to install R packages previously, rig pkg will probably feel familiar, except that you can run it from the command line instead of from within R.

This is what installing a package into an empty library looks like, if the package files are already cached:

❯ rig pkg install dplyr
▶ Downloading binary package metadata
▶ Solving dependencies
✓ Solved dependencies
15 of 15 packages to install (R 4.6.1, test: /Users/gaborcsardi/Library/R/arm64/4.6/library/__test)
Package     Version  Type    Action   Reason
cli         3.6.6    binary  install  not installed
dplyr       1.2.1    binary  install  not installed
generics    0.1.4    binary  install  not installed
glue        1.8.1    binary  install  not installed
lifecycle   1.0.5    binary  install  not installed
magrittr    2.0.5    binary  install  not installed
pillar      1.11.1   binary  install  not installed
pkgconfig   2.0.3    binary  install  not installed
R6          2.6.1    binary  install  not installed
rlang       1.3.0    binary  install  not installed
tibble      3.3.1    binary  install  not installed
tidyselect  1.2.1    binary  install  not installed
utf8        1.2.6    binary  install  not installed
vctrs       0.7.3    binary  install  not installed
withr       3.0.3    binary  install  not installed

▶ Downloading 15 packages
✓ Cached: cli
✓ Cached: dplyr
✓ Cached: generics
✓ Cached: glue
✓ Cached: lifecycle
✓ Cached: magrittr
✓ Cached: pillar
✓ Cached: pkgconfig
✓ Cached: R6
✓ Cached: rlang
✓ Cached: tibble
✓ Cached: tidyselect
✓ Cached: utf8
✓ Cached: vctrs
✓ Cached: withr
Complete: 0 downloaded, 15 cached [========================================] 15/15 packages

▶ Installing 15 packages.
Installed R6 2.6.1
Installed cli 3.6.6
Installed rlang 1.3.0
Installed magrittr 2.0.5
Installed pkgconfig 2.0.3
Installed glue 1.8.1
Installed withr 3.0.3
Installed utf8 1.2.6
Installed generics 0.1.4
Installed lifecycle 1.0.5
Installed vctrs 0.7.3
Installed tidyselect 1.2.1
Installed pillar 1.11.1
Installed tibble 3.3.1
Installed dplyr 1.2.1
✓ Installed all 15 packages successfully
✓ Installed 15 packages (R 4.6.1, test: /Users/gaborcsardi/Library/R/arm64/4.6/library/__test)

Version constraints

rig pkg install supports version constraints for packages. Request a specific version of a package with @==<version>:

❯ rig pkg install ggplot2@==4.0.2

If you are familiar with ~ and ^ version constraints from other languages, you can use them with rig pkg install as well. E.g. to install the latest version of the ggplot2 3.x.y series, you would run:

❯ rig pkg install ggplot2@^3.0.0

Dependency solver

The rig dependency solver makes sure that all package dependencies are resolved correctly before installation. If the combination of packages cannot be installed on the specified R version or platform, rig will report an error and refuse to proceed with the installation. For example:

❯ rig pkg install -r 4.0.5-x86_64 'testthat@>=3.3.0'
▶ Downloading binary package metadata
▶ Solving dependencies
✗ Cannot resolve dependencies for R 4.0.5 / x86_64-apple-darwin:
  Because testthat >=3.3.0 depends on R >=4.1.0 and you requested
  testthat >=3.3.0, the requested packages cannot be installed.

Packages from git, GitHub, URLs, paths

If you are familiar with pak’s syntax for installing packages from various non-CRAN sources, you can use the same syntax with rig. E.g. a GitHub package is simply username/repo. You can use the usual syntax for tags, branches, pull requests:

❯ rig pkg install r-lib/cli
❯ rig pkg install r-lib/cli@badcafe
❯ rig pkg install r-lib/cli@some-branch
❯ rig pkg install r-lib/cli#41

Prefix packages in git repositories with git::. To install a package from a URL simply use the full URL. For a local path use a relative path starting with ./, ../ or ~/ or an absolute path:

❯ rig pkg install .
❯ rig pkg install ./mypkg
❯ rig pkg install ../mypkg
❯ rig pkg install ~/works/mypkg
❯ rig pkg install /opt/src/mypkg
❯ rig pkg install ./mypkg_1.0.0.tar.gz

Preferring binary packages

rig prefers to install the binary package for a certain package version if it is available, instead of building and potentially compiling it from source. By default it does not choose an older version over a newer one just because a binary is available for the older version. The --prefer-binary option changes this. Use it if you prefer older binary packages over newer source packages:

❯ rig pkg install --prefer-binary tidyverse

Installing dependencies

To install the dependencies of a package, without installing the package itself, use --no-install-project. This is especially useful to install all dependencies of a package tree in the current working directory. The second variant also installs soft (development) dependencies:

❯ rig pkg install --no-install-project .
❯ rig pkg install --no-install-project --dev .

Repositories

rig pkg subcommands always use the configured repositories, see the rig repos command. They also have --with-repos and --without-repos options to include or exclude specific repositories when performing a single package operation, without modifying the configured repositories. E.g. to query the version of a package on CRAN and P3M you can run:

❯ rig pkg info otel --without-repos --with-repos=p3m --versions
otel — 2 versions (latest 0.2.0)

Version   Published    R          Deps
-------------------------------------------------------
0.1.0     2025-07-31   >= 3.6.0      0
0.2.0     2025-08-29   >= 3.6.0      0   ← latest

❯ rig pkg info otel --without-repos --with-repos=cran --versions
otel — 1 version (latest 0.2.0)

Version   Published   R          Deps
-------------------------------------------------------
0.2.0     ?           >= 3.6.0      0   ← latest

Package queries

rig pkg info prints information about a package, by default about the latest version:

❯ rig pkg info tidymodels
tidymodels 1.5.0 (RSPM)
Easily Install and Load the 'Tidymodels' Packages

The tidy modeling "verse" is a collection of packages for modeling and
statistical analysis that share the underlying design philosophy, grammar, and
data structures of the tidyverse.

Maintainer    Max Kuhn <max@posit.co>
License       MIT + file LICENSE
Published     2026-04-23 06:20:02 UTC
URL           https://tidymodels.tidymodels.org,
              https://github.com/tidymodels/tidymodels
BugReports    https://github.com/tidymodels/tidymodels/issues
Compilation   no

Depends       R (>= 4.1)
Imports       broom (>= 1.0.12), cli (>= 3.6.6), conflicted (>= 1.2.0), dials
              (>= 1.4.3), dplyr (>= 1.2.1), ggplot2 (>= 4.0.3), hardhat (>=
              1.4.3), infer (>= 1.1.0), modeldata (>= 1.5.1), parsnip (>=
              1.5.0), purrr (>= 1.2.2), recipes (>= 1.3.2), rlang (>= 1.2.0),
              rsample (>= 1.3.2), rstudioapi (>= 0.18.0), tailor (>= 0.1.0),
              tidyr (>= 1.3.2), tune (>= 2.1.0), workflows (>= 1.3.0),
              workflowsets (>= 1.1.1), yardstick (>= 1.4.0)
Suggests      covr, glue, knitr, rmarkdown, testthat (>= 3.0.0), xml2

rig pkg info --versions lists all versions of a package. This only works with P3M repositories because it needs extended metadata that plain CRAN-like repositories typically do not have:

❯ rig pkg info tidymodels --versions
tidymodels — 17 versions (latest 1.5.0)

Version   Published    R        Deps
-------------------------------------------------------
0.0.1     2018-07-27   >= 3.1     18
0.0.2     2018-11-27   >= 3.1     20
0.0.3     2019-10-04   >= 3.1     20
0.1.0     2020-02-16   >= 3.1     22
0.1.1     2020-07-14   >= 3.1     20
0.1.2     2020-11-22   >= 3.1     19
0.1.3     2021-04-19   >= 3.1     20
0.1.4     2021-10-01   >= 3.1     21
0.2.0     2022-03-19   >= 3.4     21
1.0.0     2022-07-13   >= 3.4     21
1.1.0     2023-05-01   >= 3.4     21
1.1.1     2023-08-24   >= 3.4     21
1.2.0     2024-03-25   >= 3.6     21
1.3.0     2025-02-21   >= 3.6     21
1.4.0     2025-09-05   >= 4.1     21
1.4.1     2025-09-08   >= 4.1     21
1.5.0     2026-04-23   >= 4.1     21   ← latest

rig pkg deps prints the dependencies of a package. rig pkg tree does the same, but as a dependency tree. You can use these commands to query non-CRAN-like packages. E.g. the dependency tree of the package in the current working directory is

❯ rig pkg tree --dev .
otelsdk 0.2.4.9000 (local: /Users/gaborcsardi/works/otelsdk) — 14 direct, 51 total
├── otel 0.2.0
│   └── R (>= 3.6.0) [D]
└── utils
[Suggests]
├── callr 3.8.0
│   ├── R (>= 3.4) [D]
│   ├── otel 0.2.0 (>= 0.2.0) (*)
│   ├── processx 3.9.0 (>= 3.6.1)
│   │   ├── R (>= 3.4.0) [D]
│   │   ├── ps 1.9.3 (>= 1.9.3)
│   │   │   ├── R (>= 3.4) [D]
│   │   │   └── utils
│   │   ├── R6 2.6.1
│   │   │   └── R (>= 3.6) [D]
│   │   └── utils
│   ├── R6 2.6.1 (*)
│   └── utils
├── cli 3.6.6
│   ├── R (>= 3.4) [D]
│   └── utils
├── curl 8.0.0
│   └── R (>= 3.0.0) [D]
[...]

The --dev flag includes development dependencies as well.

rig pkg search searches all CRAN packages using R-hub package search service:

❯ rig pkg search palette --short
- "palette" ----------------------------------- 177 packages in 0.009 seconds -
 #     package      version  by                     @ title
 1 100 RColorBrewer 1.1-3    Erich Neuwirth        5y ColorBrewer Palettes
 2  46 colorspace   2.1-3    Achim Zeileis         3M A Toolbox for Manipula...
 3  23 paletteer    1.7.0    Emil Hvitfeldt        9M Comprehensive Collecti...
 4  16 ggsci        5.2.0    Nan Xiao              2M Scientific Journal and...
 5  13 wesanderson  0.3.7    Karthik Ram           3y A Wes Anderson Palette...
 6   8 rcartocolor  2.1.2    Jakub Nowosad         1y 'CARTOColors' Palettes
 7   8 pals         1.10     Kevin Wright          2y Color Palettes, Colorm...
 8   8 scico        1.5.0    Thomas Lin Pedersen   3y Colour Palettes Based...
 9   6 palette      0.0.3    Christopher T. Kenny  1y Color Scheme Helpers
10   6 MetBrewer    0.2.0    Blake Robert Mills    5y Color Palettes Inspire...
11   6 palr         0.5.0    Michael D. Sumner     2M Colour Palettes for Data
12   6 blends       0.1.2    David Hodge           3M Blend Colours and Pale...
13   6 biscale      1.1.0    Christopher Prener    1y Tools and Palettes for...
14   5 palettes     0.2.2    Michael McCarthy      1y Methods for Colour Vec...
15   5 PrettyCols   1.1.0    Nicola Rennie         2y Pretty Colour Palettes
16   5 gifski       1.32.0-2 Jeroen Ooms           2y Highest Quality GIF En...
17   5 wacolors     0.3.1    Cory McCartan         5y Colorblind-Friendly Pa...
18   4 colourpicker 1.3.0    Dean Attali           3y A Colour Picker Tool f...
19   4 amerika      0.1.1    Philip Waggoner       1y American Politics-Insp...
20   4 makePalette  0.1.2    Jorge L. C. Musaja    3y Make Palette

Next 20 results: rig pkg search palette --from 21 --short

Package libraries

rig creates a user-level package library for each R installation by default. (See the rig FAQ why.) This is the package library that rig pkg commands use by default.

The rig library command supports creating other package libraries, still stored in the user’s home directory. You can refer to them by name and make them the default library for rig pkg commands with rig library default.

For example to create a temporary library and install the Shiny package from GitHub into it, including development dependencies, you can run:

rig library add test
rig library default test
rig pkg install --dev rstudio/shiny

You can also use the -l or --library option of the rig pkg commands to change the package library temporarily for one command, without changing the default.

If you are using the macOS menu bar app, then you can also manage package libraries through the app’s interface.

System requirements

On certain Linux systems rig supports automatic system requirements installation. These are currently:

• Ubuntu Linux
• RedHat Enterprise Linux and clones
• openSUSE Linux and SUSE Linux Enterprise
• Debian Linux
• Fedora Linux
• Alpine Linux

For the automatic system requirements installation to work, rig must run as the root user or with password-less sudo privileges to call the system package manager.

Note that in user mode, rig installs the self-contained manylinux R build, which does not require any system-level dependencies and also sets up Posit’s manylinux package repository, which has self-contained binaries for most CRAN and Bioconductor packages. So in user mode you typically do not need to worry about system-level dependencies.

Your feedback is welcome!

We would love to hear your thoughts, suggestions, and any issues you encounter while using rig. Please use the issue tracker for bug reports and feature requests, and the discussion forum or Posit community for general questions and discussions.

We are thrilled to announce rig 0.11.0 (and 0.10.0), with lots of new functionalities: user mode R installations, R package and project management and much more. This is the first of three blog posts about the new rig release. This post is about managing R installations and the new user mode. The second post shows how rig can help with managing R packages, and the third focuses on project management.

Full changelog on the rig web site.

New Web Site

rig has a new web site at https://rig.r-lib.org! See this web site for installation instructions, including the new user mode install; reference documentation (also in rig help!), guides about larger topics and tutorials to get started.

User Mode R Installations

rig can now install and manage R installations without requiring administrative privileges. rig calls this user mode. In user mode, R installations are installed in the user’s home directory rather than system-wide. System-wide installations are still the default currently, and they are called admin mode.

You may have both user mode and admin mode installations on one computer, but each rig command runs in either user mode or admin mode. The default is admin mode. See below for how to change it. Use --admin or --user to pick the mode for a single command, e.g. rig ls --admin always lists admin mode installations, and rig ls --user always lists user mode ones.

Advantages of using user mode:

• No admin: no need for administrative privileges.
• All Linux: works on all glibc (from glibc 2.34) and musl (from musl 1.2) based Linux systems.
• Multiple patch versions: on macOS you can install multiple patch versions side by side.
• Convenience: no need to use sudo or an admin account to install and manage R.

User mode is quite new, though. It is much less tested than admin mode, and you may encounter bugs or unexpected behavior more frequently. On Linux, the generic distro-agnostic builds may be slower for numerical work than builds specifically compiled for the distribution.

User mode (admin mode as well, actually) is slightly different on each operating system. Here is a table about where rig stores its files on each OS in both modes.

You can call the new rig system dirs command to see where rig stores its various files:

❯ rig system dirs
Mode                  user
Architecture          arm64
R root                /Users/gaborcsardi/.local/share/rig/r
Binary dir            /Users/gaborcsardi/.local/bin
Config file           /Users/gaborcsardi/Library/Application Support/com.gaborcsardi.rig/config.json
Data dir              /Users/gaborcsardi/Library/Application Support/com.gaborcsardi.rig
Cache dir             /Users/gaborcsardi/Library/Caches/com.gaborcsardi.rig
Download dir          /var/folders/h9/6ct_py4s319fxt88dxn506l00000gp/T/rig-502
Logs dir              /Users/gaborcsardi/Library/Logs/com.gaborcsardi.rig
Project library root  (in-project, .rvenv/lib)

The same in admin mode:

❯ rig system dirs --admin
Mode                  admin
Architecture          arm64
R root                /Library/Frameworks/R.framework/Versions
Binary dir            /usr/local/bin
Config file           /Users/gaborcsardi/Library/Application Support/com.gaborcsardi.rig/config.json
Data dir              /Users/gaborcsardi/Library/Application Support/com.gaborcsardi.rig
Cache dir             /Users/gaborcsardi/Library/Caches/com.gaborcsardi.rig
Download dir          /var/folders/h9/6ct_py4s319fxt88dxn506l00000gp/T/rig-502
Logs dir              /Users/gaborcsardi/Library/Logs/com.gaborcsardi.rig
Project library root  (in-project, .rvenv/lib)

macOS

On arm64 macOS rig can install both arm64 and x86_64 builds of R. The latter ones need Rosetta to run. User mode has a different, simpler naming scheme than admin mode on macOS, matching Linux and Windows. This is an example list of user mode R installations:

❯ rig ls
* name          version    aliases
------------------------------------------
  3.6.3-x86_64  (R 3.6.3)
  4.0.5-x86_64  (R 4.0.5)
  4.1.3
  4.1.3-x86_64  (R 4.1.3)
  4.2.3
  4.3.3
  4.4.3
  4.5.3                    oldrel
  4.5.3-x86_64  (R 4.5.3)  oldrel-x86_64
  4.6.0
* 4.6.1                    release
  4.6.1-x86_64  (R 4.6.1)  release-x86_64
  devel         (R 4.7.0)
  devel-x86_64  (R 4.7.0)
  next          (R 4.6.1)
  next-x86_64   (R 4.6.1)

• The native (arm64) builds are named simply with the version number.
• R-devel is simply named devel and the next version of R (R-patched, R-alpha, etc.) is named next.
• The x86_64 builds are suffixed with -x86_64.
• You can use the release alias for the most recent version of R and the oldrel alias for the previous minor release branch. E.g. the R-release quick link starts the latest version of R.

Windows

On Windows rig installs CRAN’s R builds as a regular user in user mode. This installation writes the user’s registry settings instead of the system-wide registry settings.

The names of the R installations are the same as for admin mode. rig now supports installing x86_64 and aarch64 R builds on aarch64 Windows, and similarly to macOS, the non-native builds get a -x86_64 suffix:

PS C:\Users\Gabor Csardi> rig ls
* name          version    aliases
------------------------------------------
  4.4.3
  4.4.3-x86_64  (R 4.4.3)
  4.5.3
* 4.6.1                    release
  devel         (R 4.7.0)
  next          (R 4.6.1)

The new rig rtools command lets you manage Rtools installations. In user mode they are installed into the user’s roaming profile directory, and rig configures R and Rtools to work together correctly. On aarch64 Windows, you can install both x86_64 and aarch64 versions of Rtools.

PS C:\Users\Gabor Csardi> rig rtools ls
name  version  full-version   arch     path
------------------------------------------------------
40    4.0      4.0.3.0        x86_64   C:\Users\Gabor Csardi\AppData\Roaming\rig\data\rtools\40-x86_64
44    4.4      4.4.6459.6401  aarch64  C:\Users\Gabor Csardi\AppData\Roaming\rig\data\rtools\44
45    4.5      4.5.6768.6492  aarch64  C:\Users\Gabor Csardi\AppData\Roaming\rig\data\rtools\45

Linux

On Linux rig installs a portable (manylinux) R build in user mode. These builds are self-contained, generic glibc (or musl) builds of R. These builds are intended to work on a wide range of Linux distributions. However, they are not as battle-tested as the builds rig uses in admin mode.

On glibc systems rig also configures the manylinux repository for CRAN, that pairs very well with the portable R builds. These packages are self-contained and compatible with most glibc-based Linux distributions:

> getOption("repos")
                                                         P3M-manylinux
"https://packagemanager.posit.co/cran/__linux__/manylinux_2_28/latest"
                                                                  CRAN
                                         "https://cloud.r-project.org"

Positron and RStudio configuration

rig automatically configures RStudio (when running rig add and rig default), so that the rig-default R version is also the default in RStudio. On macOS and Windows this is automatic. On Linux rig puts the rig binary directory (see the table above) on the PATH in user mode, and it is generally on the PATH in admin mode. This is enough to ensure that the rig-default R version is also the default in RStudio on Linux.

Positron does not have a generic default R version like RStudio does, but you can choose a default for each workspace. rig (in rig add and rig default) updates your Positron configuration (the current user’s config file) to make sure that Positron finds all user-mode R installations.

Switching to user mode

Call rig system user-mode to switch a previous setup from admin mode to user mode. This will reinstall your current R installations in user mode, and remove the admin mode installation. (But see its options to adjust this!)

See more about the migration at the rig homepage.

Portable Linux

Above you have seen how rig installs portable manylinux builds in user mode. It is also possible to install them in admin mode. This can be useful for Linux systems that rig does not have native builds for. To install a portable Linux build use --platform portable:

rig add --platform portable 4.6

Moreover, if rig does not support a Linux distribution natively, it will install the manylinux (or musllinux) builds instead, assuming that they are supported, even in admin mode. With this, most Linux distributions work automatically, in both admin and user modes.

Repository management

The rig release improved the rig repos subcommand substantially. These are the biggest changes.

P3M by default

rig now configures Posit Public Package Manager (P3M) as a default repository, on all platforms except aarch64 Windows. (Because P3M would serve incompatible x86_64 binaries here.) This is because we found P3M more reliable and because P3M includes older package versions and has more extensive metadata.

Nevertheless you can opt out with rig add --without-repos=p3m ... at installation time, or with rig repos setup --without-repos=p3m at any time.

Custom repositories

You can now add custom repositories to your rig configuration using rig repos add <name> <url>. For example:

rig repos add myrepo https://myrepo.example.com

You can then turn these (and also the built-in) repositories on or off using rig repos enable and rig repos disable.

Improved repository list

The output of rig repos available is now more informative and easier to read. Type rig repos available <reponame> to get more information about a repository.

❯ rig repos available
8 repositories

Name              Default   Type       Title
-----------------------------------------------------------------------------
P3M               depends   built-in   Posit Public Package Manager
P3M-manylinux     depends   built-in   Posit Package Manager manylinux portable packages
RHUB              no        built-in   R-Hub repositories
r-universe/cran   depends   built-in   R-universe CRAN mirror
r-universe/bioc   no        built-in   R-universe Bioconductor mirror
CRAN              yes       built-in   The Comprehensive R Archive Network
CRAN-archive      depends   built-in   CRAN archive
Bioconductor      no        built-in   Open source software for Bioinformatics

Use `rig repos available <name>` to see a repository's URLs.
`depends`: a default only on some platforms, architectures or R versions.

❯ rig repos available r-universe/cran
r-universe/cran (1 URL)
R-universe CRAN mirror

R-universe's mirror of CRAN. The only repository serving arm64 Windows binary
packages.

Default      only on platforms matching aarch64-*-mingw32

URL          https://cran.r-universe.dev
Platforms    windows, *-mingw32
Archs        aarch64

Repository status

Use rig repos status to check the status of all enabled repositories.

❯ rig repos status
6 repositories of R 4.6.1, index bin/macosx/sonoma-arm64/contrib/4.6

name              ping   status        types              updated            url
-----------------------------------------------------------------------------
P3M             601 ms   ok            source, win, mac   -                  https://packagemanager.posit.co/cran/latest
BioCsoft        233 ms   ok            source, win, mac   2026-10-01 17:08   https://bioconductor.org/packages/3.23/bioc
BioCann         120 ms   source only   source, win, mac   2026-08-01 16:49   https://bioconductor.org/packages/3.23/data/annotation
BioCexp         111 ms   source only   source, win, mac   2026-09-15 20:32   https://bioconductor.org/packages/3.23/data/experiment
BioCworkflows   111 ms   source only   source, win, mac   2026-08-01 16:39   https://bioconductor.org/packages/3.23/workflows
BioCbooks       120 ms   source only   source, win, mac   2026-08-01 16:25   https://bioconductor.org/packages/3.23/books

`ping` is the time the repository took to answer for the index above.
`types` is what the R installation's `repositories` file declares.
`source only`: this repository has no index for this platform and R version.

Other improvements

Avoid unneeded reinstalls

rig add now does not reinstall an R version if it is already installed. Use --reinstall to force a reinstall.

.exe quick links on Windows

On Windows rig now uses .exe quick links instead of .bat files. .bat files have a lot of issues with passing arguments, .exe files work much better.

Run R with R on Windows

The new rig system fix-r-alias subcommand deletes the R alias from PowerShell, so you can now use R to start R.

rig is now signed on Windows

Thanks to the SignPath Foundation, the rig executables and the rig installer are all signed now.

rig system blas

The new rig system blas command lets you manage the BLAS/LAPACK library used by R on macOS.

Update and uninstall

Use rig self update to update rig itself, and rig self uninstall to uninstall it. These commands only work if you installed rig using the install script. If you installed it with a package manager or ran the installer manually, you have to use the same package manager or uninstall it manually.

Your feedback is welcome!

We would love to hear your thoughts, suggestions, and any issues you encounter while using rig. Please use the issue tracker for bug reports and feature requests, and the discussion forum or Posit community for general questions and discussions.

Acknowledgments

Huge thanks to everyone who contributed to rig 0.10.0 and 0.11.0: @achubaty, @AdaemmerP, @AdrienLeGuillou, @Adrilihan, @Andryas, @bashirhamidi, @benyamins, @biocyberman, @Bisaloo, @cderv, @CGMossa, @danielloader, @dkczk, @eitsupi, @elendil95, @eliocamp, @EllaKaye, @etiennebacher, @frosforever, @gdevenyi, @ggrothendieck, @grantmcdermott, @gvelasq, @hadley, @jabenninghoff, @jameslairdsmith, @jennybc, @jeroen, @jfin4, @John15321, @jonbry, @JosiahParry, @kalenkovich, @kenahoo, @kieran-mace, @klmr, @krlmlr, @lotum-david-j, @malcolmbarrett, @mcanouil, @mhurtado13, @MilesMcBain, @mitsuki5284, @mns-nordicals, @morphatic, @Nerwosolek, @noamross, @paluigi, @robjhyndman, @rumichaska, @SaintRod, @sammieephung, @sda030, @ShixiangWang, @t-kalinowski, @tylermorganwall, @venpopov, @vikasrawal, @wzbillings, and @zivankaraman.

This post is about the R package. There is also a Python version of orbital that works on scikit-learn pipelines.

You can install both from CRAN with:

install.packages(c("orbital", "tidypredict"))

This post covers the highlights of these releases. You can see the full list of changes in the orbital release notes and the tidypredict release notes.

A lot more models

Previously orbital and tidypredict supported a handful of useful models. This included linear models, decision trees, random forests, and boosted trees models. This release adds discriminant analysis, naive Bayes, neural networks, support vector machines, partial least squares, and several more rule and ensemble methods. This fills in the gaps of most of the known models that can be made to work with orbital.

Newly supported for regression:

• bart(engine = "dbarts")
• boost_tree(engine = "h2o_gbm")
• linear_reg(engine = "glm")
• mlp(engine = "nnet")
• null_model()
• pls(engine = "mixOmics")
• rand_forest(engine = "aorsf")
• rand_forest(engine = "partykit")
• rule_fit(engine = "h2o")
• svm_linear(engine = "kernlab")
• svm_linear(engine = "LiblineaR")

Newly supported for classification:

• bag_tree(engine = "rpart") and bag_tree(engine = "C5.0")
• boost_tree(engine = "C5.0")
• boost_tree(engine = "h2o_gbm")
• C5_rules(engine = "C5.0")
• decision_tree(engine = "C5.0")
• discrim_linear(engine = "MASS"), discrim_quad(engine = "MASS")
• discrim_linear() with the "mda", "sda", and "sparsediscrim" engines
• logistic_reg(engine = "LiblineaR")
• mlp(engine = "nnet")
• multinom_reg(engine = "nnet")
• naive_Bayes(engine = "klaR") and naive_Bayes(engine = "naivebayes")
• null_model()
• pls(engine = "mixOmics")
• rule_fit(engine = "h2o")
• rule_fit(engine = "xrf")
• svm_linear(engine = "kernlab")
• svm_linear(engine = "LiblineaR")

One thing worth noting for models with “h2o” in their engine value, is that you need a running H2O cluster to build the orbital object. Prediction and SQL generation from the orbital object stays dependency free.

Here is an example of linear discriminant analysis model, which was not supported before this release, fit on the penguins data and then run in DuckDB.

library(tidymodels)
library(discrim)
library(orbital)
library(duckdb)

data(penguins, package = "modeldata")
penguins <- tidyr::drop_na(penguins)

rec_spec <- recipe(species ~ ., data = penguins) |>
  step_dummy(all_nominal_predictors()) |>
  step_normalize(all_numeric_predictors())

mod_spec <- discrim_linear(engine = "MASS")

wf_spec <- workflow(rec_spec, mod_spec) |>
  fit(penguins)

orb <- orbital(wf_spec, type = c("class", "prob"))

con <- DBI::dbConnect(duckdb::duckdb())
penguins_db <- dplyr::copy_to(con, penguins, "penguins")

predict(orb, penguins_db)

# A query:  ?? x 4
# Database: DuckDB 1.5.5 [root@Darwin 25.6.0:R 4.6.1/:memory:]
   .pred_class .pred_Adelie .pred_Chinstrap .pred_Gentoo
   <chr>              <dbl>           <dbl>        <dbl>
 1 Adelie             1.000        6.43e- 9     4.64e-25
 2 Adelie             1.000        5.25e- 6     8.81e-16
 3 Adelie             1.000        1.97e- 4     2.61e-17
 4 Adelie             1.000        2.99e- 8     1.93e-23
 5 Adelie             1.000        8.34e- 9     4.42e-29
 6 Adelie             1.000        1.60e- 6     4.14e-20
 7 Adelie             1.000        6.03e-10     1.45e-18
 8 Adelie             0.999        6.47e- 4     4.16e-20
 9 Adelie             1.000        9.56e-10     1.97e-30
10 Adelie             1.000        6.73e-14     9.13e-27
# ℹ more rows

We have documented the list of supported models. If there is a model you need that is still missing, let us know so we can prioritize it.

When a model has no probability

While we added these new methods we ran into a problem we didn’t have before. For many classification models you get both predicted probabilities and hard class predictions. This release has added models where that is not the case. A number of newly added models only produce hard class predictions.

Instead of trying to invent class probabilities, we produce an informative error for the affected models.

c5_wf <- workflow(
  species ~ .,
  decision_tree(mode = "classification", engine = "C5.0")
) |>
  fit(penguins)

orbital(c5_wf, type = "prob")

Error in `orbital()`:
! "prob" predictions are not available for this model.
ℹ It predicts a class directly, with no probability behind it.
ℹ Use `type = "class"` instead.

tidypredict is now a toolkit, not just a function

We have talked a lot about orbital so far. This is because we think that it is the ideal interface compared to tidypredict, if your goal is to generate SQL expressions for a fitted model or workflow.

orbital and tidypredict work together to generate the expression that you need. We have expanded tidypredict with a number of generics, which are all developer focused. With the goal that packages other than orbital can benefit from the work we have done. Three of them describe what a model’s fitted expressions compute:

• tidypredict_output_type() returns "numeric", "prob", "decision", or "class"
• tidypredict_outcome_levels() returns the outcome levels in model order
• tidypredict_normalized() reports whether per-level probabilities already sum to one

Below we see two of the generics in action.

library(tidypredict)

svm_fit <- svm_linear(mode = "classification", engine = "LiblineaR") |>
  fit(sex ~ bill_length_mm + body_mass_g, data = penguins)

tidypredict_output_type(svm_fit$fit)

[1] "decision"

tidypredict_outcome_levels(svm_fit$fit)

[1] "female" "male"  

The other five expose the per-tree pieces that tidypredict_fit() assembles, so a package generating its own code can split an ensemble apart and put it back together:

• tidypredict_trees() returns per-tree expressions
• tidypredict_n_trees() returns the number of trees
• tidypredict_combine_trees() turns per-tree expressions back into a prediction
• tidypredict_class_trees() returns per-tree expressions for each outcome level
• tidypredict_class_exprs() returns one finished expression per outcome level

rf <- rand_forest(mode = "regression", trees = 5, engine = "ranger") |>
  fit(body_mass_g ~ bill_length_mm + flipper_length_mm, data = penguins)

tidypredict_n_trees(rf$fit)

[1] 5

tidypredict_combine_trees(rf$fit, rlang::syms(paste0("tree_", 1:5)))

(tree_1 + tree_2 + tree_3 + tree_4 + tree_5)/5

A lot of this logic was hardcoded inside orbital on a model by model basis, it is now formalized in such a way that you can easily use just the bits you need.

Acknowledgements

Many thanks to all the people who contributed to orbital and tidypredict since the last release!

@EmilHvitfeldt, @jannikbx, and @RAMitchell.

cuda.ml is for data scientists who work primarily in R and want to use a GPU without moving their modeling workflow to Python or learning low-level GPU APIs. Version 0.4.0 is a substantial update to the package. It makes installation much simpler, expands tidymodels support, adds more ways to run tree-ensemble models, and makes it straightforward to save and restore supported fitted models.

Install from CRAN

For most users, setup is two commands:

install.packages("cuda.ml")
cuda.ml::cuda_ml_install()

The package from CRAN is a regular, portable R package. cuda_ml_install() downloads and verifies the matching compiled backend and GPU libraries, then keeps them in a cache for later R sessions. On a supported system, you do not need to compile cuda.ml from source, configure a Python environment, or assemble the GPU libraries yourself.

The result is a familiar R package workflow: install the package, prepare its supporting libraries once, and start an analysis. The extra installation call is explicit because the GPU libraries are much larger than the R package. Repeated calls reuse the completed cache, and loading cuda.ml itself is quiet and does not initialize CUDA.

Prebuilt support is available for Linux x86_64 with glibc 2.28 or newer. GPU operations require a supported NVIDIA GPU and driver 580 or newer. On Windows, install and run R inside a compatible WSL2 Linux distribution. See the installation guide for the complete system requirements and source-build options.

Use familiar modeling interfaces

cuda.ml registers parsnip engines for linear, logistic, and multinomial regression, random forests, nearest neighbors, and radial, polynomial, and linear support-vector machines. For example, this fits a random-forest classifier on the GPU using the standard parsnip interface:

library(cuda.ml)
library(parsnip)

forest_spec <- rand_forest() |>
  set_mode("classification") |>
  set_engine("cuda.ml")

forest_fit <- forest_spec |>
  fit(class ~ ., data = modeldata::hpc_data)
predict(forest_fit, modeldata::hpc_data[1:5, ], type = "prob")

# A tibble: 5 × 4
  .pred_VF .pred_F .pred_M  .pred_L
     <dbl>   <dbl>   <dbl>    <dbl>
1    0.309  0.597  0.0666  0.0275
2    0.899  0.0838 0.0138  0.00335
3    0.965  0.0261 0.00850 0.000883
4    0.973  0.0228 0.00416 0.000352
5    0.966  0.0298 0.00416 0.000352

set_engine("cuda.ml") selects the GPU-backed cuda.ml engine; the rest is a standard parsnip workflow. Recipes can learn preprocessing on the training data and carry it into resampling and prediction.

Work directly with cuda.ml

cuda.ml also provides a direct R interface. This is useful when you prefer a function-oriented workflow or want to use the package on its own.

library(cuda.ml)
library(ggplot2)

clusters <- cuda_ml_kmeans(scale(faithful), k = 2)
faithful$cluster <- factor(clusters$labels)

ggplot(faithful, aes(eruptions, waiting, color = cluster)) +
  geom_point(size = 2.5) +
  labs(
    x = "Eruption duration (minutes)",
    y = "Waiting time (minutes)"
  ) +
  theme_minimal()

The direct API covers supervised models as well as clustering and dimensionality reduction, including DBSCAN, k-means, PCA, tSVD, UMAP, and t-SNE. It also includes stochastic-gradient-descent regression, the hyperbolic-tangent SVM kernel, and external tree-ensemble inference.

Run tree ensembles on a GPU or CPU

This release expands where and how you can make predictions with tree ensembles. The new nvForest support powers prediction for random forests trained with cuda_ml_rand_forest() and can load trained XGBoost models, LightGBM text models, and Treelite checkpoints. Once a model is loaded, the API provides standard prediction along with model information, leaf identifiers, individual-tree predictions, and checkpoint import and export.

GPU inference uses the complete cuda.ml installation. A deployment that only needs CPU inference can prepare a smaller, CUDA-free backend:

cuda.ml::cuda_ml_install(device = "cpu")

The CPU backend can run a cuda.ml random forest trained on a GPU as well as a supported external tree ensemble. The complete installation can also run these models on a CPU, so the smaller backend is an optional deployment choice.

Save and deploy fitted models

cuda.ml 0.4.0 expands model persistence for training, analysis, and deployment workflows. Supported fitted models can be saved to a compressed file, restored in another R process, stored as raw bytes, or wrapped with the bundle package.

The simplest file workflow passes a path directly:

model <- cuda_ml_rand_forest(
  class ~ .,
  data = modeldata::hpc_data,
  trees = 100
)

cuda_ml_serialize(model, "hpc-runtime-forest.cuda-ml")

In another R process or deployment environment, prepare cuda.ml and restore the fitted model:

library(cuda.ml)

cuda_ml_install()
model <- cuda_ml_unserialize("hpc-runtime-forest.cuda-ml")

predict(model, modeldata::hpc_data[1:5, ], type = "class")

File paths use gzip compression. Passing connection = NULL instead returns the state as an uncompressed raw vector of bytes, which is convenient for object stores and database BLOB columns. The blob package can represent raw vectors for database workflows, and the bundle package is supported for teams that already use bundled model artifacts.

Persistence is available for linear models, logistic and multinomial regression, PCA, SVC and SVR models, UMAP, random forests, and nvForest models. cuda.ml checks the saved state and required backend before restoring it. For an nvForest-backed model, the restore call can select CPU or GPU inference.

Highlights for users upgrading from cuda.ml 0.3

This is a breaking update to the earlier package. The most visible changes are:

• The random-forest API now uses mtry for predictor sampling and sample_fraction for row sampling. trees defaults to 100, and an omitted seed draws from R’s random-number generator, so set.seed() controls the fit.
• Linear, logistic, and multinomial regression now use numeric penalty and mixture arguments that match parsnip. Logistic and multinomial regression are unregularized by default.
• normalize_input was removed from the linear-model functions. It previously requested GPU-side L2 normalization. Use explicit preprocessing such as recipes::step_normalize() when centering and scaling are appropriate; but please note, the two operations are not numerically identical.
• cuda_ml_sgd() now fits squared-loss regression only. Its loss argument was removed, and n_iters_no_change is now n_iter_no_change.
• The former FIL interface has been replaced by nvForest. Random projection and the KNN IVFSQ index have no replacement in the pinned upstream API.

See the full changelog for the complete list of API changes.

The guides cover first steps and more complete examples:

• Get started with cuda.ml
• Use cuda.ml with tidymodels
• Save and restore models
• nvForest inference and deployment

Orbital’s whole pitch is that a trained model becomes SQL, so a database can run predictions on its own, with no Python process anywhere near it. Until 0.6.0, “trained model” meant scikit-learn: pipelines, trees, linear models, all .fit() in Python and then turned into a SELECT statement. It never covered what a lot of teams are actually training now: PyTorch models, not scikit-learn pipelines.

Orbital 0.6.0 closes that gap. A torch.nn.Sequential network, trained exactly the way you already train it, now compiles to the same kind of SQL a linear regression would.

No ONNX Runtime. No model server. No separate inference service to keep alive next to the database. Just a query.

Why does that work at all, for a framework Orbital was never written for? Because of a decision made long before PyTorch was ever on the table.

Why this isn’t a bolt-on

Orbital was never really a scikit-learn tool. Underneath, it converts a scikit-learn pipeline to ONNX (Open Neural Network Exchange, a standard graph format for trained models) using the skl2onnx library, then walks that graph node by node to produce SQL. Scikit-learn was always one hop removed from what Orbital actually translates.

Adding PyTorch meant taking the same hop from a different starting point. torch.onnx.export turns a torch.nn.Sequential model into that same kind of graph. Feed it into the translator that already existed, and the translator doesn’t know or care whether the graph came from PyTorch or scikit-learn.

The proof is in how little new code that took. The entire engine for running a feed-forward network in SQL is three small classes: a Gemm translator for Linear layers, ReLU, Sigmoid. Everything else (the translator base class, the optimizer, the per-dialect SQL compiler) already existed, built earlier for trees and linear models.

When I first thought of support for PyTorch, I put it this way: “the underlying translation works on ONNX graphs… the same value proposition orbital already provides for scikit-learn models can apply as it is to pytorch networks exported to ONNX.” That sentence turned out to be the whole implementation plan.

If you want the fuller picture of how a graph becomes SQL, parser, then translator, then optimizer, the architecture docs walk through all three stages in order.

That architecture is also why I keep calling this multiple frameworks, not two frameworks. Relu, Sigmoid, Tanh, and Softmax are single translators, not one per framework. Scikit-learn’s MLPClassifier and MLPRegressor reach them through MatMul and Add, PyTorch’s nn.Sequential reaches the exact same translators through Gemm. One implementation, two entry points, and no reason it has to stop at two.

Scikit-learn and PyTorch are both real and shipping today, which is enough on its own to call this “multiple frameworks.” But the dependency story is already moving that direction: issue #113 proposes turning scikit-learn itself into an optional dependency, the same way PyTorch already is, so the core stops assuming any particular framework at all.

Making it actually usable

Neural networks are layered, and every neuron in layer two reads every output of layer one. If Orbital just inlines those outputs at each place they’re read, instead of naming them once, that repetition compounds from one layer to the next. Two layers doubles the inlined text. Five layers is a different order of magnitude.

That’s not theoretical: the issue that tracked the fix measured scikit-learn’s own default MLPClassifier(hidden_layer_sizes=(100,)), the first thing anyone reaches for, at 53MB of generated SQL and roughly 894 seconds just to generate it. A hundred neurons in one hidden layer, and the query was already unusable.

PyTorch’s Gemm translator never had this problem. It has called preserve(), materializing its output as a real SQL column, since the day it was written. Scikit-learn’s MLPClassifier and MLPRegressor compile through different ONNX ops though: MatMul then Add, because that’s what skl2onnx emits, not Gemm. Neither of those translators called preserve() at all.

The fix, Optimizer.preserve_referenced_outputs(), runs after every single node in the translation loop, for every translator, not just MatMul and Add, and checks how many times that node’s output is actually referenced downstream. Referenced more than once, it gets materialized as a named column. Referenced once or not at all, it stays inlined, no extra column, no extra noise.

None of this is new machinery either. Tree ensembles already lean on the same trick: preserve() materializes per-tree votes, or the whole ensemble’s aggregated vote so it isn’t re-emitted everywhere it’s read, as real SQL columns. preserve_referenced_outputs() generalizes that same idea automatically, for every translator, whether it’s part of an ordinary pipeline or a neural network.

Here’s what that fix was worth, measured on three shapes while it was being built:

Generation time collapsed just as hard: the 20→64→64→1 network above went from about 185 seconds to about 3.6 seconds. Running the resulting SQL got faster too, if less dramatically: on 200,000 rows, an MLP(32,32) query dropped from 0.43s to 0.35s, and an MLP(100,100) from 3.58s to 3.31s.

Same hyperparameters. Same defaults everyone actually reaches for. The difference is entirely in how the SQL gets built, not in what the network computes.

What it can do today

Neural network support in 0.6.0 covers binary classification, multiclass classification, and regression, for both scikit-learn and PyTorch. Five new or updated examples in the repo prove it out: pytorch_fraud_detector.py, pytorch_maintenance_classifier.py, pytorch_demand_regressor.py, pipeline_mlp_classifier.py, pipeline_mlp_regressor.py.

The one worth walking through is the fraud detector, since it’s the shape most teams actually need: a handful of numeric features, a binary “is this fraud” output, trained the same way this kind of model always is.

FEATURES = {
    "amount": orbital.types.DoubleColumnType(),
    "hour": orbital.types.DoubleColumnType(),
    "v1": orbital.types.DoubleColumnType(),
    "v2": orbital.types.DoubleColumnType(),
}

model = torch.nn.Sequential(
    torch.nn.Linear(len(FEATURES), 16),
    torch.nn.ReLU(),
    torch.nn.Linear(16, 8),
    torch.nn.ReLU(),
    torch.nn.Linear(8, 1),
    torch.nn.Sigmoid(),
)

# ... train model normally: Adam, BCELoss, a plain training loop ...

orbital_pipeline = orbital.parse_pytorch_model(model, FEATURES)

From that one orbital_pipeline, two engines:

duckdb_sql = orbital.export_sql("transactions", orbital_pipeline, dialect="duckdb")
postgres_sql = orbital.export_sql("transactions", orbital_pipeline, dialect="postgres")

I ran both, against a real DuckDB and a real Postgres, not just read the generated text. Same four test transactions, three ways to compute a prediction (PyTorch itself, the DuckDB query, the Postgres query), and all three agree to within 1e-5. Both queries land around 10KB, not the tens of megabytes a naive translation would have produced before 0.6.0’s optimizer fix.

That works because Sigmoid and ReLU both compile to plain arithmetic: EXP, a division, a CASE WHEN. Every SQL engine has those. It’s not an accident which activation this example uses, either: Tanh compiles to a native TANH() call instead, and not every dialect implements that the same way, so it’s the one activation in Orbital’s NN support with an actual portability caveat attached.

DuckDB and Postgres are two of the three dialects Orbital actively tests in CI. SQLite is the third. But that’s a testing choice, not an architecture boundary: Orbital doesn’t write dialect-specific SQL at all. Translation ends at ibis. export_sql just hands the finished expression to whichever of ibis’s own backend compilers matches the dialect you ask for, and ibis ships about twenty of those: Snowflake, BigQuery, Trino, MySQL, and so on.

Limits, honestly

What Orbital 0.6.0 handles is feed-forward, fully connected networks: stacks of Linear layers with ReLU, Sigmoid, Tanh, or Softmax in between. That already covers real use cases people put into production: fraud scoring, churn, demand forecasting, risk models. None of those need a CNN or a transformer.

What it doesn’t do yet is exactly what that shape excludes: no convolutions, no recurrence, no attention, no embedding layers for categorical features. If your model needs any of those, Orbital isn’t there yet.

SQL size still grows with the network. A few hidden layers of 64 to 128 neurons land comfortably in the KB range. Wider or deeper than that, it’s worth checking the generated SQL against whatever statement-size limit your engine has, before you deploy it.

That headroom exists at all thanks to Optimizer.preserve_referenced_outputs(), and that mechanism helps every translator in Orbital, not just neural networks. It’s a big enough story on its own that I’ll get back to in a future post.

Try it

pip install orbital[pytorch]

From there, the getting-started guide walks through this same fraud-detector shape end to end, and the examples directory has five more, covering classification and regression in both frameworks.

Orbital speaks PyTorch now. Scikit-learn too. Same query either way.

There are more frameworks already in the works. I won’t name them here, only that the architecture was built for exactly this.

Shared memory across processes is standard in other programming languages. In R it’s been either platform-specific (fork on Unix-alikes, unsafe from GUIs like Positron or with multi-threaded code) or file-format-specific (memory-mapped Parquet).

Two CRAN releases address those constraints directly:

• mirai 2.7.0 — daemons(memory = ...) puts a hard byte-level cap on the dispatcher queue. try_mirai() returns NULL immediately when that cap is hit, instead of blocking your session. The dispatcher itself runs as an in-process thread now; and through nanonext 1.9.0 underneath, peak memory during serialized sends has halved.
• mori 0.2.0 — graduates from experimental status, with a stable wire format underneath. Sub-lists and extracted elements now expose stable path-form names (/mori_abc[2,3]) that any process can attach to via map_shared() directly, without going through the parent.

Building out the foundations

mirai already shipped a lot of what production async needs: microsecond round-trip latency, FIFO scheduling, task cancellation, OpenTelemetry tracing, reproducible parallel RNG, custom serialization for torch / Arrow / polars, an event-driven promises method for Shiny ExtendedTask, and deployment across SSH / Slurm / Posit Workbench.

What these releases add is more foundational. R hasn’t really had a built-in memory model for asynchronous work — no way to size a task queue in bytes, and no portable way to share data across processes without paying for a full copy. mirai 2.7.0 introduces the first: a bounded queue built into the dispatcher, with try_mirai() exposing non-blocking submission under this constraint. mori 0.2.0 brings the second to a stable footing: a self-describing shared-memory format, addressable down to individual list elements by name, that any process on the machine can attach to.

Together they extend what R can credibly be used for — streaming pipelines, multi-stage data flows, and event-loop applications that previously had to reach outside the language.

mirai 2.7.0 — bounded queues and try_mirai()

daemons(memory = ...)

The dispatcher is the single point in your session where every mirai() call lands before being routed to a daemon. Each call carries its serialized arguments — the queued payload — and that payload sits in memory until a daemon is ready for it. By default the queue is unbounded, which is fine until it isn’t.

daemons() now takes a memory argument that caps the approximate total payload of queued tasks at the dispatcher, in MB:

library(mirai)
daemons(4, memory = 100)  # 100 MB queue capacity

When the queued bytes are below the cap, mirai() calls submit normally. When the cap is reached, mirai() blocks the calling R thread until daemons drain the queue enough for the new payload to fit. Existing code keeps working — the only visible difference is that your session can no longer be pushed out of memory by a runaway producer.

status() surfaces current and peak usage:

status()$memory
#     used     peak capacity 
#     12.4     87.3    100.0 

peak is the high-watermark queued payload over the lifetime of the dispatcher, which gives you an empirical way to size the cap. The recommended workflow is to run a representative workload with memory = NULL first (where capacity reports NA), observe peak, then set memory at or above that watermark.

If profiling isn’t practical, the local machine has to fit your session, the dispatcher queue, and n daemon processes (each a full R session holding an in-flight payload while executing). Half of currently available RAM (ps::ps_system_memory()[["avail"]] / 2e6, in MB) is a reasonable starting memory cap, revised down for large n or payloads. Remote daemons don’t draw on the local budget, so when most workers are remote, the cap can be more generous.

For n itself, one less than your number of CPU cores is the right starting point for CPU-bound work (one core left for your session), while I/O-bound work (API calls, database queries, file reads) can comfortably exceed core count since idle daemons cost little.

The capacity is measured in bytes, not task count. A queue capped at 100 tasks can hold either 100 small tasks or 100 huge ones, and only the second runs out of memory. A queue capped at 100 MB never runs your session out of memory regardless of the mix.

try_mirai()

A blocking submit is fine in a batch script, but not an event loop: a Shiny session that calls mirai() from inside an ExtendedTask can’t afford to block while the queue drains, because that same session is also driving the UI for all users.

try_mirai() returns immediately when the queue is full, instead of blocking. The semantics are simple:

• If the queue is below capacity, behaves identically to mirai() and returns a mirai.
• If the queue is at capacity, returns NULL immediately without blocking, and leaves the caller to decide what to do next.

m <- try_mirai({ slow_thing() }) %||% stop("queue full")

The three response strategies are: drop the task (best when the work is idempotent and frequent), retry later (queue the request behind a later::later() call), or propagate backpressure upstream — as in the example above, by raising a condition. Which is right is application-specific — try_mirai() hands you the flexibility.

If memory isn’t set, or the daemon configuration has no dispatcher, try_mirai() always returns a mirai and is identical to mirai() — so the function is safe to use unconditionally; it only does anything different in the bounded-queue case.

The two halves — daemons(memory = ...) and try_mirai() — are the right combination for any event-loop integration. Set the cap to reflect what your session can actually afford to hold, submit through try_mirai(), and the application gracefully adapts to load instead of locking up or crashing.

Under the hood

A few transport-layer changes ship alongside 2.7.0 — invisible most of the time, important when payloads are large or throughput is high:

• Dispatcher as a thread. The dispatcher is no longer a separate R process; it runs as an in-process thread, courtesy of a C-level reimplementation in nanonext. mirai round-trip latency is now in the tens of microseconds.
• Halved peak memory on sends. nanonext 1.9.0, the transport underneath mirai, now writes serialized sends directly instead of going through an intermediate buffer. A 500 MB argument going to a daemon used to need ~1 GB of momentary headroom in your session; now it needs ~500 MB. The dispatcher’s actual resident memory tracks the daemons(memory = ...) cap more accurately as a result.
• >2 GB payloads on macOS and Windows. Transfers above 2 GB previously failed silently near the INT_MAX-byte boundary on those platforms. They now go through cleanly. Linux was already fine.

mori 0.2.0 — out of experimental status

mori is a younger package — its first CRAN release was just last month — and its job is the other half of memory pressure: the data itself, rather than the queue. R processes don’t share memory, and data crosses between them through message-passing. When eight workers each need the same 200 MB data frame, that’s 1.6 GB of serialize / transfer / deserialize, plus eight separate copies of the data resident in RAM.

mori::share() writes the object once into OS-level shared memory and returns an ALTREP (alternative representation) wrapper. Multiple processes on the same machine map the same physical pages — zero-copy, lazy, and managed by R’s garbage collector. Pass the wrapper to a daemon and only the shared memory identifier (hundreds of bytes) crosses the wire; the daemon attaches and reads the pages directly.

We implemented this in mori 0.1.0, but 0.2.0 takes mori out of experimental status, landing a stable wire format. A key consequence of this is that every sub-list and extracted element of a shared region now has a stable identifier of its own. shared_name() emits path-form names (/mori_abc_1[2] for the second element of region /mori_abc_1, /mori_abc_1[2,3] for nested addressing), and map_shared() accepts the path form directly.

library(mori)
library(mirai)

lst <- share(
  list(weights = rnorm(1e6), targets = rnorm(1e6), inputs = rnorm(1e6))
)

# Sub-elements have their own path-form names

shared_name(lst)
#> [1] "/mori_abc_1"
shared_name(lst[[1]])
#> [1] "/mori_abc_1[1]"

Pass the shared object directly to a daemon — the ALTREP wrapper serializes as just the name, and the daemon attaches on deserialize:

mirai(length(weights), weights = lst[[1]])[]
#> [1] 1000000

A consumer in a separate session that only has the name as a string — from a config file, message queue, or HTTP response — attaches with map_shared():

# In a separate R session

weights <- mori::map_shared("/mori_abc_1[1]")
length(weights)
#> [1] 1000000

A path-form name is a small thing on the wire – just a string – but it enables a lot:

• Attach by name. Path-form identifiers are plain strings, so they fit anywhere a string can go: a queue payload, a config entry, an HTTP response, a database row. A consumer reads the name, calls map_shared(), and gets exactly that slice — the parent region never has to enter their session.
• Decoupled pipelines. Stage A shares a 100-column frame and tells stage B the names of the three columns to process. Stage B passes the name of its output to stage C, and so on. No stage holds more in memory than it actually uses, and no stage needs to know how its inputs were produced — only the names.
• Code symmetry. map_shared(shared_name(x)) returns an equivalent of x whether x is a root region or a single leaf element, so consumer code doesn’t need a special branch for sub-objects.
• Lifetime through one chain. Holding a reference to a leaf keeps the whole parent region alive automatically — the garbage collector walks the chain of dependencies — so a consumer never has to manually pin the root to stay safe.

Composing with daemons(memory = ...)

The two memory tools compose. mori::share() shrinks each task’s queued payload from megabytes to hundreds of bytes, which means the dispatcher’s memory cap rarely fills up at all when workers are reading shared data. The pieces are complementary:

• mori::share() removes the per-worker copy of large data.
• daemons(memory = ...) caps the dispatcher backlog when something is being sent.
• try_mirai() makes hitting that cap a non-event in an event loop.

Putting R on the concurrency map

The design of daemons(memory = ...), try_mirai(), and mori::share() draws on how async runtimes in other languages handle the same problems. Here’s the comparison:

Byte-budgeted backpressure. Async frameworks elsewhere — Python’s asyncio.Queue, Ray’s max_pending_tasks, Tokio’s bounded mpsc channels — measure queue capacity by task count. daemons(memory = ...) caps by bytes instead, which tracks what actually drives memory pressure.

Non-blocking submission on a full queue. Tokio’s try_send is the closest direct analogue: it returns immediately when the channel is full instead of blocking. try_mirai() provides the same semantics for R submission, with the additional advantage that the cap itself is byte-aware.

Integrated cross-process zero-copy. Ray’s Plasma object store offers cross-process zero-copy, but it’s a heavyweight component — you run a Ray cluster to use it, not a function. mori::share() is just a function call, producing an ALTREP wrapper that consumer processes attach to via lazy reads, down to individual columns by name.

Combined with mirai’s microsecond round-trip latency and event-driven C-level transport, these three pieces give R a concurrency story that stands on its own: byte-aware backpressure, non-blocking submission, and integrated zero-copy sharing.

Try it

Install from CRAN:

install.packages(c("mirai", "mori"))

Pointers from here:

• mirai reference vignette — the memory-management section walks through daemons(memory = ...), status()$memory, try_mirai(), and shared-memory composition.
• mirai promises vignette — Shiny ExtendedTask and event-driven async.
• mori package site — share(), shared_name() and map_shared().
• nanonext package site — the transport layer and concurrency primitives.
• The r-lib agent skill in the posit-dev-skills plugin includes an LLM-optimised mirai guide for AI coding assistants.

Issues, feedback, and questions are very welcome on GitHub: r-lib/mirai, r-lib/nanonext, shikokuchuo/mori.

Acknowledgments

A big thanks to everyone who contributed to mirai 2.7.0, mori 0.2.0, and nanonext 1.9.0 through their issues, pull requests, and discussions: @HenrikBengtsson, @kentqin-cve, @king-of-poppk, @manforkr, @mcol, @michaelmayer2, @pmac0451, @t-kalinowski, and @wlandau.

This is part of our commitment to R in production. As R applications scale – more users, more processes, more machines – you need tools to understand what’s happening across your entire system. That’s what OpenTelemetry is built for, and it’s now available for R.

What is OpenTelemetry?

OpenTelemetry (OTel) is a vendor-neutral, open source observability framework backed by the Cloud Native Computing Foundation. It has broad industry support across languages and platforms, and is already the standard in the Python, Java, JavaScript, and Go ecosystems. Now it’s available for R.

OpenTelemetry defines a standard for collecting telemetry data:

• Traces follow an operation as it moves through your system, showing exactly which functions ran, in what order, and how long each took.
• Metrics capture numerical measurements over time – things like request counts, response latencies, or memory usage.
• Logs record detailed events as they happen, providing context when you need to investigate a specific moment.

The instrumented packages described in this post all use the otel package under the hood, and focus on traces, which provide the most immediate value for understanding production behavior. You can also use otel directly to add your own metrics and logs to your application code.

Why observability matters for R in production

When you’re developing interactively in RStudio or Positron, debugging is straightforward – you can step through code, inspect objects, and add print statements. But when your R code runs in production – a Shiny app serving hundreds of users, a plumber2 API handling thousands of requests, a batch pipeline running across a cluster – the picture changes.

The core concept in OpenTelemetry is a trace: the full path of a request through your system. Each trace is made up of spans – individual units of work with a name and a duration. Spans nest inside each other, so you can follow how each operation triggered the next:

This structure gives you four things that are hard to get any other way:

• Performance: Which part of a request is slow? Span durations pinpoint where time is spent – and nesting reveals unnecessary overhead.
• Errors: In development and testing, you know where errors are – you wrote the test, you control the inputs. In production, errors surface far from their root cause, across process boundaries and async operations, triggered by conditions you never anticipated. Traces show you the full chain of real operations that led to each failure, in the context where it actually happened.
• Centralized view: When your application extends across multiple R processes or machines – a Shiny app with mirai workers, or a plumber2 API behind a load balancer – traces are aggregated into a single view across all of them.
• Real-time monitoring: OTel is designed to be left on in production, not just enabled during testing or staging. With low overhead, it runs continuously – so you see what’s happening as it happens, and dashboards and alerts catch problems before users report them.

Instrumented packages

We’ve worked across teams to add OpenTelemetry instrumentation to the R packages where it matters most:

Together, these packages cover the most common production R workloads: web applications, APIs, parallel computing, HTTP clients, AI/LLM tools, report rendering pipelines, CI test runs, and database access. Because the instrumentation is built into the packages themselves, you benefit from it automatically – no wrapper functions, nor modifications to existing code. If you already have a Shiny app or plumber2 API, your app will generate traces as soon as you enable OpenTelemetry. Your application code stays exactly as it is.

Seeing it in action

To make this concrete, let’s look at a Shiny chat app built with shinychat and ellmer that fetches weather forecasts. It uses mirai for async execution and httr2 for weather API requests.

A user asks about the weather in Atlanta and Newcastle:

With OpenTelemetry enabled, every step is captured automatically. Here are the traces from those two queries:

The trace reveals the full chain of operations from user input through to the HTTP request, across process boundaries, with no manual logging. The nesting shows how each step triggered the next, and the durations show where time was spent.

The second query failed. Without tracing, you’d see an error in your logs and start investigating. Here, the failure is immediately visible – the red span pinpoints where it occurred and the surrounding context shows why. In a production system with many concurrent users, that’s the difference between minutes and seconds of debugging.

Getting started

Getting started requires the otelsdk package and a few environment variables. No changes to your application code.

Here’s how the pieces fit together: the instrumented R packages generate telemetry data as they run. The otelsdk package collects and exports this data over HTTP to a backend – a service that stores your traces and provides a web dashboard where you can search, filter, and visualize them (like the trace screenshots above).

Step 1: Install otelsdk

install.packages("otelsdk")

Step 2: Choose a backend

OpenTelemetry is vendor-neutral, so you can send your data to any compatible backend:

• Cloud services: Logfire, Grafana Cloud, Langfuse
• Self-hosted: Jaeger, Zipkin, Prometheus

During development, you can also run an OpenTelemetry Collector on your own machine and inspect traces directly, without sending data anywhere else.

Each backend will give you an endpoint URL and an authentication token. The otelsdk collecting telemetry data guide has examples for some common backends.

Step 3: Set environment variables

Add these to your .Renviron file (use usethis::edit_r_environ() to open it), replacing the endpoint and token with the values from your chosen backend. This example uses Logfire, which offers a free tier to get started:

OTEL_TRACES_EXPORTER="http"
OTEL_EXPORTER_OTLP_ENDPOINT="https://logfire-eu.pydantic.dev"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=<YOUR-WRITE-TOKEN>"

If you’re deploying content to Posit Connect, refer to how to set environment variables on Posit Connect.

Step 4: Run your app

That’s it. Restart R, then run your Shiny app, plumber2 API, or any code that uses the instrumented packages. Traces will flow to your backend automatically. Open your backend’s web dashboard to see them – you’ll see a view like the trace screenshots shown above, with each span representing an operation in your application.

You can verify that tracing is active at any time:

otel::is_tracing_enabled()
#> [1] TRUE

OpenTelemetry is designed to be safe in production. If anything goes wrong in the telemetry code itself, it will never crash your application – errors are silently suppressed so your app keeps running.

Zero-code instrumentation

Beyond the packages that ship with built-in instrumentation, otel supports zero-code instrumentation for any R package. Set the OTEL_R_INSTRUMENT_PKGS environment variable to a comma-separated list of package names, and otel will automatically create spans for their exported functions:

OTEL_R_INSTRUMENT_PKGS=dplyr,tidyr

You can also fine-tune which functions are instrumented using include and exclude filters:

OTEL_R_INSTRUMENT_PKGS_DPLYR_INCLUDE=mutate,filter,select

This is useful for adding visibility to any package in your stack, even those without built-in OTel support. See the otel documentation for full details.

Configuration options

The otelsdk package is configured entirely through environment variables, following OpenTelemetry conventions:

See the otelsdk environment variables reference for the complete list.

Looking ahead

With instrumentation built into the packages that R users already rely on, observability becomes something you can turn on, not something you have to build.

We’re also integrating an OpenTelemetry collector into Posit Connect, giving you an end-to-end observability solution you can simply turn on or off.

OTel support across the ecosystem continues to expand. If you’d like to learn more:

• otel package documentation – the instrumentation API
• otelsdk package documentation – the SDK for collecting and exporting telemetry
• Shiny 1.12 OTel blog post – deep dive into Shiny’s OpenTelemetry support
• OpenTelemetry project – the upstream standard

We’re excited about what this opens up for the R community. Whether you’re running a Shiny dashboard for a small team, a plumber2 API serving thousands of requests, or a data pipeline distributed across a cluster – you now have the tools to see what’s happening, in real time, with no code changes required.

Students can launch Positron the same way they would open JupyterLab or a notebook. Just select it from the JupyterHub launcher and start working.

Once launched, Positron provides the full IDE experience, including:

• Rich Python and R support
• Access to the OpenVSX extension marketplace
• Built in data viewer and variables explorer
• Integrated help pane, debugger, version control and other features to help students level up when they’re ready

How it works

Positron Server is designed to integrate directly with existing JupyterHub deployments. It’s compatible with JupyterHub environments running JupyterLab 4 and Python 3.9+.

It’s installed via the jupyter-positron-server Python package, built on Jupyter Server Proxy. If you’ve configured similar services before, setup will feel familiar. This is not a standalone desktop install. Rather, it lets you bring Positron into an existing JupyterHub setup.

Who can use it?

This offering is available to academic institutions using Positron for teaching. Under a free license, institutions can provide access to enrolled students, course participants, or staff involved in the delivery or receipt of educational programming.

Full eligibility details are available in the Positron Education License Rider.

Getting started

Hosting Positron for teaching purposes requires a free license key. To get set up:

1. Review the eligibility criteria in the Positron Education License Rider.
2. Email academic-licenses@posit.co to request a teaching license.
3. Once your license is confirmed, follow the jupyter-positron-server documentation to complete setup in your JupyterHub environment.

We’re over the moon to announce the release of orbital 0.5.0. orbital lets you predict in databases using tidymodels workflows. orbital uses tidypredict under the hood to translate fitted models into expressions. This post will also cover things from tidypredict’s 1.1.0 release.

This blogpost is about the R orbital package, but there’s also a python version that works on scikit-learn models.

You can install both from CRAN with:

install.packages(c("orbital", "tidypredict"))

This blog post will cover the highlights, which are support for more models, faster performance, and more vignettes.

You can see a full list of changes in the orbital release notes and tidypredict release notes.

Newly supported models

We have added support for new models as well as more prediction types for existing supported models.

The newly supported models are.

• decision_tree(engine = "rpart")
• boost_tree(engine = "lightgbm")
• boost_tree(engine = "catboost") (More on this soon)

All of which support regression, classification and probability estimates.

The following models now also support classification and probability estimates in addition to regression.

• mars(engine = "earth")
• multinom_reg(engine = "glmnet")
• rand_forest(engine = "randomForest")
• rand_forest(engine = "ranger")

If there is a model type you specifically need please let us know so we can prioritize new additions.

Nested case_when() support

All tree based models were previously implemented as a flat case_when() statement. This means that a small tree with 3 leaves would look like this.

case_when(
  x <= 5 & y <= 3 ~ "low",
  x <= 5 & y > 3  ~ "med",
  x > 5           ~ "high"
)

And while this works, it comes with a number of downsides. In this example we have to calculate x <= 5 more than once. This might not be that big of a deal in this sized tree but it compounds very fast as the tree grows deeper.

We are also not using the information effectively. Since trees are exhaustive we shouldn’t have to calculate the last condition as all other choices have been ruled out. With these considerations in mind we have switched all trees to be expressed as nested case_when() statements.

case_when(
  x <= 5 ~ case_when(
    y <= 3 ~ "low",
    .default = "med"
  ),
  .default = "high"
)

This case_when() evaluates exactly the same as the previous flat case_when() statement. While this might be a little harder to read it provides a lot of benefit in terms of performance. Each condition is evaluated at most 1 time. This has a really big influence on the computational speed.

This also means that the R version of orbital now matches what the python version of orbital does when creating a tree.

New separate_trees argument

Some models like the ensemble tree models can be represented as a combination of multiple smaller models.This typically manifests as a single massive expression in the following format:

.pred = "(tree1) + (tree2) + (tree3) + ... + (tree100)"

This can create trouble for two main reasons. The first one is that this can cause us to hit expression nesting depth when trying to execute these in a database if we have too many trees or have too deep trees. The second related issue is that databases will not be able to recognize that these trees could be calculated in parallel and combined afterwards.

This is where the new separate_trees argument comes in. When setting separate_trees = TRUE in orbital() you change the internal representation of the orbital object to not have a single massive expression for .pred and instead split them out into multiple expressions like so.

.pred_tree_001 = "case_when(...)"
.pred_tree_002 = "case_when(...)"
.pred_tree_003 = "case_when(...)"
...
.pred = ".pred_tree_001 + .pred_tree_002 + .pred_tree_003 + ..."

This representation allows the database query optimizer to potentially evaluate trees in parallel, since each intermediate column is independent.

The separate_trees argument works for the following engines.

• xgboost
• lightgbm
• catboost
• ranger
• randomForest

This change alone allows us to work with model types previously not possible with orbital. Together with the nested tree support you can now productionize some of the most popular machine learning models.

splines support

Spline transformations are commonly used in preprocessing to capture non-linear relationships between predictors and the outcome.

With this release, orbital now supports step_spline_b(), step_spline_convex(), step_spline_monotone(), step_spline_natural(), and step_spline_nonnegative() from the recipes package. Under the hood, splines are translated into piecewise polynomial expressions that can be evaluated directly in SQL.

More vignettes

We have added a handful of new vignettes as well in this release.

• SQL expression sizes: Goes over how different hyperparameters in models affect SQL sizes. This is useful information especially when working with boosted trees as there are many different combinations of hyperparameters that produce similar performance at different SQL expression sizes. With a little effort you could pick a model that runs 10-100 times faster with minimal loss in predictive performance.

• Parallel tree evaluation in databases: A more in-depth look at how the separate_trees argument works. Also includes a section on why and when you should consider using it.

• Database deployment: Shows examples of how we can deploy an orbital model using tables and views.

• Float precision at split boundaries: Some models like xgboost and Cubist models operate on 32-bit doubles instead of on 64-bit doubles like we have in R. This can cause some problems where predictions don’t match exactly. If you use any of these models you should read this vignette to see if this issue is a dealbreaker for you or not.

Acknowledgements

A special thanks to Emily Riederer who helped workshop and benchmark these new features.

Last week, I had the pleasure of giving a talk to R-Ladies Abuja about how Positron can grow with you as you work on data that is too large for your laptop. The talk was recorded, and you can find it on YouTube here:

I opened this talk discussing how I first learned about these “beyond your laptop” technologies, typically in a organization where these technologies were already in use and seemed specific to infrastructure there. I later came to understand that these technologies are actually related to each other and understanding one can really help when you need to pick up another one. I pointed out some of the Positron features that are designed to make it easier to work with these technologies:

• Data Explorer
• Connections Pane
• Remote SSH sessions

Check out my slides

If you’d like to check out my slides, they are available as well:

sparklyr and friends have been getting some important updates in the past few months, here are some highlights:

• spark_apply() now works on Databricks Connect v2

• sparkxgb is coming back to life

• Support for Spark 2.3 and below has ended

pysparklyr 0.1.4

spark_apply() now works on Databricks Connect v2. The latest pysparklyr release uses the rpy2 Python library as the backbone of the integration.

Databricks Connect v2, is based on Spark Connect. At this time, it supports Python user-defined functions (UDFs), but not R user-defined functions. Using rpy2 circumvents this limitation. As shown in the diagram, sparklyr sends the the R code to the locally installed rpy2, which in turn sends it to Spark. Then the rpy2 installed in the remote Databricks cluster will run the R code.

A big advantage of this approach, is that rpy2 supports Arrow. In fact it is the recommended Python library to use when integrating Spark, Arrow and R. This means that the data exchange between the three environments will be much faster!

As in its original implementation, schema inferring works, and as with the original implementation, it has a performance cost. But unlike the original, this implementation will return a ‘columns’ specification that you can use for the next time you run the call.

spark_apply(
  tbl_mtcars,
  nrow,
  group_by = "am"
)

#> To increase performance, use the following schema:
#> columns = "am double, x long"

#> # Source:   table<`sparklyr_tmp_table_b84460ea_b1d3_471b_9cef_b13f339819b6`> [2 x 2]
#> # Database: spark_connection
#>      am     x
#>   <dbl> <dbl>
#> 1     0    19
#> 2     1    13

A full article about this new capability is available here: Run R inside Databricks Connect

sparkxgb

The sparkxgb is an extension of sparklyr. It enables integration with XGBoost. The current CRAN release does not support the latest versions of XGBoost. This limitation has recently prompted a full refresh of sparkxgb. Here is a summary of the improvements, which are currently in the development version of the package:

• The xgboost_classifier() and xgboost_regressor() functions no longer pass values of two arguments. These were deprecated by XGBoost and cause an error if used. In the R function, the arguments will remain for backwards compatibility, but will generate an informative error if not left NULL:

  • sketch_eps - As of XGBoost version 1.6.0 sketch_eps was replaced by max_bins
  • timeout_request_workers - Removed in XGBoost version 1.7.0 because it was no longer needed when XGBoost added barrier support

• Updates the JVM version used during the Spark session. It now uses xgboost4j-spark version 2.0.3, instead of 0.8.1. This gives us access to XGboost’s most recent Spark code.

• Updates code that used deprecated functions from upstream R dependencies. It also stops using an un-maintained package as a dependency (forge). This eliminated all of the warnings that were happening when fitting a model.

• Major improvements to package testing. Unit tests were updated and expanded, the way sparkxgb automatically starts and stops the Spark session for testing was modernized, and the continuous integration tests were restored. This will ensure the package’s health going forward.

remotes::install_github("rstudio/sparkxgb")

library(sparkxgb)
library(sparklyr)

sc <- spark_connect(master = "local")
iris_tbl <- copy_to(sc, iris)

xgb_model <- xgboost_classifier(
  iris_tbl,
  Species ~ .,
  num_class = 3,
  num_round = 50,
  max_depth = 4
)

xgb_model %>% 
  ml_predict(iris_tbl) %>% 
  select(Species, predicted_label, starts_with("probability_")) %>% 
  dplyr::glimpse()
#> Rows: ??
#> Columns: 5
#> Database: spark_connection
#> $ Species                <chr> "setosa", "setosa", "setosa", "setosa", "setosa…
#> $ predicted_label        <chr> "setosa", "setosa", "setosa", "setosa", "setosa…
#> $ probability_setosa     <dbl> 0.9971547, 0.9948581, 0.9968392, 0.9968392, 0.9…
#> $ probability_versicolor <dbl> 0.002097376, 0.003301427, 0.002284616, 0.002284…
#> $ probability_virginica  <dbl> 0.0007479066, 0.0018403779, 0.0008762418, 0.000…

sparklyr 1.8.5

The new version of sparklyr does not have user facing improvements. But internally, it has crossed an important milestone. Support for Spark version 2.3 and below has effectively ended. The Scala code needed to do so is no longer part of the package. As per Spark’s versioning policy, found here, Spark 2.3 was ’end-of-life’ in 2018.

This is part of a larger, and ongoing effort to make the immense code-base of sparklyr a little easier to maintain, and hence reduce the risk of failures. As part of the same effort, the number of upstream packages that sparklyr depends on have been reduced. This has been happening across multiple CRAN releases, and in this latest release tibble, and rappdirs are no longer imported by sparklyr.

You can install it from CRAN with:

install.packages("bundle")

Let’s walk through what bundle does, and when you might need to use it.

Saving things is hard

We often think of a trained model as a self-contained R object. The model exists in memory in R and if we have some new data, the model object can generate predictions on its own:

In reality, model objects sometimes also make use of references to generate predictions. A reference is a piece of information that a model object refers to that isn’t part of the object itself; this could be something like a connection to a server, a file on disk, or an internal function in the package used to train the model. When we call predict(), model objects know where to look to retrieve that information:

Saving model objects can sometimes disrupt those references. Thus, if we want to train a model, save it, re-load it in a production setting, and generate predictions with it, we may run into issues:

We need some way to preserve access to those references. This new package provides a consistent interface for bundling model objects with their references so that they can be safely saved and re-loaded in production:

When to bundle your model

Let’s walk through building a couple of models using data on cell body segmentation.

library(tidymodels)
data(cells, package = "modeldata")

set.seed(123)
cell_split <- cells %>% 
  select(-case) %>%
  initial_split(strata = class)

cell_train <- training(cell_split)
cell_test  <- testing(cell_split)

First, let’s train a logistic regression model:

glm_fit <- glm(class ~ ., family = "binomial", data = cell_train)

If we’re satisfied with this model and think it is ready for production, we might want to deploy it somewhere, maybe as a REST API or as a Shiny app. A typical approach would be to:

• save our model object
• start up a new R session
• load the model object into the new session
• predict on new data with the loaded model object

The callr package is helpful for demonstrating this kind of situation; it allows us to start up a fresh R session and pass a few objects in.

We’ll just make use of two of the arguments to the function r():

• func: A function that, given a model object and some new data, will generate predictions, and
• args: A named list, giving the arguments to the above function.

Let’s save our model object to a temporary file and pass it to a fresh R session for prediction, like if we had deployed the model somewhere.

library(callr)

temp_file <- tempfile()
saveRDS(glm_fit, file = temp_file)

r(
  function(temp_file, new_data) {
    model_object <- readRDS(file = temp_file)
    predict(model_object, new_data)
  },
  args = list(
    temp_file = temp_file,
    new_data = head(cell_test)
  )
)

##          1          2          3          4          5          6 
## -4.8706401 -1.8143956  2.3386470 -1.2735249 -0.3586448  2.7865270

Nice! 😀

What if instead we wanted to train a neural network using tidymodels, with keras as the modeling engine?

cell_rec <- 
  recipe(class ~ ., data = cell_train) %>%
  step_YeoJohnson(all_numeric_predictors()) %>%
  step_normalize(all_numeric_predictors())

keras_spec <- 
  mlp(penalty = 0, epochs = 10) %>% 
  set_mode("classification") %>% 
  set_engine("keras", verbose = 0) 

keras_fit <- 
  workflow(cell_rec, keras_spec) %>%
  fit(data = cell_train)

Let’s try to save this to disk and then reload it in a new session.

temp_file <- tempfile()
saveRDS(keras_fit, file = temp_file)

r(
  function(temp_file, new_data) {
    library(workflows)
    model_object <- readRDS(file = temp_file)
    predict(model_object, new_data)
  },
  args = list(
    temp_file = temp_file,
    new_data = head(cell_test)
  )
)

## Error: ! error in callr subprocess
## Caused by error in `do.call(object$predict, args)`:
## ! 'what' must be a function or character string

Oh no! 😱

It turns out that keras models need to be saved in a special way. This is true of a handful of models, like XGBoost, and even some preprocessing steps, like UMAP. These special ways to save objects, like the ones that keras provide, are often referred to as native serialization. Methods for native serialization know which references need to be brought along in order for an object to effectively do its thing in a new environment, but they are different for each model.

The bundle package provides a consistent way to deal with all these kinds of special serialization. The package provides two functions, bundle() and unbundle(), that take care of all of the minutae of preparing to save and load R objects effectively. You bundle() your model before you save it:

library(bundle)
temp_file <- tempfile()
keras_bundle <- bundle(keras_fit)
saveRDS(keras_bundle, file = temp_file)

And then you unbundle() after you read the object in a new session:

r(
  function(temp_file, new_data) {
    library(bundle)
    library(workflows)
    model_bundle <- readRDS(file = temp_file)
    model_object <- unbundle(model_bundle)
    predict(model_object, new_data)
  },
  args = list(
    temp_file = temp_file,
    new_data = head(cell_test)
  )
)

## # A tibble: 6 × 1
##   .pred_class
##   <fct>      
## 1 PS         
## 2 PS         
## 3 WS         
## 4 PS         
## 5 PS         
## 6 WS

Hooray! 🎉

We have support in bundle for a wide variety of models that require (or sometimes require) special handling for serialization, from H2O to torch luz models. Soon bundle will be integrated into vetiver, for better and more robust deployment options. If you use a model that needs special serialization and is not yet supported, let us know in an issue.

Acknowledgements

Thank you so much to everyone who contributed to this first release: @dfalbel, @juliasilge, @qiushiyan, and @simonpcouch. I would especially like to highlight Simon’s contributions, which have been central to bundle getting off the ground!

We are thrilled to announce the release of vetiver, a framework for MLOps tasks in R and Python! The goal of vetiver is to provide fluent tooling to version, share, deploy, and monitor a trained model. If you like perfume or candles, you may recognize this name; vetiver, also known as the “oil of tranquility”, is used as a stabilizing ingredient in perfumery to preserve more volatile fragrances.

You can install the released version of vetiver for R from CRAN:

install.packages("vetiver")

You can install the released version of vetiver for Python from PyPI:

pip install vetiver

We are sharing more about what vetiver is and how it works over on the RStudio blog so check that out, but we want to share here as well!

Train a model

For this example, let’s work with data on everyone’s favorite dataset on fuel efficiency for cars to predict miles per gallon. In R, we can train a decision tree model to predict miles per gallon using a tidymodels workflow:

library(tidymodels)

car_mod <-
    workflow(mpg ~ ., decision_tree(mode = "regression")) %>%
    fit(mtcars)

In Python, we can train the same kind of model using scikit-learn:

from vetiver.data import mtcars
from sklearn import tree
car_mod = tree.DecisionTreeRegressor().fit(mtcars, mtcars["mpg"])

For both R and Python, the car_mod object is a fitted model, with parameters estimated using our training data mtcars.

Create a vetiver model

We can create a vetiver_model() in R or VetiverModel() in Python from the trained model; a vetiver model object collects the information needed to store, version, and deploy a trained model.

library(vetiver)
v <- vetiver_model(car_mod, "cars_mpg")
v
#> 
#> ── cars_mpg ─ <butchered_workflow> model for deployment 
#> A rpart regression modeling workflow using 10 features

from vetiver import VetiverModel
v = VetiverModel(car_mod, model_name = "cars_mpg", 
                 save_ptype = True, ptype_data = mtcars)
v.description
#> "Scikit-learn <class 'sklearn.tree._classes.DecisionTreeRegressor'> model"

See our documentation for how to use these deployable model objects and:

• publish and version your model
• deploy your model as a REST API

Be sure to also read more on the RStudio blog.

Acknowledgements

We’d like to extend our thanks to all of the contributors who helped make these initial releases of vetiver for R and Python possible!

• R package: @cderv, @ggpinto, @isabelizimm, @juliasilge, and @mfansler

• Python package: @has2k1, and @isabelizimm

With the plumbertableau package (and its corresponding Python package, fastapitableau), you can use functions or models created in R or Python from Tableau through an API. These packages allow you to showcase cutting-edge data science results in your organization’s preferred dashboard tool.

While this post mentions R, anything possible with R and plumbertableau is also doable with Python and fastapitableau.

Foster Data Analytics Capabilities With plumbertableau

With plumbertableau, you can fully develop your model with code-first data science. The package uses plumber to create an API directly from your code. Since your model is fully developed in your data science editor, it can use all the packages and complex calculations it needs.

You can extract the best data science results using R’s capabilities as your model will not be constrained by Tableau’s environment.

Improve Data Quality With APIs for Continuous Use

Seamless integration between analytic platforms prevents issues like using outdated, inaccurate, or incomplete data. Rather than depending on a manual process, data scientists can depend on their data pipelines to ensure data integrity.

With plumbertableau, your tools are integrated through an API. The Tableau dashboard displays results without any intermediate manipulation like copy-and-pasting code or uploading datasets. You can work in confidence knowing your results are synchronized, accurate, and reproducible.

Increase Deliverability by Streamlining Data Pipelines

If your model has many dependencies or versioning requirements, it can be difficult to handle them outside of the development environment. Debugging is even more time-consuming when you need to work in separate environments to figure out what went wrong.

With RStudio Connect, you can publish directly plumbertableau extensions directly from the RStudio IDE. RStudio Connect automatically manages your API’s dependent packages and files to recreate an environment closely mimicking your local development environment. And since all your R code remains in R, you can use your usual data science techniques to efficiently resolve issues.

Read more on the Hosting page of the plumber package.

How to Use plumbertableau: XGBoost with Dynamic Model Output Example

In this walkthrough, we will be using data from the Seattle Open Data Portal to predict the paid parking occupancy percentage in various areas around the city. We will run an XGBoost model in RStudio, create a plumbertableau extension to embed into Tableau, and visualize and interact with the model in a Tableau dashboard. The code is here for reproducibility purposes; however, it will require an RStudio Connect account to complete.

The plumbertableau and fastapi packages have wonderful documentation. Be sure to read them for more information on:

• The anatomy of the extensions
• Details on setting up RStudio Connect and Tableau
• Other examples to try out in your Tableau dashboards

1. Build the model

First, we need to build a model. This walkthrough won’t be covering how to create, tune, or validate a model. If you’d like to learn more on models and machine learning, check out the tidymodels website and Julia Silge’s fantastic screencasts and tutorials.

Load Libraries

library(tidyverse)
library(RSocrata)
library(lubridate)
library(usemodels)
library(tidymodels)

Download and Clean Data

The Seattle Open Data Portal uses Socrata, a data management tool, for its APIs. We can use the RSocrata package to download the data.

parking_data <-
  RSocrata::read.socrata(
    "https://data.seattle.gov/resource/rke9-rsvs.json?$where=sourceelementkey <= 1020"
  )

parking_id <-
  parking_data %>%
  group_by(blockfacename, location.coordinates) %>%
  mutate(id = cur_group_id()) %>%
  ungroup()

parking_clean <-
  parking_id %>%
  mutate(across(c(parkingspacecount, paidoccupancy), as.numeric),
         occupancy_pct = paidoccupancy / parkingspacecount) %>%
  group_by(
    id = id,
    hour = as.numeric(hour(occupancydatetime)),
    month = as.numeric(month(occupancydatetime)),
    dow = as.numeric(wday(occupancydatetime)),
    date = date(occupancydatetime)
  ) %>%
  summarize(occupancy_pct = mean(occupancy_pct, na.rm = TRUE)) %>%
  drop_na() %>%
  ungroup()

We will also need information on the city blocks, so let’s create that dataset.

parking_information <-
  parking_id %>%
  mutate(loc = location.coordinates) %>%
  select(id, blockfacename, loc) %>%
  distinct(id, blockfacename, loc) %>%
  unnest_wider(loc, c('loc1', 'loc2'))

Create Training Data

Now, let’s create the training set from our original data.

parking_split <-
  parking_clean %>%
  arrange(date) %>%
  select(-date) %>%
  initial_time_split(prop = 0.75)

Train and Tune the Model

Here, we train and tune the model. We select the model with the best RSME to use in our dashboard.

xgboost_recipe <-
  recipe(formula = occupancy_pct ~ ., data = parking_clean) %>%
  step_zv(all_predictors())  %>%
  prep()

xgboost_folds <-
  recipes::bake(xgboost_recipe,
                new_data = training(parking_split)) %>%
  rsample::vfold_cv(v = 5)

xgboost_model <-
  boost_tree(
    mode = "regression",
    trees = 1000,
    min_n = tune(),
    tree_depth = tune(),
    learn_rate = tune(),
    loss_reduction = tune()
  ) %>%
  set_engine("xgboost", objective = "reg:squarederror")

xgboost_params <-
  parameters(min_n(),
             tree_depth(),
             learn_rate(),
             loss_reduction())

xgboost_grid <-
  grid_max_entropy(xgboost_params,
                   size = 5)

xgboost_wf <-
  workflows::workflow() %>%
  add_model(xgboost_model) %>%
  add_formula(occupancy_pct ~ .)

xgboost_tuned <- tune::tune_grid(
  object = xgboost_wf,
  resamples = xgboost_folds,
  grid = xgboost_grid,
  metrics = yardstick::metric_set(rmse, rsq, mae),
  control = tune::control_grid(verbose = TRUE)
)

xgboost_best <-
  xgboost_tuned %>%
  tune::select_best("rmse")

xgboost_final <-
  xgboost_model %>%
  finalize_model(xgboost_best)

We bundle the recipe and fitted model in an object so we can use it later.

train_processed <-
  bake(xgboost_recipe, new_data = training(parking_split))

prediction_fit <-
  xgboost_final %>%
  fit(formula = occupancy_pct ~ .,
      data    = train_processed)

model_details <- list(model = xgboost_final,
                      recipe = xgboost_recipe,
                      prediction_fit = prediction_fit)

Save Objects for the plumbertableau Extension

We’ll want to save our data and our model so that we can use them in the extension. If you have an RStudio Connect account, the pins package is a great choice for saving these objects.

rsc <-
  pins::board_rsconnect(server = Sys.getenv("CONNECT_SERVER"),
                        key = Sys.getenv("CONNECT_API_KEY"))

pins::pin_write(
  board = rsc,
  x = model_details,
  name = "seattle_parking_model",
  description = "Seattle Occupancy Percentage XGBoost Model",
  type = "rds"
)

pins::pin_write(
  board = rsc,
  x = parking_information,
  name = "seattle_parking_info",
  description = "Seattle Parking Information",
  type = "rds"
)

2. Create a plumbertableau Extension

Next, we will use our model to create a plumbertableau extension. As noted previously, the plumbertableau extension is a Plumber API with some special annotations.

Create an R script called plumber.R. At the top, we list the libraries we’ll need.

library(plumber)
library(pins)
library(tibble)
library(xgboost)
library(lubridate)
library(dplyr)
library(tidyr)
library(tidymodels)
library(plumbertableau)

We want to bring in our model details and our data. If you pinned your data, you’ll change the name of the pin below.

rsc <-
  pins::board_rsconnect(
    server = Sys.getenv("CONNECT_SERVER"),
    key = Sys.getenv("CONNECT_API_KEY")
  )

xgboost_model <-
  pins::pin_read("isabella.velasquez/seattle_parking_model", board = rsc)

Now, we add our annotations. Note that we use plumbertableau annotations, which are slightly different than the ones from plumber.

• We use tableauArg rather than params.
• We specify what is returned to Tableau with tableauReturn.
• We must use post for what is being returned.

#* @apiTitle Seattle Parking Occupancy Percentage Prediction API
#* @apiDescription Return the predicted occupancy percentage at various Seattle locations

#* @tableauArg block_id:integer numeric block ID
#* @tableauArg ndays:integer number of days in the future for the prediction

#* @tableauReturn [numeric] Predicted occupancy rate
#* @post /pred

Now, we create our function with the arguments station_id and ndays. These will have corresponding arguments in Tableau. The function will output our predicted occupancy percentage, which will be what we visualize and interact with in the dashboard.

This function takes the city block and number of days in the future to give us the predicted occupancy percentage at that time.

function(block_id, ndays) {
  times <- Sys.time() + lubridate::ddays(ndays)
  
  current_time <-
    tibble::tibble(times = times,
                   id = block_id)
  
  current_prediction  <-
    current_time %>%
    transmute(
      id = id,
      hour = hour(times),
      month = month(times),
      dow = wday(times),
      occupancy_pct = NA
    ) %>%
    bake(xgboost_model$recipe, .)
  
  parking_prediction <-
    xgboost_model$prediction_fit %>%
    predict(new_data = current_prediction)
  
  predictions <-
    parking_prediction$.pred
  
  predictions[[1]]
  
}

Finally, we finish off our script with the extension footer needed for plumbertableau extensions.

#* @plumber
tableau_extension

Here is the full plumber.R script:

library(plumber)
library(pins)
library(tibble)
library(xgboost)
library(lubridate)
library(dplyr)
library(tidyr)
library(tidymodels)
library(plumbertableau)

rsc <-
  pins::board_rsconnect(server = Sys.getenv("CONNECT_SERVER"),
                        key = Sys.getenv("CONNECT_API_KEY"))

xgboost_model <-
  pins::pin_read("isabella.velasquez/seattle_parking_model", board = rsc)

#* @apiTitle Seattle Parking Occupancy Percentage Prediction API
#* @apiDescription Return the predicted occupancy percentage at various Seattle locations

#* @tableauArg block_id:integer numeric block ID
#* @tableauArg ndays:integer number of days in the future for the prediction

#* @tableauReturn [numeric] Predicted occupancy rate
#* @post /pred

function(block_id, ndays) {
  times <- Sys.time() + lubridate::ddays(ndays)
  
  current_time <-
    tibble::tibble(times = times,
                   id = block_id)
  
  current_prediction  <-
    current_time %>%
    transmute(
      id = id,
      hour = hour(times),
      month = month(times),
      dow = wday(times),
      occupancy_pct = NA
    ) %>%
    bake(xgboost_model$recipe, .)
  
  parking_prediction <-
    xgboost_model$prediction_fit %>%
    predict(new_data = current_prediction)
  
  predictions <-
    parking_prediction$.pred
  
  predictions[[1]]
  
}

#* @plumber
tableau_extension

3. Host your API

We have to host our API so that it can be accessed in Tableau. In our case, we publish it to RStudio Connect.

Once hosted, plumbertableau automatically generates a documentation page. Notice that the SCRIPT_* value is not R code. This is a Tableau command that we will use to connect our extension and Tableau.

4. Create a calculated field in Tableau

There are a few steps you need to take so that Tableau can use your plumbertableau extension. If you are using RStudio Connect, read the documentation on how to configure RStudio Connect as an analytic extension.

Create a new workbook and upload the station_information file. Under Analysis, turn off Aggregate Measures. Drop Lat into Rows and Lon into Columns, which will create a map. Save the workbook.

Make sure your workbook knows to connect to RStudio Connect by going to Analysis > Manage Analytic Extensions Connection > Choose a Connection. Then, select your Connect account.

Drag Id into the “Detail” mark. Create a parameter called “Days in the Future”. We’re using our model to predict parking occupancy percentage for that date. Show the parameter on the worksheet.

Create a calculated field using the SCRIPT from the plumbertableau documentation page:

SCRIPT_REAL("/plumbertableau-xgboost-example/pred", block_id, ndays) 

For each tableauArg we have listed in the extension, we will replace it with its corresponding Tableau value. If you’re following along, this means block_id will become ATTR([Id]) and ndays will become ATTR([Days in the Future]).

SCRIPT_REAL("/plumbertableau-xgboost-example/pred", ATTR([Id]), ATTR([Days in the Future]))

5. Run model and visualize results in Tableau

That’s it! Once you embed your extension in Tableau’s calculated fields, you can use your model’s results in your Tableau dashboard like any other measure or dimension.

We can change the ndays argument to get new predictions from our XGBoost model and display them on our Tableau dashboard.

You can style your Tableau dashboard and then provide your users something that is not only aesthetically pleasing, but is dynamically calculating predictions based on a model you have created in R.

Conclusion

With plumbertableau, you can showcase sophisticated model results that are easy to integrate, debug, and reproduce. Your work will be at the forefront of data science while being visualized in Tableau’s easy, point-and-click interface.

Learn More

Watch James Blair showcase plumbertableau in Leveraging R & Python in Tableau with RStudio Connect:

More on how RStudio supports interoperability across tools can be found on our BI and Data Science Overview Page.

Teams often need access to key data to do their work, but have you ever opened your coworker’s script to see:

dat <-  
 read_csv("C://Users/someone_else/data/dataset.csv")

more_dat <- 
 read_csv("S://Path_to_mapped_drive_that_you_dont_have/dataset.csv")

Yikes! How will you get these files? Let’s hope you can reach your coworker before they’ve logged off for the day.

How can your code be reproducible if you have to manually change the file paths? Shudder.

What if you need to make edits to the data, will you have to keep copying CSVs and emailing files forever? Double shudder.

What if your coworker accidentally forwards your email to someone who is not supposed to have access? Oh no.

We can struggle to share data assets easily and safely, relying on emailed files to keep our analyses up to date. This makes it difficult to keep current or know what version of the data we’re using. If you’ve ever experienced any of the scenarios above, consider pins as a solution that can help you share your data assets.

What is a pin, anyway?

Pins, from the R package of the same name, are a versatile way to publish R objects on a virtual corkboard so you can share them across projects and people.

Good pins are data or assets that are a few hundred megabytes or smaller. You can pin just about any object: data, models, JSON files, feather files from the Arrow package, and more. One of the most frequent use cases is pinning small data sets — often ephemeral data or reference tables that don’t quite merit being in a database, but seemingly don’t have a good home elsewhere (until now).

Pins get published to a board, which can be an RStudio Connect server, an AWS S3 bucket or Azure Blob Storage, a shared drive like Dropbox or Sharepoint, or a variety of other options. Try it out for yourself — read in this data set we’ve pinned for you on RStudio Connect!

# Install the latest pins from CRAN
install.packages("pins")

library(pins)

# Identify the board
board <-
  board_url(c("penguins" = "https://colorado.rstudio.com/rsc/example_pin/"))

# Read the shared data
board %>%
  pin_read("penguins")

In short, if you’ve ever wondered where to put an R object that you or your colleague will need to use again, you might just want to pin it.

Pins for Sharing Across Projects and Teams

One of the greatest strengths of pins is how your pin becomes accessible directly from your R scripts and the R scripts of anyone else to whom you’ve given access. Different projects can include code that reads the same pin without creating more copies of the data:

It’s easier (and safer) to share a pin across multiple projects or people than to email files around. Pins respect the access controls of the board. Say you’ve pinned to RStudio Connect: you can control who gets to use the pin, just like any other piece of content.

Pins for Updating and Versioning

You may be wondering why use pins if you already have a shared drive with your teammates. But what happens if you need to replace the dataset with a new one? Do you email everybody to let them know? Is it dataFINALv2.csv? Or dataFINALfinal.csv?

The pins package retrieves the newest version of the pin by default. That means pin users never have to worry about getting a stale version of the pin. If you need to update your pin regularly, a scheduled R Markdown on RStudio Connect can handle this task for you, so your pin stays fresh.

But you’re not locked into losing old versions of a pin. You can version pins so that writing to an existing pin adds a new copy rather than replacing the existing data.

Here’s what versioning looks like using a temporary board:

library(pins)


board2 <- board_temp(versioned = TRUE)

board2 %>% pin_write(1:5, name = "x", type = "rds")
#> Creating new version '20210304T050607Z-ab444'
#> Writing to pin 'x'

board2 %>% pin_write(2:6, name = "x", type = "rds")
#> Creating new version '20210304T050607Z-a077a'
#> Writing to pin 'x'

board2 %>% pin_write(3:7, name = "x", type = "rds")
#> Creating new version '20210304T050607Z-0a284'
#> Writing to pin 'x'

# see all versions
board2 %>% pin_versions("x")
#> # A tibble: 3 × 3
#>   version                created             hash 
#>   <chr>                  <dttm>              <chr>
#> 1 20210304T050607Z-0a284 2021-03-04 05:06:00 0a284
#> 2 20210304T050607Z-a077a 2021-03-04 05:06:00 a077a
#> 3 20210304T050607Z-ab444 2021-03-04 05:06:00 ab444

Learn More

With pins, you and your teammates can know where your important data assets are, how to access them, and whether they are the correct version. You can work with confidence knowing you’re using the right asset, your work is reproducible, and you’re following good practices for data management.

There’s more to explore with pins. We’re excited to share how you can adopt them into your workflow.

Learn more about how and when to use pins:

• The pins package documentation
• RStudio Pro Tips: Creating Efficient Workflows with pins and RStudio Connect

See pins in action:

• Pins can pull intensive ETL processes out of your apps, improve performance, and save you the hassle of redeploying whenever the underlying data changes.
  • Watch: Deploying End-To-End Data Science with Shiny, Plumber, and Pins
• Pins can play a key role in MLOps, publishing versioned models, and monitoring model metrics.
  • Read: Model Monitoring with R Markdown, pins, and RStudio Connect

To install sparklyr.sedona from GitHub using the remotes package ¹, run

remotes::install_github(repo = "apache/incubator-sedona", subdir = "R/sparklyr.sedona")

In this blog post, we will provide a quick introduction to sparklyr.sedona, outlining the motivation behind this sparklyr extension, and presenting some example sparklyr.sedona use cases involving Spark spatial RDDs, Spark dataframes, and visualizations.

Motivation for sparklyr.sedona

A suggestion from the mlverse survey results earlier this year mentioned the need for up-to-date R interfaces for Spark-based GIS frameworks. While looking into this suggestion, we learned about Apache Sedona, a geospatial data system powered by Spark that is modern, efficient, and easy to use. We also realized that while our friends from the Spark open-source community had developed a sparklyr extension for GeoSpark, the predecessor of Apache Sedona, there was no similar extension making more recent Sedona functionalities easily accessible from R yet. We therefore decided to work on sparklyr.sedona, which aims to bridge the gap between Sedona and R.

The lay of the land²

We hope you are ready for a quick tour through some of the RDD-based and Spark-dataframe-based functionalities in sparklyr.sedona, and also, some bedazzling visualizations derived from geospatial data in Spark.

In Apache Sedona, Spatial Resilient Distributed Datasets(SRDDs) are basic building blocks of distributed spatial data encapsulating “vanilla” RDDs of geometrical objects and indexes. SRDDs support low-level operations such as Coordinate Reference System (CRS) transformations, spatial partitioning, and spatial indexing. For example, with sparklyr.sedona, SRDD-based operations we can perform include the following:

• Importing some external data source into a SRDD:

library(sparklyr)
library(sparklyr.sedona)

sedona_git_repo <- normalizePath("~/incubator-sedona")
data_dir <- file.path(sedona_git_repo, "core", "src", "test", "resources")

sc <- spark_connect(master = "local")

pt_rdd <- sedona_read_dsv_to_typed_rdd(
  sc,
  location = file.path(data_dir, "arealm.csv"),
  type = "point"
)

• Applying spatial partitioning to all data points:

sedona_apply_spatial_partitioner(pt_rdd, partitioner = "kdbtree")

• Building spatial index on each partition:

sedona_build_index(pt_rdd, type = "quadtree")

• Joining one spatial data set with another using “contain” or “overlap” as the join predicate:

polygon_rdd <- sedona_read_dsv_to_typed_rdd(
  sc,
  location = file.path(data_dir, "primaryroads-polygon.csv"),
  type = "polygon"
)

pts_per_region_rdd <- sedona_spatial_join_count_by_key(
  pt_rdd,
  polygon_rdd,
  join_type = "contain",
  partitioner = "kdbtree"
)

It is worth mentioning that sedona_spatial_join() will perform spatial partitioning and indexing on the inputs using the partitioner and index_type only if the inputs are not partitioned or indexed as specified already.

From the examples above, one can see that SRDDs are great for spatial operations requiring fine-grained control, e.g., for ensuring a spatial join query is executed as efficiently as possible with the right types of spatial partitioning and indexing.

Finally, we can try visualizing the join result above, using a choropleth map:

sedona_render_choropleth_map(
  pts_per_region_rdd,
  resolution_x = 1000,
  resolution_y = 600,
  output_location = tempfile("choropleth-map-"),
  boundary = c(-126.790180, -64.630926, 24.863836, 50.000),
  base_color = c(63, 127, 255)
)

which gives us the following:

Wait, but something seems amiss. To make the visualization above look nicer, we can overlay it with the contour of each polygonal region:

contours <- sedona_render_scatter_plot(
  polygon_rdd,
  resolution_x = 1000,
  resolution_y = 600,
  output_location = tempfile("scatter-plot-"),
  boundary = c(-126.790180, -64.630926, 24.863836, 50.000),
  base_color = c(255, 0, 0),
  browse = FALSE
)

sedona_render_choropleth_map(
  pts_per_region_rdd,
  resolution_x = 1000,
  resolution_y = 600,
  output_location = tempfile("choropleth-map-"),
  boundary = c(-126.790180, -64.630926, 24.863836, 50.000),
  base_color = c(63, 127, 255),
  overlay = contours
)

which gives us the following:

With some low-level spatial operations taken care of using the SRDD API and the right spatial partitioning and indexing data structures, we can then import the results from SRDDs to Spark dataframes. When working with spatial objects within Spark dataframes, we can write high-level, declarative queries on these objects using dplyr verbs in conjunction with Sedona spatial UDFs, e.g. ³ , the following query tells us whether each of the 8 nearest polygons to the query point contains that point, and also, the convex hull of each polygon.

tbl <- DBI::dbGetQuery(
  sc, "SELECT ST_GeomFromText(\"POINT(-66.3 18)\") AS `pt`"
)
pt <- tbl$pt[[1]]
knn_rdd <- sedona_knn_query(
  polygon_rdd, x = pt, k = 8, index_type = "rtree"
)

knn_sdf <- knn_rdd %>%
  sdf_register() %>%
  dplyr::mutate(
    contains_pt = ST_contains(geometry, ST_Point(-66.3, 18)),
    convex_hull = ST_ConvexHull(geometry)
  )

knn_sdf %>% print()

# Source: spark<?> [?? x 3]
  geometry                         contains_pt convex_hull
  <list>                           <lgl>       <list>
1 <POLYGON ((-66.335674 17.986328… TRUE        <POLYGON ((-66.335674 17.986328,…
2 <POLYGON ((-66.335432 17.986626… TRUE        <POLYGON ((-66.335432 17.986626,…
3 <POLYGON ((-66.335432 17.986626… TRUE        <POLYGON ((-66.335432 17.986626,…
4 <POLYGON ((-66.335674 17.986328… TRUE        <POLYGON ((-66.335674 17.986328,…
5 <POLYGON ((-66.242489 17.988637… FALSE       <POLYGON ((-66.242489 17.988637,…
6 <POLYGON ((-66.242489 17.988637… FALSE       <POLYGON ((-66.242489 17.988637,…
7 <POLYGON ((-66.24221 17.988799,… FALSE       <POLYGON ((-66.24221 17.988799, …
8 <POLYGON ((-66.24221 17.988799,… FALSE       <POLYGON ((-66.24221 17.988799, …

Acknowledgements

The author of this blog post would like to thank Jia Yu, the creator of Apache Sedona, and Lorenz Walthert for their suggestion to contribute sparklyr.sedona to the upstream incubator-sedona repository. Jia has provided extensive code-review feedback to ensure sparklyr.sedona complies with coding standards and best practices of the Apache Sedona project, and has also been very helpful in the instrumentation of CI workflows verifying sparklyr.sedona works as expected with snapshot versions of Sedona libraries from development branches.

The author is also grateful for his colleague Sigrid Keydana for valuable editorial suggestions on this blog post.

That’s all. Thank you for reading!

Photo by NASA on Unsplash

To install sparklyr 1.7 from CRAN, run

install.packages("sparklyr")

In this blog post, we wish to present the following highlights from the sparklyr 1.7 release:

• Image and binary data sources
• New spark_apply() capabilities
• Better integration with sparklyr extensions
• Other exciting news

Image and binary data sources

As a unified analytics engine for large-scale data processing, Apache Spark is well-known for its ability to tackle challenges associated with the volume, velocity, and last but not least, the variety of big data. Therefore it is hardly surprising to see that – in response to recent advances in deep learning frameworks – Apache Spark has introduced built-in support for image data sources and binary data sources (in releases 2.4 and 3.0, respectively). The corresponding R interfaces for both data sources, namely, spark_read_image() and spark_read_binary(), were shipped recently as part of sparklyr 1.7.

The usefulness of data source functionalities such as spark_read_image() is perhaps best illustrated by a quick demo below, where spark_read_image(), through the standard Apache Spark ImageSchema, helps connecting raw image inputs to a sophisticated feature extractor and a classifier, forming a powerful Spark application for image classifications.

The demo

In this demo, we shall construct a scalable Spark ML pipeline capable of classifying images of cats and dogs accurately and efficiently, using spark_read_image() and a pre-trained convolutional neural network code-named Inception (Szegedy et al. (2015)).

The first step to building such a demo with maximum portability and repeatability is to create a sparklyr extension that accomplishes the following:

• Specifying the required MVN dependencies of this demo (namely, the Spark Deep Learning library (Databricks, Inc. (2019)), which contains an Inception-V3-based image feature extractor accessible through the Spark ML Transformer interface)
• Bundling with itself two randomly selected ¹ and disjoint subsets of the dogs-vs-cats dataset (Elson et al. (2007)) as train and test data, which are stored in the extdata/{train,test} sub directories of the package)

A reference implementation of such a sparklyr extension can be found in here.

The second step, of course, is to make use of the above-mentioned sparklyr extension to perform some feature engineering. We will see very high-level features being extracted intelligently from each cat/dog image based on what the pre-built Inception-V3 convolutional neural network has already learned from classifying a much broader collection of images:

library(sparklyr)
library(sparklyr.deeperer)

# NOTE: the correct spark_home path to use depends on the configuration of the
# Spark cluster you are working with.
spark_home <- "/usr/lib/spark"
sc <- spark_connect(master = "yarn", spark_home = spark_home)

data_dir <- copy_images_to_hdfs()

# extract features from train- and test-data
image_data <- list()
for (x in c("train", "test")) {
  # import
  image_data[[x]] <- c("dogs", "cats") %>%
    lapply(
      function(label) {
        numeric_label <- ifelse(identical(label, "dogs"), 1L, 0L)
        spark_read_image(
          sc, dir = file.path(data_dir, x, label, fsep = "/")
        ) %>%
          dplyr::mutate(label = numeric_label)
      }
    ) %>%
      do.call(sdf_bind_rows, .)

  dl_featurizer <- invoke_new(
    sc,
    "com.databricks.sparkdl.DeepImageFeaturizer",
    random_string("dl_featurizer") # uid
  ) %>%
    invoke("setModelName", "InceptionV3") %>%
    invoke("setInputCol", "image") %>%
    invoke("setOutputCol", "features")
  image_data[[x]] <-
    dl_featurizer %>%
    invoke("transform", spark_dataframe(image_data[[x]])) %>%
    sdf_register()
}

Third step: equipped with features that summarize the content of each image well, we can build a Spark ML pipeline that recognizes cats and dogs using only logistic regression ²

label_col <- "label"
prediction_col <- "prediction"
pipeline <- ml_pipeline(sc) %>%
  ml_logistic_regression(
    features_col = "features",
    label_col = label_col,
    prediction_col = prediction_col
  )
model <- pipeline %>% ml_fit(image_data$train)

Finally, we can evaluate the accuracy of this model on the test images:

predictions <- model %>%
  ml_transform(image_data$test) %>%
  dplyr::compute()

cat("Predictions vs. labels:\n")
predictions %>%
  dplyr::select(!!label_col, !!prediction_col) %>%
  print(n = sdf_nrow(predictions))

cat("\nAccuracy of predictions:\n")
predictions %>%
  ml_multiclass_classification_evaluator(
    label_col = label_col,
    prediction_col = prediction_col,
    metric_name = "accuracy"
  ) %>%
    print()

## Predictions vs. labels:
## # Source: spark<?> [?? x 2]
##    label prediction
##    <int>      <dbl>
##  1     1          1
##  2     1          1
##  3     1          1
##  4     1          1
##  5     1          1
##  6     1          1
##  7     1          1
##  8     1          1
##  9     1          1
## 10     1          1
## 11     0          0
## 12     0          0
## 13     0          0
## 14     0          0
## 15     0          0
## 16     0          0
## 17     0          0
## 18     0          0
## 19     0          0
## 20     0          0
##
## Accuracy of predictions:
## [1] 1

New spark_apply() capabilities

Optimizations & custom serializers

Many sparklyr users who have tried to run spark_apply() or doSpark to parallelize R computations among Spark workers have probably encountered some challenges arising from the serialization of R closures. In some scenarios, the serialized size of the R closure can become too large, often due to the size of the enclosing R environment required by the closure. In other scenarios, the serialization itself may take too much time, partially offsetting the performance gain from parallelization. Recently, multiple optimizations went into sparklyr to address those challenges. One of the optimizations was to make good use of the broadcast variable construct in Apache Spark to reduce the overhead of distributing shared and immutable task states across all Spark workers. In sparklyr 1.7, there is also support for custom spark_apply() serializers, which offers more fine-grained control over the trade-off between speed and compression level of serialization algorithms. For example, one can specify

options(sparklyr.spark_apply.serializer = "qs")

,

which will apply the default options of qs::qserialize() to achieve a high compression level, or

options(sparklyr.spark_apply.serializer = function(x) qs::qserialize(x, preset = "fast"))
options(sparklyr.spark_apply.deserializer = function(x) qs::qdeserialize(x))

,

which will aim for faster serialization speed with less compression.

Inferring dependencies automatically

In sparklyr 1.7, spark_apply() also provides the experimental auto_deps = TRUE option. With auto_deps enabled, spark_apply() will examine the R closure being applied, infer the list of required R packages, and only copy the required R packages and their transitive dependencies to Spark workers. In many scenarios, the auto_deps = TRUE option will be a significantly better alternative compared to the default packages = TRUE behavior, which is to ship everything within .libPaths() to Spark worker nodes, or the advanced packages = <package config> option, which requires users to supply the list of required R packages or manually create a spark_apply() bundle.

Better integration with sparklyr extensions

Substantial effort went into sparklyr 1.7 to make lives easier for sparklyr extension authors. Experience suggests two areas where any sparklyr extension can go through a frictional and non-straightforward path integrating with sparklyr are the following:

• The dbplyr SQL translation environment
• Invocation of Java/Scala functions from R

We will elaborate on recent progress in both areas in the sub-sections below.

Customizing the dbplyr SQL translation environment

sparklyr extensions can now customize sparklyr’s dbplyr SQL translations through the spark_dependency() specification returned from spark_dependencies() callbacks. This type of flexibility becomes useful, for instance, in scenarios where a sparklyr extension needs to insert type casts for inputs to custom Spark UDFs. We can find a concrete example of this in sparklyr.sedona, a sparklyr extension to facilitate geo-spatial analyses using Apache Sedona. Geo-spatial UDFs supported by Apache Sedona such as ST_Point() and ST_PolygonFromEnvelope() require all inputs to be DECIMAL(24, 20) quantities rather than DOUBLEs. Without any customization to sparklyr’s dbplyr SQL variant, the only way for a dplyr query involving ST_Point() to actually work in sparklyr would be to explicitly implement any type cast needed by the query using dplyr::sql(), e.g.,

my_geospatial_sdf <- my_geospatial_sdf %>%
  dplyr::mutate(
    x = dplyr::sql("CAST(`x` AS DECIMAL(24, 20))"),
    y = dplyr::sql("CAST(`y` AS DECIMAL(24, 20))")
  ) %>%
  dplyr::mutate(pt = ST_Point(x, y))

.

This would, to some extent, be antithetical to dplyr’s goal of freeing R users from laboriously spelling out SQL queries. Whereas by customizing sparklyr’s dplyr SQL translations (as implemented in here and here ), sparklyr.sedona allows users to simply write

my_geospatial_sdf <- my_geospatial_sdf %>% dplyr::mutate(pt = ST_Point(x, y))

instead, and the required Spark SQL type casts are generated automatically.

Improved interface for invoking Java/Scala functions

In sparklyr 1.7, the R interface for Java/Scala invocations saw a number of improvements.

With previous versions of sparklyr, many sparklyr extension authors would run into trouble when attempting to invoke Java/Scala functions accepting an Array[T] as one of their parameters, where T is any type bound more specific than java.lang.Object / AnyRef. This was because any array of objects passed through sparklyr’s Java/Scala invocation interface will be interpreted as simply an array of java.lang.Objects in absence of additional type information. For this reason, a helper function jarray() was implemented as part of sparklyr 1.7 as a way to overcome the aforementioned problem. For example, executing

sc <- spark_connect(...)

arr <- jarray(
  sc,
  seq(5) %>% lapply(function(x) invoke_new(sc, "MyClass", x)),
  element_type = "MyClass"
)

will assign to arr a reference to an Array[MyClass] of length 5, rather than an Array[AnyRef]. Subsequently, arr becomes suitable to be passed as a parameter to functions accepting only Array[MyClass]s as inputs. Previously, some possible workarounds of this sparklyr limitation included changing function signatures to accept Array[AnyRef]s instead of Array[MyClass]s, or implementing a “wrapped” version of each function accepting Array[AnyRef] inputs and converting them to Array[MyClass] before the actual invocation. None of such workarounds was an ideal solution to the problem.

Another similar hurdle that was addressed in sparklyr 1.7 as well involves function parameters that must be single-precision floating point numbers or arrays of single-precision floating point numbers. For those scenarios, jfloat() and jfloat_array() are the helper functions that allow numeric quantities in R to be passed to sparklyr’s Java/Scala invocation interface as parameters with desired types.

In addition, while previous verisons of sparklyr failed to serialize parameters with NaN values correctly, sparklyr 1.7 preserves NaNs as expected in its Java/Scala invocation interface.

Other exciting news

There are numerous other new features, enhancements, and bug fixes made to sparklyr 1.7, all listed in the NEWS.md file of the sparklyr repo and documented in sparklyr’s HTML reference pages. In the interest of brevity, we will not describe all of them in great detail within this blog post.

Acknowledgement

In chronological order, we would like to thank the following individuals who have authored or co-authored pull requests that were part of the sparklyr 1.7 release:

• @yitao-li
• @mzorko
• @jozefhajnala
• @lresende

We’re also extremely grateful to everyone who has submitted feature requests or bug reports, many of which have been tremendously helpful in shaping sparklyr into what it is today.

Furthermore, the author of this blog post is indebted to @skeydan for her awesome editorial suggestions. Without her insights about good writing and story-telling, expositions like this one would have been less readable.

If you wish to learn more about sparklyr, we recommend visiting sparklyr.ai, spark.rstudio.com, and also reading some previous sparklyr release posts such as sparklyr 1.6 and sparklyr 1.5.

That is all. Thanks for reading!

Databricks, Inc. 2019. Deep Learning Pipelines for Apache Spark. V. 1.5.0. Released January 25. https://spark-packages.org/package/databricks/spark-deep-learning.

Elson, Jeremy, John (JD) Douceur, Jon Howell, and Jared Saul. 2007. “Asirra: A CAPTCHA That Exploits Interest-Aligned Manual Image Categorization.” Proceedings of 14th ACM Conference on Computer and Communications Security (CCS), Proceedings of 14th ACM Conference on Computer and Communications Security (CCS) Editions. https://www.microsoft.com/en-us/research/publication/asirra-a-captcha-that-exploits-interest-aligned-manual-image-categorization/.

Szegedy, Christian, Wei Liu, Yangqing Jia, et al. 2015. “Going Deeper with Convolutions.” Computer Vision and Pattern Recognition (CVPR). http://arxiv.org/abs/1409.4842.

To install sparklyr 1.6 from CRAN, run

install.packages("sparklyr")

In this blog post, we shall highlight the following features and enhancements from sparklyr 1.6:

• Weighted quantile summaries
• Power iteration clustering
• spark_write_rds() + collect_from_rds()
• Dplyr-related improvements

Weighted quantile summaries

Apache Spark is well-known for supporting approximate algorithms that trade off marginal amounts of accuracy for greater speed and parallelism. Such algorithms are particularly beneficial for performing preliminary data explorations at scale, as they enable users to quickly query certain estimated statistics within a predefined error margin, while avoiding the high cost of exact computations. One example is the Greenwald-Khanna algorithm for on-line computation of quantile summaries, as described in Greenwald and Khanna (2001). This algorithm was originally designed for efficient $\epsilon$- approximation of quantiles within a large dataset without the notion of data points carrying different weights, and the unweighted version of it has been implemented as approxQuantile() since Spark 2.0. However, the same algorithm can be generalized to handle weighted inputs, and as sparklyr user @Zhuk66 mentioned in this issue, a weighted version of this algorithm makes for a useful sparklyr feature.

To properly explain what weighted-quantile means, we must clarify what the weight of each data point signifies. For example, if we have a sequence of observations $(1, 1, 1, 1, 0, 2, -1, -1)$, and would like to approximate the median of all data points, then we have the following two options:

• Either run the unweighted version of approxQuantile() in Spark to scan through all 8 data points

• Or alternatively, “compress” the data into 4 tuples of (value, weight): $(1, 0.5), (0, 0.125), (2, 0.125), (-1, 0.25)$, where the second component of each tuple represents how often a value occurs relative to the rest of the observed values, and then find the median by scanning through the 4 tuples using the weighted version of the Greenwald-Khanna algorithm

We can also run through a contrived example involving the standard normal distribution to illustrate the power of weighted quantile estimation in sparklyr 1.6. Suppose we cannot simply run qnorm() in R to evaluate the quantile function of the standard normal distribution at $p = 0.25$ and $p = 0.75$, how can we get some vague idea about the 1st and 3rd quantiles of this distribution? One way is to sample a large number of data points from this distribution, and then apply the Greenwald-Khanna algorithm to our unweighted samples, as shown below:

library(sparklyr)

sc <- spark_connect(master = "local")

num_samples <- 1e6
samples <- data.frame(x = rnorm(num_samples))

samples_sdf <- copy_to(sc, samples, name = random_string())

samples_sdf %>%
  sdf_quantile(
    column = "x",
    probabilities = c(0.25, 0.75),
    relative.error = 0.01
  ) %>%
  print()

##        25%        75%
## -0.6629242  0.6874939

Notice that because we are working with an approximate algorithm, and have specified relative.error = 0.01, the estimated value of $-0.6629242$ from above could be anywhere between the 24th and the 26th percentile of all samples. In fact, it falls in the $25.36896$-th percentile:

pnorm(-0.6629242)

## [1] 0.2536896

Now how can we make use of weighted quantile estimation from sparklyr 1.6 to obtain similar results? Simple! We can sample a large number of $x$ values uniformly randomly from $(-\infty, \infty)$ (or alternatively, just select a large number of values evenly spaced between $(-M, M)$ where $M$ is approximately $\infty$), and assign each $x$ value a weight of $\displaystyle \frac{1}{\sqrt{2 \pi}}e^{-\frac{x^2}{2}}$, the standard normal distribution’s probability density at $x$. Finally, we run the weighted version of sdf_quantile() from sparklyr 1.6, as shown below:

library(sparklyr)

sc <- spark_connect(master = "local")

num_samples <- 1e6
M <- 1000
samples <- tibble::tibble(
  x = M * seq(-num_samples / 2 + 1, num_samples / 2) / num_samples,
  weight = dnorm(x)
)

samples_sdf <- copy_to(sc, samples, name = random_string())

samples_sdf %>%
  sdf_quantile(
    column = "x",
    weight.column = "weight",
    probabilities = c(0.25, 0.75),
    relative.error = 0.01
  ) %>%
  print()

##    25%    75%
## -0.696  0.662

Voilà! The estimates are not too far off from the 25th and 75th percentiles (in relation to our abovementioned maximum permissible error of $0.01$):

pnorm(-0.696)

## [1] 0.2432144

pnorm(0.662)

## [1] 0.7460144

Power iteration clustering

Power iteration clustering (PIC), a simple and scalable graph clustering method presented in Lin and Cohen (2010), first finds a low-dimensional embedding of a dataset, using truncated power iteration on a normalized pairwise-similarity matrix of all data points, and then uses this embedding as the “cluster indicator”, an intermediate representation of the dataset that leads to fast convergence when used as input to k-means clustering. This process is very well illustrated in figure 1 of Lin and Cohen (2010) (reproduced below)

in which the leftmost image is the visualization of a dataset consisting of 3 circles, with points colored in red, green, and blue indicating clustering results, and the subsequent images show the power iteration process gradually transforming the original set of points into what appears to be three disjoint line segments, an intermediate representation that can be rapidly separated into 3 clusters using k-means clustering with $k = 3$.

In sparklyr 1.6, ml_power_iteration() was implemented to make the PIC functionality in Spark accessible from R. It expects as input a 3-column Spark dataframe that represents a pairwise-similarity matrix of all data points. Two of the columns in this dataframe should contain 0-based row and column indices, and the third column should hold the corresponding similarity measure. In the example below, we will see a dataset consisting of two circles being easily separated into two clusters by ml_power_iteration(), with the Gaussian kernel being used as the similarity measure between any 2 points:

gen_similarity_matrix <- function() {
  # Guassian similarity measure
  guassian_similarity <- function(pt1, pt2) {
    exp(-sum((pt2 - pt1) ^ 2) / 2)
  }
  # generate evenly distributed points on a circle centered at the origin
  gen_circle <- function(radius, num_pts) {
    seq(0, num_pts - 1) %>%
      purrr::map_dfr(
        function(idx) {
          theta <- 2 * pi * idx / num_pts
          radius * c(x = cos(theta), y = sin(theta))
        })
  }
  # generate points on both circles
  pts <- rbind(
    gen_circle(radius = 1, num_pts = 80),
    gen_circle(radius = 4, num_pts = 80)
  )
  # populate the pairwise similarity matrix (stored as a 3-column dataframe)
  similarity_matrix <- data.frame()
  for (i in seq(2, nrow(pts)))
    similarity_matrix <- similarity_matrix %>%
      rbind(seq(i - 1L) %>%
        purrr::map_dfr(~ list(
          src = i - 1L, dst = .x - 1L,
          similarity = guassian_similarity(pts[i,], pts[.x,])
        ))
      )

  similarity_matrix
}

library(sparklyr)

sc <- spark_connect(master = "local")
sdf <- copy_to(sc, gen_similarity_matrix())
clusters <- ml_power_iteration(
  sdf, k = 2, max_iter = 10, init_mode = "degree",
  src_col = "src", dst_col = "dst", weight_col = "similarity"
)

clusters %>% print(n = 160)

## # A tibble: 160 x 2
##        id cluster
##     <dbl>   <int>
##   1     0       1
##   2     1       1
##   3     2       1
##   4     3       1
##   5     4       1
##   ...
##   157   156       0
##   158   157       0
##   159   158       0
##   160   159       0

The output shows points from the two circles being assigned to separate clusters, as expected, after only a small number of PIC iterations.

spark_write_rds() + collect_from_rds()

spark_write_rds() and collect_from_rds() are implemented as a less memory- consuming alternative to collect(). Unlike collect(), which retrieves all elements of a Spark dataframe through the Spark driver node, hence potentially causing slowness or out-of-memory failures when collecting large amounts of data, spark_write_rds(), when used in conjunction with collect_from_rds(), can retrieve all partitions of a Spark dataframe directly from Spark workers, rather than through the Spark driver node. First, spark_write_rds() will distribute the tasks of serializing Spark dataframe partitions in RDS version 2 format among Spark workers. Spark workers can then process multiple partitions in parallel, each handling one partition at a time and persisting the RDS output directly to disk, rather than sending dataframe partitions to the Spark driver node. Finally, the RDS outputs can be re-assembled to R dataframes using collect_from_rds().

Shown below is an example of spark_write_rds() + collect_from_rds() usage, where RDS outputs are first saved to HDFS, then downloaded to the local filesystem with hadoop fs -get, and finally, post-processed with collect_from_rds():

library(sparklyr)
library(nycflights13)

num_partitions <- 10L
sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark")
flights_sdf <- copy_to(sc, flights, repartition = num_partitions)

# Spark workers serialize all partition in RDS format in parallel and write RDS
# outputs to HDFS
spark_write_rds(
  flights_sdf,
  dest_uri = "hdfs://<namenode>:8020/flights-part-{partitionId}.rds"
)

# Run `hadoop fs -get` to download RDS files from HDFS to local file system
for (partition in seq(num_partitions) - 1)
  system2(
    "hadoop",
    c("fs", "-get", sprintf("hdfs://<namenode>:8020/flights-part-%d.rds", partition))
  )

# Post-process RDS outputs
partitions <- seq(num_partitions) - 1 %>%
  lapply(function(partition) collect_from_rds(sprintf("flights-part-%d.rds", partition)))

# Optionally, call `rbind()` to combine data from all partitions into a single R dataframe
flights_df <- do.call(rbind, partitions)

Dplyr-related improvements

Similar to other recent sparklyr releases, sparklyr 1.6 comes with a number of dplyr-related improvements, such as

• Support for where() predicate within select() and summarize(across(...)) operations on Spark dataframes
• Addition of if_all() and if_any() functions
• Full compatibility with dbplyr 2.0 backend API

select(where(...)) and summarize(across(where(...)))

The dplyr where(...) construct is useful for applying a selection or aggregation function to multiple columns that satisfy some boolean predicate. For example,

library(dplyr)

iris %>% select(where(is.numeric))

returns all numeric columns from the iris dataset, and

library(dplyr)

iris %>% summarize(across(where(is.numeric), mean))

computes the average of each numeric column.

In sparklyr 1.6, both types of operations can be applied to Spark dataframes, e.g.,

library(dplyr)
library(sparklyr)

sc <- spark_connect(master = "local")
iris_sdf <- copy_to(sc, iris, name = random_string())

iris_sdf %>% select(where(is.numeric))

iris %>% summarize(across(where(is.numeric), mean))

if_all() and if_any()

if_all() and if_any() are two convenience functions from dplyr 1.0.4 (see here for more details) that effectively ¹ combine the results of applying a boolean predicate to a tidy selection of columns using the logical and/or operators.

Starting from sparklyr 1.6, if_all() and if_any() can also be applied to Spark dataframes, .e.g.,

library(dplyr)
library(sparklyr)

sc <- spark_connect(master = "local")
iris_sdf <- copy_to(sc, iris, name = random_string())

# Select all records with Petal.Width > 2 and Petal.Length > 2
iris_sdf %>% filter(if_all(starts_with("Petal"), ~ .x > 2))

# Select all records with Petal.Width > 5 or Petal.Length > 5
iris_sdf %>% filter(if_any(starts_with("Petal"), ~ .x > 5))

Compatibility with dbplyr 2.0 backend API

Sparklyr 1.6 is fully compatible with the newer dbplyr 2.0 backend API (by implementing all interface changes recommended in here), while still maintaining backward compatibility with the previous edition of dbplyr API, so that sparklyr users will not be forced to switch to any particular version of dbplyr.

This should be a mostly non-user-visible change as of now. In fact, the only discernible behavior change will be the following code

library(dbplyr)
library(sparklyr)

sc <- spark_connect(master = "local")

print(dbplyr_edition(sc))

outputting

[1] 2

if sparklyr is working with dbplyr 2.0+, and

[1] 1

if otherwise.

Acknowledgements

In chronological order, we would like to thank the following contributors for making sparklyr 1.6 awesome:

• @yitao-li
• @pgramme
• @javierluraschi
• @andrew-christianson
• @jozefhajnala
• @nathaneastwood
• @mzorko

We would also like to give a big shout-out to the wonderful open-source community behind sparklyr, without whom we would not have benefitted from numerous sparklyr-related bug reports and feature suggestions.

Finally, the author of this blog post also very much appreciates the highly valuable editorial suggestions from @skeydan.

If you wish to learn more about sparklyr, we recommend checking out sparklyr.ai, spark.rstudio.com, and also some previous sparklyr release posts such as sparklyr 1.5 and sparklyr 1.4.

That is all. Thanks for reading!

Greenwald, Michael, and Sanjeev Khanna. 2001. “Space-Efficient Online Computation of Quantile Summaries.” SIGMOD Rec. (New York, NY, USA) 30 (2): 58–66. https://doi.org/10.1145/376284.375670.

Lin, Frank, and William Cohen. 2010. “Power Iteration Clustering.” August, 655–62.

To install sparklyr 1.5 from CRAN, run

install.packages("sparklyr")

In this blog post, we will highlight the following aspects of sparklyr 1.5:

• Better dplyr interface
• 4 useful additions to the sdf_* family of functions
• New RDS-based serialization routines along with several serialization-related improvements and bug fixes

Better dplyr interface

A large fraction of pull requests that went into the sparklyr 1.5 release were focused on making Spark dataframes work with various dplyr verbs in the same way that R dataframes do. The full list of dplyr-related bugs and feature requests that were resolved in sparklyr 1.5 can be found in here.

In this section, we will showcase three new dplyr functionalities that were shipped with sparklyr 1.5.

Stratified sampling

Stratified sampling on an R dataframe can be accomplished with a combination of dplyr::group_by() followed by dplyr::sample_n() or dplyr::sample_frac(), where the grouping variables specified in the dplyr::group_by() step are the ones that define each stratum. For instance, the following query will group mtcars by number of cylinders and return a weighted random sample of size two from each group, without replacement, and weighted by the mpg column:

mtcars %>%
  dplyr::group_by(cyl) %>%
  dplyr::sample_n(size = 2, weight = mpg, replace = FALSE) %>%
  print()

## # A tibble: 6 x 11
## # Groups:   cyl [3]
##     mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
##   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1  33.9     4  71.1    65  4.22  1.84  19.9     1     1     4     1
## 2  22.8     4 108      93  3.85  2.32  18.6     1     1     4     1
## 3  21.4     6 258     110  3.08  3.22  19.4     1     0     3     1
## 4  21       6 160     110  3.9   2.62  16.5     0     1     4     4
## 5  15.5     8 318     150  2.76  3.52  16.9     0     0     3     2
## 6  19.2     8 400     175  3.08  3.84  17.0     0     0     3     2

Starting from sparklyr 1.5, the same can also be done for Spark dataframes with Spark 3.0 or above, e.g.,:

library(sparklyr)

sc <- spark_connect(master = "local", version = "3.0.0")
mtcars_sdf <- copy_to(sc, mtcars, replace = TRUE, repartition = 3)

mtcars_sdf %>%
  dplyr::group_by(cyl) %>%
  dplyr::sample_n(size = 2, weight = mpg, replace = FALSE) %>%
  print()

# Source: spark<?> [?? x 11]
# Groups: cyl
    mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
  <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1  21       6 160     110  3.9   2.62  16.5     0     1     4     4
2  21.4     6 258     110  3.08  3.22  19.4     1     0     3     1
3  27.3     4  79      66  4.08  1.94  18.9     1     1     4     1
4  32.4     4  78.7    66  4.08  2.2   19.5     1     1     4     1
5  16.4     8 276.    180  3.07  4.07  17.4     0     0     3     3
6  18.7     8 360     175  3.15  3.44  17.0     0     0     3     2

or

mtcars_sdf %>%
  dplyr::group_by(cyl) %>%
  dplyr::sample_frac(size = 0.2, weight = mpg, replace = FALSE) %>%
  print()

## # Source: spark<?> [?? x 11]
## # Groups: cyl
##     mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
##   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1  21       6 160     110  3.9   2.62  16.5     0     1     4     4
## 2  21.4     6 258     110  3.08  3.22  19.4     1     0     3     1
## 3  22.8     4 141.     95  3.92  3.15  22.9     1     0     4     2
## 4  33.9     4  71.1    65  4.22  1.84  19.9     1     1     4     1
## 5  30.4     4  95.1   113  3.77  1.51  16.9     1     1     5     2
## 6  15.5     8 318     150  2.76  3.52  16.9     0     0     3     2
## 7  18.7     8 360     175  3.15  3.44  17.0     0     0     3     2
## 8  16.4     8 276.    180  3.07  4.07  17.4     0     0     3     3

Row sums

The rowSums() functionality offered by dplyr is handy when one needs to sum up a large number of columns within an R dataframe that are impractical to be enumerated individually. For example, here we have a six-column dataframe of random real numbers, where the partial_sum column in the result contains the sum of columns b through d within each row: