Skip to content

detectCores() is not future proof - it risks giving an error on machines with more than 125 CPU cores #22

@HenrikBengtsson

Description

@HenrikBengtsson

In

interflex/R/kernel.R

Lines 765 to 767 in e35ba4a

maxcores <- detectCores()
cores <- min(maxcores, cores)
pcl <- future::makeClusterPSOCK(cores)

you're using detectCores(). More and more users have access to machines with > 125 CPU cores, e.g. we see 128-, 192-, and 256-core machine on academic compute clusters these days. Any parallel cluster in R that use PSOCK workers (e.g. parallel::makeCluster(), future's multisession, doParallel, ...) will fail on such machines. See my Please Avoid detectCores() in your R Packages blog post on why detectCores() is problematic.

I understand that your default cores = 4 protect against this via cores <- min(maxcores, cores), so the risk is smaller, because detectCores() is just a very liberal upper limit that rarely kicks in. Also, the fact that you're using future::makeClusterPSOCK(cores) also helps - it will detect overuse and given an informative error message upfront.

Since you're already using future, which depends on parallelly, my recommendation would be to switch out detectCores() with parallelly::availableCores(). That still defaults to detectCores() if nothing else is set, but it will at least be agile to CPU limits that the current system have in place, e.g. job schedulers, CGroups, and sysadm env vars. See above blog post for examples.

PS. This is not critical, but since I stumbled upon this code when reverse-dependency checking the future package, but I figured you'd might wanna know already now, given that CRAN has given you deadline to update the package by 2025-11-15.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions