-
Notifications
You must be signed in to change notification settings - Fork 13
Description
In
Lines 765 to 767 in e35ba4a
| maxcores <- detectCores() | |
| cores <- min(maxcores, cores) | |
| pcl <- future::makeClusterPSOCK(cores) |
you're using detectCores(). More and more users have access to machines with > 125 CPU cores, e.g. we see 128-, 192-, and 256-core machine on academic compute clusters these days. Any parallel cluster in R that use PSOCK workers (e.g. parallel::makeCluster(), future's multisession, doParallel, ...) will fail on such machines. See my Please Avoid detectCores() in your R Packages blog post on why detectCores() is problematic.
I understand that your default cores = 4 protect against this via cores <- min(maxcores, cores), so the risk is smaller, because detectCores() is just a very liberal upper limit that rarely kicks in. Also, the fact that you're using future::makeClusterPSOCK(cores) also helps - it will detect overuse and given an informative error message upfront.
Since you're already using future, which depends on parallelly, my recommendation would be to switch out detectCores() with parallelly::availableCores(). That still defaults to detectCores() if nothing else is set, but it will at least be agile to CPU limits that the current system have in place, e.g. job schedulers, CGroups, and sysadm env vars. See above blog post for examples.
PS. This is not critical, but since I stumbled upon this code when reverse-dependency checking the future package, but I figured you'd might wanna know already now, given that CRAN has given you deadline to update the package by 2025-11-15.