python discovery for reticulate can cause inspection errors

[aronatkins]

related also to this issue and maybe this is just more details on this specific issue.

When you try to deploy an app that has ragnar as a dependency AND you have used ragnar in a way that generated the ephemeral python environment as described in the original issue, you get the following error during deployment to Connect.

ℹ Capturing R dependencies from renv.lock
✔ Found 72 dependencies
Error in pythonConfig():
! Failed to detect python environment
Caused by error in system2():
! error in running command
Backtrace:
▆

1. ├─rsconnect::deployApp(...)
2. │ └─rsconnect:::bundleApp(...)
3. │ └─rsconnect:::createAppManifest(...)
4. │ └─rsconnect (local) pythonConfig(appDir)
5. │ ├─base::withCallingHandlers(...)
6. │ └─rsconnect:::inferPythonEnv(appDir, python = python, forceGenerate = forceGenerate)
7. │ └─base::system2(...)
8. └─rsconnect (local) <fn>(<cmdError>)
9. └─cli::cli_abort("Failed to detect python environment", parent = err)

10. └─rlang::abort(...)
    

Execution halted

This happens because python is retrieved by the getPython function when the option of "rsconnect.python.enabled is set to TRUE.

    pythonEnabled <- getOption("rsconnect.python.enabled", !targetIsShinyapps)
    if (pythonEnabled) {
        getPython(path)

The function rsconnect::inferPythonEnv uses the value of python = getPython(), which will return NULL in this case, since it is just using the call Sys.getenv("RETICULATE_PYTHON"). The method used by reticulate described in the original issues above does not set any environment variables. Additionally, conda is not used in this case either.

The way inferPythonEnv works, is shown below:

rsconnect:::inferPythonEnv
function (workdir, python = getPython(), forceGenerate = FALSE) 
{
    env_py <- system.file("resources/environment.py", package = "rsconnect")
    args <- c(shQuote(env_py), if (forceGenerate) "-f", shQuote(workdir))
    hasConda <- is_installed("reticulate") && reticulate::py_available(initialize = FALSE) && 
        reticulate::py_config()$anaconda
    if (hasConda) {
        prefix <- getCondaEnvPrefix(python)
        conda <- getCondaExeForPrefix(prefix)
        args <- c("run", "-p", prefix, python, args)
        output <- system2(command = conda, args = args, stdout = TRUE, 
            stderr = NULL, wait = TRUE)
    }
    else {
        output <- system2(command = python, args = args, stdout = TRUE, 
            stderr = NULL, wait = TRUE)
    }

So when python is NULL and conda is not used, the system2(command = python, .... will fail with the error I pasted above.

It is interesting, since in logical case where conda exists, the python is extracted from the reticulate::py_config()$anaconda variable.

So perhaps the inferPythonEnv function should instead be using the reticulate::py_config()$python location when reticulate is installed, like this?

    hasVirtualEnv <- is_installed("reticulate") && reticulate::py_available(initialize = FALSE) && 
        !reticulate::py_config()$anaconda
  
   if(hasVirtualEnv) python <- reticulate::py_config()$python

Since reticulate is already a suggested package, and in this case, you are already checking for reticulate being installed, perhaps this logical case would properly find the python reticulate is using in a cleaner way?
Tagging @t-kalinowski, @aronatkins and @nealrichardson since I discussed this issue with them.

Originally posted by @SokolovAnatoliy in #1143

[aronatkins]

Python and its configuration function are computed during deployment here:

rsconnect/R/deployApp.R

Lines 472 to 473 in b396625

	python <- getPythonForTarget(python, accountDetails)
	pythonConfig <- pythonConfigurator(python, forceGeneratePythonEnvironment)

The getPython() function returns NULL if it cannot discover a Python installation.

rsconnect/R/bundlePython.R

Lines 38 to 54 in b396625

	getPython <- function(path = NULL) {
	if (!is.null(path)) {
	return(path.expand(path))
	}
	path <- Sys.getenv("RETICULATE_PYTHON")
	if (path != "") {
	return(path.expand(path))
	}
	path <- Sys.getenv("RETICULATE_PYTHON_FALLBACK")
	if (path != "") {
	return(path.expand(path))
	}
	NULL
	}

If pythonConfigurator ever sees python==NULL, it returns NULL:

rsconnect/R/bundlePython.R

Lines 3 to 6 in b396625

	pythonConfigurator <- function(python, forceGenerate = FALSE) {
	if (is.null(python)) {
	return(NULL)
	}

I'm guessing that RStudio is setting RETICULATE_PYTHON_FALLBACK and that is somehow causing the downstream system2 error.

https://github.com/rstudio/rstudio/blob/8921c6ac0db9f520e0b657186c6375cc4320a95e/src/cpp/session/modules/SessionRSConnect.cpp#L181-L204

You can see a similar error by passing a bogus value for python to rsconnect::writeManifest() (or deployApp()):

> rsconnect::writeManifest(python="notpython")
ℹ Capturing R dependencies
✔ Found 36 dependencies
Error in `pythonConfig()`:
! Failed to detect python environment
Caused by error in `system2()`:
! error in running command
Run `rlang::last_trace()` to see where the error occurred.
> rlang::last_trace(drop = FALSE)
<error/rlang_error>
Error in `pythonConfig()`:
! Failed to detect python environment
Caused by error in `system2()`:
! error in running command
---
Backtrace:
    ▆
 1. ├─rsconnect::writeManifest(python = "notpython")
 2. │ └─rsconnect:::createAppManifest(...)
 3. │   └─rsconnect (local) pythonConfig(appDir)
 4. │     ├─base::withCallingHandlers(...)
 5. │     └─rsconnect:::inferPythonEnv(appDir, python = python, forceGenerate = forceGenerate)
 6. │       └─base::system2(...)
 7. └─rsconnect (local) `<fn>`(`<cmdError>`)
 8.   └─cli::cli_abort("Failed to detect python environment", parent = err)
 9.     └─rlang::abort(...)

There are two problems:

First, RStudio is somehow discovering and configuring a Python that causes errors for rsconnect. Secondly, rsconnect is not showing details about that error.

The following Shiny application is enough to experiment with this problem, as it includes reticulate.

library(shiny)
library(reticulate)

ui <- fluidPage("hello")
server <- function(input, output) {}
shinyApp(ui = ui, server = server)

CC @kevinushey

[aronatkins]

@SokolovAnatoliy - Could you install the latest rsconnect from GitHub to see if that provides more information when deploying? It should indicate the targeted Python binary, which may reveal what's going wrong.

remotes::install_github("rstudio/rsconnect")

[SamEdwardes]

I am having a similar issue. Please let me know if you think this is unrelated and I will open a new ticket.

app.R

library(shiny)
library(reticulate)

reticulate::py_require("polars")
pl <- reticulate::import("polars")
print(pl$DataFrame(iris))

ui <- fluidPage("hello")
server <- function(input, output) {}
shinyApp(ui = ui, server = server)

Steps to reprex

Set up in the console:

# Set up renv
renv::init()

# Install packages
renv::install(c("reticulate", "shiny"))
renv::install("rstudio/rsconnect")

# Create a virtual environment
reticulate::virtualenv_create("./.venv")

# Configure renv to use the virtual environment
renv::use_python("./.venv/bin/python")

# Configure reticulate to use the virtual environment
# instead of uv
Sys.setenv(
  RETICULATE_USE_MANAGED_VENV = "no",
  RETICULATE_PYTHON = "./.venv"
)

# Install Python deps
reticulate::py_install("polars")

# Capture R and Python requirements
renv::snapshot()

Create the error:

> rsconnect::writeManifest()
ℹ Capturing R dependencies from renv.lock
✔ Found 37 dependencies
Error in `pythonConfig()` at rsconnect/R/bundle.R:207:5:
! Failed to detect python environment using
  "/Users/samedwardes/tmp/test2/.venv/bin/python"
Caused by error:
! parse error: premature EOF
                                       
                     (right here) ------^
Run `rlang::last_trace()` to see where the error occurred.
Warning message:
In system2(command = python, args = args, stdout = TRUE, stderr = NULL,  :
  running command ''/Users/samedwardes/tmp/test2/.venv/bin/python' '/Users/samedwardes/Library/Caches/org.R-project.R/R/renv/cache/v5/macos/R-4.4/aarch64-apple-darwin20/rsconnect/1.5.1.9000/ef719d9a99deed0f91db826c75be2d13/rsconnect/resources/environment.py' '/var/folders/x4/583bns3n4gbddql2dsv7jyc40000gp/T//Rtmpf40yvx/file17fa767a83d0c' 2>/dev/null' had status 1

> rlang::last_trace()
<error/rlang_error>
Error in `pythonConfig()` at rsconnect/R/bundle.R:207:5:
! Failed to detect python environment using
  "/Users/samedwardes/tmp/test2/.venv/bin/python"
Caused by error:
! parse error: premature EOF
                                       
                     (right here) ------^
---
Backtrace:
    ▆
 1. └─rsconnect::writeManifest()
 2.   └─rsconnect:::createAppManifest(...) at rsconnect/R/writeManifest.R:64:3
 3.     └─rsconnect (local) pythonConfig(appDir) at rsconnect/R/bundle.R:207:5
 4.       ├─base::withCallingHandlers(...) at rsconnect/R/bundlePython.R:11:5
 5.       └─rsconnect:::inferPythonEnv(appDir, python = python, forceGenerate = forceGenerate) at rsconnect/R/bundlePython.R:11:5
 6.         └─jsonlite::fromJSON(sanitizeSystem2json(output)) at rsconnect/R/bundlePython.R:95:3
 7.           └─jsonlite:::parse_and_simplify(...)
 8.             └─jsonlite:::parseJSON(txt, bigint_as_char)
 9.               └─jsonlite:::parse_string(txt, bigint_as_char)
Run rlang::last_trace(drop = FALSE) to see 4 hidden frames.

Environment

> sessionInfo()
R version 4.4.1 (2024-06-14)
Platform: aarch64-apple-darwin20
Running under: macOS 26.0

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/Vancouver
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] reticulate_1.43.0 shiny_1.11.1     

loaded via a namespace (and not attached):
 [1] crayon_1.5.3         cli_3.6.3            rlang_1.1.6         
 [4] png_0.1-8            renv_1.0.11          promises_1.3.3      
 [7] jsonlite_2.0.0       xtable_1.8-4         glue_1.8.0          
[10] htmltools_0.5.8.1    httpuv_1.6.15        sass_0.4.9          
[13] rsconnect_1.5.1.9000 grid_4.4.1           jquerylib_0.1.4     
[16] fastmap_1.2.0        lifecycle_1.0.4      memoise_2.0.1       
[19] compiler_4.4.1       fs_1.6.4             Rcpp_1.1.0          
[22] rstudioapi_0.17.1    later_1.3.2          lattice_0.22-7      
[25] digest_0.6.37        R6_2.5.1             reprex_2.1.1        
[28] magrittr_2.0.3       bslib_0.8.0          Matrix_1.7-4        
[31] tools_4.4.1          withr_3.0.2          mime_0.12           
[34] cachem_1.1.0        

[t-kalinowski]

@SamEdwardes I think RETICULATE_PYTHON should point to a python binary, not a venv root. e.g., .venv/bin/python.

[SamEdwardes]

Thank you @t-kalinowski, you were correct, I set RETICULATE_PYTHON incorrectly. Here is a working example now:

app.R

library(shiny)
library(reticulate)

reticulate::py_require("polars")
pl <- reticulate::import("polars")
print(pl$DataFrame(iris))

ui <- fluidPage("hello")
server <- function(input, output) {}
shinyApp(ui = ui, server = server)

Deployment steps:

# Set up renv
renv::init()

# Install packages
renv::install(c("reticulate", "shiny"))
renv::install("rstudio/rsconnect")

# Create a virtual environment
reticulate::virtualenv_create("./.venv", "/opt/homebrew/bin/python3.13")

# Configure renv to use the virtual environment
renv::use_python("./.venv/bin/python")

# Configure reticulate to use the virtual environment
# instead of uv
Sys.setenv(
	RETICULATE_USE_MANAGED_VENV = "no",
	RETICULATE_PYTHON = "./.venv/bin/python"
)

# Install Python deps
reticulate::py_install("polars")

# Capture R and Python requirements
renv::snapshot()

# Write the manifest
rsconnect::writeManifest()

So I think my issue was unrelated to the original issue in this ticket, apologies.

[t-kalinowski]

Thanks for pushing this along!

reticulate::virtualenv_create("./.venv", "/opt/homebrew/bin/python3.13")

# Configure renv to use the virtual environment
renv::use_python("./.venv/bin/python")

Sys.setenv(
	RETICULATE_USE_MANAGED_VENV = "no",
	RETICULATE_PYTHON = "./.venv/bin/python"
)

All of these steps together are somewhat redundant. Reticulate will automatically detect and use a .venv in the working directory without any other config. Just creating the venv is enough. renv::use_python("./.venv/bin/python") also sets RETICULATE_PYTHON, so you’re doubly covered there. If RETICULATE_PYTHON is set, or ./.venv exists, then RETICULATE_USE_MANAGED_VENV will have no impact and never be consulted.

I’d also suggest avoiding Homebrew’s Python for seeding venvs. brew can remove or update it unexpectedly. (https://justinmayer.com/posts/homebrew-python-is-not-for-you/). I recommend using install_python() and virtualenv_starter() for self-managed environments.

reticulate::py_install("polars")

This will reinstall or upgrade polars on every deployment — just checking if that’s intended.

reticulate::py_require("polars")

Note that this won’t have an effect when using a self-managed venv.