Allow manual specification of application dependencies

[maxheld83]

UPDATE

This is now a simple feature request: support DESCRIPTION files as a way to specify dependencies for rsconnect::AppDeploy().

---

After some deployment grief with #197, I am looking for a simpler / more robust way to tell shinyapps.io about its required dependencies, and I was hoping a DESCRIPTION would do it.

But when I add a DESCRIPTION like the below to the manifest, the deploy fails:

Package: accio
Title: Accio Web Frontend for Pensieve
Author: Maximilian Held
AuthorUrl: http://www.maxheld.de
License: Proprietary
DisplayMode: Showcase
Type: Package
Imports:
  pensieve
Remotes:
  maxheld83/pensieve

----- Deployment log started at  2017-08-21 13:04:17  -----
Deploy command: 
 rsconnect::deployApp(appDir = "~/GitHub/pensieve/inst/accio",      appFileManifest = "/var/folders/dn/1kkm6bgs7c38dfm943_1gmmh0000gn/T/3654-efab-0bd8-7a0b",      account = "maxheld83", server = "shinyapps.io", appName = "accio",      appTitle = "accio", launch.browser = function(url) {         message("Deployment completed: ", url)     }, lint = FALSE, metadata = list(asMultiple = FALSE, asStatic = FALSE,          ignoredFiles = ".git|LICENSE|README.md"), logLevel = "verbose") 

Session information: 
R version 3.4.1 (2017-06-30)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.6

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.4.1  rsconnect_0.8.4
Cookies: 
[1] "None"
GET /v1/applications/?filter=account_id:23594&filter=name:accio&count=100&offset=0 1047ms
----- Bundle upload started at  2017-08-21 13:04:18  -----
10: tryCatchOne(expr, names, parentenv, handlers[[1L]])
9: tryCatchList(expr, classes, parentenv, handlers)
8: tryCatch({
       performPackratSnapshot(bundleDir)
   }, error = function(e) {
       e$msg <- paste0("----- Error snapshotting dependencies (Packrat) -----\n", 
           e$msg)
       if (isTRUE(getOption("rsconnect.error.trace"))) {
           traceback(3, sys.calls())
       }
       stop(e)
   })
7: addPackratSnapshot(appDir, implicit_dependencies)
6: snapshotDependencies(appDir, inferDependencies(appMode, hasParameters))
5: createAppManifest(bundleDir, appMode, contentCategory, hasParameters, 
       appPrimaryDoc, assetTypeName, users)
4: bundleApp(target$appName, appDir, appFiles, appPrimaryDoc, assetTypeName, 
       contentCategory)
3: force(code)
2: withStatus(paste0("Uploading bundle for ", assetTypeName, ": ", 
       application$id), {
       bundlePath <- bundleApp(target$appName, appDir, appFiles, 
           appPrimaryDoc, assetTypeName, contentCategory)
       if (isShinyapps(accountDetails)) {
           bundleSize <- file.info(bundlePath)$size
           checkSum <- digest::digest(bundlePath, "md5", file = TRUE)
           bundle <- client$createBundle(application$id, "application/x-tar", 
               bundleSize, checkSum)
           if (!uploadBundle(bundle, bundleSize, bundlePath)) {
               stop("Could not upload file.")
           }
           response <- client$updateBundleStatus(bundle$id, status = "ready")
           bundle <- client$getBundle(bundle$id)
       }
       else {
           bundle <- client$uploadApplication(application$id, bundlePath)
       }
   })
1: rsconnect::deployApp(appDir = "~/GitHub/pensieve/inst/accio", 
       appFileManifest = "/var/folders/dn/1kkm6bgs7c38dfm943_1gmmh0000gn/T/3654-efab-0bd8-7a0b", 
       account = "maxheld83", server = "shinyapps.io", appName = "accio", 
       appTitle = "accio", launch.browser = function(url) {
           message("Deployment completed: ", url)
       }, lint = FALSE, metadata = list(asMultiple = FALSE, asStatic = FALSE, 
           ignoredFiles = ".git|LICENSE|README.md"), logLevel = "verbose")
----- Deployment error -----
Error: Unable to retrieve package records for the following packages:
- 'pensieve'
 
----- Error stack trace -----
12: stop(e)
11: value[[3L]](cond)
10: tryCatchOne(expr, names, parentenv, handlers[[1L]])
9: tryCatchList(expr, classes, parentenv, handlers)
8: tryCatch({
       performPackratSnapshot(bundleDir)
   }, error = function(e) {
       e$msg <- paste0("----- Error snapshotting dependencies (Packrat) -----\n", 
           e$msg)
       if (isTRUE(getOption("rsconnect.error.trace"))) {
           traceback(3, sys.calls())
       }
       stop(e)
   })
7: addPackratSnapshot(appDir, implicit_dependencies)
6: snapshotDependencies(appDir, inferDependencies(appMode, hasParameters))
5: createAppManifest(bundleDir, appMode, contentCategory, hasParameters, 
       appPrimaryDoc, assetTypeName, users)
4: bundleApp(target$appName, appDir, appFiles, appPrimaryDoc, assetTypeName, 
       contentCategory)
3: force(code)
2: withStatus(paste0("Uploading bundle for ", assetTypeName, ": ", 
       application$id), {
       bundlePath <- bundleApp(target$appName, appDir, appFiles, 
           appPrimaryDoc, assetTypeName, contentCategory)
       if (isShinyapps(accountDetails)) {
           bundleSize <- file.info(bundlePath)$size
           checkSum <- digest::digest(bundlePath, "md5", file = TRUE)
           bundle <- client$createBundle(application$id, "application/x-tar", 
               bundleSize, checkSum)
           if (!uploadBundle(bundle, bundleSize, bundlePath)) {
               stop("Could not upload file.")
           }
           response <- client$updateBundleStatus(bundle$id, status = "ready")
           bundle <- client$getBundle(bundle$id)
       }
       else {
           bundle <- client$uploadApplication(application$id, bundlePath)
       }
   })
1: rsconnect::deployApp(appDir = "~/GitHub/pensieve/inst/accio", 
       appFileManifest = "/var/folders/dn/1kkm6bgs7c38dfm943_1gmmh0000gn/T/3654-efab-0bd8-7a0b", 
       account = "maxheld83", server = "shinyapps.io", appName = "accio", 
       appTitle = "accio", launch.browser = function(url) {
           message("Deployment completed: ", url)
       }, lint = FALSE, metadata = list(asMultiple = FALSE, asStatic = FALSE, 
           ignoredFiles = ".git|LICENSE|README.md"), logLevel = "verbose")
Error: Unable to retrieve package records for the following packages:
- 'pensieve'
In addition: Warning messages:
1: In FUN(X[[i]], ...) :
  Package 'pensieve' not available in repository or locally
2: In FUN(X[[i]], ...) :
  Failed to infer source for package 'pensieve'; using latest available version on CRAN instead
Execution halted

Is this generally a supported way to specify dependencies, and if so, what am I doing wrong?
If not, could this be supported?

To be clear, this is kind of the inverse of #192; I want rsconnect to ignore dark packrat magic and just adhere to the DESCRIPTION.

[maxheld83]

(I am generally quite unhappy with the time I spend dependency-managing different build environments between packrat, TRAVIS, shinyapps.io, and, god forbid, at some point, my own RStudio Connect server.)

[kevinushey]

rsconnect is (for better or worse) currently coupled to Packrat: every deployment starts with a packrat::snapshot() on the local machine, and ends with a packrat::restore() on the deployment machine.

The error you're seeing would likely imply that you have the pensieve package installed from a local tarball on your machine:

Error: Unable to retrieve package records for the following packages:
- 'pensieve'
In addition: Warning messages:
1: In FUN(X[[i]], ...) :
  Package 'pensieve' not available in repository or locally
2: In FUN(X[[i]], ...) :
  Failed to infer source for package 'pensieve'; using latest available version on CRAN instead
Execution halted

I think this error would go away if you manually re-installed the package using devtools::install_github().

[maxheld83]

thanks!
I guess this isn't the place for the broader discussion of whether there should be an alternative to packrat, so I'm closing this won't-fix 😏 .

[kevinushey]

FWIW there's some talk internally at RStudio about:

1. Spending some serious time to overhaul Packrat,
2. Considering an entirely new solution altogether to package management.

I can't make any promises about timelines but we're aware of the frustrations users have been encountering with Packrat (sorry it's been such a frustrating experience!)

[maxheld83]

Dependency management issues for shinyapps.io (such as #197) continue to be a huge problem for me, and the "dark magic" of packrat just make this very hard to debug by myself (or at all).

If I understand the current rsconnect design correctly, its dependency management depends on the side effects of install_github() and friends, which also makes it impossible to version-control the dependencies. (In a "full" packrat mode, at least there is the packrat.lock and /src to keep track of things – but not when it's all used from inside rsconnect).

So I'd like re-open this as a simple feature request: support DESCRIPTION files as an alternative / fall-back way to specify dependencies for rsconnect::AppDeploy(), to override any packrat dark magic. The DESCRIPTION could be used when found at the root of the app.

This would be limited (though the Remotes: field addresses many use cases), but it would seem to be more (SCM-) transparent and easier to debug.

It would also copy existing ways of doing dependency management for R projects (the Travis R image and CRAN), that people are familiar with.

[dracodoc]

I want to add my vote with two different scenarios:

1. I use pacman in my shiny app, so that user can run the app with run_github directly over my github repo. It will start to install package dependencies in starting the app. However rsconnect only scan require library and full qualified names for packages discovery.

I made a PR with packrat to support it, but it was considered not suitable for packrat, so I have to just fork packrat and use my fork.

2. If some packages are delivered through drat, rsconnect can find the package name but I'm not sure if it can install it in the shiny server. I read through Local Cran repo doesn t work with rsconnect deployApp (even when Repository: ... is included in Description) #238, seemed to be doable but with ugly hack.

For the pacman case, @kevinushey said

I'd prefer to avoid adding this directly to Packrat (which would imply we'd have to maintain + update it), and instead provide a mechanism for users to register their own dependency resolution routines.

I totally agree this and really hoped to see something like OP suggested.

However I'd suggest to extend the description to at least support drat or other repo servers. Description and remote section may not enough for these cases.

Actually I don't see why it have to be description format, this is not a package, and you are not using any package related tools, and description format is too limited.

I think a better model is homebrew formula, so for certain package and version there are some instructions to install it properly in another machine. RStudio also maintained something similar for linux package installations. Of course the dependencies declarations can be separated from with the formula, so for simple cases user don't need to create formula, but advanced users can use formula to duplicate their specific environment.

[1beb]

+1 for specifying your own depends.

Python PIP has a great system for this using requirements.txt (https://pip.readthedocs.io/en/1.1/requirements.html).

Similar examples in JS come in the form of manifest.json.

Not a big fan of hidden mechanics that can't be edited or adjusted. Everything must have a configuration file! Generate it for me yes, but tell me where it is so I can review/fix/tinker with it.

[hadley]

I think #471 is now the current equivalent.