using source-overlay errs on second restore

[aronatkins]

This issue shows that packrat can restore from bundles in a source overlay directory but is unable to use those installed packages on subsequent restore. We are seeing this problem while using the packrat cache, but the steps here show that the cache is not necessary.

• Create a toy package that we will "pretend" is on GitHub.
• Create a tar.gz bundle using R CMD build completelyPrivatePackage.
• Add that tar.gz into a "sources" directory with a fake-SHA filename.
• Create a toy project that uses this package.
• Restore that project (using the sources directory).
• Restore that project again.

example `packrat.lock`

PackratFormat: 1.4
PackratVersion: 0.4.4
RVersion: 3.2.0
Repos: CRAN=http://cran.rstudio.com

Package: completelyPrivatePackage
Source: github
Version: 0.1.0
Hash: 88e8036bdbc415864d0f021ce6017997
GithubRepo: completelyPrivatePackage
GithubUsername: this-user-does-not-exist
GithubRef: master
GithubSha1: decafbaddecafbaddecafbaddecafbaddecafbad
RemoteHost: api.github.com
RemoteRepo: completelyPrivatePackage
RemoteUsername: this-user-does-not-exist
RemoteRef: master
RemoteSha: decafbaddecafbaddecafbaddecafbaddecafbad

In my toy project, I have a source-overlay directory that contains:

source-overlay/completelyPrivatePackage/decafbaddecafbaddecafbaddecafbaddecafbad.tar.gz

Here is a restore script that uses this overlay directory:

sourceOverlay <- file.path(getwd(), "source-overlay")
Sys.setenv(R_PACKRAT_SRC_OVERLAY = sourceOverlay)
Sys.setenv(R_PACKRAT_SRC_DIR = file.path(tempdir(), "packrat-src"))

options(packrat.untrusted.packages = character())
options(
  packrat.verbose.cache = TRUE,
  packrat.connect.timeout = 10
)

packrat::set_opts(
  auto.snapshot = FALSE,
  project = getwd(),
  persist = FALSE
)

packrat::restore(
  overwrite.dirty = TRUE,
  prompt = FALSE,
  restart = FALSE
)

On first run (no packrat/lib):

R --vanilla -s -f ./doit.R
Installing completelyPrivatePackage (0.1.0) ...
	OK (built source)
Warning messages:
1: In packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
  The most recent snapshot was generated using R version 3.2.0
2: In untar(src, exdir = target, compressed = "gzip") :
  argument 'compressed' is ignored for the internal method
3: In untar(src, compressed = "gzip", list = TRUE) :
  argument 'compressed' is ignored for the internal method

On second run (after that successful restore):

R --vanilla -s -f ./doit.R
Error: Unable to retrieve package records for the following packages:
- 'completelyPrivatePackage'
In addition: Warning message:
In packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
  The most recent snapshot was generated using R version 3.2.0
Execution halted

The restore works by adding the following fields to the DESCRIPTION file within that sources tar.gz file:

Source: github
GithubRepo: completelyPrivatePackage
GithubUsername: this-user-does-not-exist
GithubRef: master
GithubSHA1: decafbaddecafbaddecafbaddecafbaddecafbad
RemoteHost: api.github.com
RemoteRepo: completelyPrivatePackage
RemoteUsername: this-user-does-not-exist
RemoteRef: master
RemoteSha: decafbaddecafbaddecafbaddecafbaddecafbad

With this additional metadata, the second restore succeeds:

R --vanilla -s -f ./doit.R
Already up to date.
Warning message:
In packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
  The most recent snapshot was generated using R version 3.2.0

Note: You'll need to remove the packrat/lib when switching between the original and augmented source bundles.

It feels as if packrat is seeing the presence of a tar.gz in the sources directory:

packrat/R/restore.R

Lines 91 to 92 in 448aafd

	if (file.exists(file.path(pkgSrcDir, pkgSrcFile)))
	return(NULL)

That prevents us from adding DESCRIPTION fields associated with GitHub packages:

packrat/R/restore.R

Lines 307 to 328 in 448aafd

	ghdata <- c(
	GithubRepo = pkgRecord$gh_repo,
	GithubUsername = pkgRecord$gh_username,
	GithubRef = pkgRecord$gh_ref,
	GithubSHA1 = pkgRecord$gh_sha1,
	GithubSubdir = pkgRecord$gh_subdir
	)
	ghinfo <- as.data.frame(as.list(ghdata), stringsAsFactors = FALSE)
	appendToDcf(file.path(basedir, 'DESCRIPTION'), ghinfo)
	file.create(file.path(pkgSrcDir, pkgSrcFile))
	dest <- normalizePath(file.path(pkgSrcDir, pkgSrcFile), winslash = '/')
	# R's internal tar (which we use here for cross-platform consistency)
	# emits warnings when there are > 100 characters in the path, due to the
	# resulting incompatibility with older implementations of tar. This isn't
	# relevant for our purposes, so suppress the warning.
	in_dir(dirname(basedir),
	suppressWarnings(tar(tarfile = dest, files = basename(basedir),
	compression = 'gzip', tar = 'internal'))
	)

We add fields for other types of repositories (bitbucket, gitlab, etc). Packages from these sources probably have the same issue.

[aronatkins]

Packrat probably should not touch bundles in the source overlay directory, instead modifying them on-the-fly as it prepares to install from that file.