mz511: persist secret files

[mzihlmann]

Fixes #511

Description

We previously already added the option to persist secret files, but we never tested it as setup is a bit difficult. Now in testing, we realized that even though we do persist the secret file, we don't use the persisted copy, but the original. This is because in a for loop we update a local struct instead of the original.

I now not only fixed the bug but also added an integration test. The integration test was a bit tricky as we need to use a file the pre-exists in the kaniko binary as a secret, as we're later gonna delete it. I used the /etc/nsswitch.conf file. And it indeed can simulate the error.

Interestingly, that uncovered another diff between buildkit and kaniko, when we delete this file, buildkit will emit a whiteout file and we don't.

[mzihlmann]

The actual bug is here

https://github.com/osscontainertools/kaniko/blob/main/cmd/executor/cmd/root.go#L445

	for k, s := range opts.Secrets {
		if s.Type == "env" {
			_, ok := os.LookupEnv(s.Src)
			if !ok {
				return fmt.Errorf("environment variable for secret %q not set: %s", k, s.Src)
			}
		} else {
			// In multistage builds the original secret file might be deleted between stages.
			// We therefore safeguard it across stages by copying it into /kaniko.
			// We are not allowed to move it as it might be mounted into the container.
			destPath := filepath.Join(config.KanikoSecretsDir, k)
			err := os.MkdirAll(config.KanikoSecretsDir, 0700)
			if err != nil {
				return err
			}
			err = util.CopyFileInternal(s.Src, destPath, util.FileContext{})
			if err != nil {
				return fmt.Errorf("copying secret %s: %w", s.Src, err)
			}
			s.Src = destPath
		}
	}

As you can see we already do have handling for persisting file secrets in kanikoDir.

But note the last line s.Src = destPath where we attempt to update the Src. Because we iterate over a map of structs we actually only update the loop local copy. I now changed the map to be a map of pointers, so when we update here, we update the single original copy.