support 'path variables' in nomad job files

[kevinschoonover]

Proposal

Add support for getting absolute path to job file via a path.X variable such as in:

• https://www.packer.io/docs/templates/hcl_templates/path-variables
• https://www.terraform.io/docs/language/expressions/references.html#filesystem-and-workspace-info

Use-cases

I am running tempo in microservices mode in which each microservice shares a configuration across multiple groups/jobs. Instead of copying an pasting the configuration into a template stanza, I use the file function to share it:

# tempo.hcl
template {
  data        = file("tempo.yml.j2")
  destination = "${NOMAD_TASK_DIR}/tempo.yaml"
}

; however, this requires me to run nomad run tempo.hcl in the same directory as tempo.hcl where as something like

# tempo.hcl
template {
  data        = file("${path.job_file}/tempo.yml.j2")
  destination = "${NOMAD_TASK_DIR}/tempo.yaml"
}

would allow me to run it in any arbitrary directory such as ../../.

Attempted Solutions

I tried using the abspath function since it mentions

abspath takes a string containing a filesystem path and converts it to an absolute path. That is, if the path is not absolute, it will be joined with the job file directory.

but it returns the absolute path to the directoy I am running the nomad cli in and not to the jobfile.

# tempo.hcl
template {
  data        = file(abspath("${path.job_file}/tempo.yml.j2") # 
  destination = "${NOMAD_TASK_DIR}/tempo.yaml"
}

$ nomad run jobs/observability/tempo.hcl
tempo.hcl:877,23-28: Error in function call; Call to function "file" failed: no file exists at ${pwd}\tempo.yml.j2.
tempo.hcl:877,23-28: Unsuitable value type; Unsuitable value: value must be known

[m1keil]

Stumbled upon the same thing. I'm not sure if that's a feature request + a documentation bug or a bug with abspath.

Either way, having the ability to specify paths that depend on the nomad job path and not Nomad's CWD would be a huge plus.

[tgross]

The current abspath behavior is deliberate because we do all the HCL2 parsing in the context of the CLI's current directory. For what it's worth, I suspect that trying to use path.cwd and path.pwd and a relative path is going to be just as brittle as hard-coding the path. Maybe something like path.jobfile or path.jobdirectory would be valuable. But this seems like a totally reasonable feature request.

[m1keil]

@tgross Do you think it's reasonable to submit a documentation fix for this?

[tgross]

Sure, happy to accept a PR!

[kevinschoonover]

The current abspath behavior is deliberate because we do all the HCL2 parsing in the context of the CLI's current directory. For what it's worth, I suspect that trying to use path.cwd and path.pwd and a relative path is going to be just as brittle as hard-coding the path. Maybe something like path.jobfile or path.jobdirectory would be valuable. But this seems like a totally reasonable feature request.

path.jobdirectory (or equivalent) is exactly what I'm looking for since the configuration path should be closely related to the job file path at least for my use case.

Look forward to seeing what you come up with.