Problem using templates after Nomad update

[uranru]

Hi there,

I updated Consul on my servers from version 1.20.4 to version 1.22.3.
I updated Nomad from version 1.9.6 to version 1.11.1.

These templates stopped working for me.

      template {
        data = <<EOH
          {{ range service "server.zabbix" }}
          ZBX_SERVER_HOST={{ .Address }}:{{ .Port }}
          {{ end }}
          EOH
        destination = "local/env"
        env         = true
      }

      template {
        data = <<EOH
          {{ range service "java-gateway.zabbix" }}
          ZBX_JAVAGATEWAY     = {{ .Address }}
          ZBX_JAVAGATEWAYPORT = {{ .Port }}
          {{ end }}
          EOH
        destination = "local/env"
        env         = true
      }

At the same time, I see that the services are registered in Сosnul

consul catalog services -tags

consul
consul-dns        primary
nomad             http,rpc,serf
nomad-client      http
zabbix            balancer,db-restapi,haproxy,java-gateway,master,node0,node1,pgbouncer,postgres,primary,proxy,proxy-tools,replica,server,web

It's very sad that this functionality stopped working.

Nomad version

Nomad v1.11.1
BuildDate 2025-12-09T20:10:56Z
Revision 5b76eb0

Operating system and Environment details

Issue

Reproduction steps

Expected Result

Actual Result

Job file (if appropriate)

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[jrasell]

Hi @uranru and thanks for raising this issue. What error are you getting when you try and run the jobs with the detailed template? 1.10 introduced changes to the way Nomad integrates with Consul, did you work through the upgrade guide when you performed the Nomad upgrade? I would specifically call out the changes here.

[uranru]

Thanks for the quick response.
I checked the Nomad and Consul logs and found no errors.
I then filed a bug report.
But I missed the changes in the Identity settings ( . I'll test these settings on the test cluster today.

[uranru]

I read the documentation; the integration is tied to nomad_service.
But the task doesn't generate such a field in the JWT ​​token.
Here's an example:

{
  "aud": "consul.io",
  "iat": 1770044790,
  "jti": "96228a1a-8d85-36aa-138f-34b843be6cd3",
  "nbf": 1770044790,
  "nomad_allocation_id": "e5374d62-fe34-dbe3-6b49-e3d8ce6d8212",
  "nomad_job_id": "wiki_postgresql",
  "nomad_namespace": "default",
  "nomad_task": "wiki_postgresql",
  "sub": "tn1:default:wiki_postgresql:wiki_postgresql:wiki_postgresql:service_identity"
}

[tgross]

@uranru the nomad_service claim is for services that are being registered, not for queries over services. The service registration part is definitely getting that claim, otherwise you wouldn't see them registered in Consul.

You still haven't provided logs or error messages or a full jobspec, but I suspect you're problem would be fixed if you add an empty consul {} block to the task. This tells the template runner that it needs a Consul token, which is not supplied by default as of Nomad 1.10.0.

[uranru]

Sorry,
I'll provide my test configuration below.
Please note that I'm using Ansible templates.
I hope this doesn't confuse you.

nomad.hcl.j2:

data_dir  = "/opt/nomad/data"
bind_addr = "{{ query('community.general.dig', inventory_hostname) | reject('equalto', cluster.ip) | first }}"

# http слушает два IP, для возможности локального обращения консул к jwks.json
addresses {
  http = "{{ query('community.general.dig', inventory_hostname) | reject('equalto', cluster.ip) | first }} 127.0.0.1"
}

datacenter  = "{{ cluster.dc }}"
region      = "tn1"

server {
  enabled                = true
  bootstrap_expect       = 3
  min_heartbeat_ttl      = "5s"
  heartbeat_grace        = "10s"
  failover_heartbeat_ttl = "30s"
  job_gc_threshold       = "48h"

  server_join {
    retry_join = {{ cluster.nodes | to_json }}
    retry_max      = 0
    retry_interval = "5s"
  }
}

client {
  enabled = true
  servers = {{ cluster.nodes | to_json }}
  cni_path = "/opt/cni/bin"
  cni_config_dir = "/opt/cni/conf"

  meta {
    # Переменная rack - для распределения и наименования сервисов
    rack           = "{{ nomad_meta_rack }}"
    # Переменная cluster_peers - для настроек keepalive
    cluster_peers  = "{{ query('community.general.dig', *cluster.nodes | reject('match', inventory_hostname_short)) | reject('equalto', cluster.ip) | join(',') }}"
  }

#  host_volume "host_volume" {
#    path      = "/opt/nomad/host_volume"
#    read_only = false
#  }

}

consul {
  address = "{{ query('community.general.dig', inventory_hostname) | reject('equalto', cluster.ip) | first }}:8500"
  token = "{{ consul.token }}"
  grpc_address = "127.0.0.1:8502"

  service_identity {
    aud = ["consul.io"]
    ttl = "12h"
  }

  task_identity {
    aud = ["consul.io"]
    ttl = "12h"
  }
}


{% if vault_addr is defined and vault_backend_path is defined %}
vault {
  enabled = true
  address = "{{ vault_addr }}"
  jwt_auth_backend_path = "{{ vault_backend_path }}"
}
{% endif %}


ui {
  enabled = true
}

acl {
  enabled = true
}

plugin "docker" {
  config {
    allow_privileged = true
    allow_caps = ["NET_ADMIN","NET_BROADCAST","NET_RAW"]
    volumes {
      enabled = true
    }
    extra_labels = ["job_name", "job_id", "task_group_name", "task_name", "namespace", "node_name", "node_id"]

    # Параметры чистки образов и контейнеров
    gc {
      image       = false
      container   = true
    }
  }
}
{% if nomad_meta_rack == 'r3' %}

plugin "raw_exec" {
  config {
    enabled = true
  }
}
{% endif %}

telemetry {
  collection_interval        = "5s"
  publish_allocation_metrics = true
  publish_node_metrics       = true
  prometheus_metrics         = true
}

## Logging
enable_syslog        = false
log_level            = "INFO"
log_file             = "/var/log/nomad/nomad.log.json"
#log_file             = "/var/log/nomad/nomad.log"
log_json             = true
#log_json             = false
log_rotate_duration  = "24h"
#log_rotate_bytes    = 10485760
log_rotate_max_files = 5

wiki-postgresql.nomad.j2:

job "wiki_postgresql" {

  datacenters = ["{{ cluster.dc }}"]
  type        = "service"

  # Запускать таски всегда на разных нодах
  constraint {
    distinct_hosts = true
  }

  migrate {
    max_parallel     = 1
    health_check     = "checks"
    min_healthy_time = "10s"
    healthy_deadline = "5m"
  }

  update {
    max_parallel = 1
    healthy_deadline = "3m"
    progress_deadline = "10m"
    auto_revert = false
    canary = 0
  }

  group "wiki_postgresql" {
    count = 1

    network {
      mode = "host"

      port "postgresql" {
        to     = 5432
        static = 5432
      }
      dns {
        servers = ["198.19.128.1"]
        searches = ["service.{{ consul.domain }}", "prodnt"]
      }
    }

    task "wiki_postgresql" {
      driver = "docker"

#      vault {
#        disable_file = true
#        env          = true
#        change_mode  = "noop"
#      }

#      identity {
#        name = "vault_default"
#        aud  = ["vault-nd-zabbix.io"]
#      }

      consul {}

#      identity {
#        env  = true
#      #file = true # Сохраним в файл для удобства чтения
#        #name = "service_identity"
#        aud  = ["consul.io"]
#        ttl  = "12h"
#      }

      resources {
        cores = 2
        memory = 4096
      }

      config {
        hostname = "wiki-postgresql-node${NOMAD_ALLOC_INDEX}"
        image = "{{ docker.version_postgresql }}"
        ports = ["postgresql"]

        mount {
          type = "bind"
          source = "/opt/service/wiki-postgresql/data/"
          target = "/bitnami/postgresql/"
          readonly = false
        }

        #logging {
        #  type = "fluentd"
        #  config {
        #    fluentd-address = "localhost:24224"
        #    fluentd-async-connect = "true"
        #    tag = "nomad.patroni"
        #  }
        #}
      }

      constraint {
        attribute = "${meta.rack}"
        operator  = "set_contains_any"
        value     = "{{ app1.nomad_constraint }}"
      }

{% raw %}
      template {
        data = <<EOH
          {{ range service "web.wiki" }}
          TEST_SERVER_HOST={{ .Address }}:{{ .Port }}
          {{ end }}
          EOH
        destination = "local/env"
        env         = true
      }
{% endraw %}


{% raw %}
#      template {
#        data        = <<EOF
#Consul Services:
#{{- range services}}
#  * {{.Name}}{{end}}
#EOF
#        destination = "local/consul-info.txt"
#      }
{% endraw %}

      env {
              ALLOW_EMPTY_PASSWORD         = "no"
        POSTGRESQL_PORT_NUMBER       = "5432"
        POSTGRESQL_USERNAME          = "{{ app1.database_user }}"
        POSTGRESQL_PASSWORD          = "{{ app1.database_password_app }}"
        POSTGRESQL_DATABASE          = "{{ app1.database_name }}"
        POSTGRESQL_POSTGRES_PASSWORD = "{{ app1.database_password_app }}"
        POSTGRESQL_MAX_CONNECTIONS   = "100"
        POSTGRES_INITDB_ARGS         = "--encoding=UTF8"
      }

      service {
        name         = "{{ app1.name }}"
        provider     = "consul"
        port         = "postgresql"
        address_mode = "host"
        tags         = ["node${NOMAD_ALLOC_INDEX}", "postgresql"]
      }

    }
  }


}

If I do this in Consul, I get a 403 error when starting the task.
settings:

consul acl auth-method create \
-name 'nomad-workloads' -type 'jwt' -description 'JWT auth method for Nomad services and workloads' \
-http-addr=http://***:8500 \
-token=*** \
-config -<<EOF
{
  "JWKSURL": "http://127.0.0.1:4446/.well-known/jwks.json",
  "JWTSupportedAlgs": ["RS256"],
  "BoundAudiences": ["consul.io"],
  "ClaimMappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task",
    "nomad_service": "nomad_service"
  }
}
EOF


consul acl binding-rule create \
-method 'nomad-workloads' \
-http-addr=http://***:8500 \
-token=*** \
-description 'Binding rule for services registered from Nomad' \
-bind-type 'service' \
-bind-name '${value.nomad_service}' \
-selector '"nomad_service" in value'

log:

{"@level":"debug","@message":"warning: response content-type header not explicitly set.","@module":"agent","@timestamp":"2026-02-02T16:35:56.380449+03:00","request-path":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/v1/ac
l/token/self","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"stale=","Fragment":"","RawFragment":""}}
{"@level":"error","@message":"Request error","@module":"agent.http","@timestamp":"2026-02-02T16:35:56.381974+03:00","error":"Permission denied","from":"***:46302","method":"POST","url":"/v1/acl/login"}
{"@level":"debug","@message":"Request finished","@module":"agent.http","@timestamp":"2026-02-02T16:35:56.382239+03:00","from":"***:46302","latency":"786.904µs","method":"POST","url":"/v1/acl/login"}

If I add this there are no errors, but the templates don't work:

consul acl binding-rule create -method 'nomad-workloads' -description 'Binding rule for Nomad tasks' -bind-type 'role' -bind-name 'nomad-tasks-${value.nomad_namespace}' -selector '"nomad_service" not in value'


root@nd-dev-n01:/opt/consul/auth# consul acl token list -http-addr=http://***:8500 -token=***
AccessorID:       9b868ed3-aab9-7c78-54f1-6c369d41fb89
SecretID:         ***
Description:      token created via login: {"requested_by":"nomad_task_wiki_postgresql"}
Local:            true
Auth Method:      nomad-workloads (Namespace: )
Create Time:      2026-02-02 18:06:31.00979922
Roles:
   aba41348-d14c-8232-77b0-6884d7229199 - nomad-tasks-default

AccessorID:       352fcf45-188b-0411-84ae-498abe185d55
SecretID:         ***
Description:      token created via login: {"requested_by":"nomad_task_wiki_server"}
Local:            true
Auth Method:      nomad-workloads (Namespace: )
Create Time:      2026-02-02 17:18:55.586494948
Roles:
   aba41348-d14c-8232-77b0-6884d7229199 - nomad-tasks-default

AccessorID:       9d0946f9-6e4c-29ef-fbbc-d471d37b62bb
SecretID:         ***
Description:      token created via login: {"requested_by":"nomad_service_wiki"}
Local:            true
Auth Method:      nomad-workloads (Namespace: )
Create Time:      2026-02-02 18:06:31.003118896
Service Identities:
   wiki (Datacenters: all)

AccessorID:       00000000-0000-0000-0000-000000000002
SecretID:         anonymous
Description:      Anonymous Token
Local:            false
Create Time:      2024-09-25 11:06:53.330543169

AccessorID:       78d0fc5f-6751-83dc-30e9-497e39fc1cf5
SecretID:         ***
Description:      token created via login: {"requested_by":"nomad_service_haproxy"}
Local:            true
Auth Method:      nomad-workloads (Namespace: )
Create Time:      2026-02-02 16:06:19.013402251
Service Identities:
   haproxy (Datacenters: all)

AccessorID:       33f8a6f9-808b-32ab-b900-d6aeec1e6fcf
SecretID:         ***
Description:      token created via login: {"requested_by":"nomad_service_wiki"}
Local:            true
Auth Method:      nomad-workloads (Namespace: )
Create Time:      2026-02-02 17:18:55.580488342
Service Identities:
   wiki (Datacenters: all)

AccessorID:       cb4ec412-c95e-df23-38bb-f7778cf37e09
SecretID:         ***
Description:      Initial Management Token
Local:            false
Create Time:      2024-09-25 11:06:53.325695489
Policies:
   00000000-0000-0000-0000-000000000001 - global-management

If I delete the previous binding-rule and do this, the templates work, but I don’t like these settings:

consul acl binding-rule create \
-http-addr=http://***:8500 \
-token=*** \
-method 'nomad-workloads' \
-description 'Binding rule for services registered from Nomad' \
-bind-type 'service' \
-bind-name 'test' \
-selector ''

[tgross]

You need two binding rules for Consul. The binding rule you've created allows a service to register itself, which matches with the symptoms you're seeing where the services are registered. That's this one you created:

consul acl binding-rule create \
  -method 'nomad-workloads' \
  -description 'Binding rule for services registered from Nomad' \
  -bind-type 'service' \
  -bind-name '${value.nomad_service}' \
  -selector '"nomad_service" in value'

But you need a second binding rule for the template blocks in the tasks to read the service catalog so that you can read other services or the Consul KV store. That'll look something like:

consul acl binding-rule create \
  -method 'nomad-workloads' \
  -bind-type 'role' \
  -bind-name 'nomad-tasks-${value.nomad_namespace}' \
  -selector '"nomad_service" not in value'

You also need to define the Consul ACL Role for the workloads so that this binding rule grants the permissions you want.

See also the documentation on configuring Consul binding rules for Nomad.

[uranru]

I wrote about this above.
this rule was created
Job started, but the templates didn't work