Add support for extra template variables

'netlab config' used to pass the '-e' parameter to the Ansible
playbook. That approach no longer works after we started rendering the
Jinja2 templates in netlab.

This commit parses '-e' parameters in 'netlab config' command and adds
extra variables to node data, making them available in custom config
templates.

Other minor fixes:

* Refactor common Ansible args processing into shared functions
* Check that no extra vars are specified on config reload
* Remove an unneeded call to set_custom_config

Author: ipspace

docs/netlab/config.md

@@ -1,14 +1,16 @@
 (netlab-config)=
 # Deploying Custom Device Configurations
 
-**netlab config** uses an internal Ansible playbook (`netsim/ansible/config.ansible`) to deploy custom device configurations generated from the supplied Jinja2 template(s) to lab devices. It takes lab parameters from the _netlab_ snapshot file and Ansible inventory created by the **[netlab create](netlab-create)** or **[netlab up](netlab-up)** command.
+**netlab config** is used to deploy custom device configuration templates to lab devices. It reads lab parameters from the _netlab_ snapshot file created by the **[netlab create](netlab-create)** or **[netlab up](netlab-up)** command, renders the supplied Jinja2 template ([limitations](dev-templates)), and uses the internal `config.ansible` Ansible playbook to deploy the rendered configuration snippets.
 
-You have to use **netlab config** on a running lab. If you want to try out the configuration templates without starting the lab,  add the [**config** attribute](custom-config) to node data and run **netlab create** (to create the Ansible inventory) followed by **[netlab initial -c -o](netlab-initial)** to create the configuration files.
+You have to use **netlab config** on a running lab. If you want to try out the configuration templates without starting the lab,  add the [**config** attribute](custom-config) to node data and run **netlab create** (to generate the snapshot file), followed by **[netlab initial -c -o](netlab-initial)** to create the configuration files.
 
 ## Usage
 
 ```text
-usage: netlab config [-h] [-r] [-v] [-q] [-i INSTANCE] template
+usage: netlab config [-h] [-r] [-l LIMIT] [-e EXTRA_VARS [EXTRA_VARS ...]] [-v] [-q]
+                     [-i INSTANCE]
+                     template
 
 Deploy custom configuration template
 
@@ -18,10 +20,13 @@ positional arguments:
 options:
   -h, --help            show this help message and exit
   -r, --reload          Reload saved device configurations
+  -l, --limit LIMIT     Limit the operation to a subset of nodes
+  -e, --extra-vars EXTRA_VARS [EXTRA_VARS ...]
+                        Specify extra variables for the configuration template
   -v, --verbose         Verbose logging (add multiple flags for increased verbosity)
   -q, --quiet           Report only major errors
-  -i INSTANCE, --instance INSTANCE
-                        Specify lab instance to configure
+  -i, --instance INSTANCE
+                        Specify the lab instance to configure
 
 All other arguments are passed directly to ansible-playbook
 ```
@@ -40,19 +45,39 @@ When executed with the `-i` option, **‌netlab config** expects the configurati
 
 ## Limiting the Scope of Configuration Deployments
 
-All unrecognized parameters are passed to the internal `config.ansible` Ansible playbook. You can use **ansible-playbook** CLI parameters to modify the configuration deployment, for example:
+You can use the `-l` parameter to deploy device configurations on a subset of devices. The parameter value must be a valid _netlab_ node selection expression ([more details](netlab-inspect-node)).
 
-* `-l` parameter to deploy device configurations on a subset of devices.
-* `-C` parameter to run the Ansible playbook in dry-run mode.
+All unrecognized parameters are passed to the internal `config.ansible` Ansible playbook, allowing you to use the **ansible-playbook** CLI parameters to modify the configuration deployment. For example, you can use the `-C` parameter to run the Ansible playbook in dry-run mode.
+
+## Extra Variables
+
+You can use the `-e` parameter to specify an extra variable value in the `name=value` format (the `-e` parameter can be used multiple times). _netlab_ recognizes only the `name=value` format, not the JSON or filename formats recognized by Ansible.
+
+The extra variables are applied to all nodes and can be used in device configuration templates. For example, the following Jinja2 template uses the `df_state` variable to turn BGP default route advertisements on or off:
+
+```
+router bgp {{ bgp.as }}
+!
+{% for af in ['ipv4','ipv6'] %}
+{%   for ngb in bgp.neighbors if af in ngb %}
+{%     if loop.first %}
+  address-family {{ af }}
+{%     endif %}
+    {% if df_state|default('') == 'off' %}no {% endif %}neighbor {{ ngb[af] }} default-originate
+{%   endfor %}
+{% endfor %}
+```
+
+After saving the above template into `bgp_default.j2`, you can use `netlab config bgp_default --limit somenode` to enable BGP default route advertisement and `netlab config bgp_default --limit somenode -e df_state=off` to turn it off.
 
 ## Restoring Saved Device Configurations
 
-**netlab config --reload** implements the *reload saved device configurations* part of the **netlab initial -r** command. It waits for devices to become ready (because it's used immediately after a lab has been started) and starts the initial configuration process on devices that need more than a replay of saved configuration ([more details](netlab-up-reload)).
+**netlab config --reload** implements the *reload saved device configurations* part of the **netlab initial -r** command. It waits for devices to become ready (since it's used immediately after a lab has been started) and starts the initial configuration process on devices that need more than a replay of saved configuration ([more details](netlab-up-reload)).
 
-After that, it treats the saved device configurations as custom templates and uses the same process as the regular **netlab config** command.
+After that, it treats the saved device configurations as custom templates (using the same process as the regular **netlab config** command), allowing you to use Jinja2 expressions in saved device configurations.
 
 ## Debugging Device Configurations
 
 To display device configurations within the Ansible playbook without deploying them, use `-v --tags test` parameters after the template name. 
 
-The `-v` flag triggers a debugging printout, and the bogus `test` flag skips the configuration deployment.
+The `-v` flag enables debugging output, and the bogus `test` flag skips configuration deployment.

netsim/cli/config.py

@@ -9,7 +9,7 @@
 
 from box import Box
 
-from ..data import get_empty_box
+from ..data import get_box
 from ..utils import log
 from . import (
   _nodeset,
@@ -22,6 +22,7 @@
   parser_lab_location,
 )
 from .initial import configs as i_configs
+from .initial import utils as i_utils
 
 
 #
@@ -41,6 +42,10 @@ def custom_config_parse(args: typing.List[str]) -> typing.Tuple[argparse.Namespa
     '-l','--limit',
     dest='limit', action='store',
     help='Limit the operation to a subset of nodes')
+  parser.add_argument(
+    '-e','--extra-vars',
+    dest='extra_vars',action='store',nargs='+',
+    help='Specify extra variables for the configuration template')
   parser.add_argument(
     dest='template', action='store',
     help='Configuration template or a directory with templates')
@@ -57,14 +62,22 @@ def set_initial_args(args: argparse.Namespace, initial: bool = False) -> None:
   setattr(args,'no_refresh',False)                # ... mandatory refresh
   setattr(args,'generate',None)                   # ... and internally-generated configs
 
-def set_custom_config(topology: Box, nodeset: list, cfg_name: str) -> None:
+def set_custom_config(
+      topology: Box,
+      nodeset: list,
+      cfg_name: str,
+      extra_vars: dict = {}) -> None:
+
   for n_name in nodeset:
-    topology.nodes[n_name].config = [ cfg_name ]
+    n_data = topology.nodes[n_name]
+    n_data.config = [ cfg_name ]
+    for k,v in extra_vars.items():
+      n_data[k] = v
 
-def ansible_extra_vars(topology: Box, reload: bool = False) -> Box:
+def ansible_extra_vars(topology: Box, reload: bool = False, extra_vars: dict = {}) -> Box:
   cfg_sfx = '.cfg' if reload else ''
 
-  ev = get_empty_box()
+  ev = get_box(extra_vars)
   ev.node_files = str(Path("./node_files").resolve().absolute())
 
   ev.paths_t_files.files = "{{ config_module }}" + cfg_sfx    # Take only module file from node_files
@@ -76,6 +89,29 @@ def ansible_extra_vars(topology: Box, reload: bool = False) -> Box:
   ev.paths_custom.tasks = topology.defaults.paths.custom.tasks
   return ev
 
+def get_ansible_args(ans_vars: Box,nodeset: list,cfg_name: str) -> list:
+  args = i_utils.common_ansible_args()
+  args += ["-e",ans_vars.to_json(),"-e",f'config={cfg_name}']
+  args += ["-l",','.join(nodeset)]
+  return args
+
+def parse_extra_vars(ev_list: typing.Optional[list]) -> dict:
+  ev: dict = {}
+  if not ev_list:
+    return ev
+
+  for v_item in ev_list:
+    if '=' not in v_item:
+      error_and_exit('Extra variables have to be specified in name=value format')
+    (n,v)  = v_item.split('=',maxsplit=1)
+    try:
+      value = eval(v)
+    except:
+      value = v
+    ev[n] = value
+
+  return ev
+
 """
 Create the required configs in node_files
 """
@@ -84,11 +120,12 @@ def create_node_files(
       nodeset: list,
       args: argparse.Namespace,
       cfg_name: str,
+      extra_vars: dict = {},
       initial: bool = False,
       cfg_suffix: str = 'none') -> None:
 
   set_initial_args(args,initial=initial)              # Adjust args for 'netlab initial' processing
-  set_custom_config(topology,nodeset,cfg_name)
+  set_custom_config(topology,nodeset,cfg_name,extra_vars)
 
   i_configs.create_node_configs(                      # Create the necessary files in node_files directory
     topology=topology,
@@ -109,6 +146,9 @@ def reload_node_configs(topology: Box,nodeset: list,args: argparse.Namespace, re
   if not cfg_path.is_dir():                           # Sanity check: are we reloading from a directory?
     error_and_exit('The argument specified with the --reload option must be a directory')
 
+  if args.extra_vars:
+    error_and_exit('You cannot specify extra vars while reloading configuration')
+
   no_config = []
   for n_name in nodeset:                              # Identify nodes that have no configs
     if not list(cfg_path.glob(n_name+'.*')):          # ... in the specified directory
@@ -136,8 +176,7 @@ def reload_node_configs(topology: Box,nodeset: list,args: argparse.Namespace, re
   # Run the Ansible playbook with modified path variables and an adjusted nodeset
   #
   ans_vars = ansible_extra_vars(topology,reload=True)
-  rest_args = rest + ["-e",ans_vars.to_json(),"-e",f'config={str(cfg_path.name)}']
-  rest_args += ["-l",','.join(nodeset)]
+  rest_args = rest + get_ansible_args(ans_vars,nodeset,str(cfg_path.name))
   if not ansible.playbook('reload-config.ansible',rest_args,abort_on_error=False):
     error_and_exit('Cannot reload initial device configurations')
 
@@ -153,14 +192,13 @@ def deploy_custom_config(topology: Box,nodeset: list,args: argparse.Namespace, r
     if cfg_name.endswith('.j2'):
       cfg_name = cfg_name[:-3]
 
-  set_custom_config(topology,nodeset,cfg_name)
-  create_node_files(topology,nodeset,args,cfg_name)
+  extra_vars=parse_extra_vars(args.extra_vars)
+  create_node_files(topology,nodeset,args,cfg_name,extra_vars)
 
   # Run the Ansible playbook with modified path variables and an adjusted nodeset
   #
-  ans_vars = ansible_extra_vars(topology,reload=False)
-  rest_args = rest + ["-e",ans_vars.to_json(),"-e",f'config={cfg_name}']
-  rest_args += ["-l",','.join(nodeset)]
+  ans_vars = ansible_extra_vars(topology,reload=False,extra_vars=extra_vars)
+  rest_args = rest + get_ansible_args(ans_vars,nodeset,cfg_name)
   if not ansible.playbook('config.ansible',rest_args,abort_on_error=False):
     error_and_exit('Cannot deploy custom configuration template')
 

netsim/cli/initial/utils.py

@@ -8,6 +8,7 @@
 
 from box import Box
 
+from ...utils import log
 from .. import common_parse_args, parser_lab_location
 
 
@@ -73,23 +74,28 @@ def initial_config_parse(args: typing.List[str]) -> typing.Tuple[argparse.Namesp
 
   return parser.parse_known_args(args)
 
+def common_ansible_args() -> list:
+  rest = []
+  if log.VERBOSE:
+    rest += ['-' + 'v' * log.VERBOSE]
+
+  if log.QUIET:
+    os.environ["ANSIBLE_STDOUT_CALLBACK"] = "selective"
+
+  return rest
+
 """
 Build Ansible arguments based on 'netlab initial' parameters
 """
 def ansible_args(args: argparse.Namespace) -> list:
-  rest: typing.List[str] = []
-  if args.verbose:
-    rest = ['-' + 'v' * args.verbose] + rest
+  rest = common_ansible_args()
 
   if args.limit:
     rest = ['--limit',args.limit] + rest
 
   if args.initial:
     rest = ['-t','initial'] + rest
 
-  if args.quiet:
-    os.environ["ANSIBLE_STDOUT_CALLBACK"] = "selective"
-
   if args.module:
     if args.module != "*":
       rest = ['-e','modlist='+args.module] + rest