Draft: Add YANG modeling and validation feature for topology files

[jbemmel]

Note: This PR is a “sneak preview” of a potential feature and is not intended to be merged as-is.

This PR introduces a Netlab YANG topology validation feature, which validates a topology file against an internal YANG schema.

The schema defines the structure of a valid topology and includes declarative must statements, such as “the node name referenced in a link must exist.” The included schema is partially complete, covering all tests in tests/integration/initial to start. Currently, it has four must statements, which can be extended over time.

Additionally, the PR includes preliminary support for detecting the 'wtf' link type in errors/.

The intention of this PR is to illustrate a possible approach to declarative topology validation. Structural validation is handled in YANG, while detailed semantic rules (like type coercion for interfaces) remain in Python. This approach demonstrates how extended validation could evolve without replacing existing functionality.

Features:

• New 'netlab yang' command to validate topology files
• YANG model (netlab-topology.yang) with comprehensive topology structure
• Support for YANG MUST statement validation using yangson library
• JSON cache support for faster validation (via consolidate feature)
• All tests/integration/initial/*.yml files pass validation

YANG Model:

• Defines complete topology structure (nodes, links, defaults, etc.)
• Supports device-specific and module-specific attributes (anydata)
• Validates node roles, MTU, loopback configuration
• Validates link types, interfaces, addressing, and attributes
• Flexible structure using anydata for extensibility

CLI Command:

• netlab yang [topology-file] --model [yang-model] --output [text|json]
• Supports --json-cache option for faster validation
• Supports NETLAB_JSON_CACHE environment variable
• Detailed error reporting with full tracebacks

Dependencies:

• yangson >= 1.4.19
• jsonschema v4+ compatibility fixes in consolidate.py

Files:

• netsim/cli/yang.py: CLI command handler
• netsim/cli/yang_validator.py: YANG validation implementation
• netsim/yang/netlab-topology.yang: YANG model definition
• MANIFEST.in: Include YANG model files in package
• requirements.txt: Update yangson version requirement

[ipspace]

Apart from an interesting intellectual exercise, I don't see how this could be useful as long as there are so many "anydata" leaves in the schema.

Also, YANG is known for its inability to allow a leaf to have two data types. As you well know, we sometimes allow a str that is then automatically turned into a list, or a bool instead of an IPv4 address. Also, see https://blog.ipspace.net/2025/04/api-data-model-contract/ for an example of what happens when someone drinks too much YANG Kool-Aid.

[jbemmel]

Apart from an interesting intellectual exercise, I don't see how this could be useful as long as there are so many "anydata" leaves in the schema.

Also, YANG is known for its inability to allow a leaf to have two data types. As you well know, we sometimes allow a str that is then automatically turned into a list, or a bool instead of an IPv4 address. Also, see https://blog.ipspace.net/2025/04/api-data-model-contract/ for an example of what happens when someone drinks too much YANG Kool-Aid.

Thanks for your feedback — I understand this PR may not be the ideal format for deeper discussion.

I wanted to check if you had a chance to review line 456:

leaf ipv4 {
          type union {
            type boolean;
            type string;
          }
          description
            "IPv4 address for this interface. Can be specified as an address
             string or as a boolean 'true' for unnumbered interfaces (where
             the interface uses the node's loopback address).";
        }

This is a union type that allows either a string or a boolean, with Python handling the semantic rules (e.g., true for unnumbered interfaces). Structurally, this approach enforces topology integrity while accommodating the flexible typing you mentioned.

I thought it might be worth highlighting as an example of how YANG could handle multiple types safely. It may be we'd need to revise the point in the process where this check would fit best - curious to hear your thoughts.

[ipspace]

Also, YANG is known for its inability to allow a leaf to have two data types.

This is a union type that allows either a string or a boolean, with Python handling the semantic rules (e.g., true for unnumbered interfaces).

Interesting, and it's already in RFC 7950. I wonder where did I get the idea that this is not possible with YANG?

Thank you, I learned something new ;)

[jbemmel]

Interesting, and it's already in RFC 7950. I wonder where did I get the idea that this is not possible with YANG?

Thank you, I learned something new ;)

We both did - thank you too.

Would you like to explore further, or do you prefer to leave it at this?

[ipspace]

Would you like to explore further, or do you prefer to leave it at this?

I don't think it makes sense to switch. We built tons of little quirks into our system (like transforming "str" values into "lists") as we needed them, and we use attributes dictionaries in other parts of the code.

All that stuff would have to be replaced, and I still wonder what the gains might be.

[jbemmel]

Would you like to explore further, or do you prefer to leave it at this?

I don't think it makes sense to switch. We built tons of little quirks into our system (like transforming "str" values into "lists") as we needed them, and we use attributes dictionaries in other parts of the code.

All that stuff would have to be replaced, and I still wonder what the gains might be.

One way we could use YANG is as a kind of sanity check to verify that the user input data model is consistent before it reaches downstream transformations. Existing normalization, device quirks, and templates would remain unchanged — nothing is replaced.

The 'anydata' illustrates how it allows for incremental adoption to see if it’s actually helpful in practice. The PR only covers the tests/integration/initial labs - hence the union type for ipv4. Other modules are deliberately left without structure.

I structured it as a separate 'yang' command so you could run it easily - but eventually, this would be part of the data processing workflow for netlab create.

[ipspace]

One way we could use YANG is as a kind of sanity check to verify that the user input data model is consistent before it reaches downstream transformations.

We do all that and more in existing code, so what would we gain?

I structured it as a separate 'yang' command so you could run it easily

If you wish to invest time into this and keep it as an optional data structure, feel free to do it. However, will you maintain the YANG schema "indefinitely" or will it become obsolete the moment we add something new?

[jbemmel]

If you wish to invest time into this and keep it as an optional data structure, feel free to do it. However, will you maintain the YANG schema "indefinitely" or will it become obsolete the moment we add something new?

I can contribute it as a plugin, and only include it in select use cases (that make sense to me). That way, it doesn't bother anyone - besides me - if it gets stale. It will be my problem, not yours