diff --git a/.github/workflows/CI_test_suite.yml b/.github/workflows/CI_test_suite.yml index 2a0f718..90a7454 100644 --- a/.github/workflows/CI_test_suite.yml +++ b/.github/workflows/CI_test_suite.yml @@ -34,9 +34,9 @@ jobs: run: pip install -r python_requirements.txt working-directory: test_suite - - name: Run system tests sequentially + - name: Run system tests sequentially with NAT IP pooling id: system-test - run: ./system_tests.sh -b + run: ./system_tests.sh -b -n 2 working-directory: test_suite continue-on-error: true @@ -69,9 +69,9 @@ jobs: cache-from: type=gha cache-to: type=gha,mode=max - - name: Run system tests in parallel + - name: Run system tests in parallel with NAT IP pooling id: system-test - run: GITHUB_ACTION=true ./system_tests.sh -t 4 + run: GITHUB_ACTION=true ./system_tests.sh -t 4 -n 2 working-directory: test_suite continue-on-error: true diff --git a/test_suite/CHANGELOG.md b/test_suite/CHANGELOG.md index 45a528e..0c87228 100644 --- a/test_suite/CHANGELOG.md +++ b/test_suite/CHANGELOG.md @@ -2,6 +2,18 @@ In this file, the test suite features that have been made possible thanks to [funding from NLnet](./README.md#funding) are documented. +## NAT IP pooling (June 20, 2025) +### Added +- Explanation of NAT IP pooling and how it is implemented in the test suite in the [system test documentation](./README.md#ip-address-pooling). +- The `-n` flag to [`system_tests.sh`](system_tests.sh) and a positional parameter to [`system_test.sh`](system_test.sh),[`nat_simulation/setup_networks.sh`](nat_simulation/setup_networks.sh) and [`nat_simulation/setup_router.sh`](nat_simulation/setup_router.sh), which specify the amount of IP addresses available to the routers for NAT IP pooling. +- New expected test result `TS_PASS`, which accepts both a direct and relayed connection between the peers. This new result is necessary because for some NAT combinations with IP pooling, it is uncertain whether a direct connection can be established. +- Report on which system tests are affected by IP pooling in the [system test results](./README.md#effect-of-ip-address-pooling-on-system-test-results). + +### Changed +- The logic in [`system_tests.sh`](system_tests.sh) which decides the expected test result based on the specified NAT combination of the peers. This logic now uses the new `TS_PASS` result for certain combinations if NAT IP pooling is enabled. +- Branches on the (expected) test result in [`system_test.sh`](system_test.sh) and [`test_client/setup_client.sh`](test_client/setup_client.sh), such that they also take the new `TS_PASS` result into account. +- Older results of the system tests without NAT IP pooling. They now contain a reference to the new results, and their visualization has been changed to align with the new results. +- The implementation of the NAT mapping & filtering behaviour, respectively found in [`nat_simulation/setup_networks.sh`](nat_simulation/setup_networks.sh) and [`nat_simulation/setup_router.sh`](nat_simulation/setup_router.sh). The implementation of the mapping behaviour now also does NAT IP pooling, and the filtering behaviour had to be adjusted to take the multiple IP addresses into account. ## Parallel system tests (April 11, 2025) diff --git a/test_suite/README.md b/test_suite/README.md index 31f6017..7963c2f 100644 --- a/test_suite/README.md +++ b/test_suite/README.md @@ -10,7 +10,8 @@ A CI test suite for the eduP2P prototype. 3. [Performance Tests](#performance-tests) 4. [Integration Tests](#integration-tests) 2. [Results](#results) - 1. [System Test Results](#system-test-results) + 1. [System Test + Results](#system-test-results-without-ip-address-pooling) 2. [Performance Test Results](#performance-test-results) 3. [Integration Test Results](#integration-test-results) 3. [Bibliography](#bibliography) @@ -115,12 +116,16 @@ requirements](#system-test-specific-requirements), the tests may be run in parallel using Docker. The user can specify the amount of “threads” with the `-t` flag, which determines over how many Docker containers the tests will be distributed. The reason for using Docker is that it allows -the concurrent tests to be executed in isolated networks. Naturally, -running the tests in parallel allows the tests to run much faster. One -disadvantage of the parallel tests is that a Docker image must be built -before running the tests. This takes quite a while the first time the -image is built, but by making use of Docker’s caching, building the -image again after revisions to the code is significantly faster. +the concurrent tests to be executed in isolated networks, which improves +the clarity of the logs and avoid any interference that could be caused +by multiple tests running simultaneously in the same network. + +Naturally, running the tests in parallel allows the tests to run much +faster. One disadvantage of the parallel tests is that a Docker image +must be built before running the tests. This takes quite a while the +first time the image is built, but by making use of Docker’s caching, +building the image again after revisions to the code is significantly +faster. The parallel system tests are also being used in the CI GitHub workflow when new code is pushed to a branch. For pull requests, the sequential @@ -248,7 +253,8 @@ describes how they are implemented. To categorize different types of NAT, this test suite follows the terminology of RFC 4787 [\[4\]](#ref-rfc4787). This RFC outlines various NAT behaviours, of which the following are implemented in the test -suite: mapping behaviours, filtering behaviours and hairpinning. +suite: mapping behaviours, filtering behaviours, hairpinning and IP +address pooling. #### Mapping behaviours @@ -298,11 +304,6 @@ of behaviours: ![](./images/system_tests/nat_mapping.png) -Note that in this test suite, the NAT’s IP pooling behaviour is not -considered, as the routers in the simulated network setup only have one -IP address. Therefore, the only difference between mappings is their -ports. - The test suite implements the above three behaviours by using the nftables framework [\[3\]](#ref-man_nft) in the routers’ namespaces. For each of the three mapping behaviours, separate rules have to be applied @@ -310,12 +311,10 @@ in the `nat` table’s `postrouting` chain: 1. **EIM:** A rule is applied to all packets going to the public network with a source address from the private network. The target - of this rule is `masquerade`, with the `persistent` option. - `masquerade` is a form of Source NAT where the source IP is - automatically translated to the IP of the outgoing network - interface, which in this case is the router’s public IP address. - With the `persistent` option, the same mapping is reused for each - different endpoint. + of this rule is `snat`, with the `persistent` option. The `snat` + target causes the source address of the packets to be translated, + and the `persistent` option makes sure the same mapping is reused + for each different endpoint. The mappings created with this rule are also automatically used to translate the destination IP of packets going to the private @@ -332,8 +331,8 @@ in the `nat` table’s `postrouting` chain: 3. **ADPM:** For this mapping behaviour, only one rule has to be applied again. The rule is identical to that for EIM, except that - the `random` option is used with the `masquerade` target instead of - the `persistent` option. With the `random` option, a random port is + the `random` option is used with the `snat` target instead of the + `persistent` option. With the `random` option, a random port is selected for each different endpoint. The exact syntax of the rules can be found in [the script applying the @@ -448,6 +447,57 @@ NAT hairpinning rules](nat_simulation/setup_nat_filtering_hairpinning.sh), which is the same script that was used for applying filtering. +#### IP Address Pooling + +Until now, we have assumed that the routers applying NAT only have one +public IP address. However, it is also possible for a NAT to have +multiple, allowing it to choose from a pool of IP addresses when +translating the address of a host behind the NAT. + +RFC 4787 specifies two types of IP address pooling behaviours: + +1. **Paired**: the NAT maps all sessions with the same internal IP + address to the same public IP. +2. **Arbitrary**: the NAT may assign different public IP addresses to + sessions belonging to the same internal IP. + +We implement the second behaviour in the test suite, as it has more +potential to cause issues for P2P protocols. The implementation relies +on two nftables features: packet marking and maps, which act like +dictionaries. + +Each packet that flows through the `nat` table is marked in the +`prerouting` chain. The range of possible mark values is equal to the +size of the NAT’s IP pool, which can be specified via the `-n` flag of +[system_tests.sh](system_tests.sh). The mark values form the keys of the +nftables map, while the NAT’s public IP’s form the values. Therefore, +the packet mark decides how the packet’s source IP address is +translated. + +The packet mark is calculated by passing some of the packet’s +information to a hash function. A hash function always gives the same +output when given identical input, but almost never gives the same +output for different inputs. We can use these two properties to simulate +IP address pooling in the test suite. The type of packet information +given to the hash function depends on the NAT’s mapping behaviour: + +1. **EIM:** By passing only the source IP address and port of the + packet to the hash function, we ensure that the same internal + endpoint is always assigned the same public IP address. +2. **ADM:** By additionally adding the destination IP address to the + hash function’s inputs, we allow two packets from the same internal + endpoint to be assigned different public IPs, unless they have the + same destination IP. +3. **APDM:** By also including the destination port, two packets from + the same internal endpoint are only assigned the same public IP + address if their external endpoints are also identical. + +Although the source IP address is a part of the hash function input in +all three cases, there is always at least one other element present. +Therefore, it is possible that the NAT assigns different public IP +addresses to sessions belonging to the same source IP, which means the +pooling behaviour is Arbitrary. + ## Performance Tests The test suite contains performance tests to measure the bitrate, jitter @@ -571,10 +621,19 @@ commands in the repository’s root directory: # Results -The results are split in a separate section for the system tests, +The results are split into separate sections for the system tests, performance tests, and integration tests. -## System Test Results +The system tests results are from an older version of the test suite +that did not yet implement NAT IP address pooling. In the current +version of the test suite, these results can still be reproduced by +setting the size of the NATs’ IP address pool to 1, which effectively +disables IP address pooling. + +The old results are followed by a section in which the differences +resulting from the addition of IP address pooling are described. + +## System Test Results Without IP Address Pooling Using the test suite’s system tests, we can get an overview of whether two eduP2P peers are able to establish a direct connection using UDP @@ -610,10 +669,10 @@ is behind the NAT indicated by the cell’s column header. | NAT Type | Full Cone | Restricted Cone | Port Restricted Cone | Symmetric | |:---|:---|:---|:---|:---| -| **Full Cone** | X | X | X | X | -| **Restricted Cone** | X | X | X | X | -| **Port Restricted Cone** | X | X | X | | -| **Symmetric** | X | X | | | +| **Full Cone** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | +| **Restricted Cone** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | +| **Port Restricted Cone** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | +| **Symmetric** | :white_check_mark: | :white_check_mark: | :x: | :x: | As seen in the table, UDP hole punching succeeds unless one peer is behind a Port Restricted Cone NAT or Symmetric NAT, and the other peer @@ -724,10 +783,10 @@ are shown in the table below: | NAT Type | Full Cone | Restricted Cone | Port Restricted Cone | Symmetric | |:---|:---|:---|:---|:---| -| **Full Cone** | X | X | X | X | -| **Restricted Cone** | X | X | X | | -| **Port Restricted Cone** | X | X | X | | -| **Symmetric** | X | | | | +| **Full Cone** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | +| **Restricted Cone** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | +| **Port Restricted Cone** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | +| **Symmetric** | :white_check_mark: | :x: | :x: | :x: | Comparing this table with the one in the previous section, we see that eduP2P is not able to establish a direct connection when one peer is @@ -1022,15 +1081,15 @@ ADF, ADPF) behaviours are shown in the table below: | NAT Type | EIM-EIF | EIM-ADF | EIM-ADPF | ADM-EIF | ADM-ADF | ADM-ADPF | ADPM-EIF | ADPM-ADF | ADPM-ADPF | |:---|:---|:---|:---|:---|:---|:---|:---|:---|:---| -| **EIM-EIF** | X | X | X | X | X | X | X | X | X | -| **EIM-ADF** | X | X | X | X | X | X | X | X | X | -| **EIM-ADPF** | X | X | X | X | | | X | | | -| **ADM-EIF** | X | X | X | X | X | X | X | X | X | -| **ADM-ADF** | X | X | | X | | | X | | | -| **ADM-ADPF** | X | X | | X | | | X | | | -| **ADPM-EIF** | X | X | X | X | X | X | X | X | X | -| **ADPM-ADF** | X | X | | X | | | X | | | -| **ADPM-ADPF** | X | X | | X | | | X | | | +| **EIM-EIF** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | +| **EIM-ADF** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | +| **EIM-ADPF** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | +| **ADM-EIF** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | +| **ADM-ADF** | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | +| **ADM-ADPF** | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | +| **ADPM-EIF** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | +| **ADPM-ADF** | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | +| **ADPM-ADPF** | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | Based on these results, we can conclude that there are three (overlapping) types of NAT scenarios where the UDP hole punching process @@ -1167,6 +1226,65 @@ table where a direct connection could not be established: NAT to let through later pings sent to the STUN endpoint of the peer behind this NAT. +## Effect of IP Address Pooling on System Test Results + +We repeat both the experiment with the RFC 3489 NATs and RFC 4787 NAT +mapping & filtering behaviours. + +### Experiment with RFC 3489 NAT types + +| NAT Type | Full Cone | Restricted Cone | Port Restricted Cone | Symmetric | +|:---|:---|:---|:---|:---| +| **Full Cone** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | +| **Restricted Cone** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :question: | +| **Port Restricted Cone** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | +| **Symmetric** | :white_check_mark: | :question: | :x: | :x: | + +As seen in the table above, the only NAT combination for which the +result has changed is the Restricted Cone paired with the Symmetric NAT. +Without IP address pooling, UDP hole punching succeeded for this +combination. However, the outcome of hole punching is uncertain when +these NATs support IP pooling, as indicated by the question mark. + +Let Peer 1 be the eduP2P client behind the Restricted Cone NAT, and Peer +2 the client behind the Symmetric NAT. The outcome of the hole punching +process depends on whether the pings from Peer 1 to Peer 2 have their +source IP translated to Peer 1’s STUN IP, and vice versa. If this is the +case for both peers, the pings from Peer 1 to Peer 2 cause a session to +be created on the Restricted Cone NAT, where the destination IP is Peer +2’s STUN IP and Peer 1’s IP is translated to its own STUN IP. Then, the +pings from Peer 2 to Peer 1 will be let through by the Restricted Cone +NAT, as their source IP matches this existing session’s destination IP. +Therefore, UDP hole punching would succeed. + +However, if either of the peers have their source IP translated to an +address different from the STUN IP, the Restricted Cone NAT would filter +the pings from Peer 2 to Peer 1, because they do not originate from the +destination IP of an existing session. + +### Experiment with RFC 4787 NAT mapping & filtering behaviours + +| NAT Type | EIM-EIF | EIM-ADF | EIM-ADPF | ADM-EIF | ADM-ADF | ADM-ADPF | ADPM-EIF | ADPM-ADF | ADPM-ADPF | +|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---| +| **EIM-EIF** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | +| **EIM-ADF** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :question: | :question: | :white_check_mark: | :question: | :question: | +| **EIM-ADPF** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | +| **ADM-EIF** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | +| **ADM-ADF** | :white_check_mark: | :question: | :x: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | +| **ADM-ADPF** | :white_check_mark: | :question: | :x: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | +| **ADPM-EIF** | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | +| **ADPM-ADF** | :white_check_mark: | :question: | :x: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | +| **ADPM-ADPF** | :white_check_mark: | :question: | :x: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | + +Just like with the RFC 3489 experiment, the change in results caused by +NAT IP address pooling is fairly limited. The outcome of UDP hole +punching is only uncertain for NAT combinations where one has +Endpoint-Independent Mapping and Address-Dependent Filtering behaviour. +The UDP hole punching process for these combinations is the same as the +process described for the RFC 3489 Restricted Cone and Symmetric NAT +combination. This is because the Restricted Cone NAT corresponds to an +EIM-ADF NAT. + ## Performance Test Results The results in this section were measured on my own laptop with the @@ -1453,4 +1571,4 @@ Conservancy](https://commonsconservancy.org/). [The Commons Conservancy Logo](https://commonsconservancy.org/) The test suite features that have been made possible thanks to this -funding are described below. +funding are described in the [test suite’s changelog](CHANGELOG.md). diff --git a/test_suite/nat_simulation/setup_nat_filtering_hairpinning.sh b/test_suite/nat_simulation/setup_nat_filtering_hairpinning.sh index e1f00ff..a8079da 100755 --- a/test_suite/nat_simulation/setup_nat_filtering_hairpinning.sh +++ b/test_suite/nat_simulation/setup_nat_filtering_hairpinning.sh @@ -2,14 +2,13 @@ pub_nat_iface=$1 priv_nat_iface=$2 -pub_ip=$3 -priv_subnet=$4 -nat_filter=$5 +priv_subnet=$3 +nat_filter=$4 # Make sure all arguments have been passed, and nat_filter is between 0 and 2 -if [[ $# -ne 5 || ! ($nat_filter =~ ^[0-2]$) ]]; then +if [[ $# -ne 4 || ! ($nat_filter =~ ^[0-2]$) ]]; then echo """ -Usage: ${0} +Usage: ${0} may be one of the following numbers: 0 - Endpoint-Independent @@ -21,30 +20,33 @@ This script must be run with root permissions, and assumes the nftables postrout fi # Configure NAT filtering type with nftables -nft add chain nat prerouting { type nat hook prerouting priority -100\; } - nft add table inet filter nft add chain inet filter input { type filter hook input priority 0\; policy drop\; } nft add rule inet filter input ct state related,established counter accept # This rule is sufficient to simulate ADPF -# This pattern captures 1) the source IP, 2) the source port, 3) the destination IP, and 4) the translated source port from a conntrack event -pattern=".*src=(\S+).*sport=(\S+).*src=(\S+).*dport=(\S+)$" +# This pattern captures the following info from a conntrack event: +# 1) the source IP +# 2) the source port +# 3) the destination IP, +# 4) the translated source IP, +# 5) the translated source port. +pattern=".*src=(\S+).*sport=(\S+).*src=(\S+).*dst=(\S+).*dport=(\S+).*$" -# Hairpinning: if a mapping is created with source IP \1, source port \2 and translated source port \4: -hairpin_rule1="nat prerouting iif $priv_nat_iface ip saddr $priv_subnet ip daddr $pub_ip meta l4proto {tcp, udp} th dport \4 counter dnat to \1:\2" # All traffic from the private network destined to $pub_ip:\4 should be hairpinned pack to \1:\2 -hairpin_rule2="nat postrouting iif $priv_nat_iface ip saddr \1 ip daddr $priv_subnet meta l4proto {tcp, udp} th sport \2 counter snat to $pub_ip:\4" # For all hairpinned packets from \1:\2, the source becomes $pub_ip:\4 +# Hairpinning: if a mapping is created where source IP \1 and source port \2 are respectively translated to \4 and \5: +hairpin_rule1="nat prerouting iif $priv_nat_iface ip saddr $priv_subnet ip daddr \4 meta l4proto {tcp, udp} th dport \5 counter dnat to \1:\2" # All traffic from the private network destined to \4:\5 should be hairpinned pack to \1:\2 +hairpin_rule2="nat postrouting iif $priv_nat_iface ip saddr \1 ip daddr $priv_subnet meta l4proto {tcp, udp} th sport \2 counter snat to \4:\5" # For all hairpinned packets from \1:\2, the source becomes \4:\5 # Filtering (not necessary for ADPF because of filter rule above) case $nat_filter in 0) - # If a mapping is created with source IP \1, source port \2 and translated source port \4, all traffic destined to \4 should be DNATed to \1:\2 - filter_rule="nat prerouting iif $pub_nat_iface meta l4proto {tcp, udp} th dport \4 counter dnat to \1:\2";; + # If a mapping is created with source IP \1, source port \2 and translated source port \5, all traffic destined to \5 should be DNATed to \1:\2 + filter_rule="nat prerouting iif $pub_nat_iface meta l4proto {tcp, udp} th dport \5 counter dnat to \1:\2";; 1) - # If a mapping is created with source IP \1, source port \2, translated source IP \3 and translated source port \4, all traffic from \3 destined to \4 should be DNATed to \1:\2 - filter_rule="nat prerouting ip saddr \3 iif $pub_nat_iface meta l4proto {tcp, udp} th dport \4 counter dnat to \1:\2";; + # If a mapping is created with source IP \1, source port \2, translated source IP \3 and translated source port \5, all traffic from \3 destined to \5 should be DNATed to \1:\2 + filter_rule="nat prerouting ip saddr \3 iif $pub_nat_iface meta l4proto {tcp, udp} th dport \5 counter dnat to \1:\2";; esac -# Only monitor new source NAT connections that are created by the nftables masquerade rule +# Only monitor new source NAT connections that are created by the nftables NAT mapping rules if [[ $nat_filter -eq 2 ]]; then conntrack -En -s $priv_subnet -e NEW | sed -rn -e "s#$pattern#nft add rule $hairpin_rule1; nft add rule $hairpin_rule2#e" # No filter rule for ADPF NAT else diff --git a/test_suite/nat_simulation/setup_nat_mapping.sh b/test_suite/nat_simulation/setup_nat_mapping.sh index d533d6a..0d5dc03 100755 --- a/test_suite/nat_simulation/setup_nat_mapping.sh +++ b/test_suite/nat_simulation/setup_nat_mapping.sh @@ -1,34 +1,51 @@ #!/usr/bin/env bash nat_iface=$1 -priv_subnet=$2 -nat_map=$3 -adm_ips=$4 +pub_prefix=$2 +priv_subnet=$3 +nat_map=$4 +n_pooling_ips=$5 +adm_ips=$6 # Make sure all arguments have been passed, and nat_map and nat_filter are both integers between 0 and 2 -if [[ $# -ne 4 || ! ($nat_map =~ ^[0-2]+$ ) ]]; then +if [[ $# -ne 6 || ! ($nat_map =~ ^[0-2]+$ ) ]]; then echo """ -Usage: ${0} +Usage: ${0} and may be one of the following numbers: 0 - Endpoint-Independent 1 - Address-Dependent 2 - Address and Port-Dependent + specifies the number of IP addresses assigned to each NAT; setting this argument >1 allows NAT IP pooling to be simulated + is a string of IP addresses separated by a space that may be the destination IP of packets crossing this NAT device (also generated by setup_networks.sh), and is necessary to simulate an Address-Dependent Mapping This script must be run with root permissions""" exit 1 fi -# Configure NAT mapping type with nftables rules +# Create NAT table and chains to configure NAT mapping type nft add table ip nat +nft add chain nat prerouting { type nat hook prerouting priority -100\; } nft add chain ip nat postrouting { type nat hook postrouting priority 100\; } +# Dictionary with packet mark key and IP address value, so that we can mark packets to control how their address will be translated +nft add map nat ip_pooling { type mark : ipv4_addr\; } +for ((i=0; i<$n_pooling_ips; i++)); do + nft add element nat ip_pooling { $i : $pub_prefix.$((254-i)) } # For NAT IPs, host identifiers range from (254 - n_pooling_ips) to 254 +done + case $nat_map in 0) - nft add rule ip nat postrouting ip saddr $priv_subnet oif $nat_iface counter masquerade persistent;; - 1) + # In EIM the same host always reuses the same mapping, so we set the packet mark by hashing the source IP + UDP/TCP source port + nft add rule nat prerouting meta l4proto { tcp, udp } ct mark set jhash ip saddr . th sport mod $n_pooling_ips + + nft add rule nat postrouting ip saddr $priv_subnet oif $nat_iface counter snat to ct mark map @ip_pooling persistent;; + 1) + # In ADM the mapping is reused as long as the destination IP does not change, so we also make the hash depend on the destination IP + nft add rule nat prerouting meta l4proto { tcp, udp } ct mark set jhash ip saddr . th sport . ip daddr mod $n_pooling_ips + # Assign a block of 100 ports to each IP in adm_ips port_range_start=50000 port_range_width=100 @@ -36,9 +53,13 @@ case $nat_map in # Iterate over each IP for ip in $adm_ips; do port_range_end=$(($port_range_start + $port_range_width - 1)) - nft add rule ip nat postrouting ip protocol {tcp, udp} ip saddr $priv_subnet ip daddr $ip oif $nat_iface counter masquerade to :${port_range_start}-${port_range_end} persistent + port_range="${port_range_start}-${port_range_end}" + nft add rule ip nat postrouting ip protocol {tcp, udp} ip saddr $priv_subnet ip daddr $ip oif $nat_iface counter snat to ct mark map @ip_pooling:$port_range persistent port_range_start=$(($port_range_end+1)) done ;; 2) - nft add rule ip nat postrouting ip saddr $priv_subnet oif $nat_iface counter masquerade random;; + # In APDM each connection gets a new mapping, so the packet mark can be completely random + nft add rule nat prerouting ct mark set numgen random mod $n_pooling_ips + + nft add rule nat postrouting ip saddr $priv_subnet oif $nat_iface counter snat to ct mark map @ip_pooling random;; esac \ No newline at end of file diff --git a/test_suite/nat_simulation/setup_networks.sh b/test_suite/nat_simulation/setup_networks.sh index cdc3725..0b23240 100755 --- a/test_suite/nat_simulation/setup_networks.sh +++ b/test_suite/nat_simulation/setup_networks.sh @@ -1,17 +1,24 @@ #!/usr/bin/env bash -if [[ $1 = "-h" ]]; then - echo """ -Usage: ${0} +usage_str=""" +Usage: ${0} + + specifies the number of IP addresses assigned to each NAT; setting this argument >1 allows NAT IP pooling to be simulated Simulates a network setup containing two private networks connected via the public network. Each private network contains two peers using eduP2P To allow traffic to flow between the public and private networks, the scripts setup_nat_mapping.sh should also be executed To allow traffic to flow between peers in the same private network, the scripts setup_nat_filtering_hairpinning.sh should also be executed This script must be run with root permissions""" + +if [[ $1 = "-h" || $# -ne 1 ]]; then + echo $usage_str exit 1 fi +# Number of IPs per router to test NAT IP pooling +n_pooling_ips=$1 + # Enable IP forwarding to allow for routing between namespaces sysctl -w net.ipv4.ip_forward=1 &> /dev/null @@ -60,13 +67,14 @@ for ((i=1; i<=n_priv_nets; i++)); do priv_subnet="${priv_prefix}.0/24" router_priv_ip="${priv_prefix}.254" pub_prefix="192.168.${i}" + pub_subnet="${pub_prefix}.0/24" router_pub_ip="${pub_prefix}.254" - # Add router's public IP to list created earlier - adm_ips+=($router_pub_ip) + # Add router's public subnet to list created earlier + adm_ips+=($pub_subnet) # Setup router - ip netns exec $router_name ./setup_router.sh $router_name $priv_name $priv_subnet $router_priv_ip $router_pub_ip $switch_ip + ip netns exec $router_name ./setup_router.sh $router_name $priv_name $priv_subnet $router_priv_ip $pub_prefix $n_pooling_ips $switch_ip # Setup private network ip netns exec $priv_name ./setup_private.sh $router_name $router_pub_ip $priv_subnet diff --git a/test_suite/nat_simulation/setup_router.sh b/test_suite/nat_simulation/setup_router.sh index 0444074..19fa922 100755 --- a/test_suite/nat_simulation/setup_router.sh +++ b/test_suite/nat_simulation/setup_router.sh @@ -4,17 +4,20 @@ router_name=$1 priv_name=$2 priv_subnet=$3 priv_ip=$4 -pub_ip=$5 -switch_ip=$6 +pub_subnet_prefix=$5 +n_ips=$6 +switch_ip=$7 -if [[ $# -ne 6 ]]; then +if [[ $# -ne 7 ]]; then echo """ -Usage: ${0} +Usage: ${0} This script must be run with root permissions""" exit 1 fi +pub_subnet="$pub_subnet_prefix.0/24" + # Create veth pair to place the router's private interface in the private and router namespaces router_priv="${router_name}_priv" ip link add $router_priv type veth peer $router_name netns $priv_name @@ -25,7 +28,12 @@ ip netns exec $priv_name ip link set $router_name up # Create veth pair to place the router's public interface in the public and router namespaces router_pub="${router_name}_pub" ip link add $router_pub type veth peer $router_name netns public -ip addr add "${pub_ip}/24" dev $router_pub + +for host in $(seq $((254 - $n_ips + 1)) 254); do + ip="$pub_subnet_prefix.$host/24" + ip addr add $ip dev $router_pub +done + ip link set $router_pub up ip netns exec public ip link set $router_name up @@ -38,4 +46,4 @@ ip route add $priv_ip dev $router_priv ip route add $priv_subnet via $priv_ip dev $router_priv # Create route to router in the public network -ip netns exec public ip route add $pub_ip dev $router_name \ No newline at end of file +ip netns exec public ip route add $pub_subnet dev $router_name \ No newline at end of file diff --git a/test_suite/system_test.sh b/test_suite/system_test.sh index 43d0a82..48c8eff 100755 --- a/test_suite/system_test.sh +++ b/test_suite/system_test.sh @@ -4,7 +4,7 @@ SYSTEM_TEST_TIMEOUT=60 usage_str=""" -Usage: ${0} [OPTIONAL ARGUMENTS] [NAT CONFIGURATION 1]:[NAT CONFIGURATION 2] [WIREGUARD INTERFACE 1]:[WIREGUARD INTERFACE 2] +Usage: ${0} [OPTIONAL ARGUMENTS] [NAT CONFIGURATION 1]:[NAT CONFIGURATION 2] [WIREGUARD INTERFACE 1]:[WIREGUARD INTERFACE 2] is the expected result of the system test: 1. TS_PASS_DIRECT: the peers have established a direct connection @@ -33,6 +33,8 @@ Usage: ${0} [OPTIONAL ARGUMENTS] [NAT CO 2 - Address and Port-Dependent Examples of valid NAT configurations: 0-1:1-2 (both peers in private networks), 0-1: (peer 2 in public network), : (both peers in public network) + specifies the number of IP addresses assigned to each NAT; setting this argument >1 allows NAT IP pooling to be simulated + If [WIREGUARD INTERFACE 1] or [WIREGUARD INTERFACE 2] is not provided, the corresponding peer will use userspace WireGuard is a string of IP addresses separated by a space that may be the destination IP of packets crossing this NAT device, and is necessary to simulate an Address-Dependent Mapping @@ -95,8 +97,8 @@ done shift $((OPTIND-1)) # Make sure all required arguments have been passed -if [[ $# -ne 12 ]]; then - exit_with_error "expected 12 positional parameters, but received $#" +if [[ $# -ne 13 ]]; then + exit_with_error "expected 13 positional parameters, but received $#" fi test_target=$1 @@ -107,10 +109,11 @@ test_idx=$5 control_pub_key=$6 control_ip=$7 control_port=$8 -adm_ips=$9 -log_lvl=${10} -log_dir=${11} -repo_dir=${12} +n_pooling_ips=$9 +adm_ips=${10} +log_lvl=${11} +log_dir=${12} +repo_dir=${13} # Validate namespace configuration string ns_regex="([^-:]+)" # One or more occurence of every character except '-' and ':' (these are used to separate the namespaces) @@ -255,12 +258,12 @@ for ((i=0; i<${#router_ns_list[@]}; i++)); do router_ns=${router_ns_list[$i]} router_pub="${router_ns}_pub" router_priv="${router_ns}_priv" - router_pub_ip="192.168.$((i+1)).254" - priv_prefix="10.0.$((i+1)).0/24" + router_pub_prefix="192.168.$((i+1))" + priv_subnet="10.0.$((i+1)).0/24" - sudo ip netns exec $router_ns ./setup_nat_mapping.sh $router_pub $priv_prefix ${nat_map[$i]} "${adm_ips}" + sudo ip netns exec $router_ns ./setup_nat_mapping.sh $router_pub $router_pub_prefix $priv_subnet ${nat_map[$i]} $n_pooling_ips "${adm_ips[@]}" - sudo ip netns exec $router_ns ./setup_nat_filtering_hairpinning.sh $router_pub $router_priv $router_pub_ip $priv_prefix ${nat_filter[$i]} 2>&1 | \ + sudo ip netns exec $router_ns ./setup_nat_filtering_hairpinning.sh $router_pub $router_priv $priv_subnet ${nat_filter[$i]} 2>&1 | \ tee ${log_dir}/$router_ns.txt > /dev/null & # Combination of tee and redirect to /dev/null is necessary to avoid weird behaviour caused by redirecting a script run with sudo done @@ -302,7 +305,7 @@ else fi # Output test result -if [[ $test_target != $test_result ]]; then +if [[ ! ( $test_result =~ $test_target ) ]]; then echo -e "${RED}$test_result${NC}" clean_exit 1 fi diff --git a/test_suite/system_tests.sh b/test_suite/system_tests.sh index 719fc18..cd2bfec 100755 --- a/test_suite/system_tests.sh +++ b/test_suite/system_tests.sh @@ -28,6 +28,8 @@ The following options can be used to configure additional parameters during the -L Specifies the alphanumeric name of the directory inside system_test_logs/ where the test logs will be stored If this argument is not provided, the directory name is the current timestamp + -n + Specifies the number of IP addresses assigned to each NAT. Passing a number >1 allows NAT IP pooling to be simulated during the system tests -t Run the system tests in parallel with the specified number of threads. It is not recommended to combine this flag with -p, as multithreading will likely degrade the performance and the graphs will not be created automatically @@ -37,11 +39,12 @@ The following options can be used to configure additional parameters during the # Use functions and constants from util.sh . ./util.sh -# Default log level +# Default parameter values log_lvl="debug" +n_pooling_ips=3 # Validate optional arguments -while getopts ":c:d:ef:l:L:t:bph" opt; do +while getopts ":c:d:ef:l:L:n:t:bph" opt; do case $opt in c) connectivity=true @@ -97,6 +100,13 @@ while getopts ":c:d:ef:l:L:t:bph" opt; do b) build=true ;; + n) + n_pooling_ips=$OPTARG + + # Make sure n_pooling_ips is an integer between 1 and 9 + n_pooling_ips_regex="^[1-9]$" + validate_str $n_pooling_ips $n_pooling_ips_regex + ;; p) performance=true ;; @@ -145,7 +155,7 @@ function build_go() { function setup_networks() { cd nat_simulation/ - adm_ips=$(sudo ./setup_networks.sh) # setup_networks.sh returns an array of IPs used by hosts in the network simulation setup, this list is needed to simulate a NAT device with an Address-Dependent Mapping + adm_ips=$(sudo ./setup_networks.sh $n_pooling_ips) # setup_networks.sh returns an array of IPs used by hosts in the network simulation setup, this list is needed to simulate a NAT device with an Address-Dependent Mapping } function extract_server_pub_key() { @@ -325,7 +335,7 @@ function run_system_test() { let "n_tests++" # Run in background and wait for test to finish to allow for interrupting from the terminal - ./system_test.sh $@ $n_tests $control_pub_key $control_ip $control_port "$adm_ips" $log_lvl $log_dir $repo_dir & + ./system_test.sh $@ $n_tests $control_pub_key $control_ip $control_port $n_pooling_ips "$adm_ips" $log_lvl $log_dir $repo_dir & test_pid=$! wait $test_pid @@ -352,9 +362,16 @@ function connectivity_test_logic() { # After sending one ping, the subsequent incoming pings from the peer's STUN endpoint will be accepted, regardless of the filtering behaviour test_target="TS_PASS_DIRECT" elif [[ $nat1_mapping -eq 0 && $nat1_filter -eq 1 || $nat2_mapping -eq 0 && $nat2_filter -eq 1 ]]; then - # An EIF-ADF NAT will always let the peer's pings through after sending its first ping - # This is not a general property of EIM-ADF NATs, but holds in this test suite because each NAT only has one IP address - test_target="TS_PASS_DIRECT" + # If NAT IP pooling is disabled, the endpoints used by the peers to communicate with each other have the same IP as their STUN endpoints + # Therefore, ADF NATs behave the same as EIF NATs in this case + if [[ $n_pooling_ips -eq 1 ]]; then + test_target="TS_PASS_DIRECT" + # If NAT IP pooling is enabled, the endpoints may have different IPs from the STUN endpoints + # If this is the case for both peers, the ADF NATs will not let the pings through + # The result TS_PASS indicates that it is not certain whether a direct connection can be established, and the test will succeed for both TS_PASS_DIRECT and TS_PASS_RELAY + else + test_target="TS_PASS" + fi else test_target="TS_PASS_RELAY" fi @@ -500,7 +517,7 @@ if [[ -n $n_threads ]]; then # Containers are only used one time, now that they have finished running they can be removed docker rm ${container_ids[@]} > /dev/null -else +elif [[ -n $performance ]]; then # Create graphs for performance tests, if any were included python3 visualize_performance_tests.py $log_dir fi diff --git a/test_suite/test_client/setup_client.sh b/test_suite/test_client/setup_client.sh index 9238e23..f37e3b9 100755 --- a/test_suite/test_client/setup_client.sh +++ b/test_suite/test_client/setup_client.sh @@ -176,8 +176,8 @@ function try_connect() { try_connect "http://${peer_ipv4}" -# Peers try to establish a direct connection after initial connection; if expecting a direct connection, give them some time to establish one -if [[ $test_target == "TS_PASS_DIRECT" ]]; then +# Peers try to establish a direct connection after initial connection; if expecting a (potential) direct connection, give them some time to establish one +if [[ $test_target == "TS_PASS" || $test_target == "TS_PASS_DIRECT" ]]; then timeout 10s tail -f -n +1 -s 0.1 $out | sed -n "/ESTABLISHED direct peer connection/q" fi