diff --git a/README.md b/README.md index 8f2546e..e59f0a9 100644 --- a/README.md +++ b/README.md @@ -90,6 +90,171 @@ The `import` command also supports input in the [rosinstall file format](http:// Only for this command vcs2l supports the pseudo clients `tar` and `zip` which fetch a tarball / zipfile from a URL and unpack its content. For those two types the `version` key is optional. If specified only entries from the archive which are in the subfolder specified by the version value are being extracted. +### Import with extends functionality + +The `vcs import` command supports an `extends` key at the top level of the YAML file. The value of that key is a path or URL to another YAML file which is imported first. +This base file can itself also contain the key to chain multiple files. The extension to this base file is given precedence over the parent in case of duplicate repository entries. + +#### Normal Extension + +For instance, consider the following two files: + +- **`base.repos`**: contains three repositories `vcs2l`, `immutable/hash` and `immutable/tag`, checked out at specific versions. + + ```yaml + --- + repositories: + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: main + immutable/hash: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 377d5b3d03c212f015cc832fdb368f4534d0d583 + immutable/tag: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.3 + ``` + +- **`base_extension.repos`**: extends the base file and overrides the version of `immutable/hash` and `immutable/tag` repositories. + + ```yaml + --- + extends: base.repos + repositories: + immutable/hash: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 25e4ae2f1dd28b0efcd656f4b1c9679d8a7d6c22 + immutable/tag: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.5 + ``` +The resulting extension import would import vcs2l at version `main`, `immutable/hash` at version `25e4ae2` and `immutable/tag` at version `1.1.5`. + +#### Multiple Extensions + +The `extends` key also supports a list of files to extend from. The files are imported in the order they are specified and the precedence is given to the last file in case of duplicate repository entries. + +For instance, consider the following three files: + +- **`base_1.repos`**: contains two repositories `vcs2l` and `immutable/hash`, checked out at `1.1.3`. + + ```yaml + --- + repositories: + immutable/hash: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: e700793cb2b8d25ce83a611561bd167293fd66eb # 1.1.3 + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.3 + ``` + +- **`base_2.repos`**: contains two repositories `vcs2l` and `immutable/hash`, checked out at `1.1.4`. + + ```yaml + --- + repositories: + immutable/hash: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 2c7ff89d12d8a77c36b60d1f7ba3039cdd3f742b # 1.1.4 + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.4 + ``` + +- **`multiple_extension.repos`**: extends both base files and overrides the version of `vcs2l` repository. + + ```yaml + --- + extends: + - base_1.repos # Lower priority + - base_2.repos # Higher priority + repositories: + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.5 + ``` + +The resulting extension import would import `immutable/hash` at version `1.1.4` (from `base_2.repos`) and `vcs2l` at version `1.1.5`. + +Duplicate file names in the `extends` list are not allowed and would raise the following error: + +```bash +Duplicate entries found in extends in file: /multiple_extension.repos +``` + +#### Circular Loop Protection + +In order to avoid infinite loops in case of circular imports the tool detects already imported files and raises an error if such a file is encountered again. + +For instance, consider the following two files: + +- **`loop_base.repos`**: extends the `loop_extension.repos` file, and contains two repositories `vcs2l` and `immutable/tag`. + + ```yaml + --- + extends: loop_extension.repos + repositories: + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: main + immutable/tag: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.3 + ``` + +- **`loop_extension.repos`**: extends the `loop_base.repos` file, and modifies the version of `immutable/tag` with `1.1.5`. + + ```yaml + --- + extends: loop_base.repos + repositories: + immutable/tag: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.5 + ``` +The resulting extension import would prevent the download and raise the following error: + +```bash +Circular import detected: /loop_extension.repos +``` + +#### File path behaviour + +Currently there are two ways to specify the path to the repository file passed to `vcs import`: + +1. **Recommended**: Using `--input`. + + * For instance: `vcs import --input my.repos ` + + * The extended files are searched relative to the file containing the `extends` key. + + * You do not require to be in the same directory as `my.repos` to run the command. + +2. Using the input redirection operator `<` to pass a local file path via `stdin`. + + * For instance: `vcs import < my.repos ` + + * The extended files are searched relative to the current working directory. + + * Therefore, you have to be in the **same** directory as `my.repos` to run the command. + + The files being directly extended by the file provided through `stdin` are relative to the current working directory. + Any other file being extended is relative to the file extending it. + ### Delete set of repositories The `vcs delete` command removes all directories of repositories which are passed in via `stdin` in YAML format. diff --git a/test/import_extends.txt b/test/import_extends.txt new file mode 100644 index 0000000..cd9f1f7 --- /dev/null +++ b/test/import_extends.txt @@ -0,0 +1,67 @@ +...... +=== ./immutable/hash (git) === +Cloning into '.'... +Note: switching to 'e700793cb2b8d25ce83a611561bd167293fd66eb'. + +You are in 'detached HEAD' state. You can look around, make experimental +changes and commit them, and you can discard any commits you make in this +state without impacting any branches by switching back to a branch. + +If you want to create a new branch to retain commits you create, you may +do so (now or later) by using -c with the switch command. Example: + + git switch -c + +Or undo this operation with: + + git switch - + +Turn off this advice by setting config variable advice.detachedHead to false + +HEAD is now at e700793 1.1.3 +=== ./immutable/hash_tar (tar) === +Downloaded tarball from 'https://github.com/ros-infrastructure/vcs2l/archive/377d5b3d03c212f015cc832fdb368f4534d0d583.tar.gz' and unpacked it +=== ./immutable/hash_zip (zip) === +Downloaded zipfile from 'https://github.com/ros-infrastructure/vcs2l/archive/377d5b3d03c212f015cc832fdb368f4534d0d583.zip' and unpacked it +=== ./immutable/tag (git) === +Cloning into '.'... +Note: switching to 'tags/1.1.3'. + +You are in 'detached HEAD' state. You can look around, make experimental +changes and commit them, and you can discard any commits you make in this +state without impacting any branches by switching back to a branch. + +If you want to create a new branch to retain commits you create, you may +do so (now or later) by using -c with the switch command. Example: + + git switch -c + +Or undo this operation with: + + git switch - + +Turn off this advice by setting config variable advice.detachedHead to false + +HEAD is now at e700793 1.1.3 +=== ./vcs2l (git) === +Cloning into '.'... +Note: switching to '1.1.3'. + +You are in 'detached HEAD' state. You can look around, make experimental +changes and commit them, and you can discard any commits you make in this +state without impacting any branches by switching back to a branch. + +If you want to create a new branch to retain commits you create, you may +do so (now or later) by using -c with the switch command. Example: + + git switch -c + +Or undo this operation with: + + git switch - + +Turn off this advice by setting config variable advice.detachedHead to false + +HEAD is now at e700793 1.1.3 +=== ./without_version (git) === +Cloning into '.'... diff --git a/test/import_multiple_extends.txt b/test/import_multiple_extends.txt new file mode 100644 index 0000000..8dbc6df --- /dev/null +++ b/test/import_multiple_extends.txt @@ -0,0 +1,49 @@ +...... +=== ./immutable/hash (git) === +Cloning into '.'... +Note: switching to '2c7ff89d12d8a77c36b60d1f7ba3039cdd3f742b'. + +You are in 'detached HEAD' state. You can look around, make experimental +changes and commit them, and you can discard any commits you make in this +state without impacting any branches by switching back to a branch. + +If you want to create a new branch to retain commits you create, you may +do so (now or later) by using -c with the switch command. Example: + + git switch -c + +Or undo this operation with: + + git switch - + +Turn off this advice by setting config variable advice.detachedHead to false + +HEAD is now at 2c7ff89 1.1.4 +=== ./immutable/hash_tar (tar) === +Downloaded tarball from 'https://github.com/ros-infrastructure/vcs2l/archive/377d5b3d03c212f015cc832fdb368f4534d0d583.tar.gz' and unpacked it +=== ./immutable/hash_zip (zip) === +Downloaded zipfile from 'https://github.com/ros-infrastructure/vcs2l/archive/377d5b3d03c212f015cc832fdb368f4534d0d583.zip' and unpacked it +=== ./immutable/tag (git) === +Cloning into '.'... +Note: switching to 'tags/1.1.5'. + +You are in 'detached HEAD' state. You can look around, make experimental +changes and commit them, and you can discard any commits you make in this +state without impacting any branches by switching back to a branch. + +If you want to create a new branch to retain commits you create, you may +do so (now or later) by using -c with the switch command. Example: + + git switch -c + +Or undo this operation with: + + git switch - + +Turn off this advice by setting config variable advice.detachedHead to false + +HEAD is now at 25e4ae2 1.1.5 +=== ./vcs2l (git) === +Cloning into '.'... +=== ./without_version (git) === +Cloning into '.'... diff --git a/test/list_extension.repos b/test/list_extension.repos new file mode 100644 index 0000000..87d760a --- /dev/null +++ b/test/list_extension.repos @@ -0,0 +1,16 @@ +# Check for hash, tag, and branch imports with extends functionality with predominant tag 1.1.3 +--- +extends: list.repos +repositories: + immutable/hash: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: e700793cb2b8d25ce83a611561bd167293fd66eb # 1.1.3 + immutable/tag: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: tags/1.1.3 + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.3 diff --git a/test/list_extension_2.repos b/test/list_extension_2.repos new file mode 100644 index 0000000..053e1a2 --- /dev/null +++ b/test/list_extension_2.repos @@ -0,0 +1,16 @@ +# Check for hash, tag, and branch imports with extends functionality with predominant tag 1.1.4 +--- +extends: list.repos +repositories: + immutable/hash: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 2c7ff89d12d8a77c36b60d1f7ba3039cdd3f742b # 1.1.4 + immutable/tag: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: tags/1.1.4 + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.4 diff --git a/test/list_multiple_extension.repos b/test/list_multiple_extension.repos new file mode 100644 index 0000000..3f6282a --- /dev/null +++ b/test/list_multiple_extension.repos @@ -0,0 +1,14 @@ +# Check for multiple levels of extension in repositories files. +--- +extends: + - list_extension.repos # predominant tag is 1.1.3 # Lower priority + - list_extension_2.repos # predominant tag is 1.1.4 # Higher priority +repositories: + immutable/tag: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: tags/1.1.5 + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: main diff --git a/test/loop_base.repos b/test/loop_base.repos new file mode 100644 index 0000000..d2c50ec --- /dev/null +++ b/test/loop_base.repos @@ -0,0 +1,12 @@ +# Base repositories file that creates an extends a circular import on its extension implementation. +--- +extends: loop_extension.repos +repositories: + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: main + immutable/tag: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: tags/1.1.3 diff --git a/test/loop_extension.repos b/test/loop_extension.repos new file mode 100644 index 0000000..7debf28 --- /dev/null +++ b/test/loop_extension.repos @@ -0,0 +1,8 @@ +# Repositories extension file that creates an extends a circular import on its base. +--- +extends: loop_base.repos +repositories: + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.3 diff --git a/test/test_commands.py b/test/test_commands.py index 39c75a6..32338d5 100644 --- a/test/test_commands.py +++ b/test/test_commands.py @@ -14,6 +14,13 @@ REPOS_FILE = os.path.join(os.path.dirname(__file__), 'list.repos') REPOS_FILE_URL = file_uri_scheme + REPOS_FILE REPOS2_FILE = os.path.join(os.path.dirname(__file__), 'list2.repos') +REPOS_EXTENDS_FILE = os.path.join(os.path.dirname(__file__), 'list_extension.repos') +REPOS_MULTIPLE_EXTENDS_FILE = os.path.join( + os.path.dirname(__file__), 'list_multiple_extension.repos' +) +REPOS_EXTENDS_LOOP_FILE = os.path.join( + os.path.dirname(__file__), 'loop_extension.repos' +) BAD_REPOS_FILE = os.path.join(os.path.dirname(__file__), 'bad.repos') TEST_WORKSPACE = os.path.join( os.path.dirname(os.path.dirname(__file__)), 'test_workspace' @@ -277,18 +284,24 @@ def test_import_shallow(self): finally: rmtree(workdir) - def test_import_url(self): - workdir = os.path.join(TEST_WORKSPACE, 'import-url') + def import_common(self, import_file, repos_file): + """Common test function for import operations + + Args: + import_file: Assertion expected output file name (without .txt) + repos_file: path to the .repos file to use + """ + workdir = os.path.join(TEST_WORKSPACE, import_file) os.makedirs(workdir) try: output = run_command( - 'import', ['--input', REPOS_FILE_URL, '.'], subfolder='import-url' + 'import', ['--input', repos_file, '.'], subfolder=import_file ) # the actual output contains absolute paths output = output.replace( b'repository in ' + workdir.encode() + b'/', b'repository in ./' ) - expected = get_expected_output('import') + expected = get_expected_output(import_file) # newer git versions don't append ... after the commit hash assert output == expected or output == expected.replace(b'... ', b' ') finally: @@ -327,6 +340,24 @@ def test_deletion(self): finally: rmtree(workdir) + def test_import_file_url(self): + """Test import from file URL.""" + self.import_common('import', REPOS_FILE_URL) + + def test_import_extends(self): + """Test import with extends functionality.""" + self.import_common('import_extends', REPOS_EXTENDS_FILE) + + def test_import_extends_loop(self): + """Test import with extends functionality that creates a circular import.""" + with self.assertRaises(subprocess.CalledProcessError) as e: + run_command('import', ['--input', REPOS_EXTENDS_LOOP_FILE, '.']) + self.assertIn(b'Circular import detected:', e.exception.output) + + def test_import_multiple_extends(self): + """Test import with multiple extends functionality.""" + self.import_common('import_multiple_extends', REPOS_MULTIPLE_EXTENDS_FILE) + def test_validate(self): output = run_command('validate', ['--input', REPOS_FILE]) expected = get_expected_output('validate') diff --git a/vcs2l/commands/import_.py b/vcs2l/commands/import_.py index b1f0c8b..5b4ae0b 100644 --- a/vcs2l/commands/import_.py +++ b/vcs2l/commands/import_.py @@ -10,6 +10,7 @@ from vcs2l.clients import vcs2l_clients from vcs2l.clients.vcs_base import run_command from vcs2l.commands.command import Command, add_common_arguments +from vcs2l.errors import CircularImportError from vcs2l.executor import ansi, execute_jobs, output_repositories, output_results from vcs2l.streams import set_streams @@ -86,23 +87,94 @@ def file_or_url_type(value): return request.Request(value, headers={'User-Agent': 'vcs2l/' + vcs2l_version}) -def get_repositories(yaml_file): +def load_yaml_file(yaml_file): + """Load and parse a YAML file.""" try: - root = yaml.safe_load(yaml_file) + return yaml.safe_load(yaml_file) except yaml.YAMLError as e: - raise RuntimeError('Input data is not valid yaml format: %s' % e) + raise RuntimeError('Input data is not valid yaml format: %s' % e) from e + +def get_repositories_from_root(root): + """Extract repositories from the parsed YAML root object.""" try: repositories = root['repositories'] return get_repos_in_vcs2l_format(repositories) except KeyError as e: - raise RuntimeError('Input data is not valid format: %s' % e) + raise RuntimeError('Input data is not valid format: %s' % e) from e except TypeError as e: # try rosinstall file format try: return get_repos_in_rosinstall_format(root) except Exception: - raise RuntimeError('Input data is not valid format: %s' % e) + raise RuntimeError('Input data is not valid format: %s' % e) from e + + +def get_repositories(yaml_file, visited_files=None): + """Recursively get repositories from a YAML file, handling inheritance.""" + if visited_files is None: + visited_files = set() + + # Get absolute path to handle relative paths consistently + current_file_path = os.path.abspath(yaml_file.name) + + if current_file_path in visited_files: + raise CircularImportError(f'Circular import detected: {current_file_path}') + + visited_files.add(current_file_path) + + try: + root = load_yaml_file(yaml_file) + + combined_repos = {} + + if 'extends' in root: + parent_files = root['extends'] + # Convert single file to list for consistent processing + if isinstance(parent_files, str): + parent_files = [parent_files] + + # Check for duplicate entries in extends + if len(parent_files) != len(set(parent_files)): + raise RuntimeError( + f'Duplicate entries found in extends in file: {current_file_path}' + ) + + for parent_file in parent_files: + # If absolute path is not valid, try relative to current file + if not os.path.isabs(parent_file): + current_dir = os.path.dirname(current_file_path) + parent_file = os.path.join(current_dir, parent_file) + + if not os.path.exists(parent_file): + raise RuntimeError(f'Parent file not found: {parent_file}') + + try: + # Recursively get repositories from parent file + with open(parent_file, 'r', encoding='utf-8') as parent_f: + parent_repos = get_repositories(parent_f, visited_files.copy()) + combined_repos.update(parent_repos) + + except CircularImportError: + raise + + except Exception as e: + raise RuntimeError( + f'Error reading parent file {parent_file}: \n{str(e)}' + ) from e + + current_repos = get_repositories_from_root(root) + combined_repos.update(current_repos) + + return combined_repos + + except FileNotFoundError as e: + raise RuntimeError(f'File not found: {yaml_file}') from e + except yaml.YAMLError as e: + raise RuntimeError(f'Error parsing YAML file {yaml_file}: {str(e)}') from e + finally: + # Remove current file from visited set when leaving this call + visited_files.discard(current_file_path) def get_repos_in_vcs2l_format(repositories): diff --git a/vcs2l/errors.py b/vcs2l/errors.py index fc7ba30..fe88a1b 100644 --- a/vcs2l/errors.py +++ b/vcs2l/errors.py @@ -21,3 +21,10 @@ def __init__(self, min_version: str = '3.6'): f'vcs2l requires Python {min_version} or higher.' ) super().__init__(message) + + +class CircularImportError(Vcs2lError): + """Raised when a circular import is detected.""" + + def __init__(self, message: str = 'Circular import detected.'): + super().__init__(message)