Skip to content

[Nexthop] Distro CLI: Artifact store #795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
raghav-nexthop wants to merge 14 commits into from
Closed

[Nexthop] Distro CLI: Artifact store #795

raghav-nexthop wants to merge 14 commits into from
+12,961 −0

Conversation

@raghav-nexthop
Copy link

@raghav-nexthop raghav-nexthop commented Jan 9, 2026 •
edited
Loading

Pre-submission checklist

  • I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • pre-commit run

Summary

This PR introduces artifact storage and caching system to avoid redundant downloads and builds.

  • ArtifactStore class for managing build artifacts with file caching
  • Support for metadata and data file separation in storage
  • Cache evaluation with caller-provided functions for flexible caching strategies

This is part of a series. The previous one is #794 The next one is #796

Test Plan

Added unit tests

PYTHONPATH=fboss-image:$PYTHONPATH python3 -m pytest fboss-image/distro_cli/tests/ -v
PYTHONPATH=fboss-image:$PYTHONPATH python3 -m pytest fboss-image/distro_cli/tests/ -v
===================================================== test session starts ======================================================
platform linux -- Python 3.10.12, pytest-8.4.1, pluggy-1.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /work/raghav/private-fboss
collected 45 items                                                                                                             

fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_data_metadata_separation PASSED                   [  2%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_fetch_fn_receives_stored_files PASSED             [  4%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_fetch_fn_store_miss_updates_store PASSED          [  6%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_get_store_hit PASSED                              [  8%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_get_store_miss PASSED                             [ 11%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_multiple_data_files PASSED                        [ 13%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_store_data_and_metadata PASSED                    [ 15%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_store_data_only PASSED                            [ 17%]
fboss-image/distro_cli/tests/build_test.py::TestBuildCommand::test_build_all_stub PASSED                                 [ 20%]
fboss-image/distro_cli/tests/build_test.py::TestBuildCommand::test_build_command_exists PASSED                           [ 22%]
fboss-image/distro_cli/tests/build_test.py::TestBuildCommand::test_build_specific_components_stub PASSED                 [ 24%]
fboss-image/distro_cli/tests/cli_test.py::ValidatePathTest::test_validate_path_converts_to_path PASSED                   [ 26%]
fboss-image/distro_cli/tests/cli_test.py::ValidatePathTest::test_validate_path_existing_file PASSED                      [ 28%]
fboss-image/distro_cli/tests/cli_test.py::ValidatePathTest::test_validate_path_nonexistent_file_raises PASSED            [ 31%]
fboss-image/distro_cli/tests/cli_test.py::ValidatePathTest::test_validate_path_nonexistent_file_without_check PASSED     [ 33%]
fboss-image/distro_cli/tests/cli_test.py::CommandGroupTest::test_add_command_to_group PASSED                             [ 35%]
fboss-image/distro_cli/tests/cli_test.py::CommandGroupTest::test_add_command_with_arguments PASSED                       [ 37%]
fboss-image/distro_cli/tests/cli_test.py::CommandGroupTest::test_command_group_creation PASSED                           [ 40%]
fboss-image/distro_cli/tests/cli_test.py::CommandGroupTest::test_command_group_with_arguments PASSED                     [ 42%]
fboss-image/distro_cli/tests/cli_test.py::CLITest::test_add_command PASSED                                               [ 44%]
fboss-image/distro_cli/tests/cli_test.py::CLITest::test_add_command_group PASSED                                         [ 46%]
fboss-image/distro_cli/tests/cli_test.py::CLITest::test_add_command_with_arguments PASSED                                [ 48%]
fboss-image/distro_cli/tests/cli_test.py::CLITest::test_cli_creation PASSED                                              [ 51%]
fboss-image/distro_cli/tests/cli_test.py::CLITest::test_cli_with_verbose_flag PASSED                                     [ 53%]
fboss-image/distro_cli/tests/cli_test.py::CLITest::test_cli_without_verbose_flag PASSED                                  [ 55%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_device_commands_exist PASSED                       [ 57%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_getip_stub PASSED                                  [ 60%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_image_stub PASSED                                  [ 62%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_image_upstream_stub PASSED                         [ 64%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_reprovision_stub PASSED                            [ 66%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_ssh_stub PASSED                                    [ 68%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_update_stub PASSED                                 [ 71%]
fboss-image/distro_cli/tests/docker_image_test.py::TestShouldBuildImage::test_custom_expiration_time PASSED              [ 73%]
fboss-image/distro_cli/tests/docker_image_test.py::TestShouldBuildImage::test_image_exists_and_not_expired PASSED        [ 75%]
fboss-image/distro_cli/tests/docker_image_test.py::TestShouldBuildImage::test_image_expired PASSED                       [ 77%]
fboss-image/distro_cli/tests/docker_image_test.py::TestShouldBuildImage::test_image_not_found PASSED                     [ 80%]
fboss-image/distro_cli/tests/docker_test.py::TestDockerInfrastructure::test_run_simple_container PASSED                  [ 82%]
fboss-image/distro_cli/tests/image_builder_test.py::TestImageBuilder::test_build_all_stub PASSED                         [ 84%]
fboss-image/distro_cli/tests/image_builder_test.py::TestImageBuilder::test_build_components_stub PASSED                  [ 86%]
fboss-image/distro_cli/tests/image_builder_test.py::TestImageBuilder::test_builder_initialization PASSED                 [ 88%]
fboss-image/distro_cli/tests/manifest_test.py::TestImageManifest::test_has_component PASSED                              [ 91%]
fboss-image/distro_cli/tests/manifest_test.py::TestImageManifest::test_load_manifest PASSED                              [ 93%]
fboss-image/distro_cli/tests/manifest_test.py::TestImageManifest::test_missing_components PASSED                         [ 95%]
fboss-image/distro_cli/tests/manifest_test.py::TestImageManifest::test_resolve_path_relative PASSED                      [ 97%]
fboss-image/distro_cli/tests/manifest_test.py::TestImageManifest::test_resolve_path_url PASSED                           [100%]

====================================================== 45 passed in 0.83s ===

@raghav-nexthop
[Nexthop] Add FBOSS-OSS Kernel
cb436fa 
Summary:

Here an RPM specfile and surrounding machinery is added to build an
FBOSS-compatible Linux kernel from vanilla Linux sources which is
compatible with CentOS. On top of this reference configuration,
additional local changes from `fboss-local-overrides.yaml` are applied
prior to building.

To use, inside an FBOSS build container, run:

    # fboss-image/kernel/scripts/build_kernel.sh 6.4.3 out
    ...
    Kernel RPM build complete!
    Output files:
    out/x86_64/kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm
    out/x86_64/kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm
    out/x86_64/kernel-core-6.4.3-1.fboss.el9.x86_64.rpm
    out/x86_64/kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm
    out/BUILD/linux-6.4.3/kernel/config_data.gz
    out/SRPMS/kernel-6.4.3-1.fboss.el9.src.rpm
    out/kernel-6.4.3.rpms.tar.gz
    out/kernel-6.4.3-1.fboss.el9.src.rpm
    out/RPMS/x86_64/kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm
    out/RPMS/x86_64/kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm
    out/RPMS/x86_64/kernel-core-6.4.3-1.fboss.el9.x86_64.rpm
    out/RPMS/x86_64/kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm
    # ls fboss-image/kernel/dist/build-6.4.3/out/
    kernel-6.4.3-1.fboss.el9.src.rpm  kernel-core-6.4.3-1.fboss.el9.x86_64.rpm   kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm
    kernel-6.4.3.rpms.tar.gz          kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm  kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm
@raghav-nexthop
Merge upstream/main into raghav.oss-kernel to update base
ecf6352
@raghav-nexthop
[Nexthop] CLI Skeleton for FBOSS Image Builder
7ff9abf
@raghav-nexthop
[Nexthop] Core Infrastructure for FBOSS Image Builder
aaf9292 
Add foundational infrastructure for distro_cli:
- Custom exception classes for error handling
- Shared constants for Docker and build configuration
- Path resolution utilities for project structure
- ImageManifest class for build configuration parsing
- CMake test infrastructure for Python unit tests

Test Plan:
- Pre-commit checks: PASSED
- Files staged: 5 files, +209 lines
- Ready for unit test execution via CMake (ctest)
@raghav-nexthop
[Nexthop] Docker Support for FBOSS Image Builder
0918753 
Add Docker container and image management:
- Docker container wrapper with exec/run capabilities
- Docker image wrapper with build/pull/tag operations
- Bazel build configuration for docker module
- Comprehensive unit tests for container and image operations

Dependencies: Uses constants from PR #1

Test Plan:
- Pre-commit checks: PASSED
- Files staged: 5 files, +502 lines
- Unit tests: docker_test.py, docker_image_test.py
@raghav-nexthop
[Nexthop] Artifact Handling for FBOSS Image Builder
d98f6af
@raghav-nexthop
[Nexthop] Execute & Download for FBOSS Image Builder
2a82e88 
Add command execution and file download utilities:
- Execute module for running commands in Docker containers
- Download module for fetching files from URLs
- Support for HTTP/HTTPS downloads with progress tracking
- Comprehensive unit tests (6 tests)

Dependencies: Uses docker, artifact, and exceptions from previous PRs

Test Plan:
- Pre-commit checks: PASSED
- Files staged: 3 files, +557 lines
- Unit tests: download_test.py (6 tests)
@raghav-nexthop
[Nexthop] Core Infrastructure for FBOSS Image Builder
705ef4b 
Add foundational infrastructure for distro_cli:
- Custom exception classes for error handling
- Shared constants for Docker and build configuration
- Path resolution utilities for project structure
- ImageManifest class for build configuration parsing
- CMake test infrastructure for Python unit tests

Test Plan:
- Pre-commit checks: PASSED
- Files staged: 5 files, +209 lines
- Ready for unit test execution via CMake (ctest)
@raghav-nexthop
[Nexthop] Docker Support for FBOSS Image Builder
e126c00 
Add Docker container and image management:
- Docker container wrapper with exec/run capabilities
- Docker image wrapper with build/pull/tag operations
- Bazel build configuration for docker module
- Comprehensive unit tests for container and image operations

Dependencies: Uses constants from PR #1

Test Plan:
- Pre-commit checks: PASSED
- Files staged: 5 files, +502 lines
- Unit tests: docker_test.py, docker_image_test.py
@raghav-nexthop
[Nexthop] Component System for FBOSS Image Builder
2810480 
Add abstract build component system:
- AbstractComponent base class for build components
- Component lifecycle management (prepare, build, extract)
- Integration with artifact store, download, and execute modules
- Test helpers for component testing

Dependencies: Uses artifact, download, execute, constants, and exceptions

Test Plan:
- Pre-commit checks: PASSED
- Files staged: 2 files, +373 lines
- Test helpers available for component testing
@raghav-nexthop
[Nexthop] Add FBOSS-OSS Kernel
de68ddb 
Summary:

Here an RPM specfile and surrounding machinery is added to build an
FBOSS-compatible Linux kernel from vanilla Linux sources which is
compatible with CentOS. On top of this reference configuration,
additional local changes from `fboss-local-overrides.yaml` are applied
prior to building.

To use, inside an FBOSS build container, run:

    # fboss-image/kernel/scripts/build_kernel.sh 6.4.3 out
    ...
    Kernel RPM build complete!
    Output files:
    out/x86_64/kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm
    out/x86_64/kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm
    out/x86_64/kernel-core-6.4.3-1.fboss.el9.x86_64.rpm
    out/x86_64/kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm
    out/BUILD/linux-6.4.3/kernel/config_data.gz
    out/SRPMS/kernel-6.4.3-1.fboss.el9.src.rpm
    out/kernel-6.4.3.rpms.tar.gz
    out/kernel-6.4.3-1.fboss.el9.src.rpm
    out/RPMS/x86_64/kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm
    out/RPMS/x86_64/kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm
    out/RPMS/x86_64/kernel-core-6.4.3-1.fboss.el9.x86_64.rpm
    out/RPMS/x86_64/kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm
    # ls fboss-image/kernel/dist/build-6.4.3/out/
    kernel-6.4.3-1.fboss.el9.src.rpm  kernel-core-6.4.3-1.fboss.el9.x86_64.rpm   kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm
    kernel-6.4.3.rpms.tar.gz          kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm  kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm
@raghav-nexthop
Merge updated raghav.oss-kernel (clean base on latest upstream/main)
6110afe
@raghav-nexthop
Merge updated raghav.distro-cli-core
0c17023
@raghav-nexthop
Merge updated raghav.distro-cli-docker
5176a21
@meta-cla meta-cla bot added the CLA Signed label Jan 9, 2026
This was referenced Jan 9, 2026
Closed
Closed
@raghav-nexthop raghav-nexthop force-pushed the raghav.distro-cli-artifact branch 3 times, most recently from 9957d52 to 5176a21 Compare January 9, 2026 13:39
@raghav-nexthop
Copy link
Author

raghav-nexthop commented Jan 9, 2026

New PR will be created

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@raghav-nexthop