Skip to content

[Nexthop] Add download support for FBOSS Image Builder #801

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
raghav-nexthop wants to merge 6 commits into
base: main
Choose a base branch
from
Open

[Nexthop] Add download support for FBOSS Image Builder #801

raghav-nexthop wants to merge 6 commits into from
+8,549 −4

Conversation

@raghav-nexthop
Copy link

@raghav-nexthop raghav-nexthop commented Jan 9, 2026 •
edited
Loading

Pre-submission checklist

  • I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • pre-commit run

Summary

Add file download capabilities for fetching build dependencies.

  • Implement download module for HTTP/HTTPS file retrieval
  • Add support for progress tracking during downloads

Test Plan

Include comprehensive unit tests for download operations

cd /work/raghav/private-fboss && git checkout raghav.distro-cli-download-part5 && cd fboss-image && PYTHONPATH=/work/raghav/private-fboss/fboss-image:$PYTHONPATH python3 -m pytest distro_cli/tests/ -v
Switched to branch 'raghav.distro-cli-download-part5'
=================================================================== test session starts ====================================================================
platform linux -- Python 3.10.12, pytest-8.4.1, pluggy-1.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /work/raghav/private-fboss/fboss-image
collected 50 items                                                                                                                                         

distro_cli/tests/artifact_test.py::TestArtifactStore::test_data_metadata_separation PASSED                                                           [  2%]
distro_cli/tests/artifact_test.py::TestArtifactStore::test_fetch_fn_receives_stored_files PASSED                                                     [  4%]
distro_cli/tests/artifact_test.py::TestArtifactStore::test_fetch_fn_store_miss_updates_store PASSED                                                  [  6%]
distro_cli/tests/artifact_test.py::TestArtifactStore::test_get_store_hit PASSED                                                                      [  8%]
distro_cli/tests/artifact_test.py::TestArtifactStore::test_get_store_miss PASSED                                                                     [ 10%]
distro_cli/tests/artifact_test.py::TestArtifactStore::test_multiple_data_files PASSED                                                                [ 12%]
distro_cli/tests/artifact_test.py::TestArtifactStore::test_store_data_and_metadata PASSED                                                            [ 14%]
distro_cli/tests/artifact_test.py::TestArtifactStore::test_store_data_only PASSED                                                                    [ 16%]
distro_cli/tests/build_test.py::TestBuildCommand::test_build_all_stub PASSED                                                                         [ 18%]
distro_cli/tests/build_test.py::TestBuildCommand::test_build_command_exists PASSED                                                                   [ 20%]
distro_cli/tests/build_test.py::TestBuildCommand::test_build_specific_components_stub PASSED                                                         [ 22%]
distro_cli/tests/cli_test.py::ValidatePathTest::test_validate_path_converts_to_path PASSED                                                           [ 24%]
distro_cli/tests/cli_test.py::ValidatePathTest::test_validate_path_existing_file PASSED                                                              [ 26%]
distro_cli/tests/cli_test.py::ValidatePathTest::test_validate_path_nonexistent_file_raises PASSED                                                    [ 28%]
distro_cli/tests/cli_test.py::ValidatePathTest::test_validate_path_nonexistent_file_without_check PASSED                                             [ 30%]
distro_cli/tests/cli_test.py::CommandGroupTest::test_add_command_to_group PASSED                                                                     [ 32%]
distro_cli/tests/cli_test.py::CommandGroupTest::test_add_command_with_arguments PASSED                                                               [ 34%]
distro_cli/tests/cli_test.py::CommandGroupTest::test_command_group_creation PASSED                                                                   [ 36%]
distro_cli/tests/cli_test.py::CommandGroupTest::test_command_group_with_arguments PASSED                                                             [ 38%]
distro_cli/tests/cli_test.py::CLITest::test_add_command PASSED                                                                                       [ 40%]
distro_cli/tests/cli_test.py::CLITest::test_add_command_group PASSED                                                                                 [ 42%]
distro_cli/tests/cli_test.py::CLITest::test_add_command_with_arguments PASSED                                                                        [ 44%]
distro_cli/tests/cli_test.py::CLITest::test_cli_creation PASSED                                                                                      [ 46%]
distro_cli/tests/cli_test.py::CLITest::test_cli_with_verbose_flag PASSED                                                                             [ 48%]
distro_cli/tests/cli_test.py::CLITest::test_cli_without_verbose_flag PASSED                                                                          [ 50%]
distro_cli/tests/device_test.py::TestDeviceCommands::test_device_commands_exist PASSED                                                               [ 52%]
distro_cli/tests/device_test.py::TestDeviceCommands::test_getip_stub PASSED                                                                          [ 54%]
distro_cli/tests/device_test.py::TestDeviceCommands::test_image_stub PASSED                                                                          [ 56%]
distro_cli/tests/device_test.py::TestDeviceCommands::test_image_upstream_stub PASSED                                                                 [ 58%]
distro_cli/tests/device_test.py::TestDeviceCommands::test_reprovision_stub PASSED                                                                    [ 60%]
distro_cli/tests/device_test.py::TestDeviceCommands::test_ssh_stub PASSED                                                                            [ 62%]
distro_cli/tests/device_test.py::TestDeviceCommands::test_update_stub PASSED                                                                         [ 64%]
distro_cli/tests/docker_image_test.py::TestShouldBuildImage::test_custom_expiration_time PASSED                                                      [ 66%]
distro_cli/tests/docker_image_test.py::TestShouldBuildImage::test_image_exists_and_not_expired PASSED                                                [ 68%]
distro_cli/tests/docker_image_test.py::TestShouldBuildImage::test_image_expired PASSED                                                               [ 70%]
distro_cli/tests/docker_image_test.py::TestShouldBuildImage::test_image_not_found PASSED                                                             [ 72%]
distro_cli/tests/docker_test.py::TestDockerInfrastructure::test_run_simple_container PASSED                                                          [ 74%]
distro_cli/tests/download_test.py::TestDownloadArtifact::test_download_file_url PASSED                                                               [ 76%]
distro_cli/tests/download_test.py::TestDownloadArtifact::test_download_file_url_mtime_caching PASSED                                                 [ 78%]
distro_cli/tests/download_test.py::TestDownloadArtifact::test_metadata_file_structure PASSED                                                         [ 80%]
distro_cli/tests/download_test.py::TestDownloadHTTP::test_download_http_caching PASSED                                                               [ 82%]
distro_cli/tests/download_test.py::TestDownloadHTTP::test_download_http_url PASSED                                                                   [ 84%]
distro_cli/tests/image_builder_test.py::TestImageBuilder::test_build_all_stub PASSED                                                                 [ 86%]
distro_cli/tests/image_builder_test.py::TestImageBuilder::test_build_components_stub PASSED                                                          [ 88%]
distro_cli/tests/image_builder_test.py::TestImageBuilder::test_builder_initialization PASSED                                                         [ 90%]
distro_cli/tests/manifest_test.py::TestImageManifest::test_has_component PASSED                                                                      [ 92%]
distro_cli/tests/manifest_test.py::TestImageManifest::test_load_manifest PASSED                                                                      [ 94%]
distro_cli/tests/manifest_test.py::TestImageManifest::test_missing_components PASSED                                                                 [ 96%]
distro_cli/tests/manifest_test.py::TestImageManifest::test_resolve_path_relative PASSED                                                              [ 98%]
distro_cli/tests/manifest_test.py::TestImageManifest::test_resolve_path_url PASSED                                                                   [100%]

==================================================================== 50 passed in 1.32s ====================================================================

@raghav-nexthop
[Nexthop] Add FBOSS-OSS Kernel
9157523 
Summary:

Here an RPM specfile and surrounding machinery is added to build an
FBOSS-compatible Linux kernel from vanilla Linux sources which is
compatible with CentOS. On top of this reference configuration,
additional local changes from `fboss-local-overrides.yaml` are applied
prior to building.

To use, inside an FBOSS build container, run:

    # fboss-image/kernel/scripts/build_kernel.sh 6.4.3 out
    ...
    Kernel RPM build complete!
    Output files:
    out/x86_64/kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm
    out/x86_64/kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm
    out/x86_64/kernel-core-6.4.3-1.fboss.el9.x86_64.rpm
    out/x86_64/kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm
    out/BUILD/linux-6.4.3/kernel/config_data.gz
    out/SRPMS/kernel-6.4.3-1.fboss.el9.src.rpm
    out/kernel-6.4.3.rpms.tar.gz
    out/kernel-6.4.3-1.fboss.el9.src.rpm
    out/RPMS/x86_64/kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm
    out/RPMS/x86_64/kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm
    out/RPMS/x86_64/kernel-core-6.4.3-1.fboss.el9.x86_64.rpm
    out/RPMS/x86_64/kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm
    # ls fboss-image/kernel/dist/build-6.4.3/out/
    kernel-6.4.3-1.fboss.el9.src.rpm  kernel-core-6.4.3-1.fboss.el9.x86_64.rpm   kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm
    kernel-6.4.3.rpms.tar.gz          kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm  kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm
@meta-cla meta-cla bot added the CLA Signed label Jan 9, 2026
@raghav-nexthop
[Nexthop] Add CMake test infrastructure for FBOSS Image Builder
a0ee0b6 
Integrate distro_cli Python unit tests into the FBOSS CMake build system to enable automated testing in CI/CD pipelines.

- Add FbossImageDistroCliTests.cmake to discover and register Python unit tests
- Configure test discovery for distro_cli modules (builder, cmds, lib, tools)
- Update root CMakeLists.txt to include distro_cli test suite
- Enable distro_cli tests in GitHub Actions workflow
- Update distro_cli README with build and test instructions

The CMake configuration automatically discovers all *_test.py files and registers them as CTest targets, allowing tests to run via 'ctest' or 'make test'.
@raghav-nexthop
[Nexthop] Add basic utilities for FBOSS Image Builder
c62063d 
Provide basic tools and utiulities for the distro_cli build system.

- Add custom exception classes for structured error handling
- Define shared constants for Docker configuration and build settings
- Implement path resolution utilities for components
- Define test data fixture for image manifest validation

These core utilities provide the base infrastructure required by all downstream components including Docker integration, artifact handling, and build orchestration.
@raghav-nexthop
[Nexthop] Add Docker support for FBOSS Image Builder
13f89df 
Add Docker container and image management capabilities.

- Implement Docker container wrapper with exec and run operations
- Implement Docker image wrapper with build, pull, and tag operations
- Add comprehensive unit tests for container and image functionality
- Include test helper utilities for unit test infrastructure
- Add build_docker.sh script for Docker build automation

Added unit tests for docker infrastructure and image building.
@raghav-nexthop
[Nexthop] Add artifact handling for FBOSS Image Builder
03aa3c0 
Add artifact storage and caching capabilities for build outputs.

- Implement artifact store for managing build artifacts
- Add artifact caching and retrieval functionality
- Include unit tests for artifact operations
- Update test helpers to support artifact testing

Added unit tests for artifact storage and caching.
@raghav-nexthop raghav-nexthop force-pushed the raghav.distro-cli-download-part5 branch 2 times, most recently from e077125 to 9b7170a Compare January 12, 2026 18:07
@raghav-nexthop
[Nexthop] Add download support for FBOSS Image Builder
153f4e4 
Add file download capabilities for fetching build dependencies.

- Implement download module for HTTP/HTTPS file retrieval
- Add support for progress tracking during downloads
- Include comprehensive unit tests for download operations
@raghav-nexthop raghav-nexthop force-pushed the raghav.distro-cli-download-part5 branch from 9b7170a to 153f4e4 Compare January 12, 2026 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@raghav-nexthop