[Nexthop] Execute & Download for FBOSS Image Builder #796

raghav-nexthop · 2026-01-09T05:24:40Z

Pre-submission checklist

I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
pre-commit run

Summary

This PR introduces command execution and file download utilities for the FBOSS Image Builder, building on the artifact storage system from the previous PR.

Execute module for running commands in Docker containers
Download module for fetching files from URLs with progress tracking
Integration with ArtifactStore for caching downloaded files
HTTP/HTTPS download support with metadata caching

This is part of a series. The previous one is #795

Test Plan

Added new Unit Tests.

PYTHONPATH=fboss-image:$PYTHONPATH python3 -m pytest fboss-image/distro_cli/tests/ -v
===================================================== test session starts ======================================================
platform linux -- Python 3.10.12, pytest-8.4.1, pluggy-1.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /work/raghav/private-fboss
collected 50 items                                                                                                             

fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_data_metadata_separation PASSED                   [  2%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_fetch_fn_receives_stored_files PASSED             [  4%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_fetch_fn_store_miss_updates_store PASSED          [  6%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_get_store_hit PASSED                              [  8%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_get_store_miss PASSED                             [ 10%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_multiple_data_files PASSED                        [ 12%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_store_data_and_metadata PASSED                    [ 14%]
fboss-image/distro_cli/tests/artifact_test.py::TestArtifactStore::test_store_data_only PASSED                            [ 16%]
fboss-image/distro_cli/tests/build_test.py::TestBuildCommand::test_build_all_stub PASSED                                 [ 18%]
fboss-image/distro_cli/tests/build_test.py::TestBuildCommand::test_build_command_exists PASSED                           [ 20%]
fboss-image/distro_cli/tests/build_test.py::TestBuildCommand::test_build_specific_components_stub PASSED                 [ 22%]
fboss-image/distro_cli/tests/cli_test.py::ValidatePathTest::test_validate_path_converts_to_path PASSED                   [ 24%]
fboss-image/distro_cli/tests/cli_test.py::ValidatePathTest::test_validate_path_existing_file PASSED                      [ 26%]
fboss-image/distro_cli/tests/cli_test.py::ValidatePathTest::test_validate_path_nonexistent_file_raises PASSED            [ 28%]
fboss-image/distro_cli/tests/cli_test.py::ValidatePathTest::test_validate_path_nonexistent_file_without_check PASSED     [ 30%]
fboss-image/distro_cli/tests/cli_test.py::CommandGroupTest::test_add_command_to_group PASSED                             [ 32%]
fboss-image/distro_cli/tests/cli_test.py::CommandGroupTest::test_add_command_with_arguments PASSED                       [ 34%]
fboss-image/distro_cli/tests/cli_test.py::CommandGroupTest::test_command_group_creation PASSED                           [ 36%]
fboss-image/distro_cli/tests/cli_test.py::CommandGroupTest::test_command_group_with_arguments PASSED                     [ 38%]
fboss-image/distro_cli/tests/cli_test.py::CLITest::test_add_command PASSED                                               [ 40%]
fboss-image/distro_cli/tests/cli_test.py::CLITest::test_add_command_group PASSED                                         [ 42%]
fboss-image/distro_cli/tests/cli_test.py::CLITest::test_add_command_with_arguments PASSED                                [ 44%]
fboss-image/distro_cli/tests/cli_test.py::CLITest::test_cli_creation PASSED                                              [ 46%]
fboss-image/distro_cli/tests/cli_test.py::CLITest::test_cli_with_verbose_flag PASSED                                     [ 48%]
fboss-image/distro_cli/tests/cli_test.py::CLITest::test_cli_without_verbose_flag PASSED                                  [ 50%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_device_commands_exist PASSED                       [ 52%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_getip_stub PASSED                                  [ 54%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_image_stub PASSED                                  [ 56%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_image_upstream_stub PASSED                         [ 58%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_reprovision_stub PASSED                            [ 60%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_ssh_stub PASSED                                    [ 62%]
fboss-image/distro_cli/tests/device_test.py::TestDeviceCommands::test_update_stub PASSED                                 [ 64%]
fboss-image/distro_cli/tests/docker_image_test.py::TestShouldBuildImage::test_custom_expiration_time PASSED              [ 66%]
fboss-image/distro_cli/tests/docker_image_test.py::TestShouldBuildImage::test_image_exists_and_not_expired PASSED        [ 68%]
fboss-image/distro_cli/tests/docker_image_test.py::TestShouldBuildImage::test_image_expired PASSED                       [ 70%]
fboss-image/distro_cli/tests/docker_image_test.py::TestShouldBuildImage::test_image_not_found PASSED                     [ 72%]
fboss-image/distro_cli/tests/docker_test.py::TestDockerInfrastructure::test_run_simple_container PASSED                  [ 74%]
fboss-image/distro_cli/tests/download_test.py::TestDownloadArtifact::test_download_file_url PASSED                       [ 76%]
fboss-image/distro_cli/tests/download_test.py::TestDownloadArtifact::test_download_file_url_mtime_caching PASSED         [ 78%]
fboss-image/distro_cli/tests/download_test.py::TestDownloadArtifact::test_metadata_file_structure PASSED                 [ 80%]
fboss-image/distro_cli/tests/download_test.py::TestDownloadHTTP::test_download_http_caching PASSED                       [ 82%]
fboss-image/distro_cli/tests/download_test.py::TestDownloadHTTP::test_download_http_url PASSED                           [ 84%]
fboss-image/distro_cli/tests/image_builder_test.py::TestImageBuilder::test_build_all_stub PASSED                         [ 86%]
fboss-image/distro_cli/tests/image_builder_test.py::TestImageBuilder::test_build_components_stub PASSED                  [ 88%]
fboss-image/distro_cli/tests/image_builder_test.py::TestImageBuilder::test_builder_initialization PASSED                 [ 90%]
fboss-image/distro_cli/tests/manifest_test.py::TestImageManifest::test_has_component PASSED                              [ 92%]
fboss-image/distro_cli/tests/manifest_test.py::TestImageManifest::test_load_manifest PASSED                              [ 94%]
fboss-image/distro_cli/tests/manifest_test.py::TestImageManifest::test_missing_components PASSED                         [ 96%]
fboss-image/distro_cli/tests/manifest_test.py::TestImageManifest::test_resolve_path_relative PASSED                      [ 98%]
fboss-image/distro_cli/tests/manifest_test.py::TestImageManifest::test_resolve_path_url PASSED                           [100%]

====================================================== 50 passed in 1.44s ======================================================

Summary: Here an RPM specfile and surrounding machinery is added to build an FBOSS-compatible Linux kernel from vanilla Linux sources which is compatible with CentOS. On top of this reference configuration, additional local changes from `fboss-local-overrides.yaml` are applied prior to building. To use, inside an FBOSS build container, run: # fboss-image/kernel/scripts/build_kernel.sh 6.4.3 out ... Kernel RPM build complete! Output files: out/x86_64/kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm out/x86_64/kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm out/x86_64/kernel-core-6.4.3-1.fboss.el9.x86_64.rpm out/x86_64/kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm out/BUILD/linux-6.4.3/kernel/config_data.gz out/SRPMS/kernel-6.4.3-1.fboss.el9.src.rpm out/kernel-6.4.3.rpms.tar.gz out/kernel-6.4.3-1.fboss.el9.src.rpm out/RPMS/x86_64/kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm out/RPMS/x86_64/kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm out/RPMS/x86_64/kernel-core-6.4.3-1.fboss.el9.x86_64.rpm out/RPMS/x86_64/kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm # ls fboss-image/kernel/dist/build-6.4.3/out/ kernel-6.4.3-1.fboss.el9.src.rpm kernel-core-6.4.3-1.fboss.el9.x86_64.rpm kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm kernel-6.4.3.rpms.tar.gz kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm

Add foundational infrastructure for distro_cli: - Custom exception classes for error handling - Shared constants for Docker and build configuration - Path resolution utilities for project structure - ImageManifest class for build configuration parsing - CMake test infrastructure for Python unit tests Test Plan: - Pre-commit checks: PASSED - Files staged: 5 files, +209 lines - Ready for unit test execution via CMake (ctest)

Add Docker container and image management: - Docker container wrapper with exec/run capabilities - Docker image wrapper with build/pull/tag operations - Bazel build configuration for docker module - Comprehensive unit tests for container and image operations Dependencies: Uses constants from PR #1 Test Plan: - Pre-commit checks: PASSED - Files staged: 5 files, +502 lines - Unit tests: docker_test.py, docker_image_test.py

Add command execution and file download utilities: - Execute module for running commands in Docker containers - Download module for fetching files from URLs - Support for HTTP/HTTPS downloads with progress tracking - Comprehensive unit tests (6 tests) Dependencies: Uses docker, artifact, and exceptions from previous PRs Test Plan: - Pre-commit checks: PASSED - Files staged: 3 files, +557 lines - Unit tests: download_test.py (6 tests)

Add foundational infrastructure for distro_cli: - Custom exception classes for error handling - Shared constants for Docker and build configuration - Path resolution utilities for project structure - ImageManifest class for build configuration parsing - CMake test infrastructure for Python unit tests Test Plan: - Pre-commit checks: PASSED - Files staged: 5 files, +209 lines - Ready for unit test execution via CMake (ctest)

Add Docker container and image management: - Docker container wrapper with exec/run capabilities - Docker image wrapper with build/pull/tag operations - Bazel build configuration for docker module - Comprehensive unit tests for container and image operations Dependencies: Uses constants from PR #1 Test Plan: - Pre-commit checks: PASSED - Files staged: 5 files, +502 lines - Unit tests: docker_test.py, docker_image_test.py

Add abstract build component system: - AbstractComponent base class for build components - Component lifecycle management (prepare, build, extract) - Integration with artifact store, download, and execute modules - Test helpers for component testing Dependencies: Uses artifact, download, execute, constants, and exceptions Test Plan: - Pre-commit checks: PASSED - Files staged: 2 files, +373 lines - Test helpers available for component testing

Summary: Here an RPM specfile and surrounding machinery is added to build an FBOSS-compatible Linux kernel from vanilla Linux sources which is compatible with CentOS. On top of this reference configuration, additional local changes from `fboss-local-overrides.yaml` are applied prior to building. To use, inside an FBOSS build container, run: # fboss-image/kernel/scripts/build_kernel.sh 6.4.3 out ... Kernel RPM build complete! Output files: out/x86_64/kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm out/x86_64/kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm out/x86_64/kernel-core-6.4.3-1.fboss.el9.x86_64.rpm out/x86_64/kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm out/BUILD/linux-6.4.3/kernel/config_data.gz out/SRPMS/kernel-6.4.3-1.fboss.el9.src.rpm out/kernel-6.4.3.rpms.tar.gz out/kernel-6.4.3-1.fboss.el9.src.rpm out/RPMS/x86_64/kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm out/RPMS/x86_64/kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm out/RPMS/x86_64/kernel-core-6.4.3-1.fboss.el9.x86_64.rpm out/RPMS/x86_64/kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm # ls fboss-image/kernel/dist/build-6.4.3/out/ kernel-6.4.3-1.fboss.el9.src.rpm kernel-core-6.4.3-1.fboss.el9.x86_64.rpm kernel-headers-6.4.3-1.fboss.el9.x86_64.rpm kernel-6.4.3.rpms.tar.gz kernel-devel-6.4.3-1.fboss.el9.x86_64.rpm kernel-modules-6.4.3-1.fboss.el9.x86_64.rpm

Add CMake configuration and CI workflow for FBOSS Image Builder: - CMake test configuration for distro_cli unit tests - GitHub Actions workflow for automated testing - README documentation for the distro_cli tool Test Plan: - CMake configuration tested with pytest integration - CI workflow validates on pull requests

Add foundational infrastructure for distro_cli: - Custom exception classes for error handling - Shared constants for Docker and build configuration - Path resolution utilities for locating components - ImageManifest class for build configuration parsing - Unit tests for ImageManifest with test data Test Plan: - Pre-commit checks: PASSED - Unit tests: 5/5 passing $ PYTHONPATH=fboss-image python3 -m pytest fboss-image/distro_cli/tests/manifest_test.py -v test_has_component PASSED test_load_manifest PASSED test_missing_components PASSED test_resolve_path_relative PASSED test_resolve_path_url PASSED

Add Docker container and image management: - Docker container wrapper with exec/run capabilities - Docker image wrapper with build/pull/tag operations - Build script for FBOSS builder Docker image - Comprehensive unit tests for Docker operations Dependencies: Uses constants from PR facebook#792 Test Plan: - All unit tests pass (docker_test.py, docker_image_test.py) - Docker image builds successfully with all build dependencies

Add CMake configuration and CI workflow for FBOSS Image Builder: - CMake test configuration for distro_cli unit tests - GitHub Actions workflow for automated testing - README documentation for the distro_cli tool Test Plan: - CMake configuration tested with pytest integration - CI workflow validates on pull requests

Add foundational infrastructure for distro_cli: - Custom exception classes for error handling - Shared constants for Docker and build configuration - Path resolution utilities for locating components - ImageManifest class for build configuration parsing - Unit tests for ImageManifest with test data Test Plan: - All unit tests pass (manifest_test.py) - Pre-commit checks pass

raghav-nexthop · 2026-01-09T15:32:35Z

New PR will be created.

raghav-nexthop added 20 commits December 31, 2025 11:34

Merge upstream/main into raghav.oss-kernel to update base

ecf6352

[Nexthop] CLI Skeleton for FBOSS Image Builder

7ff9abf

[Nexthop] Artifact Handling for FBOSS Image Builder

d98f6af

Merge updated raghav.oss-kernel (clean base on latest upstream/main)

6110afe

Merge updated raghav.distro-cli-core

0c17023

Merge updated raghav.distro-cli-docker

5176a21

[Nexthop] CLI Skeleton for FBOSS Image Builder

3a4da29

meta-cla bot added the CLA Signed label Jan 9, 2026

raghav-nexthop mentioned this pull request Jan 9, 2026

[Nexthop] Distro CLI: Artifact store #795

Closed

2 tasks

raghav-nexthop added 6 commits January 9, 2026 10:15

Merge remote-tracking branch 'upstream/main' into raghav.oss-kernel

653070a

Merge branch 'raghav.oss-kernel' into raghav.distro-cli-skeleton

7a7392a

Merge branch 'raghav.distro-cli-skeleton' into raghav.distro-cli-docker

c5c1821

Merge branch 'raghav.distro-cli-skeleton' into raghav.distro-cli-core

f07b7c3

Merge branch 'raghav.distro-cli-core' into raghav.distro-cli-docker

716cefd

Merge raghav.distro-cli-docker into raghav.distro-cli-artifact

b50b241

raghav-nexthop force-pushed the raghav.distro-cli-download branch from f37da62 to b50b241 Compare January 9, 2026 10:39

fixup! [Nexthop] Artifact Handling for FBOSS Image Builder

8996bac

raghav-nexthop closed this Jan 9, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Nexthop] Execute & Download for FBOSS Image Builder #796

[Nexthop] Execute & Download for FBOSS Image Builder #796

raghav-nexthop commented Jan 9, 2026 •

edited

Loading

Uh oh!

raghav-nexthop commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

[Nexthop] Execute & Download for FBOSS Image Builder #796

[Nexthop] Execute & Download for FBOSS Image Builder #796

Conversation

raghav-nexthop commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

raghav-nexthop commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

raghav-nexthop commented Jan 9, 2026 •

edited

Loading