diff --git a/cov_docker_script/README.md b/cov_docker_script/README.md new file mode 100644 index 0000000..4a46c2c --- /dev/null +++ b/cov_docker_script/README.md @@ -0,0 +1,1459 @@ +# πŸ”§ Coverity Native Build System for RDK-B Components + +**Generic, reusable build system enabling Coverity static analysis for any RDK-B component.** + +[![Docker](https://img.shields.io/badge/Docker-Enabled-blue)](https://github.com/rdkcentral/docker-rdk-ci) +[![GitHub Actions](https://img.shields.io/badge/CI-GitHub_Actions-green)](https://github.com/rdkcentral/moca-agent/blob/feature/cov_native_build/.github/workflows/native-build.yml) +[![Coverity](https://img.shields.io/badge/Coverity-Ready-orange)](https://www.synopsys.com/software-integrity/security-testing/static-analysis-sast.html) + +--- + +## πŸ“‹ Table of Contents + +- [Overview](#overview) +- [Prerequisites](#prerequisites) +- [Getting Started](#getting-started) + - [Quick Start](#quick-start) + - [Docker Environment Setup](#docker-environment-setup) + - [Component Integration](#component-integration) +- [Configuration Guide](#configuration-guide) + - [Scripts Overview](#scripts-overview) + - [JSON Configuration](#json-configuration) + - [Build Types](#build-types) + - [Environment Variables](#environment-variables) +- [Advanced Usage](#advanced-usage) + - [Advanced Features](#advanced-features) + - [Troubleshooting](#troubleshooting) +- [CI/CD Integration](#cicd-integration) + - [GitHub Actions](#github-actions-integration) + - [Coverity Enablement](#coverity-enablement-flow) +- [Migration Guide](#migration-guide) +- [Governance & Rules](#governance--rules) +- [References](#references) + +--- + +## 🎯 Overview + +### What is This? + +This build system provides a **standardized native build workflow** that enables Coverity static analysis for RDK-B components by: + +- βœ… Building components **outside the Yocto environment** +- βœ… Explicitly declaring **all dependencies in JSON** +- βœ… Validating builds in **Docker containers** +- βœ… Automating validation through **GitHub Actions** + +**The validated native build is a mandatory prerequisite for enabling Coverity scanning.** + +### Purpose + +| Goal | Description | +|------|-------------| +| πŸ›‘οΈ **Enable Coverity** | Standardized pathway to static analysis | +| πŸ”„ **Build Reproducibility** | Consistent builds across environments | +| πŸ“¦ **Dependency Management** | JSON-driven dependency resolution | +| πŸš€ **Component Onboarding** | Scalable integration process | + +### System Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Docker │────▢│ GitHub Actions │────▢│ Coverity β”‚ +β”‚ (Build Env) β”‚ β”‚ (Validation) β”‚ β”‚ (Analysis) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ β”‚ β”‚ + β”‚ β”‚ β”‚ + Provides Validates Consumes + consistent build validated + environment success configuration +``` + +#### Component Responsibilities + +| Component | Responsibilities | +|-----------|------------------| +| **Docker** | β€’ Consistent build environment
β€’ Pre-installed toolchain
β€’ System-level dependencies | +| **GitHub Actions** | β€’ Automated build triggers
β€’ PR validation
β€’ Quality gate enforcement | +| **Coverity** | β€’ Static analysis
β€’ Security scanning
β€’ Code quality metrics | + +### High-Level Workflow + +1. **Setup** β†’ Docker environment prepared using [docker-rdk-ci](https://github.com/rdkcentral/docker-rdk-ci) +2. **Integrate** β†’ Component adds `cov_docker_script` directory +3. **Configure** β†’ Dependencies declared in JSON +4. **Build** β†’ Native build executes locally and in CI +5. **Validate** β†’ GitHub Actions confirms build stability +6. **Enable** β†’ Component eligible for Coverity scanning +7. **Onboard** β†’ CMFSUPPORT ticket raised + +--- + +## πŸ“¦ Prerequisites + +### Required Tools + +| Tool | Purpose | Minimum Version | +|------|---------|-----------------| +| `bash` | Shell scripting | 4.0+ | +| `git` | Repository cloning | - | +| `jq` | JSON parsing | - | +| `gcc`/`g++` | C/C++ compiler | - | +| `make` | Build automation | - | +| `python3` | Patching & scripts | - | + +### Optional Tools (Dependency-Based) + +| Tool | When Required | +|------|---------------| +| `autoconf`, `automake`, `libtool` | Autotools builds | +| `cmake` | CMake builds | +| `meson`, `ninja` | Meson builds | +| `pkg-config` | Dependency detection | + +### Docker Access + +- Docker installed and running +- Access to [docker-rdk-ci](https://github.com/rdkcentral/docker-rdk-ci) +- Permissions to create/run containers + +--- + +## πŸš€ Getting Started + +### Quick Start + +**For existing components with `cov_docker_script` already integrated:** + +```bash +# Navigate to component root +cd /path/to/your-component + +# Run complete build pipeline +./cov_docker_script/common_external_build.sh + +# Clean build (removes previous artifacts) +CLEAN_BUILD=true ./cov_docker_script/common_external_build.sh +``` + +**What happens:** +1. βš™οΈ **Setup Dependencies** β†’ Clones repos, copies headers, builds libraries +2. πŸ”¨ **Build Component** β†’ Applies patches, builds component, installs libraries + +--- + +### Docker Environment Setup + +#### Step 1: Clone docker-rdk-ci + +```bash +git clone https://github.com/rdkcentral/docker-rdk-ci.git +cd docker-rdk-ci +``` + +#### Step 2: Update Dockerfile (If Needed) + +⚠️ **Only required when system-level packages are missing** + +**When to update:** +- Native build fails due to missing system packages +- Required RDK-B dependency not in standard image + +**Process:** +1. Update Dockerfile locally +2. Validate build completes successfully +3. Raise PR to [docker-rdk-ci](https://github.com/rdkcentral/docker-rdk-ci) + +```dockerfile +# Example: Adding missing packages +RUN apt-get update && apt-get install -y \ + libdbus-1-dev \ + libssl-dev \ + your-missing-package +``` + +> πŸ’‘ **Note:** Component-specific headers/libraries must be declared in `component_config.json`, **not** in Dockerfile. + +#### Step 3: Build or Pull Docker Image + +**Option A: Build Locally (After Dockerfile Updates)** + +```bash +sudo docker build --network=host -t rdkb-native-build . + +# Verify +sudo docker images +``` + +**Option B: Use Official Image (No Changes Needed)** + +```bash +docker pull ghcr.io/rdkcentral/docker-rdk-ci:latest +``` + +#### Step 4: Create and Run Container + +```bash +# Create volume mount directory +mkdir -p $HOME/docker_files + +# Run container +sudo docker run \ + --name=rdkb-builder \ + --volume=$HOME/docker_files:/home/$USER \ + --restart=no \ + --runtime=runc \ + -t -d \ + rdkb-native-build + +# Start if not running +sudo docker start rdkb-builder +``` + +#### Step 5: Configure User Permissions + +**Prevent file permission issues between host and container:** + +```bash +# Add group +sudo docker exec rdkb-builder groupadd $USER --gid=$(id -g $USER) + +# Add user +sudo docker exec rdkb-builder useradd -m $USER -G users \ + --uid=$(id -u $USER) --gid=$(id -g $USER) +``` + +#### Step 6: Access Container + +```bash +sudo docker exec --user $USER -it rdkb-builder /bin/bash +``` + +βœ… **Container is now ready for native builds!** + +--- + +### Component Integration + +#### Step 1: Clone Your Component + +```bash +git clone https://github.com/rdkcentral/your-component.git +cd your-component +``` + +#### Step 2: Add cov_docker_script Directory + +**Copy the directory structure from reference:** + +```bash +# Example structure +your-component/ +β”œβ”€β”€ cov_docker_script/ +β”‚ β”œβ”€β”€ common_build_utils.sh # Utility functions +β”‚ β”œβ”€β”€ setup_dependencies.sh # Dependency setup +β”‚ β”œβ”€β”€ build_native.sh # Component build +β”‚ β”œβ”€β”€ common_external_build.sh # Orchestrator +β”‚ β”œβ”€β”€ component_config.json # Configuration +β”‚ └── configure_options.conf # Build flags (optional) +β”œβ”€β”€ source/ +└── ... (component files) +``` + +**Reference:** [moca-agent/cov_docker_script](https://github.com/rdkcentral/moca-agent/blob/feature/cov_native_build/cov_docker_script) + +> ⚠️ **Important:** Scripts must remain unchanged. Only JSON/conf files are modifiable. + +#### Step 3: Configure Dependencies + +**Edit `component_config.json` to declare all dependencies:** + +```json +{ + "dependencies": { + "repos": [ + { + "name": "your-dependency", + "repo": "https://github.com/org/dependency.git", + "branch": "main", + "header_paths": [ + { "source": "include", "destination": "$HOME/usr/include/rdkb" } + ], + "build": { + "type": "cmake", + "cmake_flags": "-DCMAKE_INSTALL_PREFIX=$HOME/usr" + } + } + ] + }, + "native_component": { + "name": "your-component", + "include_path": "$HOME/usr/include/rdkb/", + "lib_output_path": "$HOME/usr/local/lib/", + "build": { + "type": "autotools", + "configure_options": [ + "CPPFLAGS=-I$HOME/usr/include/rdkb", + "LDFLAGS=-L$HOME/usr/local/lib" + ] + } + } +} +``` + +#### Step 4: Run Native Build + +```bash +./cov_docker_script/common_external_build.sh +``` + +#### Step 5: Resolve Build Failures + +**Build fails? Follow this process:** + +1. πŸ” **Identify** β†’ Missing headers or libraries +2. πŸ“ **Update** β†’ JSON configuration +3. πŸ”„ **Re-run** β†’ Build again + +```bash +# Check build logs +ls -la $HOME/build/ + +# Verify headers copied +ls -la $HOME/usr/include/rdkb/ + +# Verify libraries built +ls -la $HOME/usr/local/lib/ +``` + +> ❌ **No script modifications allowed!** + +--- + +## πŸ“– Configuration Guide + +### Scripts Overview + +#### 1️⃣ common_build_utils.sh + +**Shared utility library** with common functions for all build scripts. + +| Function | Purpose | +|----------|---------| +| `log()`, `ok()`, `warn()`, `err()`, `step()` | Color-coded logging | +| `expand_path()` | Expands `$HOME` variables | +| `check_dependencies()` | Validates system tools | +| `clone_repo()` | Git repository cloning | +| `copy_headers()` | Header file copying | +| `copy_python_files_generic()` | Python file copying | +| `apply_patch()` | Patch application (replace/create) | +| `build_autotools()`, `build_cmake()`, `build_meson()` | Build functions | +| `execute_commands()` | Custom command execution | +| `copy_libraries()` | Library installation | +| `print_banner()`, `print_section()` | Formatting utilities | + +**Usage:** Sourced by other scripts (not run directly) + +```bash +source common_build_utils.sh +``` + +**Auto-configured:** +- Validates presence of git, jq, gcc, make +- Sets up color-coded terminal output +- Exports all functions for use in other scripts + +--- + +#### 2️⃣ setup_dependencies.sh + +**Dependency setup script** that clones, builds, and installs dependencies. + +**Process:** +1. Reads `component_config.json` +2. Clones repos to `$HOME/build/` +3. Copies headers to `$HOME/usr/include/rdkb/` +4. Builds libraries (if configured) +5. Installs to `$HOME/usr/local/lib/` +6. Configures PKG_CONFIG_PATH and LD_LIBRARY_PATH + +**Usage:** + +```bash +# Default config +./setup_dependencies.sh + +# Custom config +./setup_dependencies.sh /path/to/config.json + +# Clean build +CLEAN_BUILD=true ./setup_dependencies.sh + +# Custom directories +BUILD_DIR=/tmp/build USR_DIR=/opt/rdkb ./setup_dependencies.sh +``` + +**Environment Variables:** +- `BUILD_DIR` - Where to clone repos (default: `$HOME/build`) +- `USR_DIR` - Install directory (default: `$HOME/usr`) +- `CLEAN_BUILD` - Set to `true` to remove previous artifacts + +**Outputs:** +- Headers: `$HOME/usr/include/rdkb/` +- Libraries: `$HOME/usr/local/lib/`, `$HOME/usr/lib/` + +--- + +#### 3️⃣ build_native.sh + +**Component build script** that builds the native component. + +**Process:** +1. Processes component headers +2. Applies patches (replace or create) +3. Copies Python files (for code generation) +4. Runs pre-build commands +5. Configures build environment +6. Executes build (autotools/cmake) +7. Installs libraries + +**Usage:** + +```bash +# Default +./build_native.sh + +# Custom paths +./build_native.sh /path/to/config.json /path/to/component + +# With overrides +HEADER_PATH=/custom/include ./build_native.sh +``` + +**Prerequisites:** +- `setup_dependencies.sh` completed +- Headers/libraries in `$HOME/usr/` + +**Output:** +- Component libraries in path specified by `native_component.lib_output_path` +- Default: `$HOME/usr/local/lib/` + +--- + +#### 4️⃣ common_external_build.sh + +**Orchestrator script** that runs the complete pipeline. + +**Process:** +1. Validates configuration +2. Runs `setup_dependencies.sh` (Step 1/2) +3. Runs `build_native.sh` (Step 2/2) +4. Displays status + +**Usage:** + +```bash +# Complete build +./common_external_build.sh + +# Custom paths +./common_external_build.sh /path/to/config.json /path/to/component + +# Clean build +CLEAN_BUILD=true ./common_external_build.sh +``` + +βœ… **This is the recommended entry point** + +**Output:** +- Complete dependency setup +- Built component with all libraries +- Success/failure status for entire pipeline + +--- + +#### 5️⃣ component_config.json + +**JSON configuration** defining dependencies and build settings. + +**Key Sections:** +- `dependencies.repos[]` β†’ Dependency repositories +- `native_component` β†’ Component build configuration +- `source_patches[]` β†’ Patches to apply +- `pre_build_commands[]` β†’ Commands before build + +**Not a script, but required by all build scripts.** + +See [JSON Configuration](#json-configuration) section for details. + +--- + +#### 6️⃣ configure_options.conf + +**Optional configuration file** for complex autotools builds. + +**Format:** + +```properties +# Section headers in square brackets +[CPPFLAGS] +-I$HOME/usr/include/rdkb/ +-DFEATURE_FLAG + +[CFLAGS] +-ffunction-sections +-fdata-sections + +[LDFLAGS] +-L$HOME/usr/local/lib/ +-Wl,--allow-shlib-undefined +``` + +**Benefits:** +- βœ… One flag per line with comments +- βœ… Section-based organization (preprocessor, compiler, linker) +- βœ… Better version control diffs +- βœ… Cleaner JSON configuration + +**Sections:** +- `[CPPFLAGS]` - Preprocessor flags (includes with `-I`, defines with `-D`) +- `[CFLAGS]` - C compiler flags (optimization, warnings, debugging) +- `[LDFLAGS]` - Linker flags (library paths with `-L`, linker options with `-Wl`) + +**Usage in JSON:** + +```json +{ + "native_component": { + "build": { + "type": "autotools", + "configure_options_file": "cov_docker_script/configure_options.conf" + } + } +} +``` + +> πŸ’‘ **Note:** Use `configure_options_file` **OR** `configure_options` array, not both. + +--- + +### JSON Configuration + +#### Dependencies Section + +```json +{ + "dependencies": { + "repos": [ + { + "name": "repo-name", + "repo": "https://github.com/org/repo.git", + "branch": "main", + "header_paths": [ + { + "source": "include", + "destination": "$HOME/usr/include/rdkb" + } + ], + "source_patches": [ + { + "file": "source/header.h", + "type": "replace", + "search": "old text", + "replace": "new text" + }, + { + "file": "$HOME/usr/include/rdkb/header.h", + "type": "replace", + "search": "old text", + "replace": "new text" + } + ], + "build": { + "type": "autotools|cmake|meson|commands|script", + "configure_flags": "--prefix=$HOME/usr", + "parallel_make": true + } + } + ] + } +} +``` + +**Field Reference:** + +| Field | Required | Description | +|-------|----------|-------------| +| `name` | βœ… | Repository name | +| `repo` | βœ… | Git repository URL | +| `branch` | βœ… | Branch to clone | +| `header_paths[]` | ⬜ | Headers to copy | +| `source_patches[]` | ⬜ | Patches to apply to dependency | +| `build` | ⬜ | Build configuration (omit for header-only) | + +--- + +#### Native Component Section + +```json +{ + "native_component": { + "name": "component-name", + "include_path": "$HOME/usr/include/rdkb/", + "lib_output_path": "$HOME/usr/local/lib/", + "header_sources": [ + { + "source": "source/include", + "destination": "$HOME/usr/include/rdkb" + } + ], + "source_patches": [ + { + "file": "$HOME/usr/include/rdkb/header.h", + "type": "replace", + "search": "old text", + "replace": "new text" + }, + { + "file": "$HOME/usr/include/rdkb/new.h", + "type": "create", + "content": "#ifndef NEW_H\n#define NEW_H\n#endif" + } + ], + "pre_build_commands": [ + { + "description": "Generate code from XML", + "command": "python3 $HOME/usr/include/rdkb/generator.py input.xml output.c" + } + ], + "build": { + "type": "autotools", + "configure_options_file": "cov_docker_script/configure_options.conf" + } + } +} +``` + +**Field Reference:** + +| Field | Description | +|-------|-------------| +| `header_sources[]` | Component headers to copy (relative paths) | +| `source_patches[]` | Patches to apply (types: `replace`, `create`) | +| `pre_build_commands[]` | Commands before build (e.g., code generation) | +| `build` | Build configuration | + +**Configuration Details:** +- `header_sources[]` - Component headers to copy before building. Source paths are relative to component directory. +- `source_patches[]` - Patches to apply after headers are copied. Supports two types: + - `type: "replace"` - Replace text in existing file (requires `search` and `replace` fields) + - `type: "create"` - Create new file with content (requires `content` field) + - Use absolute paths with `$HOME` for files in install directories +- `pre_build_commands[]` - Commands to run after patches but before build (e.g., code generation). Each has `description` and `command`. + - Automatically copies all Python files from `$PYTHON_SRC_DIR` to `$PYTHON_DST_DIR` before running commands + - Useful for code generators, data transformers, or custom preprocessing +- `include_path` - Colon-separated include paths for building +- `lib_output_path` - Where to install built libraries + +--- + +### Build Types + +#### πŸ”¨ Autotools + +**Option 1: Inline Flags** + +```json +{ + "build": { + "type": "autotools", + "configure_flags": "--prefix=$HOME/usr --enable-feature" + } +} +``` + +**Option 2: Options Array** + +```json +{ + "build": { + "type": "autotools", + "configure_options": [ + "CPPFLAGS=-I$HOME/usr/include/rdkb", + "CFLAGS=-ffunction-sections", + "LDFLAGS=-L$HOME/usr/local/lib" + ] + } +} +``` + +**Option 3: External File (Recommended for Complex Builds)** + +```json +{ + "build": { + "type": "autotools", + "configure_options_file": "cov_docker_script/configure_options.conf" + } +} +``` + +**Advanced Options:** + +```json +{ + "build": { + "type": "autotools", + "configure_flags": "--prefix=$HOME/usr", + "make_targets": ["all", "install"], + "parallel_make": true + } +} +``` + +| Option | Default | Description | +|--------|---------|-------------| +| `make_targets` | `["all"]` | Make targets to build | +| `parallel_make` | `true` | Enable `-j$(nproc)` | + +**When to use configure_options_file:** +- Complex builds with 20+ compiler flags +- Many preprocessor defines (-D flags) +- Multiple include/library paths +- Better maintainability and version control +- Cleaner JSON configuration + +--- + +#### πŸ—οΈ CMake + +```json +{ + "build": { + "type": "cmake", + "build_dir": "build", + "cmake_flags": "-DCMAKE_INSTALL_PREFIX=$HOME/usr -DBUILD_SHARED_LIBS=ON", + "make_targets": ["all", "install"], + "parallel_make": true + } +} +``` + +| Option | Default | Description | +|--------|---------|-------------| +| `build_dir` | `"build"` | CMake build directory | +| `cmake_flags` | - | CMake configuration flags | +| `make_targets` | `["all"]` | Make targets | +| `parallel_make` | `true` | Parallel compilation | + +--- + +#### πŸ”© Meson + +```json +{ + "build": { + "type": "meson", + "build_dir": "builddir", + "meson_flags": "--prefix=$HOME/usr --buildtype=release", + "ninja_targets": ["all", "install"] + } +} +``` + +| Option | Default | Description | +|--------|---------|-------------| +| `build_dir` | `"builddir"` | Meson build directory | +| `meson_flags` | - | Meson setup flags | +| `ninja_targets` | `["all"]` | Ninja targets | + +--- + +#### ⚑ Custom Commands + +```json +{ + "build": { + "type": "commands", + "commands": [ + "meson setup build --prefix=$HOME/usr", + "meson compile -C build", + "meson install -C build" + ] + } +} +``` + +Execute custom build commands in sequence. + +--- + +#### πŸ“œ Custom Script + +```json +{ + "build": { + "type": "script", + "script": "cov_docker_script/custom_build.sh" + } +} +``` + +Execute a custom build script. Used for nested builds or complex build logic. + +--- + +### Environment Variables + +**Automatically configured by scripts:** + +| Variable | Default | Description | +|----------|---------|-------------| +| `BUILD_DIR` | `$HOME/build` | Repository clone location | +| `USR_DIR` | `$HOME/usr` | Install directory | +| `PKG_CONFIG_PATH` | Auto-configured | Dependency detection | +| `LD_LIBRARY_PATH` | Auto-configured | Runtime linking | +| `CPPFLAGS` | Auto-configured | Include paths | +| `LDFLAGS` | Auto-configured | Library paths | +| `CLEAN_BUILD` | `false` | Set to `true` for clean build | + +**Advanced variables:** + +| Variable | Default | Description | +|----------|---------|-------------| +| `PYTHON_SRC_DIR` | `$HOME/build` | Python files source | +| `PYTHON_DST_DIR` | `$HOME/usr/include/rdkb` | Python files destination | +| `PARENT_BUILD_DIR` | - | Nested build coordination | +| `PARENT_USR_DIR` | - | Nested build coordination | + +--- + +## πŸš€ Advanced Usage + +### Advanced Features + +#### 🐍 Automatic Python File Copy + +**Before `pre_build_commands`, Python files (`.py`) are auto-copied from dependencies.** + +**How it works:** +1. Scans `$PYTHON_SRC_DIR` recursively for `.py` files +2. Copies to `$PYTHON_DST_DIR` +3. Makes utilities available for code generation + +**Example:** + +```json +{ + "pre_build_commands": [ + { + "description": "Generate code from XML using dependency's Python script", + "command": "python3 $HOME/usr/include/rdkb/dm_pack_code_gen.py config/data.xml output.c" + } + ] +} +``` + +**Customize directories:** + +```bash +PYTHON_SRC_DIR=/custom/src PYTHON_DST_DIR=/custom/dst ./build_native.sh +``` + +--- + +#### πŸ”— Nested Build Scripts + +**When dependencies use nested scripts:** + +```json +{ + "name": "common-library", + "build": { + "type": "script", + "script": "cov_docker_script/common_external_build.sh" + } +} +``` + +**Parent exports:** +- `PARENT_BUILD_DIR` β†’ Parent's BUILD_DIR value +- `PARENT_USR_DIR` β†’ Parent's USR_DIR value + +This allows nested scripts to coordinate their build locations. + +--- + +#### 🩹 Patch Types + +**1. Replace Patch (Modify Existing Files)** + +Replaces text in an existing file: + +```json +{ + "file": "$HOME/usr/include/rdkb/header.h", + "type": "replace", + "search": "typedef struct _OLD", + "replace": "typedef struct DBusLoop DBusLoop;\n\ntypedef struct _OLD" +} +``` + +**2. Create Patch (New Files)** + +Creates a new file with specified content: + +```json +{ + "file": "$HOME/usr/include/rdkb/generated.h", + "type": "create", + "content": "#ifndef GENERATED_H\n#define GENERATED_H\n\n// Auto-generated\n\n#endif" +} +``` + +--- + +#### 🎯 Build Target Customization + +All build types support custom targets: + +**Autotools/CMake:** + +```json +{ + "build": { + "type": "autotools", + "make_targets": ["all", "install", "check"] + } +} +``` + +**Meson:** + +```json +{ + "build": { + "type": "meson", + "ninja_targets": ["all", "install", "test"] + } +} +``` + +--- + +#### βš™οΈ Parallel Build Control + +Control parallel compilation (default: enabled): + +```json +{ + "build": { + "type": "cmake", + "parallel_make": false + } +} +``` + +**When to disable:** +- Build systems with race conditions +- Low-memory environments +- Debugging build issues + +--- + +### Troubleshooting + +#### ❌ Build fails with "command not found" + +**For Docker Container Environments:** + +If you're running builds in a Docker container and encounter missing tools, add the required packages to the Docker image: + +1. Update the [docker-rdk-ci Dockerfile](https://github.com/rdkcentral/docker-rdk-ci/blob/main/Dockerfile) +2. Add missing packages to the appropriate `RUN apt-get install` section +3. Rebuild and use the updated Docker image + +**For Native/Local Builds:** + +Install required tools directly: + +```bash +sudo apt-get install git jq gcc make autoconf automake libtool cmake python3 +``` + +--- + +#### ❌ Dependencies fail to build + +**Check:** +- `$HOME/build//` for build logs +- `configure_flags` in JSON are correct +- System packages installed (cmake, meson, etc.) + +--- + +#### ❌ Headers not found during component build + +**Verify:** +- `setup_dependencies.sh` completed successfully +- `$HOME/usr/include/rdkb/` contains expected headers +- `header_paths` in JSON point to correct source directories + +--- + +#### ❌ Libraries not found + +**Check library directories:** +- `$HOME/usr/local/lib/` (primary location) +- `$HOME/usr/lib/` (secondary location) + +**Verify:** +- Dependencies built successfully (look for `.so`, `.a` files) +- Build logs for `make install` errors + +--- + +#### ❌ Patches fail to apply + +**Common issues:** +- **File not found:** Verify file path is relative to component directory +- **Use `../`** for files outside component (e.g., `../usr/include/rdkb/header.h`) +- **Exact match required:** Search string must exactly match file content +- **Python3 required:** Ensure Python3 is installed + +--- + +#### 🧹 Clean Build + +Remove all previous build artifacts: + +```bash +CLEAN_BUILD=true ./common_external_build.sh +``` + +--- + +#### βœ… Validate Configuration + +```bash +# Check JSON syntax +jq . component_config.json + +# List all dependencies +jq '.dependencies.repos[].name' component_config.json +``` + +--- + +#### πŸ“‚ Directory Structure After Build + +``` +$HOME/ +β”œβ”€β”€ build/ # Cloned repositories +└── usr/ + β”œβ”€β”€ include/ + β”‚ └── rdkb/ # All dependency headers + β”œβ”€β”€ lib/ # Secondary library location + └── local/ + └── lib/ # Primary library location (.so, .a) +``` + +--- + +## πŸ”„ CI/CD Integration + +### GitHub Actions Integration + +The GitHub Actions workflow provides automated validation of the native build: + +**Workflow Responsibilities:** +- βœ… Triggers the native build automatically on PR creation and updates +- βœ… Fails on compilation errors +- βœ… Exposes logs for troubleshooting +- βœ… Ensures consistent validation across components + +**Successful CI validation is required before Coverity onboarding.** + +**Example Workflow:** + +```yaml +# .github/workflows/native-build.yml +name: Native Build + +on: + pull_request: + push: + branches: [main, develop] + +jobs: + build: + runs-on: ubuntu-latest + container: + image: ghcr.io/rdkcentral/docker-rdk-ci:latest + steps: + - uses: actions/checkout@v3 + - name: Run Native Build + run: ./cov_docker_script/common_external_build.sh +``` + +**The workflow validates that:** +1. All dependencies are correctly declared in JSON +2. The component builds successfully in a clean environment +3. No hardcoded paths or undeclared dependencies exist + +**Reference Implementation:** +- [moca-agent native-build.yml](https://github.com/rdkcentral/moca-agent/blob/feature/cov_native_build/.github/workflows/native-build.yml) + +--- + +### Coverity Enablement Flow + +Once the native build is validated: + +#### Step 1: Validate Build Stability + +- βœ… Native build succeeds locally +- βœ… GitHub Actions workflow passes consistently +- βœ… All dependencies declared in JSON + +#### Step 2: Raise CMFSUPPORT Ticket + +**Include:** +- Component repository URL +- Reference validated native build configuration +- Link to successful GitHub Actions runs + +#### Step 3: Coverity Integration + +- Coverity scanning is enabled using the validated configuration +- Static analysis runs using the approved build flow +- Results are available in Coverity dashboard + +**Key Principle:** +Native build validation enables Coverity integration. + +--- + +## πŸ“š Migration Guide + +### Adopting for Another Component + +**These scripts are 100% generic and component-agnostic!** + +#### Step 1: Copy the Scripts + +```bash +# Copy all scripts to your component's build directory +cp -r /reference/cov_docker_script /path/to/new-component/ + +# Make executable +chmod +x /path/to/new-component/cov_docker_script/*.sh +``` + +#### Step 2: Create component_config.json + +Create a new `component_config.json` for your component: + +```json +{ + "_comment": "Component Build Configuration", + "_version": "2.0", + + "dependencies": { + "repos": [ + { + "name": "your-dependency", + "repo": "https://github.com/org/your-dependency.git", + "branch": "main", + "header_paths": [ + { "source": "include", "destination": "$HOME/usr/include/rdkb" } + ], + "build": { + "type": "cmake", + "cmake_flags": "-DCMAKE_INSTALL_PREFIX=$HOME/usr" + } + } + ] + }, + + "native_component": { + "name": "your-component-name", + "include_path": "$HOME/usr/include/rdkb/", + "lib_output_path": "$HOME/usr/local/lib/", + "source_patches": [], + "build": { + "type": "autotools", + "configure_options": [ + "CPPFLAGS=-I$HOME/usr/include/rdkb", + "LDFLAGS=-L$HOME/usr/local/lib" + ] + } + } +} +``` + +#### Step 3: Run the Build + +```bash +cd /path/to/new-component/cov_docker_script +./common_external_build.sh +``` + +**That's it!** No script modifications needed. The scripts automatically: +- Read component name from JSON +- Find component directory (parent of script directory) +- Clone dependencies listed in JSON +- Copy headers from paths specified in JSON +- Build using build type specified in JSON +- Apply patches listed in JSON + +--- + +### What Makes These Scripts Generic? + +| Feature | Implementation | +|---------|----------------| +| βœ… No hardcoded paths | All from JSON/environment | +| βœ… No hardcoded names | Read from JSON | +| βœ… No hardcoded dependencies | Defined in JSON | +| βœ… No hardcoded builds | Type/options from JSON | +| βœ… Flexible build systems | Autotools/CMake/Meson/Custom | +| βœ… Configurable patches | All in JSON | + +--- + +### Example: Migrating from Utopia to CcspPandM + +```bash +# 1. Copy scripts to CcspPandM +cp -r utopia/cov_docker_script ccsp-p-and-m/ + +# 2. Create ccsp-p-and-m/cov_docker_script/component_config.json +# Update: component name, dependencies, build settings + +# 3. Run build +cd ccsp-p-and-m/cov_docker_script +./common_external_build.sh +``` + +**Scripts remain unchanged - only JSON changes!** + +--- + +### Complete Example: Component with All Features + +**component_config.json:** + +```json +{ + "dependencies": { + "repos": [ + { + "name": "rbus", + "repo": "https://github.com/rdkcentral/rbus.git", + "branch": "v2.7.0", + "header_paths": [ + { "source": "include", "destination": "$HOME/usr/include/rdkb/rbus" } + ], + "build": { + "type": "cmake", + "build_dir": "build", + "cmake_flags": "-DCMAKE_INSTALL_PREFIX=$HOME/usr" + } + }, + { + "name": "common-library", + "repo": "https://github.com/rdkcentral/common-library.git", + "branch": "main", + "header_paths": [ + { "source": "source/ccsp/include", "destination": "$HOME/usr/include/rdkb" } + ], + "source_patches": [ + { + "file": "$HOME/usr/include/rdkb/ccsp_message_bus.h", + "type": "replace", + "search": "typedef struct _CCSP_MESSAGE_BUS_CONNECTION", + "replace": "typedef struct DBusLoop DBusLoop;\n\ntypedef struct _CCSP_MESSAGE_BUS_CONNECTION" + }, + { + "file": "$HOME/usr/include/rdkb/custom_config.h", + "type": "create", + "content": "#ifndef CUSTOM_CONFIG_H\n#define CUSTOM_CONFIG_H\n#define CUSTOM_FEATURE_ENABLED\n#endif" + } + ], + "build": { + "type": "script", + "script": "cov_docker_script/common_external_build.sh" + } + } + ] + }, + + "native_component": { + "name": "moca-agent", + "include_path": "$HOME/usr/include/rdkb/", + "lib_output_path": "$HOME/usr/local/lib/", + "header_sources": [ + { "source": "source/include", "destination": "$HOME/usr/include/rdkb" } + ], + "pre_build_commands": [ + { + "description": "Generate dm_pack_datamodel.c from XML", + "command": "python3 $HOME/usr/include/rdkb/dm_pack_code_gen.py config/TR181-MoCA.XML source/MoCASsp/dm_pack_datamodel.c" + } + ], + "build": { + "type": "autotools", + "configure_options_file": "cov_docker_script/configure_options.conf" + } + } +} +``` + +**configure_options.conf:** + +```properties +# MoCA Agent Configure Options +[CPPFLAGS] +# Include paths +-I$HOME/usr/include/rdkb/ +-I/usr/include/dbus-1.0 + +# Core system defines +-DSAFEC_DUMMY_API +-D_COSA_HAL_ +-DCONFIG_SYSTEM_MOCA + +# CCSP/Component defines +-DCCSP_SUPPORT_ENABLED +-D_CCSP_CWMP_TCP_CONNREQ_HANDLER + +# Product/Platform defines +-D_XB6_PRODUCT_REQ_ +-D_XB7_PRODUCT_REQ_ + +# Features +-DFEATURE_SUPPORT_WEBCONFIG +-DMOCA_HOME_ISOLATION + +[CFLAGS] +-ffunction-sections +-fdata-sections +-fno-strict-aliasing + +[LDFLAGS] +-L$HOME/usr/local/lib/ +-Wl,--allow-shlib-undefined +``` + +**Build execution:** + +```bash +cd /path/to/moca-agent/cov_docker_script +./common_external_build.sh +``` + +**What happens:** +1. Clones rbus and common-library repositories +2. Copies headers from dependencies +3. Applies patches to common-library headers +4. Builds rbus (cmake) and common-library (nested build script) +5. Copies component headers +6. Runs pre-build command (generates code from XML) +7. Reads configure options from configure_options.conf +8. Runs autotools build with all flags +9. Installs libraries to output path + +--- + +## βš–οΈ Governance & Rules + +### Configuration Rules + +| Rule | Status | +|------|--------| +| Only JSON files may be modified | βœ… Allowed | +| All dependencies must be declared | βœ… Required | +| Script modifications | ❌ Forbidden | + +### Script Governance + +- πŸ”’ Scripts are **generic, shared, and immutable** +- 🌐 Any script change must apply **globally** to all components +- πŸ“ Component-specific logic belongs in **JSON only** + +### Standardization + +| Benefit | Description | +|---------|-------------| +| βœ… Scalable Onboarding | Consistent process across components | +| βœ… Build Reproducibility | Same configuration = same results | +| βœ… Maintainability | Single source of truth in JSON | +| βœ… Quality Assurance | Automated validation via CI/CD | + +### Key Principles + +> **Native build validation enables Coverity integration.** +> +> Scripts are immutable; configuration is component-specific. + +**This approach ensures:** +- Scalable Coverity onboarding +- Consistent build behavior +- Long-term maintainability across RDK-B components + +--- + +## πŸ”— References + +### Docker RDK CI Repository + +Docker image for consistent RDK-B native builds: +- **Repository:** https://github.com/rdkcentral/docker-rdk-ci +- **Official Image:** `ghcr.io/rdkcentral/docker-rdk-ci:latest` + +### Native Build Reference Implementation + +Complete example with cov_docker_script integration: +- **Component:** https://github.com/rdkcentral/moca-agent/tree/feature/cov_native_build +- **Scripts Directory:** https://github.com/rdkcentral/moca-agent/blob/feature/cov_native_build/cov_docker_script +- **README:** https://github.com/rdkcentral/moca-agent/blob/feature/cov_native_build/cov_docker_script/README.md + +### GitHub Actions Workflow + +Reference CI/CD workflow for native build validation: +- **Workflow:** https://github.com/rdkcentral/moca-agent/blob/feature/cov_native_build/.github/workflows/native-build.yml + +### Component Configuration Examples + +- **JSON Configuration:** https://github.com/rdkcentral/moca-agent/blob/feature/cov_native_build/cov_docker_script/component_config.json +- **Configure Options:** https://github.com/rdkcentral/moca-agent/blob/feature/cov_native_build/cov_docker_script/configure_options.conf + +--- + +## πŸ“„ License + +This build system is part of the RDK-B project. See component repository for license details. + +--- + +## 🀝 Contributing + +For issues, improvements, or questions: + +1. πŸ› **Issues** β†’ Raise in component repository +2. πŸ’‘ **Script Changes** β†’ Must apply globally to all components +3. πŸ“ **Documentation** β†’ Update this README +4. 🐳 **Docker Changes** β†’ Raise PR to [docker-rdk-ci](https://github.com/rdkcentral/docker-rdk-ci) + +--- + +## πŸ“ž Support + +- **CMFSUPPORT Tickets** β†’ For Coverity enablement +- **GitHub Issues** β†’ For build system questions +- **Component Maintainers** β†’ For component-specific issues + +--- + +**Made with ❀️ for RDK-B Community** diff --git a/cov_docker_script/build_native.sh b/cov_docker_script/build_native.sh new file mode 100755 index 0000000..07177d5 --- /dev/null +++ b/cov_docker_script/build_native.sh @@ -0,0 +1,417 @@ +#!/usr/bin/env bash +set -e + +################################################################################ +# Generic Native Component Build Script +# Usage: ./build_native.sh [config_file] [component_dir] +################################################################################ + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CONFIG_FILE="${1:-$SCRIPT_DIR/component_config.json}" +COMPONENT_DIR="${2:-$(cd "$SCRIPT_DIR/.." && pwd)}" + +# Source common utilities +source "$SCRIPT_DIR/common_build_utils.sh" + +# Validate environment +if [[ ! -f "$CONFIG_FILE" ]]; then + err "Config file not found: $CONFIG_FILE" + exit 1 +fi + +if [[ ! -d "$COMPONENT_DIR" ]]; then + err "Component directory not found: $COMPONENT_DIR" + exit 1 +fi + +check_dependencies || exit 1 + +# Read component configuration +COMPONENT_NAME=$(jq -r '.native_component.name' "$CONFIG_FILE") +BUILD_TYPE=$(jq -r '.native_component.build.type' "$CONFIG_FILE") +HEADER_PATH=$(expand_path "$(jq -r '.native_component.include_path' "$CONFIG_FILE")") +LIB_PATH=$(expand_path "$(jq -r '.native_component.lib_output_path' "$CONFIG_FILE")") + +# Configure environment +configure_environment() { + print_banner "Building Native Component: $COMPONENT_NAME" + + log "Component: $COMPONENT_NAME" + log "Build type: $BUILD_TYPE" + log "Component directory: $COMPONENT_DIR" + log "Header path: $HEADER_PATH" + log "Library path: $LIB_PATH" + echo "" + + # Setup PKG_CONFIG_PATH and LD_LIBRARY_PATH + export PKG_CONFIG_PATH="$LIB_PATH/pkgconfig:${PKG_CONFIG_PATH:-}" + export LD_LIBRARY_PATH="$LIB_PATH:${LD_LIBRARY_PATH:-}" + + # Add common include and lib paths + export CPPFLAGS="${CPPFLAGS:-} -I$HEADER_PATH" + export CFLAGS="${CFLAGS:-} -I$HEADER_PATH" + export LDFLAGS="${LDFLAGS:-} -L$LIB_PATH" + + log "Environment configured" + log " PKG_CONFIG_PATH=$PKG_CONFIG_PATH" + log " LD_LIBRARY_PATH=$LD_LIBRARY_PATH" + echo "" +} + +# Apply source patches +apply_source_patches() { + local patch_count + patch_count=$(jq -r '.native_component.source_patches // [] | length' "$CONFIG_FILE") + + if [[ "$patch_count" -eq 0 ]]; then + log "No source patches configured" + return 0 + fi + + step "Applying source patches ($patch_count patches)" + + local i=0 + while [[ $i -lt $patch_count ]]; do + local file search replace type content + file=$(jq -r ".native_component.source_patches[$i].file" "$CONFIG_FILE") + type=$(jq -r ".native_component.source_patches[$i].type // \"replace\"" "$CONFIG_FILE") + search=$(jq -r ".native_component.source_patches[$i].search // \"\"" "$CONFIG_FILE") + replace=$(jq -r ".native_component.source_patches[$i].replace // \"\"" "$CONFIG_FILE") + content=$(jq -r ".native_component.source_patches[$i].content // \"\"" "$CONFIG_FILE") + + # Expand $HOME in file path, then resolve relative paths from COMPONENT_DIR + local expanded_file=$(expand_path "$file") + local target_file + if [[ "$expanded_file" = /* ]]; then + # Absolute path - use as is + target_file="$expanded_file" + else + # Relative path - prepend COMPONENT_DIR + target_file="$COMPONENT_DIR/$expanded_file" + fi + + if ! apply_patch "$target_file" "$search" "$replace" "$type" "$content"; then + err "Failed to apply patch $((i+1))/$patch_count" + return 1 + fi + + i=$((i + 1)) + done + + ok "All patches applied successfully" + echo "" + return 0 +} + +# Process native headers +process_native_headers() { + local header_count + header_count=$(jq -r '.native_component.header_sources // [] | length' "$CONFIG_FILE") + + if [[ "$header_count" -eq 0 ]]; then + log "No header sources configured" + return 0 + fi + + step "Processing native component headers ($header_count sources)" + + local i=0 + while [[ $i -lt $header_count ]]; do + local src dst + src=$(jq -r ".native_component.header_sources[$i].source" "$CONFIG_FILE") + dst=$(jq -r ".native_component.header_sources[$i].destination" "$CONFIG_FILE") + + # Expand paths + src="$COMPONENT_DIR/$src" + dst=$(expand_path "$dst") + + copy_headers "$src" "$dst" + i=$((i + 1)) + done + + ok "All headers processed successfully" + echo "" + return 0 +} + +# Parse configure options from external file +parse_configure_options_file() { + local conf_file="$1" + local -n options_array=$2 + + if [[ ! -f "$conf_file" ]]; then + err "Configure options file not found: $conf_file" + return 1 + fi + + local current_section="" + local cppflags="" + local cflags="" + local ldflags="" + + while IFS= read -r line || [[ -n "$line" ]]; do + # Skip empty lines and comments + [[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue + + # Detect section headers + if [[ "$line" =~ ^\[([A-Z_]+)\] ]]; then + current_section="${BASH_REMATCH[1]}" + continue + fi + + # Trim whitespace + line=$(echo "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + [[ -z "$line" ]] && continue + + # Append to appropriate section + case "$current_section" in + CPPFLAGS) + cppflags+="$line " + ;; + CFLAGS) + cflags+="$line " + ;; + LDFLAGS) + ldflags+="$line " + ;; + esac + done < "$conf_file" + + # Expand environment variables in the flags using envsubst or manual replacement + # Use manual replacement for better control and to avoid shell interpretation issues + cppflags="${cppflags//\$HOME/$HOME}" + cflags="${cflags//\$HOME/$HOME}" + ldflags="${ldflags//\$HOME/$HOME}" + + # Build final options array + [[ -n "$cppflags" ]] && options_array+=("CPPFLAGS=${cppflags% }") + [[ -n "$cflags" ]] && options_array+=("CFLAGS=${cflags% }") + [[ -n "$ldflags" ]] && options_array+=("LDFLAGS=${ldflags% }") +} + +# Build with autotools +build_component_autotools() { + cd "$COMPONENT_DIR" + + # Read configure options as array + local configure_options=() + + # Check if using external configure options file + local config_file_path + config_file_path=$(jq -r '.native_component.build.configure_options_file // empty' "$CONFIG_FILE") + + if [[ -n "$config_file_path" ]]; then + # Using external configuration file + config_file_path=$(expand_path "$config_file_path") + # If relative path, make it relative to component dir + if [[ ! "$config_file_path" = /* ]]; then + config_file_path="$COMPONENT_DIR/$config_file_path" + fi + + step "Reading configure options from: $config_file_path" + if ! parse_configure_options_file "$config_file_path" configure_options; then + err "Failed to parse configure options file" + return 1 + fi + ok "Loaded configure options from file" + else + # Using inline configure_options array (legacy support) + local opt_count + opt_count=$(jq -r '.native_component.build.configure_options // [] | length' "$CONFIG_FILE") + + local i=0 + while [[ $i -lt $opt_count ]]; do + local option + option=$(jq -r ".native_component.build.configure_options[$i]" "$CONFIG_FILE") + option=$(expand_path "$option") + configure_options+=("$option") + i=$((i + 1)) + done + fi + + # Run autogen if exists + if [[ -f "./autogen.sh" ]]; then + step "Running autogen.sh" + chmod +x ./autogen.sh + # Set NOCONFIGURE to prevent autogen.sh from automatically running configure + if ! NOCONFIGURE=1 ./autogen.sh; then + err "autogen.sh failed" + return 1 + fi + ok "autogen.sh completed" + echo "" + fi + + # Configure + step "Running configure" + + # Export configure options as environment variables + for option in "${configure_options[@]}"; do + export "$option" + done + + if ! ./configure; then + err "Configure failed" + return 1 + fi + ok "Configure completed" + echo "" + + # Make + local make_targets + make_targets=$(jq -r '.native_component.build.make_targets[]? // "all"' "$CONFIG_FILE" | tr '\n' ' ') + + local parallel_make + parallel_make=$(jq -r '.native_component.build.parallel_make // true' "$CONFIG_FILE") + + local make_jobs="" + [[ "$parallel_make" == "true" ]] && make_jobs="-j$(nproc)" + + step "Running make $make_jobs $make_targets" + if ! make $make_jobs $make_targets; then + err "Make failed" + return 1 + fi + ok "Make completed" + echo "" + + return 0 +} + +# Run pre-build commands from native_component.pre_build_commands[] +run_pre_build_commands() { + + log "copying python files generic..." + copy_python_files_generic + + log "Running pre-build commands..." + + if [[ -z "$CONFIG_FILE" ]] || [[ ! -f "$CONFIG_FILE" ]]; then + err "CONFIG_FILE not set or file missing" + return 1 + fi + + if [[ -z "$COMPONENT_DIR" ]] || [[ ! -d "$COMPONENT_DIR" ]]; then + err "COMPONENT_DIR not set or directory missing" + return 1 + fi + + local cmd_count + cmd_count=$(jq '.native_component.pre_build_commands // [] | length' "$CONFIG_FILE") + + if [[ "$cmd_count" -eq 0 ]]; then + log "No pre-build commands to run" + return 0 + fi + + pushd "$COMPONENT_DIR" >/dev/null || { + err "Failed to enter component root: $COMPONENT_DIR" + return 1 + } + + local i description command + for ((i=0; i/dev/null + return 1 + fi + done + + popd >/dev/null + ok "Pre-build commands completed" + return 0 +} + +# Build with CMake +build_component_cmake() { + cd "$COMPONENT_DIR" + + local build_dir cmake_flags make_targets parallel_make + build_dir=$(jq -r '.native_component.build.build_dir // "build"' "$CONFIG_FILE") + cmake_flags=$(jq -r '.native_component.build.cmake_flags // empty' "$CONFIG_FILE") + cmake_flags=$(expand_path "$cmake_flags") + make_targets=$(jq -r '.native_component.build.make_targets[]? // "all"' "$CONFIG_FILE" | tr '\n' ' ') + parallel_make=$(jq -r '.native_component.build.parallel_make // true' "$CONFIG_FILE") + + build_cmake "$COMPONENT_DIR" "$build_dir" "$cmake_flags" "$make_targets" "$parallel_make" || return 1 + + return 0 +} + +# Install libraries +install_libraries() { + step "Installing libraries to $LIB_PATH" + mkdir -p "$LIB_PATH" + + # Find and copy all library files (shared objects, static, libtool archives) + find "$COMPONENT_DIR" \( -name "*.so*" -o -name "*.a" -o -name "*.la*" \) \( -type f -o -type l \) -exec cp -Pv {} "$LIB_PATH/" \; 2>/dev/null || true + + ok "Libraries installed" + echo "" +} + +# Main execution +main() { + configure_environment + + # Process native headers + if ! process_native_headers; then + err "Header processing failed" + exit 1 + fi + + # Apply patches + if ! apply_source_patches; then + err "Patch application failed" + exit 1 + fi + + # Run pre-build commands + if ! run_pre_build_commands; then + err "Pre-build commands failed" + exit 1 + fi + + # Build based on type + case "$BUILD_TYPE" in + autotools) + if ! build_component_autotools; then + err "Autotools build failed" + exit 1 + fi + ;; + + cmake) + if ! build_component_cmake; then + err "CMake build failed" + exit 1 + fi + ;; + + *) + err "Unsupported build type: $BUILD_TYPE" + exit 1 + ;; + esac + + # Install libraries + install_libraries + + print_banner "Native Component Build Completed Successfully" + log "Component: $COMPONENT_NAME" + log "Headers: $HEADER_PATH" + log "Libraries: $LIB_PATH" + echo "" +} + +main diff --git a/cov_docker_script/common_build_utils.sh b/cov_docker_script/common_build_utils.sh new file mode 100755 index 0000000..4f3e732 --- /dev/null +++ b/cov_docker_script/common_build_utils.sh @@ -0,0 +1,312 @@ +#!/usr/bin/env bash + +################################################################################ +# Common Build Utilities +# Shared functions for dependency and component builds +################################################################################ + +# Colors +RED="\e[31m"; GREEN="\e[32m"; YELLOW="\e[33m" +BLUE="\e[34m"; CYAN="\e[36m"; BOLD="\e[1m"; NC="\e[0m" + +# Logging functions +log() { echo -e "${CYAN}[INFO]${NC} $1"; } +ok() { echo -e "${GREEN}[OK]${NC} $1"; } +warn() { echo -e "${YELLOW}[WARN]${NC} $1"; } +err() { echo -e "${RED}[ERROR]${NC} $1"; } +step() { echo -e "${BLUE}[STEP]${NC} $1"; } + +# Expand $HOME in paths +expand_path() { + echo "${1//\$HOME/$HOME}" +} + +# Validate required tools +check_dependencies() { + local required_tools=("git" "jq" "gcc" "make") + local missing=() + + for tool in "${required_tools[@]}"; do + if ! command -v "$tool" &> /dev/null; then + missing+=("$tool") + fi + done + + if [ ${#missing[@]} -gt 0 ]; then + err "Missing required tools: ${missing[*]}" + err "Please install them before continuing" + return 1 + fi + return 0 +} + +# Clone a git repository +clone_repo() { + local name="$1" repo="$2" branch="$3" dest="$4" + + if [[ -d "$dest" ]]; then + warn "$name already exists, skipping clone" + return 0 + fi + + log "Cloning $name (branch: $branch)" + if ! git clone --branch "$branch" "$repo" "$dest" --depth 1; then + err "Failed to clone $name" + return 1 + fi + ok "$name cloned successfully" + return 0 +} + +# Copy headers from source to destination +copy_headers() { + local src="$1" dst="$2" + + src=$(expand_path "$src") + dst=$(expand_path "$dst") + + mkdir -p "$dst" + + if [[ -d "$src" ]]; then + log "Copying headers: $src β†’ $dst" + if ! find "$src" -maxdepth 1 -name "*.h" -exec cp {} "$dst/" \; 2>/dev/null; then + warn "No headers found in $src" + fi + else + warn "Header source not found: $src" + fi +} + +# Generic API to copy all Python files from a source directory (recursively) to a destination directory (flat, no subdirs) +copy_python_files_generic() { + local src_dir="${PYTHON_SRC_DIR:-$HOME/build}" + local dst_dir="${PYTHON_DST_DIR:-$HOME/usr/include/rdkb}" + if [[ -n "$src_dir" && -n "$dst_dir" ]]; then + log "[PYTHON COPY] Scanning for Python files in: $src_dir" + mkdir -p "$dst_dir" + local py_files + py_files=$(find "$src_dir" -type f -name "*.py") + local count=0 + if [[ -n "$py_files" ]]; then + log "[PYTHON COPY] Copying Python files to: $dst_dir" + while IFS= read -r file; do + cp "$file" "$dst_dir/" + count=$((count+1)) + done <<< "$py_files" + ok "[PYTHON COPY] $count Python file(s) copied to $dst_dir" + else + warn "[PYTHON COPY] No Python files found in $src_dir" + fi + else + warn "[PYTHON COPY] Source or destination directory not set. Skipping copy." + fi +} + +# Apply source patches +apply_patch() { + local file="$1" search="$2" replace="$3" type="${4:-replace}" content="$5" + + if [[ "$type" == "create" ]]; then + log "Creating file: $file" + local dir=$(dirname "$file") + mkdir -p "$dir" + echo -e "$content" > "$file" + if [[ $? -ne 0 ]]; then + err "Failed to create file: $file" + return 1 + fi + ok "File created successfully" + return 0 + fi + + if [[ ! -f "$file" ]]; then + err "Patch target not found: $file" + return 1 + fi + + log "Patching: $file ($type)" + + # Use python for safe string replacement with literal matching + if ! python3 -c " +import sys +with open('$file', 'r') as f: + content = f.read() +content = content.replace('''$search''', '''$replace''') +with open('$file', 'w') as f: + f.write(content) +"; then + err "Failed to apply patch to $file" + return 1 + fi + + ok "Patch applied successfully" + return 0 +} + +# Build with autotools +build_autotools() { + local repo_dir="$1" configure_flags="$2" make_targets="$3" parallel_make="${4:-true}" + + pushd "$repo_dir" >/dev/null || return 1 + + # Run autogen or autoreconf if needed + if [[ -f "autogen.sh" ]]; then + step "Running autogen.sh" + chmod +x autogen.sh + # Set NOCONFIGURE to prevent autogen.sh from automatically running configure + if ! NOCONFIGURE=1 ./autogen.sh; then + err "autogen.sh failed" + popd >/dev/null + return 1 + fi + elif [[ -f "configure.ac" ]] || [[ -f "configure.in" ]]; then + step "Running autoreconf" + if ! autoreconf -fi; then + err "autoreconf failed" + popd >/dev/null + return 1 + fi + fi + + # Configure + step "Running configure" + # Ensure PKG_CONFIG_PATH is set for configure + export PKG_CONFIG_PATH="${HOME}/usr/local/lib/pkgconfig:${HOME}/usr/lib/pkgconfig:${PKG_CONFIG_PATH:-}" + if ! eval "./configure $configure_flags"; then + err "Configure failed" + popd >/dev/null + return 1 + fi + + # Make + local make_jobs="" + [[ "$parallel_make" == "true" ]] && make_jobs="-j$(nproc)" + + step "Running make $make_jobs $make_targets" + if ! make $make_jobs $make_targets; then + err "Make failed" + popd >/dev/null + return 1 + fi + + popd >/dev/null + ok "Autotools build completed" + return 0 +} + +# Build with CMake +build_cmake() { + local repo_dir="$1" build_dir="$2" cmake_flags="$3" make_targets="$4" parallel_make="${5:-true}" + + pushd "$repo_dir" >/dev/null || return 1 + mkdir -p "$build_dir" + + step "Running cmake" + if ! eval "cmake -S . -B $build_dir $cmake_flags"; then + err "CMake configuration failed" + popd >/dev/null + return 1 + fi + + local make_jobs="" + [[ "$parallel_make" == "true" ]] && make_jobs="-j$(nproc)" + + step "Building with make $make_jobs $make_targets" + if ! make $make_jobs -C "$build_dir" $make_targets; then + err "Make failed" + popd >/dev/null + return 1 + fi + + popd >/dev/null + ok "CMake build completed" + return 0 +} + +# Build with Meson +build_meson() { + local repo_dir="$1" build_dir="$2" meson_flags="$3" ninja_targets="$4" + + pushd "$repo_dir" >/dev/null || return 1 + + step "Running meson setup" + if ! eval "meson setup $build_dir $meson_flags"; then + err "Meson setup failed" + popd >/dev/null + return 1 + fi + + step "Running ninja -C $build_dir $ninja_targets" + if ! ninja -C "$build_dir" $ninja_targets; then + err "Ninja build failed" + popd >/dev/null + return 1 + fi + + popd >/dev/null + ok "Meson build completed" + return 0 +} + +# Execute custom commands +execute_commands() { + local repo_dir="$1" config_file="$2" index="$3" + + pushd "$repo_dir" >/dev/null || return 1 + + local cmd_count + cmd_count=$(jq ".dependencies.repos[$index].build.commands | length" "$config_file") + + local i=0 + while [[ $i -lt $cmd_count ]]; do + local cmd + cmd=$(jq -r ".dependencies.repos[$index].build.commands[$i]" "$config_file") + step "Executing: $cmd" + if ! eval "$cmd"; then + err "Command failed: $cmd" + popd >/dev/null + return 1 + fi + i=$((i + 1)) + done + + popd >/dev/null + ok "Commands executed successfully" + return 0 +} + +# Copy shared libraries to destination +copy_libraries() { + local src_dir="$1" dst_dir="$2" + + dst_dir=$(expand_path "$dst_dir") + mkdir -p "$dst_dir" + + log "Copying libraries to $dst_dir" + find "$src_dir" \( -name "*.so*" -o -name "*.a" -o -name "*.la*" \) \( -type f -o -type l \) -exec cp -Pv {} "$dst_dir/" \; 2>/dev/null || true +} + +# Print banner +print_banner() { + local title="$1" + echo "" + echo -e "${BLUE}================================================${NC}" + echo -e "${BLUE} $title${NC}" + echo -e "${BLUE}================================================${NC}" + echo "" +} + +# Print section header +print_section() { + local title="$1" + echo "" + echo -e "${BLUE}------------------------------------------------${NC}" + echo -e "${BOLD}${CYAN}β–Ά $title${NC}" + echo -e "${BLUE}------------------------------------------------${NC}" +} + +# Export this file's functions +export -f log ok warn err step +export -f expand_path check_dependencies clone_repo copy_headers apply_patch +export -f build_autotools build_cmake build_meson execute_commands copy_libraries +export -f print_banner print_section diff --git a/cov_docker_script/common_external_build.sh b/cov_docker_script/common_external_build.sh new file mode 100755 index 0000000..3826f78 --- /dev/null +++ b/cov_docker_script/common_external_build.sh @@ -0,0 +1,92 @@ +#!/usr/bin/env bash +set -e + +################################################################################ +# Common External Build Script +# Orchestrates the complete build process: dependencies + native component +# Usage: ./common_external_build.sh [config_file] [component_dir] +################################################################################ + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CONFIG_FILE="${1:-$SCRIPT_DIR/component_config.json}" +COMPONENT_DIR="${2:-$(cd "$SCRIPT_DIR/.." && pwd)}" + +# Source common utilities +source "$SCRIPT_DIR/common_build_utils.sh" + +# Validate inputs +if [[ ! -f "$CONFIG_FILE" ]]; then + err "Config file not found: $CONFIG_FILE" + exit 1 +fi + +if [[ ! -d "$COMPONENT_DIR" ]]; then + err "Component directory not found: $COMPONENT_DIR" + exit 1 +fi + +# Get component name from config +COMPONENT_NAME=$(jq -r '.native_component.name' "$CONFIG_FILE") + +# Print main banner +echo "" +echo -e "${BOLD}${BLUE}================================================================${NC}" +echo -e "${BOLD}${BLUE} Complete Build Pipeline for: ${COMPONENT_NAME}${NC}" +echo -e "${BOLD}${BLUE}================================================================${NC}" +echo "" +log "Configuration: $CONFIG_FILE" +log "Component directory: $COMPONENT_DIR" +echo "" + +# Step 1: Setup Dependencies +print_banner "Step 1/2: Setting Up Dependencies" +log "Running dependency setup script..." +echo "" + +if ! "$SCRIPT_DIR/setup_dependencies.sh" "$CONFIG_FILE"; then + err "Dependency setup failed" + exit 1 +fi + +echo "" +ok "Dependencies setup completed successfully" +echo "" + +# Step 2: Build Native Component +print_banner "Step 2/2: Building Native Component" +log "Running native component build script..." +echo "" + +if ! "$SCRIPT_DIR/build_native.sh" "$CONFIG_FILE" "$COMPONENT_DIR"; then + err "Native component build failed" + exit 1 +fi + +echo "" +ok "Native component build completed successfully" +echo "" + +# Final summary +echo "" +echo -e "${BOLD}${GREEN}================================================================${NC}" +echo -e "${BOLD}${GREEN} Complete Build Pipeline Completed Successfully!${NC}" +echo -e "${BOLD}${GREEN}================================================================${NC}" +echo "" +log "Component: ${BOLD}$COMPONENT_NAME${NC}" +log "All dependencies built and installed" +log "Native component compiled successfully" +echo "" + +# Display installation paths +HEADER_PATH=$(jq -r '.native_component.include_path' "$CONFIG_FILE") +LIB_PATH=$(jq -r '.native_component.lib_output_path' "$CONFIG_FILE") +HEADER_PATH="${HEADER_PATH//\$HOME/$HOME}" +LIB_PATH="${LIB_PATH//\$HOME/$HOME}" + +echo -e "${CYAN}Installation Locations:${NC}" +log " Headers: $HEADER_PATH" +log " Libraries: $LIB_PATH" +echo "" + +echo -e "${GREEN}βœ“ Ready for Coverity analysis or deployment${NC}" +echo "" diff --git a/cov_docker_script/setup_dependencies.sh b/cov_docker_script/setup_dependencies.sh new file mode 100755 index 0000000..a4835f8 --- /dev/null +++ b/cov_docker_script/setup_dependencies.sh @@ -0,0 +1,241 @@ +#!/usr/bin/env bash +set -e + +################################################################################ +# Generic Dependency Setup Script +# Usage: ./setup_dependencies.sh [config_file] +################################################################################ + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CONFIG_FILE="${1:-$SCRIPT_DIR/component_config.json}" + +# Source common utilities +source "$SCRIPT_DIR/common_build_utils.sh" + +# Default directories +BUILD_DIR="${BUILD_DIR:-$HOME/build}" +USR_DIR="${USR_DIR:-$HOME/usr}" + +# Validate environment +if [[ ! -f "$CONFIG_FILE" ]]; then + err "Config file not found: $CONFIG_FILE" + exit 1 +fi + +check_dependencies || exit 1 + +# Initialize environment +initialize_environment() { + print_banner "Dependency Setup" + + log "Configuration: $CONFIG_FILE" + log "Build directory: $BUILD_DIR" + log "Install directory: $USR_DIR" + echo "" + + # Clean if requested + if [[ "${CLEAN_BUILD:-false}" == "true" ]]; then + warn "Cleaning previous build artifacts" + [[ -d "$BUILD_DIR" ]] && rm -rf "$BUILD_DIR" + [[ -d "$USR_DIR" ]] && rm -rf "$USR_DIR" + fi + + # Create directories + mkdir -p "$BUILD_DIR" + mkdir -p "$USR_DIR/include/rdkb" + mkdir -p "$USR_DIR/local/lib" + mkdir -p "$USR_DIR/local/lib/pkgconfig" + mkdir -p "$USR_DIR/lib" + + # Setup PKG_CONFIG_PATH and LD_LIBRARY_PATH for dependencies + export PKG_CONFIG_PATH="$USR_DIR/local/lib/pkgconfig:$USR_DIR/lib/pkgconfig:${PKG_CONFIG_PATH:-}" + export LD_LIBRARY_PATH="$USR_DIR/local/lib:$USR_DIR/lib:${LD_LIBRARY_PATH:-}" + export CPPFLAGS="${CPPFLAGS:-} -I$USR_DIR/include" + export LDFLAGS="${LDFLAGS:-} -L$USR_DIR/local/lib" + + log "PKG_CONFIG_PATH=$PKG_CONFIG_PATH" + log "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" + echo "" + + ok "Environment initialized" +} + +# Process header paths for a repository +process_headers() { + local index="$1" + local repo_dir="$2" + local name="$3" + + local count + count=$(jq ".dependencies.repos[$index].header_paths | length" "$CONFIG_FILE") + + if [[ "$count" -eq 0 ]]; then + log "No headers configured for $name" + return 0 + fi + + local i=0 + while [[ $i -lt $count ]]; do + local src dst + src=$(jq -r ".dependencies.repos[$index].header_paths[$i].source" "$CONFIG_FILE") + dst=$(jq -r ".dependencies.repos[$index].header_paths[$i].destination" "$CONFIG_FILE") + + copy_headers "$repo_dir/$src" "$dst" + i=$((i + 1)) + done + + return 0 +} + +# Build a repository +build_repository() { + local index="$1" + local repo_dir="$2" + local name="$3" + + local build_type + build_type=$(jq -r ".dependencies.repos[$index].build.type // empty" "$CONFIG_FILE") + + if [[ -z "$build_type" ]]; then + log "No build configuration for $name (headers only)" + return 0 + fi + + step "Building $name (type: $build_type)" + + case "$build_type" in + autotools) + local configure_flags make_targets parallel_make + configure_flags=$(jq -r ".dependencies.repos[$index].build.configure_flags // empty" "$CONFIG_FILE") + make_targets=$(jq -r ".dependencies.repos[$index].build.make_targets[]? // \"all\"" "$CONFIG_FILE" | tr '\n' ' ') + parallel_make=$(jq -r ".dependencies.repos[$index].build.parallel_make // true" "$CONFIG_FILE") + + build_autotools "$repo_dir" "$configure_flags" "$make_targets" "$parallel_make" || return 1 + ;; + + cmake) + local build_dir cmake_flags make_targets parallel_make + build_dir=$(jq -r ".dependencies.repos[$index].build.build_dir // \"build\"" "$CONFIG_FILE") + cmake_flags=$(jq -r ".dependencies.repos[$index].build.cmake_flags // empty" "$CONFIG_FILE") + make_targets=$(jq -r ".dependencies.repos[$index].build.make_targets[]? // \"all\"" "$CONFIG_FILE" | tr '\n' ' ') + parallel_make=$(jq -r ".dependencies.repos[$index].build.parallel_make // true" "$CONFIG_FILE") + + build_cmake "$repo_dir" "$build_dir" "$cmake_flags" "$make_targets" "$parallel_make" || return 1 + ;; + + meson) + local build_dir meson_flags ninja_targets + build_dir=$(jq -r ".dependencies.repos[$index].build.build_dir // \"builddir\"" "$CONFIG_FILE") + meson_flags=$(jq -r ".dependencies.repos[$index].build.meson_flags // empty" "$CONFIG_FILE") + ninja_targets=$(jq -r ".dependencies.repos[$index].build.ninja_targets[]? // \"all\"" "$CONFIG_FILE" | tr '\n' ' ') + + build_meson "$repo_dir" "$build_dir" "$meson_flags" "$ninja_targets" || return 1 + ;; + + commands) + execute_commands "$repo_dir" "$CONFIG_FILE" "$index" || return 1 + ;; + + script) + local script_path + script_path=$(jq -r ".dependencies.repos[$index].build.script" "$CONFIG_FILE") + local full_script="$repo_dir/$script_path" + + if [[ -f "$full_script" ]]; then + step "Executing build script: $script_path" + chmod +x "$full_script" + + export PARENT_BUILD_DIR="$BUILD_DIR" + export PARENT_USR_DIR="$USR_DIR" + + pushd "$repo_dir" >/dev/null || return 1 + if ! "$full_script"; then + err "Build script failed" + popd >/dev/null + return 1 + fi + popd >/dev/null + + unset PARENT_BUILD_DIR PARENT_USR_DIR + else + err "Build script not found: $full_script" + return 1 + fi + ;; + + *) + err "Unknown build type: $build_type" + return 1 + ;; + esac + + # Copy libraries + copy_libraries "$repo_dir" "$USR_DIR/local/lib" + copy_libraries "$repo_dir" "$USR_DIR/lib" + + ok "$name build completed" + return 0 +} + +# Process a single dependency +process_dependency() { + local index="$1" + + local name repo branch + name=$(jq -r ".dependencies.repos[$index].name" "$CONFIG_FILE") + repo=$(jq -r ".dependencies.repos[$index].repo" "$CONFIG_FILE") + branch=$(jq -r ".dependencies.repos[$index].branch" "$CONFIG_FILE") + + local repo_dir="$BUILD_DIR/$name" + + print_section "Processing: $name" + + # Clone repository + if ! clone_repo "$name" "$repo" "$branch" "$repo_dir"; then + err "Failed to process $name" + return 1 + fi + + # Copy headers + if ! process_headers "$index" "$repo_dir" "$name"; then + err "Failed to copy headers for $name" + return 1 + fi + + # Build if needed + if ! build_repository "$index" "$repo_dir" "$name"; then + err "Failed to build $name" + return 1 + fi + + ok "$name processed successfully" + return 0 +} + +# Main execution +main() { + initialize_environment + + local count + count=$(jq ".dependencies.repos | length" "$CONFIG_FILE") + + log "Found $count dependencies to process" + echo "" + + local i=0 + while [[ $i -lt $count ]]; do + if ! process_dependency "$i"; then + err "Dependency setup failed" + exit 1 + fi + i=$((i + 1)) + done + + echo "" + print_banner "Dependencies Setup Completed Successfully" + log "Headers installed: $USR_DIR/include/rdkb" + log "Libraries installed: $USR_DIR/local/lib and $USR_DIR/lib" + echo "" +} + +main