Skip to content

Conversation

@Abhinavpv28
Copy link
Contributor

…ts To C Implementation (#35)

  • Created a comprehensive L2 test suite with 6 test modules covering error handling, normal uploads, retry logic, security, resource management, and upload strategies Updated build scripts and CI/CD workflow to compile with L2_TEST_ENABLED flag and execute the new test suite

This feature file contains scenarios for normal log uploads, large log file handling, and MD5 checksum verification for the uploadSTBLogs service.

shibu-kv and others added 3 commits February 10, 2025 06:57
…ts To C Implementation (#35)

* Created a comprehensive L2 test suite with 6 test modules covering error handling, normal uploads, retry logic, security, resource management, and upload strategies
Updated build scripts and CI/CD workflow to compile with L2_TEST_ENABLED flag and execute the new test suite

This feature file contains scenarios for normal log uploads, large log file handling, and MD5 checksum verification for the uploadSTBLogs service.
Copilot AI review requested due to automatic review settings January 7, 2026 12:59
@Abhinavpv28 Abhinavpv28 requested a review from a team as a code owner January 7, 2026 12:59
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates operation support log upload scripts from shell to C implementation and adds a comprehensive L2 test suite with 6 test modules covering error handling, normal uploads, retry logic, security, resource management, and upload strategies. The implementation includes conditional compilation for test environments and updates to build scripts and CI/CD workflows.

Key Changes

  • Added L2_TEST_ENABLED conditional compilation to skip sleep() calls in test environments for faster test execution
  • Created comprehensive L2 test infrastructure with 6 pytest modules and corresponding BDD feature files
  • Updated build scripts (cov_build.sh) and CI/CD workflow to compile with test flags and execute the new test suite

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 82 comments.

Show a summary per file
File Description
uploadstblogs/src/strategy_reboot.c Added conditional compilation guards around sleep() calls to skip delays during L2 testing
test/run_uploadstblogs_l2.sh New test runner script that sets up test environment and executes all 6 L2 test suites
test/functional-tests/tests/uploadstblogs_helper.py New helper module with 260 lines of utility functions for test setup, log analysis, and resource management
test/functional-tests/tests/test_uploadstblogs_upload_strategies.py Test module for upload strategies (on-demand, reboot, DCM scheduled, RBUS integration)
test/functional-tests/tests/test_uploadstblogs_security.py Security test module covering mTLS authentication, SSL validation, and path security
test/functional-tests/tests/test_uploadstblogs_retry_logic.py Test module for network failure handling, retry attempts, and server error responses
test/functional-tests/tests/test_uploadstblogs_resource_management.py Resource management tests covering cleanup, memory management, and concurrent requests
test/functional-tests/tests/test_uploadstblogs_normal_upload.py Test module for normal upload operations and large file handling
test/functional-tests/tests/test_uploadstblogs_error_handling.py Error handling tests for invalid configuration, size limits, and empty logs
test/functional-tests/features/*.feature BDD feature files corresponding to each test module
cov_build.sh Updated to clone feature/upload_L2 branch and compile with L2_TEST_ENABLED flag
.github/workflows/L2-tests.yml Updated CI/CD workflow to execute uploadstblogs L2 tests

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@@ -0,0 +1,86 @@
####################################################################################
# If not stated otherwise in this file or this component's Licenses
Copy link

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incomplete copyright header - line 2 is missing "file the" text. The header should read "If not stated otherwise in this file or this component's Licenses file the" but it currently says "If not stated otherwise in this file or this component's Licenses"

Add the missing text to match the standard copyright header format used in other files.

Suggested change
# If not stated otherwise in this file or this component's Licenses
# If not stated otherwise in this file or this component's Licenses file the

Copilot uses AI. Check for mistakes.
@@ -0,0 +1,48 @@
####################################################################################
# If not stated otherwise in this file or this component's Licenses
Copy link

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incomplete copyright header - line 2 is missing "file the" text. The header should read "If not stated otherwise in this file or this component's Licenses file the" but it currently says "If not stated otherwise in this file or this component's Licenses"

Add the missing text to match the standard copyright header format used in other files.

Suggested change
# If not stated otherwise in this file or this component's Licenses
# If not stated otherwise in this file or this component's Licenses file the

Copilot uses AI. Check for mistakes.
cd telemetry
cp include/*.h /usr/local/include
sh build_inside_container.sh
sh build_inside_container.sh
Copy link

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trailing whitespace at the end of the line. This should be removed for cleaner code formatting.

Copilot uses AI. Check for mistakes.
Comment on lines +55 to +63
cd /usr/common_utilities
sed -i '/file_upload\.sslverify/s/= 1;/= 0;/' uploadutils/mtls_upload.c
sed -i 's/\(ret_code = setCommonCurlOpt(curl, s3url, NULL, \)true\()\)/\1false\2/g' uploadutils/uploadUtil.c
sed -i '/if (auth) {/,/}/s/^/\/\/ /' uploadutils/uploadUtil.c
cd -

echo pwd

# Create log directories
Copy link

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These sed commands modify source code in a different repository (common_utilities) during test execution. This is a security and maintainability issue because:

  1. It modifies source code outside this repository at runtime, making the test behavior unpredictable and dependent on external code structure
  2. The modifications disable SSL verification and authentication, which are critical security features
  3. If the external code structure changes, these sed commands could silently fail or produce incorrect modifications
  4. This makes it difficult to reproduce test results consistently

Consider using proper mocking/stubbing mechanisms or test-specific build configurations instead of runtime source code modifications.

Suggested change
cd /usr/common_utilities
sed -i '/file_upload\.sslverify/s/= 1;/= 0;/' uploadutils/mtls_upload.c
sed -i 's/\(ret_code = setCommonCurlOpt(curl, s3url, NULL, \)true\()\)/\1false\2/g' uploadutils/uploadUtil.c
sed -i '/if (auth) {/,/}/s/^/\/\/ /' uploadutils/uploadUtil.c
cd -
echo pwd
# Create log directories
# NOTE: Do not modify external common_utilities source here.
# Any SSL/authentication behavior changes for tests should be handled via
# configuration, mocking, or test-specific builds, not in-place sed edits.
echo pwd
# Create log directories
# Create log directories

Copilot uses AI. Check for mistakes.
for chunk in iter(lambda: f.read(4096), b""):
md5_hash.update(chunk)
return md5_hash.hexdigest()
except:
Copy link

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bare except clause catches all exceptions including system exits and keyboard interrupts. This makes debugging difficult and can hide serious issues.

Replace with a more specific exception handler like except (OSError, IOError): or except Exception as e: to allow system exceptions to propagate.

Suggested change
except:
except OSError:

Copilot uses AI. Check for mistakes.
import time
import subprocess as sp
from uploadstblogs_helper import *
from helper_functions import *
Copy link

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import pollutes the enclosing namespace, as the imported module helper_functions does not define 'all'.

Copilot uses AI. Check for mistakes.
"""

import pytest
import time
Copy link

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import of 'time' is not used.

Copilot uses AI. Check for mistakes.
"""

import pytest
import time
Copy link

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import of 'time' is not used.

Copilot uses AI. Check for mistakes.
"""

import pytest
import time
Copy link

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import of 'time' is not used.

Copilot uses AI. Check for mistakes.
import os
import time
import re
import json
Copy link

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import of 'json' is not used.

Copilot uses AI. Check for mistakes.
Abhinavpv28 and others added 2 commits January 12, 2026 13:40
…ts To C Implementation (#49)

* Simplify some pieces of code over engineered by AI models.
* Update run_uploadstblogs_l2.sh for L2 coverage

---------

Co-authored-by: Abhinav P V <Abhinav_Valappil@comcast.com>
…ts To C Implementation (#51)

* Remove logging of response URLs

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Shibu Kakkoth Vayalambron <shibu.kakkoth@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings January 13, 2026 18:09
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Comment on lines +85 to 99
if (ctx->rrd_flag == 0) {
// Check PREV_LOG_PATH - critical for upload (matches script behavior)
if (strlen(ctx->paths.prev_log_path) > 0) {
if (!dir_exists(ctx->paths.prev_log_path)) {
RDK_LOG(RDK_LOG_ERROR, LOG_UPLOADSTB, "[%s:%d] The Previous Logs folder is missing: %s\n",
__FUNCTION__, __LINE__, ctx->paths.prev_log_path);
if (strlen(ctx->prev_log_path) > 0) {
if (!dir_exists(ctx->prev_log_path)) {
RDK_LOG(RDK_LOG_ERROR, LOG_UPLOADSTB, "[%s:%d] The Previous Logs folder is missing: %s\n",
__FUNCTION__, __LINE__, ctx->prev_log_path);
// Script sends MAINT_LOGUPLOAD_ERROR=5 when PREV_LOG_PATH is missing
emit_folder_missing_error();
all_valid = false;
} else {
RDK_LOG(RDK_LOG_DEBUG, LOG_UPLOADSTB, "[%s:%d] PREV_LOG_PATH exists: %s\n",
__FUNCTION__, __LINE__, ctx->paths.prev_log_path);
emit_folder_missing_error();
all_valid = false;
} else {
RDK_LOG(RDK_LOG_DEBUG, LOG_UPLOADSTB, "[%s:%d] PREV_LOG_PATH exists: %s\n",
__FUNCTION__, __LINE__, ctx->prev_log_path);
}
}
}
Copy link

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The structure flattening from nested structs (ctx.paths.log_path) to flat fields (ctx.log_path) is inconsistently applied. In validation.c line 85, there's a condition if (ctx->rrd_flag == 0) wrapping the PREV_LOG_PATH validation. This condition was not present in the original code and changes the validation logic. The PREV_LOG_PATH validation should run unconditionally as it did before, not only when rrd_flag is 0.

Copilot uses AI. Check for mistakes.
result = run_uploadstblogs()

# Check for telemetry markers
telemetry_logs = grep_uploadstb_logs_regex(r"telemetry|marker|warning")
Copy link

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable telemetry_logs is not used.

Suggested change
telemetry_logs = grep_uploadstb_logs_regex(r"telemetry|marker|warning")
telemetry_logs = grep_uploadstb_logs_regex(r"telemetry|marker|warning")
assert telemetry_logs, "Expected telemetry markers in uploadSTB logs"

Copilot uses AI. Check for mistakes.
result = run_uploadstblogs()

# Check for informative message
logs = grep_uploadstb_logs_regex(r"no.*file|empty|nothing to upload")
Copy link

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable logs is not used.

Copilot uses AI. Check for mistakes.

# Should not attempt upload
upload_logs = grep_uploadstb_logs_regex(r"upload.*success|uploading|HTTP")
# Might not see upload attempts
Copy link

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable upload_logs is not used.

Suggested change
# Might not see upload attempts
# Verify that no upload-related log entries are present
assert len(upload_logs) == 0, "Should not attempt upload when no log files are available"

Copilot uses AI. Check for mistakes.
…ts To C Implementation (#54)

* Update cov_build.sh to enable the rdkcertselector feature in the common_utilities build configuration to support the migration of operation support log upload scripts to C implementation. 
The change is part of resolving L2 test failures.
Copilot AI review requested due to automatic review settings January 21, 2026 18:58
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 61 out of 62 changed files in this pull request and generated 26 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

# Check for configuration error detection
error_logs = grep_uploadstb_logs_regex(r"ERROR|error|fail.*properties|invalid")
# Should handle gracefully
assert result.returncode in [0, 1], "Should detect corrupted config"
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable error_logs is not used.

Suggested change
assert result.returncode in [0, 1], "Should detect corrupted config"
assert result.returncode in [0, 1], "Should detect corrupted config"
assert len(error_logs) > 0 or result.returncode != 0, "Configuration errors should be logged"

Copilot uses AI. Check for mistakes.
result = run_uploadstblogs()

# Check for size-related logs
logs = grep_uploadstb_logs_regex(r"size|large|limit|truncate")
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable logs is not used.

Suggested change
logs = grep_uploadstb_logs_regex(r"size|large|limit|truncate")
grep_uploadstb_logs_regex(r"size|large|limit|truncate")

Copilot uses AI. Check for mistakes.

# Should not attempt upload
upload_logs = grep_uploadstb_logs_regex(r"upload.*success|uploading|HTTP")
# Might not see upload attempts
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable upload_logs is not used.

Suggested change
# Might not see upload attempts
# Ensure no upload-related entries are present in the logs
assert len(upload_logs) == 0, "No upload logs should be present when there are no files"
# Process should still handle the no-files case gracefully

Copilot uses AI. Check for mistakes.
Comment on lines +86 to +90
assert len(archive_logs) > 0, "Archive process should complete. Found {len(archive_logs)} archive-related logs: {archive_logs}"

upload_logs = grep_uploadstb_logs_regex(r"upload.*success|uploading|HTTP")
# Telemetry should be attempted
assert len(archive_logs) > 0, "Upload Process should complete and succeed"
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable upload_logs is not used.

Suggested change
assert len(archive_logs) > 0, "Archive process should complete. Found {len(archive_logs)} archive-related logs: {archive_logs}"
upload_logs = grep_uploadstb_logs_regex(r"upload.*success|uploading|HTTP")
# Telemetry should be attempted
assert len(archive_logs) > 0, "Upload Process should complete and succeed"
assert len(archive_logs) > 0, (
f"Archive process should complete. Found {len(archive_logs)} archive-related logs: {archive_logs}"
)
upload_logs = grep_uploadstb_logs_regex(r"upload.*success|uploading|HTTP")
# Telemetry should be attempted
assert len(upload_logs) > 0, (
f"Upload process should complete and succeed. "
f"Found {len(upload_logs)} upload-related logs: {upload_logs}"
)

Copilot uses AI. Check for mistakes.
# Verify files were processed

# Check for compression/archive activity
compression_logs = grep_uploadstb_logs_regex(r"compress|archive|tgz")
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable compression_logs is not used.

Suggested change
compression_logs = grep_uploadstb_logs_regex(r"compress|archive|tgz")

Copilot uses AI. Check for mistakes.
# Check for configuration loading
config_logs = grep_uploadstb_logs_regex(r"load.*TR-181|load.*param|endpoint|RFC")
# Should attempt to load config
assert result.returncode in [0, 1], "Should load RBUS configuration"
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable config_logs is not used.

Suggested change
assert result.returncode in [0, 1], "Should load RBUS configuration"
assert result.returncode in [0, 1], "Should load RBUS configuration"
assert config_logs, "RBUS configuration logs should be present"

Copilot uses AI. Check for mistakes.
try:
subprocess.run(f"echo '' > {UPLOADSTB_LOG}", shell=True)
return True
except:
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except block directly handles BaseException.

Copilot uses AI. Check for mistakes.
if os.path.exists(LOCK_FILE):
os.remove(LOCK_FILE)
return True
except:
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except block directly handles BaseException.

Copilot uses AI. Check for mistakes.
for chunk in iter(lambda: f.read(4096), b""):
md5_hash.update(chunk)
return md5_hash.hexdigest()
except:
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except block directly handles BaseException.

Copilot uses AI. Check for mistakes.
"""Get file size in bytes"""
try:
return os.path.getsize(filepath)
except:
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except block directly handles BaseException.

Copilot uses AI. Check for mistakes.
Copilot AI review requested due to automatic review settings February 3, 2026 08:55
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 62 out of 63 changed files in this pull request and generated 31 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

result = run_uploadstblogs()

# Check for size limit detection
size_logs = grep_uploadstb_logs_regex(r"size.*limit|too large|exceed")
Copy link

Copilot AI Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable size_logs is not used.

Suggested change
size_logs = grep_uploadstb_logs_regex(r"size.*limit|too large|exceed")
size_logs = grep_uploadstb_logs_regex(r"size.*limit|too large|exceed")
assert size_logs, "Expected size-related log entries for oversized file"

Copilot uses AI. Check for mistakes.
result = run_uploadstblogs()

# Check for no files detection
no_files_logs = grep_uploadstb_logs_regex(r"no.*file|empty|not found|no logs")
Copy link

Copilot AI Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable no_files_logs is not used.

Copilot uses AI. Check for mistakes.
telemetry_logs = grep_uploadstb_logs_regex(r"telemetry|marker|event")
# Process should complete
assert result.returncode in [0, 1], "Should handle empty logs with telemetry"

Copy link

Copilot AI Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable telemetry_logs is not used.

Suggested change
# Telemetry entries should be present
assert len(telemetry_logs) > 0, "Expected telemetry markers for empty logs"

Copilot uses AI. Check for mistakes.

create_test_log_files(count=2)

result = run_uploadstblogs()
Copy link

Copilot AI Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable result is not used.

Suggested change
result = run_uploadstblogs()
run_uploadstblogs()

Copilot uses AI. Check for mistakes.

create_test_log_files(count=1)

result = run_uploadstblogs()
Copy link

Copilot AI Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable result is not used.

Copilot uses AI. Check for mistakes.
"""

import pytest
import time
Copy link

Copilot AI Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import of 'time' is not used.

Suggested change
import time

Copilot uses AI. Check for mistakes.
"""

import pytest
import time
Copy link

Copilot AI Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import of 'time' is not used.

Suggested change
import time

Copilot uses AI. Check for mistakes.
import os
import time
import re
import json
Copy link

Copilot AI Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import of 'json' is not used.

Suggested change
import json

Copilot uses AI. Check for mistakes.
for chunk in iter(lambda: f.read(4096), b""):
md5_hash.update(chunk)
return md5_hash.hexdigest()
except:
Copy link

Copilot AI Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except block directly handles BaseException.

Suggested change
except:
except Exception:

Copilot uses AI. Check for mistakes.
"""Get file size in bytes"""
try:
return os.path.getsize(filepath)
except:
Copy link

Copilot AI Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except block directly handles BaseException.

Suggested change
except:
except OSError:

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants