[pull] main from intelowlproject:main #25
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* feat: make feed license configurable via environment variable - Move FEEDS_LICENSE from hardcoded constant to optional environment variable - Update settings.py to read FEEDS_LICENSE from environment - Add FEEDS_LICENSE configuration to env_file_template with example - Modify API views to only include license field when FEEDS_LICENSE is set - Update tests to handle both scenarios (with/without license configured) - This allows self-hosted instances to use different licenses or none at all Fixes #599 * fix: update FEEDS_LICENSE import in command_sequence and cowrie_session views * refactor: simplify license handling to use FEEDS_LICENSE constant directly * test: add explicit tests for FEEDS_LICENSE populated and empty scenarios * fix: use settings.FEEDS_LICENSE instead of direct import for @override_settings compatibility The @override_settings decorator only works when accessing settings through django.conf.settings, not with direct imports. This fixes test failures where FEEDS_LICENSE was imported directly from greedybear.settings. Changes: - api/views/utils.py: Import settings and use settings.FEEDS_LICENSE - api/views/command_sequence.py: Import settings and use settings.FEEDS_LICENSE - api/views/cowrie_session.py: Import settings and use settings.FEEDS_LICENSE - tests/test_views.py: Import settings and use settings.FEEDS_LICENSE This ensures tests with @override_settings(FEEDS_LICENSE="...") work correctly.
* feat: add IOC type filter to Feeds API and page - Add optional ioc_type parameter (ip/domain/all) to FeedRequestParams - Update FeedsRequestSerializer with ioc_type field validation - Modify get_queryset to filter IOCs by type when ioc_type is specified - Add IOC type dropdown in Feeds page UI with three options - Update frontend to include ioc_type in API calls and URL generation - Add test data with domain IOC for comprehensive testing - Add test cases for IP-only, domain-only, and all IOC type filters This enhancement allows users to filter feeds specifically by IP addresses or domains, making it easier to showcase domains extracted from payload requests separately from IP addresses. Closes #551 * fix: wrap Prioritize field in Col component to fix JSX syntax error * test: add frontend tests for IOC type filter * Fix test_valid_fields by adding missing required ioc_type field * Fix failing tests and bugs: Update assertions, fix feeds filtering, improve validation 1. Synced Tests with Existing Test Data: Updated assertions to expect 3 IOCs for Heralding instead of 2, matching the setupTestData. Refactored tests to find IOCs by value. 2. Fixed Feed Filtering Bug: Updated feeds view to correctly pass query parameters (like ioc_type) to FeedRequestParams. 3. Improved Input Validation: Added check in serializers to reject invalid IP strings that were being accepted as domains. * refactor: adjust form column widths and consolidate form groups in the Feeds component.
* Refactor: Externalize Random Forest Hyperparameters to Settings #614 * refactor: rename ML_CONFIG_PATH to ML_CONFIG_FILE * refactor: Replace `ML_CONFIG_PATH` with `ML_CONFIG_FILE` for consistency. * test: add tests for externalized random forest configuration
- Enforce non-negative predictions in Regressor.predict using np.maximum - Add test case in TestRegressor to verify negative values are clipped - Fixes issue with negative expected_interactions causing API errors
* add repositories for abstraction of data access * add tests for repositories * update constant with fields required for extraction * delete file with tests for time windows calculation (this is now tested together with the elastic repository) * add class to process incoming IOC records * collect utility functions in a separate file * add test fixtures for new tests * adapt existing tests * add tests for IOC processor * add tests for extraction utils * rename argument * add extraction strategies * add tests for extraction strategies * add pipeline class for orchestration of the extraction process with configurable initial extraction interval * add extraction job * replace old extraction job with new one in the scheduler * remove old files * remove unused ElasticJob class * remove obsolete test * fix broken f-string * catch and log failing extractions * improve logging in ExtractionPipeline * add description of the extraction process
…643) * Refactor: Rename models to singular form and fix admin registration * Fix: Remove trailing whitespace in sensor.py * Revert: Remove configurable log directory setting
Bumps library/nginx from 1.29.3-alpine to 1.29.4-alpine. --- updated-dependencies: - dependency-name: library/nginx dependency-version: 1.29.4-alpine dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [celery](https://github.com/celery/celery) from 5.5.3 to 5.6.0. - [Release notes](https://github.com/celery/celery/releases) - [Changelog](https://github.com/celery/celery/blob/main/Changelog.rst) - [Commits](celery/celery@v5.5.3...v5.6.0) --- updated-dependencies: - dependency-name: celery dependency-version: 5.6.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [slack-sdk](https://github.com/slackapi/python-slack-sdk) from 3.37.0 to 3.39.0. - [Release notes](https://github.com/slackapi/python-slack-sdk/releases) - [Commits](slackapi/python-slack-sdk@v3.37.0...v3.39.0) --- updated-dependencies: - dependency-name: slack-sdk dependency-version: 3.39.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [datasketch](https://github.com/ekzhu/datasketch) from 1.7.0 to 1.8.0. - [Release notes](https://github.com/ekzhu/datasketch/releases) - [Commits](ekzhu/datasketch@v1.7.0...v1.8.0) --- updated-dependencies: - dependency-name: datasketch dependency-version: 1.8.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [scikit-learn](https://github.com/scikit-learn/scikit-learn) from 1.7.2 to 1.8.0. - [Release notes](https://github.com/scikit-learn/scikit-learn/releases) - [Commits](scikit-learn/scikit-learn@1.7.2...1.8.0) --- updated-dependencies: - dependency-name: scikit-learn dependency-version: 1.8.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [django-ses](https://github.com/django-ses/django-ses) from 4.4.0 to 4.5.0. - [Release notes](https://github.com/django-ses/django-ses/releases) - [Changelog](https://github.com/django-ses/django-ses/blob/main/CHANGES.md) - [Commits](django-ses/django-ses@v4.4.0...v4.5.0) --- updated-dependencies: - dependency-name: django-ses dependency-version: 4.5.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Refactor CowrieExtractionStrategy and add comprehensive tests * refactor: address code review feedback - Consolidate cowrie_parser.py into cowrie.py - Remove CowrieSessionData class, use models directly - Add helper functions and comprehensive tests (25 tests) - Use defaultdict, removeprefix, and clean up comments All 265 tests passing * refactor: address additional code review feedback - Add warning log when IOCs missing in _add_fks - Remove redundant duplicate checks (Django handles it) - Remove redundant scanner tracking in _get_url_downloads - Optimize payload extraction from O(N*M) to O(M) - Document 1024 char truncation reason - Add test for empty hits edge case All 266 tests passing
* feat: add auto extraction of FireHol lists for classification purposes * feat: integrate FireHol lists with API and admin - Add Celery beat schedule for weekly FireHol extraction - Register FireHolList model in Django admin - Expose firehol_categories field in Feeds API responses - Add firehol_categories to IOC admin list display - Improve error handling with specific exception types - Simplify verbose comments in firehol.py - Merge conflicting migrations from develop branch - Update serializer tests for new firehol_categories field All 267 tests passing * refactor: address code review feedback - Extract base_path variable for FireHol URLs - Narrow RequestException scope to only wrap network call - Use raise_for_status() for cleaner HTTP error handling - Only enrich recently added IOCs (within 24h) instead of all existing ones - Add cleanup routine to delete FireHolList entries older than 30 days - Update tests to match new enrichment behavior All 267 tests passing * Move FireHol enrichment to IOC creation and add CIDR support - Move FireHol category enrichment from separate job step to iocs_from_hits() where IOCs are created, ensuring only fresh data is applied at extraction time - Add support for CIDR network ranges (netsets) using ipaddress library - Remove _enrich_recent_iocs() method as enrichment now happens at IOC creation - Update enrichment logic to handle both exact IP matches (.ipset) and network range membership (.netset) for proper dshield.netset support - Update test to reflect new behavior where FireHolCron only downloads data, enrichment happens automatically during IOC creation * Add comprehensive tests for FireHol enrichment in iocs_from_hits - Test exact IP match enrichment (for .ipset files) - Test CIDR network range match enrichment (for .netset files) - Test no match scenario returns empty categories - Test mixed exact and network range matches - Test deduplication of FireHol sources * Refactor: Extract FireHol enrichment logic into separate function - Move FireHol category lookup logic to dedicated get_firehol_categories() function - Simplify iocs_from_hits() to single-line call as requested - Improves code readability and reduces function complexity * Fix migration conflicts: Renumber to 0024 and 0025 - Renamed 0023_ioc_firehol_categories to 0024 (after upstream 0023_rename_massscanners) - Renamed 0024_merge to 0025 - Updated dependencies to point to correct parent migrations - All tests passing (277/277)
…ponse. Closes #629 (#662) * Optimize API performance by removing redundant serialization in feeds_response - Remove per-item FeedsResponseSerializer instantiation and validation - Directly append data_ dictionary to json_list (50-90x speedup) - Benchmarking shows response time reduction from ~1.8s to ~0.03s for 5000 IOCs - All existing tests pass (78/78 in test_views, 4/4 in test_serializers) - API contract remains unchanged Closes #629 * Remove unused FeedsResponseSerializer import * Add documentation to FeedsResponseSerializer explaining why it's retained - Keep serializer as schema definition and documentation - Add comprehensive docstring explaining performance optimization context - Clarifies serializer is not used in production but valuable for: * API contract documentation * Unit test validation * Future reference and potential re-enabling * Developer/consumer understanding of response format Related to #629
Bumps [joblib](https://github.com/joblib/joblib) from 1.5.2 to 1.5.3. - [Release notes](https://github.com/joblib/joblib/releases) - [Changelog](https://github.com/joblib/joblib/blob/main/CHANGES.rst) - [Commits](joblib/joblib@1.5.2...1.5.3) --- updated-dependencies: - dependency-name: joblib dependency-version: 1.5.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [celery](https://github.com/celery/celery) from 5.6.0 to 5.6.1. - [Release notes](https://github.com/celery/celery/releases) - [Changelog](https://github.com/celery/celery/blob/main/Changelog.rst) - [Commits](celery/celery@v5.6.0...v5.6.1) --- updated-dependencies: - dependency-name: celery dependency-version: 5.6.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [django](https://github.com/django/django) from 5.2.8 to 5.2.9. - [Commits](django/django@5.2.8...5.2.9) --- updated-dependencies: - dependency-name: django dependency-version: 5.2.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [numpy](https://github.com/numpy/numpy) from 2.3.5 to 2.4.0. - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst) - [Commits](numpy/numpy@v2.3.5...v2.4.0) --- updated-dependencies: - dependency-name: numpy dependency-version: 2.4.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [elasticsearch8](https://github.com/elastic/elasticsearch8-py) from 8.19.2 to 8.19.3. - [Commits](https://github.com/elastic/elasticsearch8-py/commits) --- updated-dependencies: - dependency-name: elasticsearch8 dependency-version: 8.19.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Created a 192x192 pixel logo image from the existing greedybear.png to serve as the Apple Touch Icon. This resolves the 404 error when the frontend tries to load the missing asset. Fixes #627
* Squashed '.github/' changes from aa1fe1a..5a78179 5a78179 Merge pull request #223 from certego/develop 76df2c2 added ruff and fixed a bug (#221) 59bfe83 Explicitly disabled "xpack.security" in Elasticsearch container 0c262e2 Updated CHANGELOG 0599640 Fixed create_python_cache workflow (#222) 4f21023 Added documentation - part 2 (#220) 0d2f931 updated github actions versions (#218) 013f31a Python caching revisited (#217) 548235b Linter requirements reconciliated (#215) b6fd709 Updated changelog 0cfa137 Ecr (#201) ed2dd16 Updated codeQL action to v3 (#216) 5f44be8 APT caching revisited (#214) cf7c16d Updated linters and added changes detection exclusions (#213) a492676 Deprecation of license check `table-headers` (#212) 0a6db48 Updated python linters also in '_python.yml' workflow git-subtree-dir: .github git-subtree-split: 5a78179ab0cbea826c416f8975251b519c2541fc * Simplify pre-commit to use only Ruff - Removed black, isort, flake8, pylint, bandit, autoflake from pre-commit - Ruff provides equivalent functionality for all of these - Faster pre-commit execution - Avoids conflicting linter rules * Update documentation to reflect Ruff migration - Replaced black and isort badges with Ruff badge in README - Updated PR template checklist to mention Ruff instead of Black/Flake/Isort * Run Ruff to fix linting and formatting issues - Fixed 37 import sorting and unused import issues - Reformatted 5 files with ruff format - Fixed pre-commit ruff args (removed invalid 'check' argument) - 14 wildcard import warnings remain (F403) which require manual review * Add .ruff_cache to .gitignore - Exclude Ruff's cache directory from version control * Silence F403 wildcard import warnings As per maintainer feedback, silenced F403 warnings for wildcard imports in __init__.py files since they are acceptable for this project. * Add pyproject.toml and fix migration file imports - Created pyproject.toml to extend ruff config for easier CLI usage - Fixed import sorting in 5 Django migration files - Now 'ruff check .' works without explicit --config flag * Apply ruff formatting to remaining files - Reformatted 30 Python files with ruff format - This is the result of running 'ruff format .' after creating pyproject.toml - No logic changes, only formatting (line breaks, spacing) * Add empty packages.txt for certego/.github v2.0.0 compatibility The new certego/.github v2.0.0 APT cache workflow requires a packages file, even if empty. This is a workaround for the workflow's strict validation that was introduced in v2.0.0. * Add packages_path to workflow configuration Pass packages.txt to the workflow to fix APT cache restoration step * Make packages.txt truly empty Remove comments that were being interpreted as package names by apt-get * Specify RabbitMQ version for CI Use rabbitmq:4-management-alpine to fix Docker image pull error. The default 'latest' is not a valid tag for management-alpine images. * Fix RabbitMQ version tag duplication Use version '4' instead of '4-management-alpine' since the workflow automatically appends '-management-alpine' suffix * Expand Ruff rule coverage with comprehensive linting modules Added comprehensive Ruff rule modules as requested: - E/W: Full pycodestyle error and warning coverage - N: pep8-naming for naming conventions - UP: pyupgrade for modern Python syntax - B: flake8-bugbear for common Python bugs - C4: flake8-comprehensions for list/dict improvements - DJ: flake8-django for Django-specific linting All rules are documented with inline comments and links to official docs. Fixed 43 auto-fixable violations (imports, annotations, etc). Added comprehensive ignore list for intentional code patterns: - Test helpers (mutable defaults, classmethods) - ML conventions (X, X_train naming) - Django patterns (null=True on CharField, models without __str__) - Legacy naming (viewType, iocType, migration functions) All checks passing ✅ * Fix easy E/N/UP/C violations as requested Applied Ruff auto-fixes for straightforward style improvements: - N804: Renamed 'self' → 'cls' in test classmethods (6 occurrences) - UP008: Simplified super() calls (3 occurrences) - UP031/UP032: Modernized string formatting (2 occurrences) - C401: Fixed set comprehensions (7 occurrences) Total: 18 violations fixed across 7 files. Invasive changes (N801/N802 model/function renames, N803/N806 ML naming) deferred to follow-up issue for dedicated testing and review.
…Closes #671 (#676) * refactor: rename viewType→ViewType and iocType→IocType (PEP8 N801) - Renamed class viewType to ViewType - Renamed class iocType to IocType - Updated all imports and usages across ~20 files - Updated models, API views, tests, and cronjobs - Fixes naming convention violations per PEP8 N801 * refactor: rename ML variables to follow PEP8 (N803/N806) and update Ruff config - Renamed X → x, X_train → x_train, X_test → x_test in ML code - Updated ml_model.py: all method parameters and local variables - Updated random_forest.py: train() method variables - Removed N801, N802, N803, N806 from Ruff ignore list - Migration files already excluded via extend-exclude - Fixes PEP8 N803/N806 violations in ML scoring code * refactor: fix remaining PEP8 naming violations (N802/N806) - Renamed generalHoneypots → general_honeypots (3 occurrences) - Renamed checkAuthentication → check_authentication - Renamed checkConfiguration → check_configuration - Updated authentication/urls.py imports and URL patterns - Renamed X_train, X_test → x_train, x_test in test_rf_models.py - All PEP8 N801/N802/N803/N806 violations now resolved - Migration files remain excluded via extend-exclude in .ruff.toml * chore: update frontend dependencies * chore: update frontend dependencies
* feat: ntfy alerts for monitor logs. Closes #664 * refactor: changed function name to send_slack_message to improve code readability * added formatting for ntfy alerts * added a test for ntfy alerts feature * removed redundant dependency * added comments in env template * formatting changes * fix formatting --------- Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> Co-authored-by: tim <tim.leonhard@posteo.de>
…loses #677 (#679) * refactor: fix mutable default arguments (B006/B008) - Replace empty list defaults with None in _create_mock_ioc() - Replace datetime.now() call in default argument with None - Initialize mutable defaults inside function to avoid shared state - Fixes flake8-bugbear violations B006 and B008 Phase 1 of bugbear violations fix. All tests passing (282/282). * refactor: fix exception handling (B017/B904) - Add 'from None' to LoginSerializer to suppress exception context when re-raising ValidationError (intentionally hiding user existence) - Add 'from e' to CowrieSessionRepository to preserve exception chain when raising descriptive ValueError - Replace bare Exception with IntegrityError in test for database constraint violations Phase 2 of bugbear violations fix. All tests passing (282/282). * refactor: fix lambda loop variable binding (B023) - Add default argument to lambda in multi_label_encode to capture loop variable correctly - Prevents late binding issue where all lambdas would reference the final loop value instead of capturing each iteration's value Phase 3 of bugbear violations fix. All tests passing (282/282). * refactor: enable bugbear rules in ruff config - Remove B006, B008, B017, B023, and B904 from ignore list - All bugbear violations have been fixed in previous commits - Enforces proper exception handling, mutable defaults, and lambda patterns Phase 4 (final) of bugbear violations fix. All tests passing (282/282). All ruff checks passing. * refactor: use ternary operators for cleaner code Use ternary operators directly in mock assignments instead of separate if-else blocks for a more concise and Pythonic approach. Co-authored-by: regulartim
… (#684) * refactor: add __str__ methods and fix field ordering (DJ008/DJ012) - Add __str__ to UserProfile, Sensor, FireHolList, CowrieSession, Statistics, MassScanner, and WhatsMyIPDomain models - Move UserProfile fields before Meta class (DJ012) - Improves admin interface usability and debugging Phase 1 of Django violations fix. All tests passing (282/282). * refactor: fix CharField null=True violations (DJ001) - Replace null=True with blank=True, default='' on FireHolList.source and MassScanner.reason fields - Filter empty source strings in get_firehol_categories to prevent empty values in firehol_categories list - Create database migration to update schema Phase 2 of Django violations fix. All tests passing (282/282). * refactor: enable Django violations checks in ruff config - Remove DJ001, DJ008, and DJ012 from ignore list - All Django model violations have been fixed in previous commits - Enforces Django best practices for CharField, __str__, and field ordering Phase 3 (final) of Django violations fix. All tests passing (282/282). All ruff checks passing.
…quests in Firehol.py. (#711) * Adding validation methods for cidr and validation for incoming get requests in firehol * fix merge * fix extra import
Co-authored-by: Shivraj Suman <shivraj.24bcg10030@vitbhopal.ac.in>
Bumps [datasketch](https://github.com/ekzhu/datasketch) from 1.8.0 to 1.9.0. - [Release notes](https://github.com/ekzhu/datasketch/releases) - [Commits](ekzhu/datasketch@v1.8.0...v1.9.0) --- updated-dependencies: - dependency-name: datasketch dependency-version: 1.9.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* feat: Add test coverage reporting (#701) - Add .coveragerc configuration file - Define source paths (greedybear/) - Exclude migrations, tests, and generated files - Configure report formatting (show_missing, precision) - Set output formats for HTML and XML reports - Update CI workflow to generate coverage artifacts - Generate XML coverage report for download - Upload coverage reports as GitHub Actions artifacts - Add coverage summary to job summary for quick visibility - Set 30-day retention for coverage artifacts - Update .gitignore to exclude coverage artifacts - htmlcov/, .coverage, coverage.xml, etc. This implements Phase 1 (Local Coverage Setup) and Phase 2 (CI Integration) as discussed in issue #701. Coverage is already enabled in the CI workflow and will now generate detailed reports. Local usage: coverage run --source=greedybear manage.py test tests coverage report # Terminal output coverage html # Browser-viewable report in htmlcov/ Related: #701 * feat: Auto-install coverage in local development - Add coverage auto-installation to local.override.yml - Coverage is now automatically installed when running docker compose - Developers no longer need to manually 'pip install coverage' - Add requirements/test.txt for non-Docker environments - Provides easy installation: pip install -r requirements/test.txt - Documents test dependencies This implements Option 4 as approved by @mlodic. Related: #701 * fix: Correct regex pattern in .coveragerc - Fixed if __name__ == __main__ pattern - Changed from dots (.__main__.:) to proper quotes ("__main__":) - Copilot suggestion applied Co-authored-by: GitHub Copilot * fix: Upload only coverage.xml artifact - Removed .coverage file from artifact upload - Prevents overwrites between different Python versions in matrix - XML report is sufficient for coverage analysis - Binary .coverage files are version-specific Co-authored-by: GitHub Copilot * feat: Add dev-requirements.txt with conditional Dockerfile install Implements Option B as approved by @mlodic: - Create requirements/dev-requirements.txt with coverage>=7.3.2 - Add INSTALL_DEV build argument to Dockerfile - Update local.override.yml to use INSTALL_DEV=true - Remove requirements/test.txt (replaced by dev-requirements.txt) Benefits: - Dependabot tracks dev dependencies automatically - Clean separation between dev and production - Production image stays lean (no coverage) - Dev dependencies installed at build time, not runtime Usage: docker compose up --build # Local dev (includes coverage) docker build . # Production (no dev deps) Related: #701 * fix: Replace periods with hyphens in artifact name - Artifact name 'coverage-report-3.13' becomes 'coverage-report-3-13' - Prevents GitHub Actions naming conflicts with periods Co-authored-by: GitHub Copilot * fix: Use valid GitHub Actions expression for artifact name * chore: Trigger CI with temporary debug log * Revert "chore: Trigger CI with temporary debug log" This reverts commit a7aa941.
#725) * Refactor: Remove hardcoded Cowrie and Log4j fields from IOC model - Migrated existing data to GeneralHoneypot M2M relationship - Removed boolean fields from IOC model - Updated repositories and extraction strategies to use GeneralHoneypot - Cleaned up API views and removed legacy Enums - Updated frontend to dynamically load honeypots - Added comprehensive tests for feed types and backward compatibility * Fix frontend tests: update useDataTable mock * Fix frontend linting issues in Feeds.jsx * refactor: address review comments - Remove legacy log4j alias and normalization logic - Simplify queries by removing Q() wrappers - Use list comprehensions for cleaner code - Update tests to use log4pot instead of log4j - Verify general_honeypot_name in extraction tests --------- Co-authored-by: tim <46972822+regulartim@users.noreply.github.com>
…loses #727 (#730) * fix: standardize line endings and integrate frontend linters into pre-commit Fixes #729: Fix inconsistent line endings causing CI failures on Windows - Update Ruff config to enforce LF line endings (line-ending = 'lf') - Add .gitattributes to normalize line endings via Git Fixes #727: Integrate Frontend Linters into Pre-commit Workflow - Add Prettier hook using local npm script (npm run formatter) - Add ESLint hook using local npm script (npm run lint) - Uses same linter versions as CI to ensure consistency * feat: skip frontend hooks if node_modules not installed Frontend pre-commit hooks (prettier, eslint) now gracefully skip if frontend/node_modules doesn't exist. This allows backend-only contributors to use pre-commit without needing to run npm install in frontend/.
#728) * feat: Add Tor exit node extraction with separate TorExitNode model (#547) * fix: use CustomTestCase and add task scheduling Thanks for the review! Made both changes: - Switched to CustomTestCase for test consistency - Added weekly Celery Beat schedule (Sundays at 4:30 AM) Addresses feedback from @regulartim * fix: address review feedback for tor exit node feature - Added migration file for TorExitNode model (0032) - Fixed case sensitivity issue - changed 'Tor Exit Node' to lowercase for API filtering - Registered TorExitNode in admin panel for easy management --------- Co-authored-by: SUMIT DAS <sumitdas@SUMITs-MacBook-Air.local>
Bumps [pandas](https://github.com/pandas-dev/pandas) from 2.3.3 to 3.0.0. - [Release notes](https://github.com/pandas-dev/pandas/releases) - [Commits](pandas-dev/pandas@v2.3.3...v3.0.0) --- updated-dependencies: - dependency-name: pandas dependency-version: 3.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* feat: Add end-to-end tests for ExtractionPipeline (PR 1/2) #636 * test: verify search time window in execute flow * test: verify grouping logic ensures strategies receive correct hits * chore: remove unused mock assignments in extraction tests * test: verify IOC accumulation from multiple strategies * refactor: standardize TestExecuteEmptyResponse with mock helper * refactor: use common ExtractionTestCase base class * test: check for whitespace-only src_ip skipping * test: check for whitespace-only type skipping * refactor: deduplicate _create_pipeline_with_mocks into base class * test: verify exception logging in pipeline strategy execution * test: explicit patch LEGACY_EXTRACTION in sensor test * refactor: move MockElasticHit to tests/__init__.py for shared usage
- Split required_fields into base_fields and verbose_fields - Base fields always returned (value, first_seen, attack_count, etc.) - Verbose fields only returned when verbose=true: - days_seen - destination_ports - honeypots - firehol_categories - destination_port_count only calculated when destination_ports exists This ensures /api/feeds endpoints return concise responses by default, with verbose data only when explicitly requested via verbose=true parameter. Co-authored-by: SUMIT DAS <sumitdas@SUMITs-MacBook-Air.local>
…744) (#745) - honeypots was redundant with feed_type (same data, different casing) - Still fetch honeypots from DB to calculate feed_type - Remove honeypots from response in both verbose and non-verbose modes - Users only see feed_type which has all necessary information Co-authored-by: SUMIT DAS <sumitdas@SUMITs-MacBook-Air.local>
* add v3 announcement * change version number format to match other posts
…oses #636 (#740) * Add strategy-specific E2E tests and edge cases for ExtractionPipeline. Closes #636 * fix: address PR feedback - improve test assertions and remove unused mocks - Replace weak assertGreaterEqual(result, 0) with specific mock.call_count assertions - Fix E2E tests to use proper ExtractionStrategyFactory mocking pattern - Remove unnecessary UpdateScores patch decorators from factory tests - Remove unused mock_scores parameters * refactor: split pipeline tests and use real factory/strategies in E2E - Split monolithic test file into 4 focused files - E2E tests now use real ExtractionStrategyFactory and strategies - Only mock repositories at the boundary - Tests actual integration path as it runs in production * test: add back edge cases for pipeline tests - test_honeypot_skipped_when_not_ready (grouping file) - test_strategy_returns_empty_ioc_records (E2E file) - test_partial_strategy_success (E2E file) - test_large_batch_of_hits (E2E file) * Add IOC content verification tests and reorganize test files - Add TestIocContentVerification class with 3 tests for IOC content verification - Move E2ETestCase class to tests/__init__.py for shared usage (reviewer feedback) - Split edge cases into test_extraction_pipeline_edge_cases.py Edge cases now clearly document when mocking is required: - test_partial_strategy_success: Mocks factory (needs to force exception) - test_large_batch_of_hits_with_real_strategy: Uses REAL strategy Tests added: - test_cowrie_ioc_content_verified: Verifies IOC has correct IP - test_multiple_honeypots_ioc_content_verified: Verifies multiple IOCs - test_ioc_scanner_field_contains_honeypot_type: Verifies scanner field Addresses reviewer feedback to: 1. Verify actual IOC content, not just count 2. Move shared test infrastructure to tests/__init__.py 3. Keep test files focused and manageable in size 4. Use real strategies where possible in tests * Fix misleading comment in large batch test * test: explicitly assert IOC extraction count before verifying scoring call in e2e pipeline test
* test(migrations): add migrations test * resolve linter issue * make CI use requirements-dev.txt and remove redundant coverage entry * refactor/add testcase * remove code snippet * add test_runner & update migration test flow * run migration tests in CI --------- Co-authored-by: tim <46972822+regulartim@users.noreply.github.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
12 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )