Fix conflicts oonirun v2 1 #1056
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* First steps * Next steps * Use right source for clickhouse image * Add requests library used in tests * Add docker related operations * Added profiles to select which services to run * Add context comment * Remove dependency on clickhouse to allow fastpath to run standalone * binding to 0.0.0.0 to accept traffic from outside docker * Using simpler dockerfile configuration * Using simpler dockerfile configuration * Fixing dependency betwen fastpath and clickhouse * Improving usability of makefile * Creating multi stage docker file * Remove deprecated pkg_resources dependency * Skip broken tests * Upgrade deprecated upload-artifact action in test_fastpath.yml * Move fastpath.conf to the top dir * remove useless print * Working on adding integration tests for the docker deployment * Added integration tests with docker * Added requests to simplify testing * Add more integration tests * Add better healthcheck to docker compose * Pin docker container version * Remove hardcoded reload config --------- Co-authored-by: Luis Díaz <ldiazn98@gmail.com> Co-authored-by: Luis Diaz <41093870+LDiazN@users.noreply.github.com>
* Add docker build command * Add push to dockerhub command * Add buildspec file * Add service name variable * Remove unnecessary port 5000 * Update package version in makefile * Add version parsing in fastpath makefile
* Remove unused imports * Add open report endpoint * Moved reports to its own module * Moved generate report id function to utils file * timer function should go before router.post * Add collector ID to mock settings * port measurement submission endpoint to ECS * Add parameter for fastpath url * add binary data as input receive msmnt * Fixed measurement upload endpoint * Added /close endpoint * Black reformat * Add metrics for dropped measurements * Add health check for fastpath rechability * Add retry to fastpath for failed measurement upload * Add boto3 as dependency * Add s3 upload for failed reports * Add s3 upload of failed reports * Trigger ci * Trigger ci * Add fastpath mock for tests * Add reports tag to close report * Black reformat * Add function to write measurements to disk * Add measurement spool dir to fastpath configs * Add fixture fixing bad docker compose path * git status * Add parsing of measurement spool dir setting * Add write-to-disk feature flag * remove out dated comment * Remove file creation in ooniprobe * Remove unused import * Move write_to_disk check to msm_processor * Add metrics for sending to fastpath and sending to s3 failures * Change write-to-disk parameter to flag argument * Add explicit names to timers * Make metrics decorator work with or without calling it * Add exponential back off with jitter to retry * Add async request to fastpath * Update fastpath/fastpath/core.py * Update fastpath changelog * Add test-helpers to probe services This endpoint is required to test miniooni with the dev api, since this endpoint is not present in dev at all --------- Co-authored-by: Arturo Filastò <arturo@filasto.net>
* Add msm_failure mapping * Cast probe_asn to str to match MeasurementMeta model * Add tests for format validation
* Add msm_failure mapping * Cast probe_asn to str to match MeasurementMeta model * Add tests for format validation * Add limit 1 to test query
* Fix bad bucket name * Set dev bucket as default bucket * Add prod bucket comment
Add more detailed comments about how the filtering works
* Don't specify protocol for other collectors * Fix rewriting msm uid The fastpath was rewriting the measurement uid, generating slight variations in the timestamp that led to 404 errors when trying to retrieve the measurement from the explorer This error was introduced when the feature for writing measurements to the disk from the fastpath was added. This commit should not affect the monolith * Fix bad hour calculation in write measurement to disk * Simplify hour parsing * Downgrade to python 3.12 to prevent clickh error There seems to be an issue with clickhouse when running as a docker container where it will crash if running in an user other than root. The problem is that in 3.13 the python standard library they changed the errors risen by the get user function see: mymarilyn/clickhouse-driver#462
* Fix missing comma * Add test for group_by failure
* Simplify the analysis aggregation SQL query * Focus on only returning blocking loni values * Include results in the API * Avoid duplicating entries * Also include the blocked protocol maximum likelyhood value * Move format_aggregate_query into the aggregate_analysis route * Fix bugs in SQL query * Add analysis description * Bump debian version in build_deb_package CI action * Add loni-outcome-space docs * Add link to chart as svg * Remove extra WHERE clause * Fix return values
* Workaround for missing arrayElementOrNull function
This function was introduced only in clickhouse 24.10+
* Cast the unset blocked_max_protocol to tuple('none', 0.0)
* Fix missing comma * Add test for group_by failure * Add fastpath user and setup fastpath directories * trigger ci --------- Co-authored-by: Arturo Filastò <arturo@filasto.net>
* Fix bad argument specification in raw_measurement * Add comment about report_id and measurement_uid * Add monkeypatch to test files from s3 without access to s3 * remove outdated comment * black reformat * Download measurement tar in cache dir * Move download of measurement to maybe_download_fixtures * fix bad path configuration * Add comment to make fixture download more clear * simplify fixture download
…analysis * Distinguish between dns_other and dns_isp in aggregate observations * Fix tests for observations aggregation
* Remove test_start_time ordering field, hard cap limit to 100 * validating order fields and limiting page size in monolith * Update changelog * clarify changelog * Fix type error
* Remove test_start_time ordering field, hard cap limit to 100 * validating order fields and limiting page size in monolith * Update changelog * clarify changelog * Fix type error * Set new debops-ci version to test the backend CI * reverted branch * Bump changelog
* Remove fingerprints bytes metrics * Bump fastpath version
* Change type validation * changing api limits * Fix bad default value * Add test for validating AggregatioinResult model * remove bad docstriing * reformat * Add integration test for checking that axis_y=probe_asn works * Add type validation * Add dimension count check
* Fix bug with filtering by probe_asn without the AS prefix probe_asn filtering is supported without the AS prefix, but the return value always includes it Closes: #1008 * reflow
* Working on test to reproduce bug
* Remove useless date parsing; add test for broken query
* Fix bad comparision between model and {}
* Measurement body is already str, no need for translation
* Emulate monolith behavior of returning probe_asn as int
* Add fixture to test measurement body fetch
* Port monolith measurement_meta endpoint function
* Port basic usage test for measurements meta
* Add test from old version
* Add test to validate report id
* Black reformat
* Port more tests from the monolith
* black reformat
* Remove unused imports
* Add since parameter and fix bad date filtering
* Add test to check that default filters don't retrieve anything older than a month ago
* Fix datetime parsing
* Black reformat
* trigger ci
* trigger ci
* Add test for missing report_id or measurement_uid
* move validate function closer to where it's needed
* Add test for bad report_id detection
* Extend measurement_meta test to include query by uid
* Add logging to geoip match detection * Add logging for bad reporting
* in test_settings, add clickhouse_server fixture * delay and decrease probe frequency of is_clickhouse_running to reduce logspam * remove docker_ip and docker_services included by clickhouse_server
* add collectors to ooniapi * Make method "GET" on route for list_collectors * Use SettingsDep from ooniprobe.dependencies
* add geolookup to ooniapi probe services this includes a small refactor to make the probe_geoip method re-usable * Replace class docstring with pydantic.Field annotation * use pydantic field validation for geolookup endpoint * probe_services: validate probe_geoip; add type GeoLookupResult * monkeypatch lookup_probe_cc and lookup_probe_network to provide static responses
* add bouncer/net-tests to ooniapi under new router bouncer * fix pylint import complaints * let fastapi validate input * exclude unset parameters in bouncer/net-tests * add bouncer tests from api/tests/integ/test_probe_services.py I modified the tests to expect any error (other than http response 200) because FastAPI returns HTTP error 422 Unprocessable Entity instead of 400, as the tests do not supply a request that the model can validate * raise HTTPException with status 400 for backwards-compatibility * check status code on error is 400 for backwards compatibility with old probes
* Add logging to failing comparision function * Trigger CI * Trigger CI * Fix bad db_asn logging * Improve logging of geoip mismatches * Fix extract_probe_ipaddr
* Add initial endpoint shape for event detector changepoints * Add cusum changepoint list endpoint * Remove unused request model * Add event detector changepoint * Fix event detector table migration for tests * Add basic test for changepoint endpoint * Add basic filtering test to changepoint endpoint * Remove unused log * Add tests to check filtering by date * Add utils module with useful types * Remove unused types * Remove unused import * Add enum for change direction * Black reformat * Add testing data from the DB * Remove out dated comment * Trigger ci * fix typo in changepoint api endpoint * trigger ci * Modify changepoints table to add block_type column * replace bad quotes * Add block_type field to changepoints API
* Add prioritization CRUD methods /api/_/show_countries_prioritization /api/_/debug_prioritization * rename routers/prio to routers/prio_crud * use raw string in regex
* Add config bucket setting and function to read file from bucket * trigger ci
* add test-list/tor-targets to ooniapi probe services this adds a new environment variable "TOR_TARGETS" which points to a json file * Add config bucket setting and function to read file from bucket * trigger ci * fetch tor-targets .json from s3 bucket * add TorTargetsDep to load and cache tor_targets.json from S3 * remove unused imports; fix import order * replace @lru_cache of read_file with simple cache in dict this is because s3_client will be a new instance each invocation and will not be cached. CacheDep does not implement LRU or any eviction policy, but it's only used for a few static paths so it should not grow. --------- Co-authored-by: Luis Díaz <ldiazn98@gmail.com>
* Working on porting test list endpoint * Working on porting test-list/urls * Adding tests to test-list endpoint * Adding more tests * Add metric for test list count * black reformat * Add tests for test lists * Port test lists.1 (#1047) * in test_settings, add clickhouse_server fixture (#1028) * in test_settings, add clickhouse_server fixture * delay and decrease probe frequency of is_clickhouse_running to reduce logspam * remove docker_ip and docker_services included by clickhouse_server * add collectors to ooniapi (#1032) * add collectors to ooniapi * Make method "GET" on route for list_collectors * Use SettingsDep from ooniprobe.dependencies * add geolookup to ooniapi probe services (#1031) * add geolookup to ooniapi probe services this includes a small refactor to make the probe_geoip method re-usable * Replace class docstring with pydantic.Field annotation * use pydantic field validation for geolookup endpoint * probe_services: validate probe_geoip; add type GeoLookupResult * monkeypatch lookup_probe_cc and lookup_probe_network to provide static responses * add bouncer/net-tests to ooniapi under new router bouncer (#1036) * add bouncer/net-tests to ooniapi under new router bouncer * fix pylint import complaints * let fastapi validate input * exclude unset parameters in bouncer/net-tests * add bouncer tests from api/tests/integ/test_probe_services.py I modified the tests to expect any error (other than http response 200) because FastAPI returns HTTP error 422 Unprocessable Entity instead of 400, as the tests do not supply a request that the model can validate * raise HTTPException with status 400 for backwards-compatibility * check status code on error is 400 for backwards compatibility with old probes * Fix bad geoip reporting (#1022) * Add logging to failing comparision function * Trigger CI * Trigger CI * Fix bad db_asn logging * Improve logging of geoip mismatches * Fix extract_probe_ipaddr * Cusum changepoint api (#1027) * Add initial endpoint shape for event detector changepoints * Add cusum changepoint list endpoint * Remove unused request model * Add event detector changepoint * Fix event detector table migration for tests * Add basic test for changepoint endpoint * Add basic filtering test to changepoint endpoint * Remove unused log * Add tests to check filtering by date * Add utils module with useful types * Remove unused types * Remove unused import * Add enum for change direction * Black reformat * Add testing data from the DB * Remove out dated comment * Trigger ci * fix typo in changepoint api endpoint * trigger ci * Modify changepoints table to add block_type column * replace bad quotes * Add block_type field to changepoints API * Add prioritization CRUD methods (#1040) * Add prioritization CRUD methods /api/_/show_countries_prioritization /api/_/debug_prioritization * rename routers/prio to routers/prio_crud * use raw string in regex * S3 config parameter (#1039) * Add config bucket setting and function to read file from bucket * trigger ci --------- Co-authored-by: Luis Diaz <41093870+LDiazN@users.noreply.github.com> --------- Co-authored-by: aagbsn <aagbsn@extc.org>
… longer available (#1026) * Update build_runner to debian stable (trixie) * use python builtin json.JSONEncoder; deprecated in Flask * update pytest.yield_fixture to pytest.fixture as former is deprecated * update deprecated sqlalchemy.ext.declarative.declarative_base this is now available as sqlalchemy.orm.declarative_base * override references deprecated name and issue #42 is not a bug
…#1052) Reserved addresses in geoip database return None for the AS Name, but default fields cc=ZZ and as=AS0. This fixes server error where AS Name was None and not validated.
The function was incorrectly creating 5 test helpers (0-4) when it should only create 4 (0-3) based on the intended distribution: - 0.th.ooni.org: 10% of traffic - 1.th, 2.th, 3.th: 30% each Changes: - Adjust shift calculation: q % 4 + 1 -> q % 3 + 1 - Use range(4) instead of range(5) - Use % 4 instead of % 5 for rotation Fixes #973
* properly load the json * rename variable
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## oonirun-v2-1 #1056 +/- ##
===============================================
Coverage ? 93.82%
===============================================
Files ? 47
Lines ? 4792
Branches ? 322
===============================================
Hits ? 4496
Misses ? 233
Partials ? 63
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes merge conflicts with main and moves the clickhouse and postgres dependencies into common for other services.