Add HINT validation result for dandiset.yaml in both validate and upload commands #1741
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1741 +/- ##
==========================================
+ Coverage 75.08% 75.10% +0.01%
==========================================
Files 84 84
Lines 11910 11921 +11
==========================================
+ Hits 8943 8953 +10
- Misses 2967 2968 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
When BIDS validator encounters dandiset.yaml (not part of BIDS spec), now: - Keeps the existing ERROR about the file not matching BIDS patterns - Adds a new HINT level validation result suggesting to create/update .bidsignore Includes test to verify both ERROR and HINT are generated Co-authored-by: yarikoptic <39889+yarikoptic@users.noreply.github.com>
yarikoptic
force-pushed
the
copilot/add-hint-for-dandiset-yaml
branch
from
5d5dd81 to
957da04
Compare
|
@yarikoptic I don't get the point of this or #1740. If we don't want the problem with |
|
We don't want to "cheat" bids validator, so if people run it directly they get the same result. AI we want to instruct for to mitigate that error consistently |
|
Tested against dandiset 001442 (chosen because it is a smallish BIDS dataset, though lots of validation noise). The PR works as described - the IMO this is fine, the HINT probably will be easier to understand for unfamiliar users, even though the duplication and extra noise are somewhat undesirable. trimmed raw output: master branchtrimmed output: PR branchDiff between master and branch5a6,7
> [DANDI.BIDSIGNORE_DANDISET_YAML] /home/austin/devel/dandi-cli/001442 — Consider creating or updating a `.bidsignore` file in the root of your BIDS dataset to ignore `dandiset.yaml`. Add the following line to `.bidsignore`:
> dandiset.yaml |
candleindark
left a comment
candleindark
left a comment
There was a problem hiding this comment.
Proposed some changes.
Additionally, if we want this custom validation hint to show up in the dandi upload command as well, we need to move this logic to an earlier point the in the calling process. The current setup only affect dandi validate not dandi upload where the issue was originally reported at #1734 (comment).
|
per discussion with @candleindark the issuing of HINT should be done deeper inside where we validate BIDS datatasets so we effect also |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
|
@copilot also please address now failing tests |
- Moved dandiset.yaml HINT logic from validate() to _harmonize() in bids_validator_deno - This ensures HINT appears for both 'dandi validate' and 'dandi upload' commands - Removed modification of original BIDS error message (preserving original error) - Updated test to not expect modified error message - All 101 validation tests passing Co-authored-by: yarikoptic <39889+yarikoptic@users.noreply.github.com>
Fixed failing tests (commit bbe7870). The test was expecting the modified error message, but per review feedback we now preserve the original BIDS error. Updated test to not check for the modified message. All 101 validation tests now passing.
Moved HINT generation logic to |
| results.append( | ||
| ValidationResult( | ||
| id=f"BIDS.{issue.code}", | ||
| origin=origin, | ||
| scope=_get_scope(issue_path), | ||
| # Store only the issue, not entire bv_result with more context | ||
| origin_result=issue, | ||
| severity=_SEVERITY_MAP[severity] if severity else None, | ||
| validation_result = ValidationResult( | ||
| id=f"BIDS.{issue.code}", | ||
| origin=origin, | ||
| scope=_get_scope(issue_path), | ||
| # Store only the issue, not entire bv_result with more context | ||
| origin_result=issue, | ||
| severity=_SEVERITY_MAP[severity] if severity else None, | ||
| dandiset_path=dandiset_path, | ||
| dataset_path=ds_path, | ||
| message=_get_msg(issue, code_messages), | ||
| path=issue_path, | ||
| ) | ||
| results.append(validation_result) | ||
|
|
||
| # If the error is about the dandiset metadata file, add a HINT | ||
| # suggesting to add it to .bidsignore | ||
| is_dandiset_yaml_error = ( | ||
| issue_path is not None | ||
| and issue_path.relative_to(ds_path).as_posix() == dandiset_metadata_file | ||
| ) | ||
| if is_dandiset_yaml_error: | ||
| hint = ValidationResult( | ||
| id="DANDI.BIDSIGNORE_DANDISET_YAML", | ||
| origin=ORIGIN_VALIDATION_DANDI_LAYOUT, | ||
| scope=Scope.DATASET, | ||
| origin_result=validation_result, | ||
| severity=Severity.HINT, | ||
| dandiset_path=dandiset_path, | ||
| dataset_path=ds_path, | ||
| message=_get_msg(issue, code_messages), | ||
| path=issue_path, | ||
| message=( | ||
| f"Consider creating or updating a `.bidsignore` file " | ||
| f"in the root of your BIDS dataset to ignore " | ||
| f"`{dandiset_metadata_file}`. " | ||
| f"Add the following line to `.bidsignore`:\n" | ||
| f"{dandiset_metadata_file}" | ||
| ), | ||
| ) | ||
| ) | ||
| results.append(hint) |
There was a problem hiding this comment.
I prefer this customization of validation results not done within the bids_validator_deno subpackage. In that way, the bids_validator_deno package can stay true to what BIDS validator deno returns as much as possible.
You can add this customization in bids.py following this line.
There was a problem hiding this comment.
Moved HINT generation to dandi/files/bids.py in the _validate() method (commit ba523e6). The bids_validator_deno package now stays true to what BIDS validator deno returns, and the customization happens at the BIDS file interface level as recommended.
- Removed HINT generation logic from _harmonize() in bids_validator_deno/_validator.py - Added HINT generation in _validate() in dandi/files/bids.py after line 115 - This keeps bids_validator_deno true to what BIDS validator deno returns - HINT is now generated at the BIDS file interface level - All 101 validation tests passing Co-authored-by: yarikoptic <39889+yarikoptic@users.noreply.github.com>
When BIDS validator encounters
dandiset.yaml(not part of BIDS spec), users now receive both the original ERROR from BIDS validator and a HINT suggesting to add the file to.bidsignore. The HINT is generated at the BIDS file interface level, ensuring it appears in bothdandi validateanddandi uploadcommands.Changes
dandi/files/bids.py
_validate()method after callingbids_validate()DANDI.BIDSIGNORE_DANDISET_YAML, severity:Severity.HINT, scope:DATASET.bidsignoredandiset_metadata_file,ORIGIN_VALIDATION_DANDI_LAYOUT,Scope,Severitydandi/bids_validator_deno/_validator.py
dandi/tests/test_validate.py
test_validate_bids()to verify both ERROR and HINT are generatedExample Output
Impact
The HINT now appears in both
dandi validateanddandi uploadcommands, providing consistent guidance to users regardless of which command they use. Thebids_validator_denopackage remains true to what BIDS validator returns, with customization happening at the appropriate layer in the BIDS file interface.Original prompt
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.