Conflicting logging levels for BIDS validation errors

[kabilar]

During a recent upload to https://lincbrain.org/ by a collaborator, we received the following errors/warnings with dandi==0.71.2 and dandischema==0.11.0. One ERROR was noted from the dandi upload command. When looking at the log file, all 5 BIDS validation errors were logged as WARNING but then the last line says the dataset_description.json was actually an ERROR. Perhaps errors 1-4 should be ERROR.

...
Summary:                                                                                               20.5 GB   1 with errors   8.3 kB/s 5416 skipped          1 should be edited online
                                                                                                                                          1 ERROR               1 failed validation
                                                                                                                                          1 done                5415 file exists
2025-10-30 15:06:20,115 [ WARNING] One or more assets failed validation.  Consult the logfile for details.
2025-10-30 15:06:25,227 [    INFO] Logs saved in /home/bglickman/.local/state/dandi-cli/log/2025.10.30-18.58.29Z-21478.log
Error: failed validation

(linc) bglickman@DESKTOP-JN732VL:~/I74_uCT_Lichtman/000049$ grep "ERROR" /home/bglickman/.local/state/dandi-cli/log/2025.10.30-18.58.29Z-21478.log

2025-10-30T14:58:37-0400 [WARNING ] dandi 21478:126210317428416  Error 1: id='BIDS.JSON_SCHEMA_VALIDATION_ERROR' origin=Origin(type=<OriginType.VALIDATION: 'VALIDATION'>, validator=<Validator.bids_validator_deno: 'bids-validator-deno'>, validator_version='2.2.1', standard=<Standard.BIDS: 'BIDS'>, standard_version=None, standard_schema_version='1.1.0') scope=<Scope.FILE: 'file'> origin_result=Issue(code='JSON_SCHEMA_VALIDATION_ERROR', subCode='Authors', severity=<Severity.error: 'error'>, location='/dataset_description.json', issueMessage='must be array\n\nField description: List of individuals who contributed to the creation/curation of the dataset.\n', suggestion=None, affects=None, rule='rules.dataset_metadata.dataset_description', line=None, character=None) severity=<Severity.ERROR: 40> asset_paths=None within_asset_paths=None dandiset_path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049') dataset_path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049') message='Invalid JSON sidecar file. The sidecar is not formatted according the schema.\nsubCode: Authors\nissueMessage: must be array\n\nField description: List of individuals who contributed to the creation/curation of the dataset.\n' metadata=None path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049/dataset_description.json') path_regex=None
2025-10-30T14:58:37-0400 [WARNING ] dandi 21478:126210317428416  Error 2: id='BIDS.JSON_SCHEMA_VALIDATION_ERROR' origin=Origin(type=<OriginType.VALIDATION: 'VALIDATION'>, validator=<Validator.bids_validator_deno: 'bids-validator-deno'>, validator_version='2.2.1', standard=<Standard.BIDS: 'BIDS'>, standard_version=None, standard_schema_version='1.1.0') scope=<Scope.FILE: 'file'> origin_result=Issue(code='JSON_SCHEMA_VALIDATION_ERROR', subCode='Funding', severity=<Severity.error: 'error'>, location='/dataset_description.json', issueMessage='must be array\n\nField description: List of sources of funding (grant numbers).\n', suggestion=None, affects=None, rule='rules.dataset_metadata.dataset_description', line=None, character=None) severity=<Severity.ERROR: 40> asset_paths=None within_asset_paths=None dandiset_path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049') dataset_path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049') message='Invalid JSON sidecar file. The sidecar is not formatted according the schema.\nsubCode: Funding\nissueMessage: must be array\n\nField description: List of sources of funding (grant numbers).\n' metadata=None path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049/dataset_description.json') path_regex=None
2025-10-30T14:58:37-0400 [WARNING ] dandi 21478:126210317428416  Error 3: id='BIDS.JSON_SCHEMA_VALIDATION_ERROR' origin=Origin(type=<OriginType.VALIDATION: 'VALIDATION'>, validator=<Validator.bids_validator_deno: 'bids-validator-deno'>, validator_version='2.2.1', standard=<Standard.BIDS: 'BIDS'>, standard_version=None, standard_schema_version='1.1.0') scope=<Scope.FILE: 'file'> origin_result=Issue(code='JSON_SCHEMA_VALIDATION_ERROR', subCode='ReferencesAndLinks', severity=<Severity.error: 'error'>, location='/dataset_description.json', issueMessage='must be array\n\nField description: List of references to publications that contain information on the dataset.\nA reference may be textual or a\n[URI](SPEC_ROOT/common-principles.md#uniform-resource-indicator).\n', suggestion=None, affects=None, rule='rules.dataset_metadata.dataset_description', line=None, character=None) severity=<Severity.ERROR: 40> asset_paths=None within_asset_paths=None dandiset_path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049') dataset_path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049') message='Invalid JSON sidecar file. The sidecar is not formatted according the schema.\nsubCode: ReferencesAndLinks\nissueMessage: must be array\n\nField description: List of references to publications that contain information on the dataset.\nA reference may be textual or a\n[URI](SPEC_ROOT/common-principles.md#uniform-resource-indicator).\n' metadata=None path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049/dataset_description.json') path_regex=None
2025-10-30T14:58:37-0400 [WARNING ] dandi 21478:126210317428416  Error 4: id='BIDS.JSON_SCHEMA_VALIDATION_ERROR' origin=Origin(type=<OriginType.VALIDATION: 'VALIDATION'>, validator=<Validator.bids_validator_deno: 'bids-validator-deno'>, validator_version='2.2.1', standard=<Standard.BIDS: 'BIDS'>, standard_version=None, standard_schema_version='1.1.0') scope=<Scope.FILE: 'file'> origin_result=Issue(code='JSON_SCHEMA_VALIDATION_ERROR', subCode='Authors', severity=<Severity.error: 'error'>, location='/dataset_description.json', issueMessage='must be array\n\nField description: List of individuals who contributed to the creation/curation of the dataset.\n', suggestion=None, affects=None, rule='rules.dataset_metadata.dataset_authors', line=None, character=None) severity=<Severity.ERROR: 40> asset_paths=None within_asset_paths=None dandiset_path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049') dataset_path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049') message='Invalid JSON sidecar file. The sidecar is not formatted according the schema.\nsubCode: Authors\nissueMessage: must be array\n\nField description: List of individuals who contributed to the creation/curation of the dataset.\n' metadata=None path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049/dataset_description.json') path_regex=None
2025-10-30T14:58:37-0400 [WARNING ] dandi 21478:126210317428416  Error 5: id='BIDS.NOT_INCLUDED' origin=Origin(type=<OriginType.VALIDATION: 'VALIDATION'>, validator=<Validator.bids_validator_deno: 'bids-validator-deno'>, validator_version='2.2.1', standard=<Standard.BIDS: 'BIDS'>, standard_version=None, standard_schema_version='1.1.0') scope=<Scope.FILE: 'file'> origin_result=Issue(code='NOT_INCLUDED', subCode=None, severity=<Severity.error: 'error'>, location='/dandiset.yaml', issueMessage=None, suggestion=None, affects=None, rule=None, line=None, character=None) severity=<Severity.ERROR: 40> asset_paths=None within_asset_paths=None dandiset_path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049') dataset_path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049') message='Files with such naming scheme are not part of BIDS specification. This error is most commonly caused by typos in file names that make them not BIDS compatible. Please consult the specification and make sure your files are named correctly. If this is not a file naming issue (for example when including files not yet covered by the BIDS specification) you should include a ".bidsignore" file in your dataset (see https://github.com/bids-standard/bids-validator#bidsignore for details). Please note that derived (processed) data should be placed in /derivatives folder and source data (such as DICOMS or behavioural logs in proprietary formats) should be placed in the /sourcedata folder.' metadata=None path=PosixPath('/home/bglickman/I74_uCT_Lichtman/000049/dandiset.yaml') path_regex=None
2025-10-30T14:58:37-0400 [ERROR   ] dandi 21478:126210317428416 Error uploading /home/bglickman/I74_uCT_Lichtman/000049/dataset_description.json:

[yarikoptic]

@candleindark please have a look at this

[candleindark]

In the first pass, the errors seem to be failures in validating the dataset as a BIDS dataset. The 5 errors coming from bids_validator_deno are logged as a warning through the Python log system. The bids_validator_deno we put in awhile back is more restrictive than the previous validator.

@kabilar Is it possible to have the collaborator to validate the dataset as a BIDS dataset at https://bids-standard.github.io/bids-validator/ or using https://pypi.org/project/bids-validator-deno/?

Note (to self): The complain about dandiset.yaml shouldn't be there. The BIDS validator should be configured to ignore that in dandi-cli.

[satra]

@candleindark and @yarikoptic - just a bit of a note for the bids datasets that for many of these projects (not this one specifically), data are being generated in pieces at different institutions on the same samples (similar to dandiset 26). so many sites will be doing partial uploads. i just want to make sure that the cli will understand this. i can't remember now if this was implemented or not. i.e it should assume validity online (we can force a check on the archive in the future), and then ask if the files that are being uploaded are valid in the context of the dataset. this is more complicated, but most folks for these large datasets will not be able to download (or better should not have to download the dataset).

at present i believe @kabilar has been instructing people to skip validation, which is not a good thing.

[candleindark]

for many of these projects (not this one specifically), data are being generated in pieces at different institutions on the same samples (similar to dandiset 26). so many sites will be doing partial uploads. i just want to make sure that the cli will understand this. i can't remember now if this was implemented or not.

Partial upload is implemented according to the documentation. One an upload specific files in a Dandiset. There is an --existing option in the upload subcommand to handle the situation in which a file to be uploaded already exists on the server.

at present i believe @kabilar has been instructing people to skip validation, which is not a good thing.

The upload subcommand in dandi-cli has the --validation skip option. Is this what you are referring to, @satra? How would you do it otherwise if you are to assume the validity of the data to be valid at the time of upload?

[satra]

The upload subcommand in dandi-cli has the --validation skip option. Is this what you are referring to

yes, that. i would expect that a virtual dataset is created and the validation of names/paths is done over that virtual dataset. the current bids validator doesn't do that (i believe) and will also need access to the content, which makes this more tricky.

there was an effort in the cli for a light-weight validator that just checked paths (which is where most of the people make their mistakes).

so ideally when --existing is passed, the validation step will treat them together as a virtual dataset.

[candleindark]

@kabilar I may have gotten the intend of this issue wrong initially.

The issue you are reporting is about the apparent inconsistency in the log levels of the logged warnings/errors and not about the warnings/errors themselves, right?

[yarikoptic]

yes, originally we were validating ONLY the filepaths to be legit using bidsschematools, now we run bids-validator-deno, thus pretty much relying on having a full bids dataset locally. With --validation skip we would skip all validations. I initiated a dedicated

• upload,validate: Add --validators option #1737

as for actual original errors:

• those warnings are legit WARNINGS (not ERRORS!) and point to deficiencies (not problems per se) where RECOMMENDED (not REQUIRED) fields of the dataset_description.json are missing which I guess all of them have but all of them have it incomplete
• the ERROR one is about the dandiset.yaml which is not part of the bids spec . Ideally we should instruct bids-validator to ignore that file alone! ATM we only instruct (is that part of the log?)

❯ git grep -5 .bidsignore
dandi/validate.py-                        == dandiset_metadata_file
dandi/validate.py-                    ):
dandi/validate.py-                        r.message = (
dandi/validate.py-                            f"The dandiset metadata file, `{dandiset_metadata_file}`, "
dandi/validate.py-                            f"is not a part of BIDS specification. Please include a "
dandi/validate.py:                            f"`.bidsignore` file with specification to ignore the "
dandi/validate.py-                            f"metadata file in your dataset. For more details, see "
dandi/validate.py-                            f"https://github.com/bids-standard/bids-specification/"
dandi/validate.py-                            f"issues/131#issuecomment-461060166."
dandi/validate.py-                        )
dandi/validate.py-                    df_results.append(r)

but I think we should take approach similar to openneuro to provide custom configuration file for bids-validator , see https://github.com/bids-standard/bids-validator?tab=readme-ov-file#configuration-file, which would ignore such an error. But I wonder how were we validating all other BIDS dandisets @candleindark ?

[kabilar]

The issue you are reporting is about the apparent inconsistency in the log levels of the logged warnings/errors and not about the warnings/errors themselves, right?

Yes, thanks, the issue is about the inconsistency.

[kabilar]

at present i believe @kabilar has been instructing people to skip validation, which is not a good thing.

In the past, I was indeed instructing people to skip validation if the data were not named correctly and suggested that we name the files correctly in the future. But I have been moving away from this approach. In this case, I worked with the team to organize and name the files correctly, and we did not skip validation.

[kabilar]

Note (to self): The complain about dandiset.yaml shouldn't be there. The BIDS validator should be configured to ignore that in dandi-cli.

the ERROR one is about the dandiset.yaml which is not part of the bids spec . Ideally we should instruct bids-validator to ignore that file alone! ATM we only instruct (is that part of the log?) but I think we should take approach similar to openneuro to provide custom configuration file for bids-validator , see https://github.com/bids-standard/bids-validator?tab=readme-ov-file#configuration-file, which would ignore such an error.

+1