Skip to content

Handle large json failures #666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ryanbrandenburg wants to merge 2 commits into
base: master
Choose a base branch
from
+21 −13
Open

Handle large json failures #666

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
34624da
Handle large json failures
ryanbrandenburg Dec 19, 2025
8125198
Cleanup
ryanbrandenburg Dec 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 21 additions & 13 deletions providers/process/scancode.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ class ScanCodeProcessor extends AbstractProcessor {
maxBuffer: 5 * 1024 * 1024
})
} catch (error) {
this.logger.error(error, request.meta)
// TODO see if the new version of ScanCode has a better way of differentiating errors
if (this._isRealError(error) || this._hasRealErrors(file.name)) {
request.markDead('Error', error ? error.message : 'ScanCode run failed')
Expand Down Expand Up @@ -91,19 +92,26 @@ class ScanCodeProcessor extends AbstractProcessor {
// Scan the results file for any errors that are not just timeouts or other known errors
// TODO do we need to do this anymore
_hasRealErrors(resultFile) {
const results = JSON.parse(fs.readFileSync(resultFile))
return results.files.some(
file =>
file.scan_errors &&
file.scan_errors.some(error => {
return !(
error.includes('ERROR: Processing interrupted: timeout after') ||
error.includes('ValueError:') ||
error.includes('package.json') ||
error.includes('UnicodeDecodeError')
)
})
)
try{
const results = JSON.parse(fs.readFileSync(resultFile))
return results.files.some(
file =>
file.scan_errors &&
file.scan_errors.some(error => {
return !(
error.includes('ERROR: Processing interrupted: timeout after') ||
error.includes('ValueError:') ||
error.includes('package.json') ||
error.includes('UnicodeDecodeError')
)
})
)
} catch (e) {
// This might happen if the results file is >512mb, but regardless our error handling should not fail.
this.logger.error(`Could not parse ScanCode results file ${resultFile}: ${e.message}`)
// Assume it's broken if we could not parse it
return true
}
}

_detectVersion() {
Expand Down