RAT-533: Use smaller text samples for charset detection #612
+8
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ottlinger
changed the title
ottlinger
reviewed
Jan 18, 2026
| */ | ||
| private static Charset detectCharset(final InputStream stream, final DocumentName documentName) throws IOException, UnsupportedCharsetException { | ||
| CharsetDetector encodingDetector = new CharsetDetector(); | ||
| final int bytesForCharsetDetection = 256; |
Contributor
There was a problem hiding this comment.
Should we extract this as a constant and document it properly?
Contributor
There was a problem hiding this comment.
Or use 256 implicitly in the constructor instead?
Author
There was a problem hiding this comment.
Or use 256 implicitly in the constructor instead?
This would have been my preference, but it triggered a Checkstyle failure.
ottlinger
requested changes
Jan 18, 2026
| */ | ||
| private static Charset detectCharset(final InputStream stream, final DocumentName documentName) throws IOException, UnsupportedCharsetException { | ||
| CharsetDetector encodingDetector = new CharsetDetector(); | ||
| final int bytesForCharsetDetection = 256; |
Contributor
There was a problem hiding this comment.
Or use 256 implicitly in the constructor instead?
Tika by default uses the first 12,000 bytes of a document for charset detection. This is an extremely computationally intensive process that checks every byte of the sample against every supported charset, and which also performs ngram-based natural language detection for ISO-8859-1. As a result, the majority of apache-rat runtime is actually spent performing charset detection. Reducing the sample size to 256 bytes reduces the cost of charset detection by over 95%. On my machine, this single change cuts the total runtime of `apache-rat:check` in half.
Author
|
I assume this SonarQube failure is unrelated? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tika by default uses the first 12,000 bytes of a document for charset detection. This is an extremely computationally intensive process that checks every byte of the sample against every supported charset, and which also performs ngram-based natural language detection for ISO-8859-1. As a result, the majority of apache-rat runtime is actually spent performing charset detection.
Reducing the sample size to 256 bytes reduces the cost of charset detection by over 95%. On my machine, this single change cuts the total runtime of
apache-rat:checkin half.