Distribution code included when comparing files without extensions

[CarterZenke]

The workaround to comparing .txt files is currently to remove their file extensions. Doing so, however, seems to break the exclusion of distribution code from matches.

For example, including this file as distribution code:

compare50 indicates that the distribution file has been found:

Yet the resulting matches still include the text in the distribution code file as matched:

[ivanharvard]

I believe this happens because plaintext files are either stored as one big token or as several large, broad tokens. So, to compare50, there are practically no overlapping fingerprints between the distro code and the provided submissions.

Consider an example in which plaintext files are stored as an array, and each element of the array are different lines. Now take two identical plaintext files, test1.txt and test2.txt, that both are stored as [A, B, C]. Let's assume the hash of a token is just the length of the token, for simplicity's sake. Since our plaintext lexer reads this as just one token, then the token for both test1.txt and test2.txt is Token([A, B, C]) with a hash of 3.

However, if we had a distro code (distro.txt) with lines [A, B], then we would store this token as Token([A, B]) with a hash of 2. Since the hash of these two tokens are not the same, they are not considered to be equal. As such, the distro code never gets ignored, because compare50 believes the distro code isn't even in test1.txt or test2.txt in the first place. You can confirm this by setting the distro code to be one of test1.txt or test2.txt, which causes a "no similarities found" message to appear, despite the fact that [A, B] exists in [A, B, C].

We can try to limit the frequency of this by interpreting the submitted files as code first (see #119). When we interpret .txt files as code, then so long as the distro code is interpreted as the correct language, then the files in the same (inferred) language do not produce matching groups if these groups were also in the distro code.