alternative output format based on JSONlines

[oepen]

Background

I would like to suggest an alternative output mode that replaces BASE64 encoding with a representation based on JSONlines. This would lower storage footprint and allow adding annotations on each segment. Additionally, it could support downstream workflows that avoid the creation of large numbers of smallish (language-specific files) and would likely reduce compute for encoding and decoding.

One of my motivations for suggesting JSONLines-based output is to add in-line annotations on each output ‘text’ (i.e. each line in the files written by warc2text, which I understand corresponds to one HTML document). An obvious candidate would be the result of language detection (possibly including an indication of confidence, if available). One could maybe also record the value of the HTML lang attribute on the document, if deemed relevant. Having available both downstream could enable filtering to comparatively ‘pure’ mono-lingual corpora.

Another aspect in which we might want to preserve more information from text extraction relates to HTML parsing. For different downstream use cases – e.g. preparing mono-lingual corpora for LM training vs. discovery of parallel texts for downstream MT work – I can imagine different heuristics in treating various HTML elements. Currently, my understanding is that warc2text outputs newline-separated ‘segments’, for example for <li>, <input>, <select>, and probably <td> elements. Many of these will prototypically be very short ‘utterances’ with linguistic properties quite unlike regular ‘sentences’.

In selecting text for LM training, one might want to capitalize on ‘running’ text. Length thresholding seems an obvious candidate heuristic, but I could also imagine selectively discarding certain HTML elements, seeking to extract smaller but higher-quality mono-lingual text corpora. To enable downstream experimentation along these lines, warc2text would have to record the HTML element corresponding to each newline-separated segment in its output text.

Observations

I picked a random sample of 137 WARC files from the WIDE15 copy on Cirrus: in October 2022, there were 21340 corresponding text files. I uncompressed and concatenated the text files into one large file, called base64, which comes to 314 gb. Running base64 -d on this data (in my Cirrus home directory, which appears a comparatively slow file system) takes 34m11s (of which 24m21s cpu and 6m38s system time, respectively) and reduces file size to 236 gb (75%); call this file txt. Compression using gzip on these two variants yields

• base64: 404m3s (311m42s, 4m27s), for a resulting file base64.gz of 116 gb (36.9%)
• txt: 272m57s (249m2s, 3m18s), for a resulting file txt.gz of 82 gb (34.7%)

All of the above were on cirrus-login1 (which appeared very lightly loaded while I was running these experiments). This suggests that BASE64 encoding adds substantially in space and time, some 33% in volume and some 49% in (wall clock) compression time.

I also tried gauging the cost associated with locating and opening many small files on the Cirrus file system. Using cat > /dev/null on either the 21340 individual compressed files or the combined base64.gz gave timings of 29m15s (with virtually no cpu and system time) and 4m4s, respectively. I believe there is a substantive overhead introduced by using the file system to bifurcate into language-specific files, potentially multiplying our tens of thousands of WARC files by a factor of around 200. In my experience, HPC operators tend to be weary of data collections organized as millions of smallish files.

JSONlines

My suggestion is to add an alternative line-oriented output format based on JSONLines. Escaping of newlines in this scheme would be dealt with according to standard ANSI C string conventions (as \n, meaning that also literal backslashes and double quotes need to be backslash-escaped; in principle this also applies to other control characters, but I am guessing warc2text may not output any but rather normalize e.g. tabs or form-feeds into plain whitespace).

A candidate enriched output from warc2text could look somewhat like the following:

{"o": 42, "l": "nob", "cld2": ["nob", -0.1681], "h": ["p", "li", "li"], "s": "\"foo\"\n\\bar\\\nbaz"}
{"o": 4711, "l": "nno", "cld2": ["nob", -1.7369], "h": ["p"]" "s": "Eg skriver nynorsk."}

In addition to everything discussed in the introduction above (the lang attribute, CLD2 prediction, including log-probabilities, and a list of HTML contexts), the example introduces one additional key, o, to byte-offsets into the underlying WARC file, i.e. the starting position of the record from which each segment was extracted. This could serve as a (simple and compact) unique key pointing back to the full WARC record, in case one eventually wanted to extract additional meta-information, say the corresponding URL and time of capture.

As a side note, if warc2text outputs no longer would be bifurcated by language codes, one may also consider leaving language identification to a later stage (or asking and interpolating the opinion of multiple tools). In principle, one might want to experiment with different setups here, which would likely become practically easier, either way, by avoiding that warc2text output is organized into separate, language-specific files.

Discussion

For various downstream scenarios, one can discuss the trade-offs in recording additional information in-line, together with the text, vs. writing it into a separate file, where corresponding records could be sequentially aligned, i.e. by running line index. For current consumers of warc2text outputs, much of the additional information proposed above would likely be superfluous, i.e. unnecessary ‘fluff’ that each consumer would have to read over and discard.

In terms of total volume, the additional information may in fact still add less than the current BASE64 encoding (which limits itself to an unnecessarily narrow code book). Nevertheless, it might offer better scalability and greater flexibility over time (e.g. to communicate even more information from the original web page into downstream processing) to keep the actual text and JSON annotations as separate, aligned files.

Even if so, there may be benefits in shifting to a JSONlines encoding for the text too, notably reduced space (a more compact encoding) and an increased choice of tools for decoding warc2text outputs, e.g. either using off-the-shelf JSONlines readers or a ‘leaner’ decoder (making some specific assumptions about JSON conventions in the file) that one may hope will reduce processing time, certainly when compared to the current BASE64 decoding. I attach a code snippet below to illustrate that point (sadly purporting to be a text file, to work around limitations on possible attachments).

jsonl.txt

[akutuzov]

I strongly support moving to jsonl, given that most LM training code uses jsonl anyway. Thus, it would make life easier to everyone, I guess.

[jelmervdl]

I'm not opposed moving to jsonl but do see downsides.

My argument against jsonl (or any metadata) in the same file would be that you always have to parse the entire json payload, store arbitrary types and structure, and stringify it again at the end to maintain that metadata per entry.

One of the upsides of the base64 encoding has been that we can get away with barely any dynamic allocations. It also allows us to only deal with any metadata or faulty encoding issues when we choose to since the base64 encoding doesn't care about it. Invalid Unicode is tolerated, it isn't inside json.

I'm definitely in favour of doing lang id outside of warc2text. Langid could benefit from knowing about the structure of the page (ie deal with pages with boilerplate in English but content in another language, like forums would have) but either we ignore that for now or we could somehow store that in metadata as well?

What I'm missing in this discussion is what is the output of warc2text? Some sort of structure (this is a document, this is a paragraph, this is a list item so don't concatenate it to the next line) and some metadata (mine type, offset in warc, url, more?)

I'm more fearful of moving all the metadata into a single file, and stopping column storage. But as far as I understood that wasn't the plan.

[oepen]

Many thanks for your thoughts, @jelmervdl!

My argument against jsonl (or any metadata) in the same file would be that you always have to parse the entire json payload, store arbitrary types and structure, and stringify it again at the end to maintain that metadata per entry.

Truth be told, I have not thought that much about various types of downstream processing of these files. But I think I can readily see that point. I recall you apply sentence splitting not long after warc2text has done its jobs, presumably serializing the result back into the same file format, but with additional newlines inserted. Yes, in this kind of scenario, one would have to write out all metadata again, alongside the modified string.

But one would not have to parse the full JSON for that. We could impose additional conventions on our use of JSONLines, for example that the s element always comes first on each line. On this view, any metadata would just be a string suffix on the line, which could be read and written as an unstructured character sequence.

One of the upsides of the base64 encoding has been that we can get away with barely any dynamic allocations. It also allows us to only deal with any metadata or faulty encoding issues when we choose to since the base64 encoding doesn't care about it. Invalid Unicode is tolerated, it isn't inside json.

The code snippet I had attached was intended to demonstrate that one could read and unescape the actual string elements from JSONLines in a very lightweight approach, with barely any dynamic allocations and avoiding full JSON parsing. I expect that part would likely be computationally cheaper than full BASE64 decoding.

Regarding invalid strings, yes, (in general) those could not be represented in JSONLines. I realize I have not nearly seen the full breadth of problematic data that we will encounter in WARC files. But could one prescribe that warc2text has to be robust to invalid inputs but shall not generate invalid outputs? In other words, do we want to extract 'text' that we cannot encode as valid UTF-8?

I'm definitely in favour of doing lang id outside of warc2text. Langid could benefit from knowing about the structure of the page (ie deal with pages with boilerplate in English but content in another language, like forums would have) but either we ignore that for now or we could somehow store that in metadata as well?

What I'm missing in this discussion is what is the output of warc2text? Some sort of structure (this is a document, this is a paragraph, this is a list item so don't concatenate it to the next line) and some metadata (mine type, offset in warc, url, more?)

My tentative proposal was to devote a little bit of storage to representing some skeletal structure from the HTML document, i.e. immediate block-level context, the HTML element that established each segment: things like <p>, <li>, <code>, <td>, but not in-line elements like <em> or <span>. For example, I imagine this much could be sufficient to approximate paragraph-like text chunks (for example concatenating consecutive <li> or <option> instances) if we wanted to experiment with paragraph-level language identification. I am assuming we need to compromise and limit ourselves to relatively minimal information that will likely be valuable downstream. The complete HTML tree, for example, to me would seem like overkill (unless proven otherwise :-).

I'm more fearful of moving all the metadata into a single file, and stopping column storage. But as far as I understood that wasn't the plan.

I would be happy to focus discussion on a candidate universe were warc2text outputs pairs of line-aligned files (i.e. column storage), where at least one is encoded as JSONLines and provides a home for relevant metadata from the text extraction phase. Even in this universe, BASE64 encoding the raw strings still seems unnecessarily 'rich' to me. An alternative, also in this dual universe, might be just as JSON strings …

[jelmervdl]

Aaah! I missed the example program you included in your initial post! Indeed, if we're willing to compromise on compatibility (and we can, we're only ingesting our own stuff downstream!) parsing is much less of an issue. Also getting rid of base64 encoding would be quite a nice benefit.

But could one prescribe that warc2text has to be robust to invalid inputs but shall not generate invalid outputs? In other words, do we want to extract 'text' that we cannot encode as valid UTF-8?

This sounds more sane to me than what we did in Paracrawl. warc2text already does some encoding detection & conversion, and we can make that a bit more strict to also validate that documents that say they're utf-8 are indeed entirely valid utf-8.

would be happy to focus discussion on a candidate universe were warc2text outputs pairs of line-aligned files (i.e. column storage), where at least one is encoded as JSONLines and provides a home for relevant metadata from the text extraction phase. Even in this universe, BASE64 encoding the raw strings still seems unnecessarily 'rich' to me. An alternative, also in this dual universe, might be just as JSON strings …

If warc2text produces valid unicode, (and all downstream processing should as well), I think we can safely throw out base64 encoding. We can use null byte delimiters or json strings or whatever. Plenty of options that are more efficient.

I think for me it comes down to: let warc2text produce jsonl with a bit of metadata in there because others might want to use the immediate output of warc2text: that's okay.

If there is the expectation of carrying all the metadata across processing steps: doesn't seem feasible to me¹.

Carrying just enough metadata to reconstruct the rest of it if we need it at some point: a compromise?

My tentative proposal was to devote a little bit of storage to representing some skeletal structure from the HTML document, i.e. immediate block-level context, the HTML element that established each segment: things like <p>, <li>, <code>, <td>, but not in-line elements like <em> or <span>.

That makes sense. I'd like to know what a downstream use case would look like to find the balance between giving them (hypothetical "them") something that's next to useless, and something that requires a lot of effort in implementing² and carrying along all the metadata between processing steps, duplicating it across files and then not being used anywhere.

Footnotes

1. Taking Paracrawl as inspiration, we'd end up with a file containing the original text, sentence split text, tokenized text, translated text, document structure, mimetype, warc origin, url. And all the intermediate files as well, so one without the sentence splitting & tokenisation, one without the translation. After that we do document matching, so for all documents that match carrying over all that metadata for both sides of the match. ↩

2. Work might include replacing/extending the html parser in warc2text to do corrective HTML parsing. Right now it doesn't even keep track of a stack and just blindly trusts open and close tags match up in a document. I think its good enough for the example you gave, not enough for a tree. ↩

[jelmervdl]

Since we keep coming up with more (meta)data we'd like to track, I'm trying to keep a summary.

Currently produced:

• content-type: lower-case mime type according to HTTP response header, or warc header if that's not available.
• url: warc-target-uri of the warc record.
• text content: converted to UTF-8, block elements become new-lines
• raw html content: option of warc2text right now, exports it in its original encoding, not sure if anything in the pipeline uses it or if it is just there because it makes writing tag filters for MT easier

Proposed stuff:

• origin filename, byte offset: so you can track the exact record the data came from, somewhat implemented in Add --jsonl option #35
• scrape date: timestamp recorded by the scraper that's stored in the warc record. Knowing how old your data is sounds like something that could be pretty nice to know, I'd recon.
• language: currently kept as part of the output path. There's partial code switching support, but only what CLD2 already provides out of the box (which is max of 3 languages, and I believe a limitation on possible combinations). Should we try to store this per paragragh?
• paragraph breaks: stored as line breaks in the output currently. Each paragraph could have its own metadata, such as…
• document structure: where was this text found? block-level tags would be sufficient I assume, I don't think there's a reason to separate or store inline elements. Note that warc2text also reads ODF and office files as XML files, so not not exclusively HTML tags here.
• maybe boilerplate: per block level whether this is (likely) part of the website layout, or actual text. Related to the jusText discussion we had at some point. Sampo mentioned he'd rather have the info and make the selection afterwards than that we throw it out at this stage.
• maybe machine translation: currently detected based on whether certain tags appear anywhere in the document. The current approach that just detects certain popular CMS plugins has a low recall I'd imagine.