fread should not create data.tables with more than 2^31 rows

[ecoRoland2]

I just imported a huge file with more than 2^31 rows using fread on a machine with 512 GB RAM. It worked great. Unfortunately, I couldn't do anything with the resulting data.table. Autoprinting or subsetting returned a "long vectors not supported yet" error, which isn't surprising.

I suggest to stop fread before the data.table gets too large. Maybe it could return (with a warning) as many rows as R supports.

The fread documentation mentions field width limits but I don't see a mention that R has a limit regarding the number of rows.

My solution was to use nrows and skip to import as two data.tables but that was after I wasted time with reading the whole file.

[MichaelChirico]

Related: #570, #3957.

Rather than turn off this support, I'd rather add support -- could you be more specific about which operations are failing?

[ecoRoland2]

Happy to do so. Feel free to ask me to test other operations. However, I'm using the server too much right now to install package versions from github. That would have to wait until I don't have looming deadlines.

I'm using R 4.5.1 and data.table 1.17.8 via RStudio Server 2024.12.1+563 "Kousa Dogwood" Release (27771613951643d8987af2b2fb0c752081a3a853, 2025-02-02) for Ubuntu Jammy.

The data.table has about 3e9 rows and 4 numeric columns. Looks like the main issue is in dim.data.table but there might be others down the line.

DT <- fread("<file path>", nThread = 5)

DT
#Error in dim.data.table(x) : 
#  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:551

print(DT)
#Error in dim.data.table(x) : 
#  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:551

nrow(DT)
#Error in dim.data.table(x) : 
#  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:551

DT[1:10]
#Error in dim.data.table(x) : 
#  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:551

setnames(DT, 1, "test")
#works

DT[, .N, by = test] #result would have about 36e6 rows
#Error in dim.data.table(x) : 
#  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:551

DT[, lapply(.SD, class)]
#Error in dim.data.table(x) : 
#  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:551

[ben-schwen]

Fixing dim opens the can of worms that dim can return either integer or double depending on the size. Might even fully switch to double if we want long vector support.

[jangorecki]

There is a value in following base R convention on that, although it may not be very robust. If we go with always double then we could provide transition option, disable long vector, which keeps everything as it was.

[aitap]

If we could pull this off, this could be a great competitive advantage, because none of the other data.frames can handle long vectors:

> data.frame(raw(2^31))
Error in if (mirn && nrows[i] > 0L) { :
  missing value where TRUE/FALSE needed
In addition: Warning message:
In attributes(.Data) <- c(attributes(.Data), attrib) :
  NAs introduced by coercion to integer range
> S4Vectors::DataFrame(x = raw(2^31))
Error in S4Vectors::DataFrame(x = raw(2^31)) :
  cannot coerce class "raw" to a DataFrame
In addition: Warning message:
In new_DataFrame(setNames(list(from), "X"), nrows = length(from)) :
  NAs introduced by coercion to integer range
> tibble::tibble(raw(2^31))
Error in `new_tibble()`:
! `nrow` must be a nonnegative whole number smaller than 2^31.
Run `rlang::last_trace()` to see where the error occurred.

Introducing long vector support will require replacing the approximately grep -rPic '(?<!x|TRUE|SET)length *\(' src/ | awk '{sub(".*:","");a+=$0}END{print a}' = 634 uses of length() with xlength() and R_len_t/int used as an index type with R_xlen_t. Eventually we'll bump against R's own limits (a large portion of R itself is not ready for dim() to return anything but an INTSXP vector, especially for matrices), but it could be interesting to see how far can this be taken.