Use file descriptors and not c-lib FILE pointers on xBSD

[sonofmom]

Expected behavior

RocksDB should be able to open more then 32767 files on xBSD.

Actual behavior

RocksDB will fail when opening file pointer with number higher then 32767. This is due to following code in FreeBSD (and other xBSD incarnation) C-Lib: https://github.com/freebsd/freebsd-src/blob/main/lib/libc/stdio/fdopen.c#L64-L67.

And this is the line of RocksDB code that invokes fdopen: https://github.com/facebook/rocksdb/blob/master/env/fs_posix.cc#L185

The problem is not caused by OS limits, this is the issue with xBSD not being able to handle a FP with number higher then 32767.

Steps to reproduce the behavior

The problem has been reproduced on FreeBSD v12.2 and v13.0 using latest RocksDB master and large dataset with > 32767 files. Any attempt to open such dataset will result in EMFILE (too many open files) error.

Suggested change

On FreeBSD Forum change from file pointers to file descriptors has been suggested: https://forums.FreeBSD.org/threads/opening-more-then-32767-file-descriptors.79780/post-505682

[zhichao-cao]

@akankshamahajan15 You may take a look? Looks like it tis indicating the blob?

[akankshamahajan15]

Looks like we can open the file using file descriptor instead of file pointer for FreeBSD. @sonofmom It would be great if you can contribute the fix.

[sonofmom]

Looks like we can open the file using file descriptor instead of file pointer for FreeBSD. @sonofmom It would be great if you can contribute the fix.

Thank you, will do.

[sonofmom]

Hi @akankshamahajan15 ,

After looking at the code and general usage of raw FDs with sequential IO I must admit that I am very reluctant to do changes. My C experience is all in embedded field where high performance I/O is of little interest while here we are dealing with advanced C I/O coding. I am afraid that anything I would do would cause serious performance degradation on xBSD systems, if not worse.

Maybe we should list this issue as "up-for-grabs" after all, I will continue looking into this but there is little chance that I will be able to implement anything useful and stable within reasonable timeframe.

[svenski123]

I've opened this PR which replaces the use of stdio in PosixSequentialFile with system call I/O.
I would expect this to work better on all platforms as it eliminates one round of buffer copying.
Users of PosixSequentialFile already provide their own buffers which the OS will now return to,
rather than first filling the stdio buffer and then copying to the application buffer.

More importantly, this allows rocksdb to work with more than 32,767 open files on platforms with broken stdio implementations such as FreeBSD.

#14229

@sunpoet CC'ing you I believe you're the current maintainer of the rocksdb port in FreeBSD and thought this issue & pull request would be of interest to you.