Skip to content

Conversation

@svenski123
Copy link

PosixSequentialFile as currently implemented opens a file using posix system calls, then uses buffered libc stdio to wrap the file descriptor and perform all subsequent I/O.

As users of PosixSequentialFile handle their own buffering, the additional buffering provided by stdio introduces needless additional copying of memory on I/O. Notably none of the other Posix*File classes (PosixWritableFile, PosixRandomAccessFile, PosixMmapFile, etc.) make any use of libc stdio calls relying exclusively on file descriptors and posix system calls.

Finally some platforms' libc stdio implementations are limited and fail unexpectedly in the face of large numbers of open file description, most notably FreeBSD's libc stdio which causes rocksdb to fail when more than 32,767 files are open.

This pull request removes the FILE file member from PosixSequentialFile and replaces use of stdio fdopen/fread/fseek/fclose calls with the equivalent -/read/lseek/close system calls instead.

Note there is additional code in function PosixHelper::GetQueueSysfsFileValueOfFd() in env/io_posix.cc that uses libc stdio fopen() and friends to read files under /sys/dev/block however this code is all under an #ifdef OS_LINUX condition compile block and so is not present in FreeBSD builds hence was left alone for the purposes of this pull request.

During development the following trivial bugs arose and were fixed in this pull request as well:

  • fix unreferenced parameter error in util/compression.cc when compiling without snappy
  • fix needless loop variable value copying error in utilities/write_batch_with_index/write_batch_with_index_test.cc
  • fix mismapped operation type string in tools/db_bench_tool.cc

…ng double buffering of data and libc stdio limitations on maximum number of open FILEs on certain BSD platforms

- fix unreferenced parameter error in util/compression.cc when compiling without snappy
- fix needless loop variable value copying error in utilities/write_batch_with_index/write_batch_with_index_test.cc
- fix mismapped operation type string in tools/db_bench_tool.cc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant