Add "OpenFileIO" proposal for discussion

[foundrytom]

This proposal calls for the creation of a minimal, open source (C/C++) library based on the prior art of the OpenImageIO IOProxy and Houdini File System APIs. Providing a common cross-platform, cross-application extensible abstraction of a random access byte stream. It's purpose is to aid the migration away from fopen et. al. as the de-facto low-level data access mechanism within digital content creation tooling and infrastructure. This is required to allow secure data access via other means without the need for complex user-space file system emulations or elaborate data orchestration techniques.

👉 Rendered version here for 👀 .

[elliotcmorris]

SAMWR

[lgritz]

I like and definitely support this.

I'll be curious to see the proposed C interface, and hoping it isn't too awkward or foreitgn looking for C++ users. OTOH, I suppose I could keep IOProxy but its implementation would change to be just a C++ friendly wrapper around the new thing.

I know it's a placeholder, but I think we should find a better name than OpenFileIO, simply because the "Open*IO" space is getting very crowded and leading to confusion (especially when compressed to "O?IO").

[foundrytom]

I like and definitely support this.

🙏 Thanks Larry.

I'll be curious to see the proposed C interface, and hoping it isn't too awkward or foreitgn looking for C++ users. OTOH, I suppose I could keep IOProxy but its implementation would change to be just a C++ friendly wrapper around the new thing.

Yeah, sorry this wasn't meant to sound set in stone, just a few others had asked about ensuring there was a C layer, for FFIs and such. I'll update to be less committal as to what the exact nature of things might be.

I know it's a placeholder, but I think we should find a better name than OpenFileIO, simply because the "Open*IO" space is getting very crowded and leading to confusion (especially when compressed to "O?IO").

1000% with you on that one! Any ideas?

[e4lam]

I've said this in private conversation but I'll repeat for the record:

For the out of scope part, I agree that resource iteration/discovery should be out of scope and the desire here to make sure that this project should remain small in scope. However, I think directories should be considered as part of the API. Without directories, one cannot sensibly present a file chooser for the artist to choose from many URI schemes (that do use directory schemes) since path components are a natural part of URIs. I understand that database file systems are directory-less, but they should still be able to be subsumed as a file system that merely has no directories. It would be really good if all of the current HDK FS plugin functionality (link to example) can be implemented in terms of this new API.

[foundrytom]

For the out of scope part, I agree that resource iteration/discovery should be out of scope and the desire here to make sure that this project should remain small in scope. However, I think directories should be considered as part of the API. Without directories, one cannot sensibly present a file chooser for the artist to choose from many URI schemes (that do use directory schemes) since path components are a natural part of URIs. I understand that database file systems are directory-less, but they should still be able to be subsumed as a file system that merely has no directories. It would be really good if all of the current HDK FS plugin functionality (link to example) can be implemented in terms of this new API.

Yeah, definitely. Hopefully we can figure a way to layer this so it doesn't over-complicate scenarios such as blind content addressed stores. A great discussion point for the meeting.

@e4lam Is the source for the stock http implementation available? It'd be a great example to see how to handle the mapping, and servers that provide no indexes.

[e4lam]

@foundrytom For the http FS plugin, we don't provide support for listing directories so there's no example for that. The plugin's getIsDirectory()/getContents() implementations merely return false.

[foundrytom]

Capturing some notes: Need to think about the capabilities of some backends - ie: some may not support random access writes, etc...