[List] Offset & Limit

[auniverseaway]

As a developer, I would like the ability to set an offset or limit for long lists.

Criteria of acceptance

1. Ability to set an offset.
2. Ability to set a limit.
3. Move to a top level object (currently is an array)
4. API matches query index offset / limit patterns
5. New route to not break existing usage.
6. Overall list limit is 5,000

[hannessolo]

This is not straight forward to implement because of the way directories are implemented. Each directory has both a .props file as well as the actual prefix.

Passing maxKeys to the s3 client counts both prefixes and .props files to get the number of keys. So worst case (listing only directories) you actually get only limit / 2 keys, but usually somewhere in-between. It affects the offset parameter too.

The way get around this would be to limit the user's limit param to the physical limit of 1000 divided by 2, so 500, but fetch max 1000 keys. Then limit it to 500 client-side.

This leads to some complexity in the code though, because the cursor that the S3 client returns will be >500, so we need to reconcile when the next page is requested.

[hannessolo]

Same goes for hidden files. Actually, super edge case, but if someone has let's say 10 hidden files, and fetches with limit=5, even the fetch 2x limit trick wouldn't work anymore - they would get 0 results.

So best option seems to be to fetch in a loop, until the requested limit is reached.