This repository was archived by the owner on Oct 30, 2024. It is now read-only.
This repository was archived by the owner on Oct 30, 2024. It is now read-only.
enhance: unpack archives before ingestion #100
Description
Currently, handling errors coming from ingesting documents within an archive can get pretty nasty.
Also, the way we do it right now cannot be parallelized for speedup and first collects all documents/chunks before doing the embedding. Missed opportunity for concurrency here and highly error-prone.