[docs] refresh workflows and roadmap

rodrigopitanga · rodrigopitanga · commit 6c847a653c23 · 2025-11-16T15:39:11.000-03:00
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -1,62 +1,81 @@
 # 👾 Contributing to PatchVec
 
-We welcome PRs, issues, and feedback!
+Patchvec accepts code and docs from people who ship patches. Follow the steps below and keep PRs focused.
 
----
+## Environment setup
 
-## 🛠 Dev Setup
 ```bash
-python -m venv .venv
-source .venv/bin/activate
-python -m pip install --upgrade pip
-pip install -r requirements-test.txt   # includes runtime + test deps
-```
+# clone and enter the repo first
+git clone https://github.com/patchvec/patchvec.git
+cd patchvec
 
-## 🧪 Testing
-```bash
-pytest -q
+# GPU deps by default; add USE_CPU=1 if you do not have a GPU
+make install-dev
+
+# copy local config overrides if you need to tweak behaviour
+cp config.yml.example config.yml
+cp tenants.yml.example tenants.yml
+
+# optional: run the service right away
+USE_CPU=1 make serve
 ```
-CI/CD blocks releases if tests fail.
 
-## ▶️ Dev Server
+Run the test suite before pushing (`USE_CPU=1` if you installed CPU wheels):
+
 ```bash
-# CPU-only deps by default
-make serve
-# or explicitly
-HOST=0.0.0.0 PORT=8080 RELOAD=1 WORKERS=1 LOG_LEVEL=info ./pavesrv.sh
+# USE_CPU=1 if you installed CPU deps
+make test
 ```
 
+Need to inspect behaviour without reloads? After tweaking `config.yml` / `tenants.yml`, run `AUTH_MODE=static PATCHVEC_AUTH__GLOBAL_KEY=<your-secret> DEV=0 make serve` for an almost production-like stack, or call the wrapper script directly: `PATCHVEC_AUTH__GLOBAL_KEY=<your-secret> ./pavesrv.sh`.
+
+## Workflow
+
+1. Fork and clone the repository.
+2. Create a branch named after the task (`feature/tenant-search`, `fix/csv-metadata`, etc.).
+3. Make the change, keep commits scoped, and include tests when possible.
+4. Run `make test` and `make check` if you touched deployment or packaging paths.
+5. Open a pull request referencing the issue you claimed.
+
+Use imperative, lowercase commit messages (`docs: clarify docker quickstart`).
+
+## Issues and task claims
+
+- `ROADMAP.md` lists chores that need owners.
+- To claim a task, open an issue titled `claim: <task>` and describe the approach.
+- Good first issues live under the `good-first-issue` label. Submit a draft PR within a few days of claiming.
+
+## Code style
+
+- Prefer direct, readable Python. Keep imports sorted and avoid wildcard imports.
+- Follow PEP 8 defaults, keep line length ≤ 88 characters, and run `ruff` locally if you have it installed.
+- Do not add framework abstractions unless they solve a concrete problem.
+- Avoid adding dependencies without discussing them in an issue first.
+
+## Pull request checklist
+
+- [ ] Tests pass locally (`make test`, add `USE_CPU=1` if you installed CPU wheels).
+- [ ] Packaged stack still works (`make check` on a clean checkout).
+- [ ] Docs updated when behavior changes.
+- [ ] PR description states what changed and why.
+
+Ship code, not questions. If you need help, post logs and the failing command instead of asking for permission.
+
+## Architecture
+
+- Stores live under `pave/stores/*` (default txtai/FAISS today, Qdrant stub ready).
+- Embedding adapters reside in `pave/embedders/*` (txtai, sentence-transformers, OpenAI, etc.).
+- `pave/service.py` wires the FastAPI application and injects the store into `app.state`.
+- CLI entrypoints are defined in `pave/cli.py`; shell shims `pavecli.sh`/`pavesrv.sh` wrap the same commands for repo contributors.
+
 ## 🧰 Makefile Targets
-- `make install` — install runtime deps (CPU default; `USE_GPU=1` for GPU)
-- `make install-dev` — runtime + test deps
-- `make serve` — start FastAPI app (uvicorn) with autoreload
-- `make test` — run tests
-- `make build` — build sdist/wheel (includes ABOUT.md)
-- `make package` — create .zip and .tar.gz in ./artifacts
-- `make release VERSION=x.y.z` — update versions (setup.py, main.py, Dockerfile, compose, README tags), prepend CHANGELOG with sorted commits since last tag, run tests/build (must pass), tag & push
-- `make clean` / `make clean-dist` — cleanup
-
-## 🚢 Release Notes
-- Version bumps also update Docker-related version strings where applicable.
-- The release target **will not push** if tests/build fail.
-- Ensure `PYPI_API_TOKEN` is set in CI to publish on tag.
-
-## 🔐 Secrets & Config
-- Don’t commit secrets. Use `.env` (ignored) or env vars in CI.
-- For per-tenant keys, use an untracked `tenants.yml` and reference it from `config.yml` (`auth.tenants_file`).
-- Config precedence: code defaults < `config.yml` < `tenants.yml` < `PATCHVEC__*` env.
-
-## 📝 Commit Style
-Keep commit messages short and scoped:
-```
-search: fix filter parsing
-docs: add curl examples
-build: include ABOUT.md in sdist
-arch: refactored StoreFactory
-```
 
-## 🧩 Architecture (brief)
-- Stores: `pave/stores/*` (default txtai/FAISS, qdrant stub)
-- Embedders: `pave/embedders/*` (default/txtai, sbert, openai)
-- Pure-ish orchestration: `pave/service.py`
-- Dependency injection: `build_app()` wires store via `app.state`
+- `make install` — install runtime deps (CPU wheels by default; `USE_GPU=1` for GPU builds).
+- `make install-dev` — runtime + test deps for contributors.
+- `make serve` — start the FastAPI app (uvicorn) with autoreload (`USE_CPU=1` for CPU-only setups).
+- `make test` — run the pytest suite.
+- `make check` — build and smoke-test the container image with the demo corpus.
+- `make build` — build sdist/wheel (includes ABOUT.md).
+- `make package` — create `.zip`/`.tar.gz` artifacts under `./artifacts`.
+- `make release VERSION=x.y.z` — sync version strings, regenerate the changelog, run tests/build, tag & push.
+- `make clean` / `make clean-dist` — remove caches and build outputs.
diff --git a/README.md b/README.md
@@ -1,87 +1,120 @@
 # 🍰 PatchVec — Lightweight, Pluggable Vector Search Microservice
 
-Upload → chunk → index (with metadata) → search via REST and CLI.
+Patchvec is a compact vector store built for people who want provenance and fast iteration on RAG plumbing. No black boxes, no hidden pipelines: every chunk records document id, page, and byte offsets, and you can swap embeddings or storage backends per collection.
 
----
+## ⚙️ Core capabilities
 
-## 🚀 Quickstart
+- **Docker images** — prebuilt CPU/GPU images published to the GitLab Container Registry.
+- **Tenants and collections** — isolation by tenant with per-collection configuration.
+- **Pluggable embeddings** — choose the embedding adapter per collection; wire in local or hosted models.
+- **REST and CLI** — production use over HTTP, quick experiments with the bundled CLI.
+- **Deterministic provenance** — every hit returns doc id, page, offset, and snippet for traceability.
 
-### 🖥️ CPU-only Dev (default)
-```bash
-python -m venv .venv
-source .venv/bin/activate
-python -m pip install --upgrade pip
-pip install -r requirements-cpu.txt
-./pavesrv.sh
-```
+## 🧭 Workflows
+
+### 🐳 Docker workflow (prebuilt images)
+
+Pull the image that fits your hardware from the [https://gitlab.com/flowlexi](Flowlexi) Container Registry on Gitlab (CUDA builds publish as `latest`, CPU-only as `latest-cpu`).
 
-### 📦 Install from PyPI
 ```bash
-python -m venv .venv
-source .venv/bin/activate
-python -m pip install --upgrade pip
-pip install patchvec[cpu]   # or patchvec[gpu] for CUDA
+docker pull registry.gitlab.com/flowlexi/patchvec/patchvec:latest
+docker pull registry.gitlab.com/flowlexi/patchvec/patchvec:latest-cpu
 ```
 
-### ▶️ Run the Server
-From source:
+Run the service by choosing the tag you need and mapping the API port locally:
+
 ```bash
-./pavesrv.sh
+docker run -d --name patchvec \
+  -p 8086:8086 \
+  registry.gitlab.com/flowlexi/patchvec/patchvec:latest-cpu
 ```
-From PyPI:
+
+Use the bundled CLI inside the container to create a tenant/collection, ingest a demo document, and query it:
+
 ```bash
-pavesrv
+docker exec patchvec pavecli create-collection demo books
+docker exec patchvec pavecli upload demo books /app/demo/20k_leagues.txt \
+  --docid=verne-20k --metadata='{"lang":"en"}'
+docker exec patchvec pavecli search demo books "captain nemo" -k 3
 ```
 
-### ⚙️ Minimal Config
-For production (static auth), set env vars (do not commit secrets):
-```env
-PATCHVEC_AUTH__MODE=static
-PATCHVEC_AUTH__GLOBAL_KEY=sekret-passwod
-```
-(Optional: copy `config.yml.example` to an untracked `config.yml` and tweak as needed)
-(Tip: use an untracked `tenants.yml` and point `auth.tenants_file` to it in `config.yml`.)
+See below for REST and UI.
 
----
+Stop the container when you are done:
 
-## 🔧 Overriding Server Settings (uvicorn)
-You can override a few server knobs via environment variables:
 ```bash
-HOST=127.0.0.1 PORT=9000 RELOAD=1 WORKERS=4 LOG_LEVEL=debug pavesrv
+docker rm -f patchvec
 ```
-> Note: Full configuration uses the `PATCHVEC_...` env scheme (e.g., `PATCHVEC_SERVER__PORT=9000`).
 
----
+### 🐍 PyPI workflow
 
-## 🌐 REST API Examples
+Install Patchvec from PyPI inside an isolated virtual environment and point it at a local configuration directory:
 
-**Create a collection**
 ```bash
-curl -X POST "http://localhost:8086/collections/acme/invoices"
-```
+mkdir -p ~/pv && cd ~/pv #or wherever
+python -m venv .venv-pv
+source .venv-pv/bin/activate
+python -m pip install --upgrade pip
+python -m pip install "patchvec[cpu]"   # or "patchvec[gpu]" for CUDA
 
-**Upload a TXT/PDF/CSV document**
-```bash
-curl -X POST "http://localhost:8086/collections/acme/invoices/documents"   -F "file=@sample.txt"   -F "docid=DOC-1"   -F 'metadata={"lang":"pt"}'
-```
+# grab the default configs
+curl -LO https://raw.githubusercontent.com/patchvec/patchvec/main/config.yml.example
+curl -LO https://raw.githubusercontent.com/patchvec/patchvec/main/tenants.yml.example
+cp config.yml.example config.yml
+cp tenants.yml.example tenants.yml
 
-**Search (GET, no filters)**
-```bash
-curl "http://localhost:8086/collections/acme/invoices/search?q=garantia&k=5"
-```
+# sample demo corpus
+curl -LO https://raw.githubusercontent.com/patchvec/patchvec/main/demo/20k_leagues.txt
 
-**Search (POST with filters)**
-```bash
-curl -X POST "http://localhost:8086/collections/acme/invoices/search"   -H "Content-Type: application/json"   -d '{"q": "garantia", "k": 5, "filters": {"docid": "DOC-1"}}'
+# point Patchvec at the config directory and set a local admin key
+export PATCHVEC_CONFIG="$HOME/pv/config.yml"
+export PATCHVEC_GLOBAL_KEY=super-sekret
+
+# option A: run the service (stays up until you stop it)
+pavesrv
+
+# option B: operate entirely via the CLI (no server needed)
+pavecli create-collection demo books
+pavecli upload demo books 20k_leagues.txt --docid=verne-20k --metadata='{"lang":"en"}'
+pavecli search demo books "captain nemo" -k 3
 ```
 
-**Health / Metrics**
+Deactivate the virtual environment with `deactivate` when finished.
+
+### 🌐 REST API and Web UI usage
+
+When the server is running (either via Docker or `pavesrv`), the API listens on `http://localhost:8086`. The following `curl` commands mirror the CLI sequence above—adjust the file path to wherever you stored the corpus (`/app/demo/20k_leagues.txt` in Docker, `~/pv/20k_leagues.txt` for PyPI installs) and reuse the bearer token exported earlier:
+
 ```bash
-curl "http://localhost:8086/health"
-curl "http://localhost:8086/metrics"
+# create collection
+curl -H "Authorization: Bearer $PATCHVEC_GLOBAL_KEY" \
+  -X POST http://localhost:8086/collections/demo/books
+
+# ingest document
+curl -H "Authorization: Bearer $PATCHVEC_GLOBAL_KEY" \
+  -X POST http://localhost:8086/collections/demo/books/documents \
+  -F "file=@20k_leagues.txt" \
+  -F 'metadata={"lang":"en"}'
+
+# run search
+curl -H "Authorization: Bearer $PATCHVEC_GLOBAL_KEY" \
+  "http://localhost:8086/collections/demo/books/search?q=captain+nemo&k=3"
 ```
 
----
+There is a simple Swagger UI available at the root of the server. Just point your browser to `http://localhost:8086/`
+
+Health and metrics endpoints are available at `/health` and `/metrics`.
+
+Configuration files copied in either workflow can be customised. Runtime options are also accepted via the `PATCHVEC_*` environment variable scheme (`PATCHVEC_SERVER__PORT`, `PATCHVEC_AUTH__MODE`, etc.), which precedes conf files.
+
+### 🛠️ Developer workflow
+
+Building from source relies on the `Makefile` shortcuts (`make install-dev`, `USE_CPU=1 make serve`, `make test`, etc.). The full contributor workflow, target reference, and task claiming rules live in [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## 🗺️ Roadmap
+
+Short & mid-term chores are tracked in [`ROADMAP.md`](ROADMAP.md). Pick one, open an issue titled `claim: <task>`, and ship a patch.
 
 ## 📜 License
+
 GPL-3.0-or-later — (C) 2025 Rodrigo Rodrigues da Silva
diff --git a/ROADMAP.md b/ROADMAP.md
@@ -0,0 +1,45 @@
+# Roadmap
+
+Immediate chores worth tackling now. Claim a task by opening an issue titled `claim: <task>` and link your branch or PR when ready.
+
+## 0.5.x Series — Adoption & Stability
+
+### v0.5.7 — Search, Filters, Traceability
+- Expand partial filter support (`*` prefix/fuzzy matching).
+- Return a `match_reason` field alongside every hit.
+- Persist configurable logging per tenant and collection.
+- Provide REST/CLI endpoints to delete a document by id.
+- Expose latency histograms (p50/p95/p99) via `/metrics` for search and ingest requests.
+
+### v0.5.8 — Persistence, Infrastructure
+- Ship an internal metadata/content store (SQLite or TinyDB) with migrations.
+- Serve `/metrics` and `/collections` using the internal store as the source of truth.
+- Emit structured logs with `request_id`, tenant, and latency, and allow rolling retention per tenant/collection.
+- Support renaming collections through the API and CLI.
+- Provide per-tenant and per-operation API rate limits
+
+### v0.5.9 — Ranking, Model Quality
+- Add hybrid reranking (vector similarity + BM25/token matching).
+- Honor `meta.priority` boosts during scoring.
+- Improve multilingual relevance and evaluation fixtures.
+
+## 🚀 Towards 1.0
+
+### 0.6 — Per-Collection Embeddings, Data Types
+- Configure embeddings per collection via `config.yml`.
+- Maintain per-collection hot caches with isolation guarantees.
+- List tenants and collections via the API (CLI parity included).
+- Lay ground to support new data types (audio, video, image)
+
+### 0.7 — API Utilities & Observability
+- Default tenant/collection selectors in Swagger UI.
+- Export indexes, enforce collection-level logging toggles, add rate limiting, and finalize per-collection embedding configuration flows.
+
+### 0.8 — Reliability & Governance
+- Deliver the internal DB for persistence, document versioning, rebuild tooling, persistent metrics, surfacing metrics in the UI, JWT auth, per-tenant quotas, and transactional rollback with safe retries.
+
+### 0.9 — Scale & Multi-Tenant Search
+- Async ingest, parallel purge, horizontal scalability, tenant groups, and shared/group search with sub-index routing.
+
+### 1.0 — API Freeze
+- Lock routes, publish the final OpenAPI spec, and ship an SDK client ready for long-term support.
