Bulk embed and pools

[techiejd]

Add bulk embedding functionality with a new job, run-tracking collection, endpoint, and admin "Embed all" button to enable efficient backfilling and flexible ingest modes.

• New payloadcms-vectorize:bulk-embed-all task handles selecting missing-version docs, delegating to provider callbacks (or the in-process fallback), polling, and reconciling embeddings.
• Ingest mode now supports bulk to skip realtime vectorization, clear stale embeddings on updates, and rely on the bulk pipeline.
• Knowledge pool collections render an admin "Embed all" control that posts to /api/vector-bulk-embed to queue a run.
• Registered the new run collection vector-bulk-embeddings-runs, added the bulk endpoint, updated dev config/helpers for bulk callbacks, and documented ingestMode + bulk callbacks.

---

 

[cursor]

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
_{Learn more about Cursor Agents}

[techiejd]

@cursoragent please start with changes. Looking forward to seeing your work.

[cursor]

Thanks for the go-ahead! I’ve implemented the bulk-embed feature set:

• Added bulk ingest mode and job (payloadcms-vectorize:bulk-embed-all) with provider callback support and in-process fallback.
• New run-tracking collection (vector-bulk-embeddings-runs) plus /api/vector-bulk-embed endpoint and admin “Embed all” button on knowledge pool lists.
• Ingest routing now respects ingestMode (realtime vs bulk), clearing stale embeddings on updates in bulk mode.
• Docs updated for ingestMode and bulk callbacks; added bulk ingest tests and updated config specs.

Tests run: pnpm vitest dev/specs/config.spec.ts (DB-dependent bulk test added but not run here).

If you’d like, I can run the full DB-backed bulk spec suite or prepare a commit/PR summary.

---

 

_{Learn more about Cursor Agents}