-
Notifications
You must be signed in to change notification settings - Fork 4
Add initial Python client #66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
peopleig
wants to merge
1
commit into
sdslabs:refactor
Choose a base branch
from
peopleig:python-client
base: refactor
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,3 +7,9 @@ | |
| .TODO | ||
| /databases | ||
| .env | ||
| __pycache__/ | ||
| build/ | ||
| .venv/ | ||
| venv/ | ||
| .pytest_cache/ | ||
| *.egg-info/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,299 @@ | ||
| # VortexDB Python Client | ||
|
|
||
| This is the official Python client for **VortexDB**, a vector database exposed via a gRPC API. | ||
|
|
||
| The client provides a thin, typed, Pythonic wrapper over the VortexDB gRPC interface, handling: | ||
| - connection setup | ||
| - authentication | ||
| - request/response mapping | ||
| - error translation | ||
| - resource cleanup | ||
|
|
||
| The client is designed to be minimal, explicit, and easy to extend. | ||
|
|
||
| --- | ||
|
|
||
| ## Requirements | ||
|
|
||
| - Python 3.9+ | ||
| - A running VortexDB gRPC server | ||
|
|
||
| --- | ||
|
|
||
| ## Installation (local / development) | ||
|
|
||
| At the moment, the client lives inside the main VortexDB repository. | ||
|
|
||
| From `client/python`: | ||
|
|
||
| ```bash | ||
| python -m venv .venv | ||
| source .venv/bin/activate | ||
|
|
||
| pip install -e . | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Configuration and Authentication | ||
|
|
||
| The client communicates with VortexDB over gRPC and requires: | ||
|
|
||
| - gRPC endpoint (host:port) | ||
| - API key (maps to `GRPC_ROOT_PASSWORD` on the server) | ||
|
|
||
| These can be provided either: | ||
|
|
||
| - explicitly when constructing the client, or | ||
| - via environment variables | ||
|
|
||
| --- | ||
|
|
||
| ## Basic Usage | ||
|
|
||
| All examples and methods of using the client are in the `examples` directory, in `*_usage.py` files | ||
| A full working example of the basic ways to use the client is available in: | ||
| `examples/basic_usage.py` | ||
|
|
||
| ### Context Manager Support | ||
|
|
||
| The client supports usage as a context manager, which automatically closes the underlying gRPC channel connection | ||
| Example available in: | ||
| ```examples/context_manager_usage.py``` | ||
|
|
||
| --- | ||
|
|
||
| ## Client API | ||
|
|
||
| ### `VortexDB` | ||
|
|
||
| Main client class for interacting with the VortexDB gRPC server. | ||
|
|
||
| #### **Constructor** | ||
|
|
||
| ``` | ||
| VortexDB( | ||
| grpc_url: str | None = None, | ||
| api_key: str | None = None, | ||
| timeout: float | None = None, | ||
| ) | ||
| ``` | ||
| `grpc_url`: gRPC server address (`host:port`) | ||
| `api_key`: API key for authentication | ||
| `timeout`: per-request timeout in seconds | ||
|
|
||
| --- | ||
|
|
||
| #### **Insert** | ||
|
|
||
| Insert a vector with an associated payload | ||
| ``` | ||
| insert(*, vector: DenseVector, payload: Payload) -> str | ||
| ``` | ||
|
|
||
| Returns | ||
| - `point_id` (UUID string) | ||
|
|
||
| Raises | ||
| - `TypeError` if `vector` is not a `DenseVector` | ||
| - gRPC-mapped errors (see Error Handling) | ||
|
|
||
| --- | ||
|
|
||
| #### **Get** | ||
|
|
||
| Fetch a point by its ID | ||
| ``` | ||
| get(*, point_id: str) -> Point | None | ||
| ``` | ||
|
|
||
| Returns | ||
| - `Point` if found | ||
| - `None` if the point does not exist | ||
|
|
||
| --- | ||
|
|
||
| #### **Search** | ||
|
|
||
| Search for nearest neighbours to a query vector | ||
| ``` | ||
| search( | ||
| *, | ||
| vector: DenseVector, | ||
| similarity: Similarity, | ||
| limit: int, | ||
| ) -> list[str] | ||
| ``` | ||
|
|
||
| Returns | ||
| - List of `point_id` strings | ||
|
|
||
| Raises | ||
| - `TypeError` if `vector` is not a `DenseVector` | ||
| - `InvalidArgumentError` for invalid parameters | ||
|
|
||
| --- | ||
|
|
||
| #### **Delete** | ||
|
|
||
| Delete a point by its ID | ||
| ``` | ||
| delete(*, point_id: str) -> None | ||
| ``` | ||
|
|
||
| Raises | ||
| - `NotFoundError` if the point does not exist | ||
|
|
||
| --- | ||
|
|
||
| #### **Close** | ||
|
|
||
| Close the underlying gRPC channel | ||
| ``` | ||
| close() -> None | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Models | ||
|
|
||
| The client exposes typed models thatt represent VortexDB concepts and handle | ||
| validation and protobuf conversion internally | ||
|
|
||
| ### `DenseVector` | ||
|
|
||
| ``` | ||
| DenseVector(values: list[float] | tuple[float, ...]) | ||
| ``` | ||
| - Validates numeric input | ||
| - Normalizes values to `float` | ||
| - Immutable (`frozen=True`) | ||
|
|
||
| --- | ||
|
|
||
| ### `Payload` | ||
|
|
||
| ``` | ||
| Payload(content_type: ContentType, content:str) | ||
| ``` | ||
| Factory Helpers: | ||
| - `Payload.text(content: str)` | ||
| - `Payload.image(content: str)` | ||
|
|
||
| --- | ||
|
|
||
| ### `Point` | ||
|
|
||
| ``` | ||
| Point( | ||
| id: str, | ||
| vector: DenseVector, | ||
| payload: Payload, | ||
| ) | ||
| ``` | ||
| Additional `pretty()` method provided to properly format output | ||
| All fields are directly accessible: | ||
| - `point.id` | ||
| - `point.vector` | ||
| - `point.payload` | ||
|
|
||
| --- | ||
|
|
||
| ### `Similarity` | ||
|
|
||
| Enum representing distance functions: | ||
| - `EUCLIDEAN` | ||
| - `MANHATTAN` | ||
| - `HAMMING` | ||
| - `COSINE` | ||
|
|
||
| --- | ||
|
|
||
| ### `ContentType` | ||
|
|
||
| Enum representing payload type: | ||
| - `TEXT` | ||
| - `IMAGE` | ||
|
|
||
| --- | ||
|
|
||
| ## Error Handling | ||
|
|
||
| The client maps gRPC status codes to Python exceptions to provide a clean, Pythonic error-handling experience. | ||
| All client exceptions inherit from `VortexDBError` | ||
|
|
||
| ### Exception Mapping | ||
|
|
||
| | gRPC Status Code | Python Exception | | ||
| | :--- | :--- | | ||
| | `UNAUTHENTICATED` | `AuthenticationError` | | ||
| | `NOT_FOUND` | `NotFoundError` | | ||
| | `INVALID_ARGUMENT` | `InvalidArgumentError` | | ||
| | `DEADLINE_EXCEEDED` | `TimeoutError` | | ||
| | `UNAVAILABLE` | `ServiceUnavailableError` | | ||
| | Any other error | `InternalServerError` | | ||
|
|
||
| --- | ||
|
|
||
| ## Testing | ||
|
|
||
| Tests are written using **pytest** | ||
| Test coverage includes: | ||
| - models | ||
| - configuration loading | ||
| - gRPC connection layer | ||
| - client API | ||
|
|
||
| Tests live in the `tests/` directory | ||
|
|
||
| To run the tests: `pytest -v` | ||
|
|
||
| --- | ||
|
|
||
| ## Proto and gRPC Stubs | ||
|
|
||
| The gRPC interface is defined using a Protocol Buffers (`.proto`) file, from which Python gRPC stubs are generated. | ||
|
|
||
| ### Proto | ||
|
|
||
| The `.proto` file is kept here for transparency and reproducibility reasons. | ||
|
|
||
| **Location:** `proto/vector-db.proto` | ||
|
|
||
|
|
||
| Even though the gRPC server is already running and exposes these methods, the client still needs the proto to: | ||
| - Generate strongly-typed request / response classes | ||
| - Generate the gRPC client stub (`VectorDBStub`) | ||
|
|
||
| --- | ||
|
|
||
| ### Generated Python stubs | ||
|
|
||
| **Location:** `vortexdb/grpc/` | ||
| - `vector_db_pb2.py` | ||
| - `vector_db_pb2_grpc.py` | ||
|
|
||
| These files are **auto-generated** from `vector-db.proto` and should not be edited manually. | ||
| The client internally wraps this stub. End users never interact with it directly. | ||
|
|
||
| --- | ||
|
|
||
| ### Regenerating the stubs | ||
|
|
||
| Regeneration will be only required if the `.proto` file is changed. For example, if: | ||
| - a new RPC is added | ||
| - enums are updated | ||
|
|
||
| From the client's top-level directory, run: | ||
|
|
||
| ```bash | ||
| python -m grpc_tools.protoc \ | ||
| -I proto \ | ||
| --python_out=vortexdb/grpc \ | ||
| --grpc_python_out=vortexdb/grpc \ | ||
| proto/vector-db.proto | ||
| ``` | ||
|
|
||
| After running this: | ||
| - `vector_db_pb2_grpc.py` and `vector_db_pb2.py` will be updated | ||
| - No other client code should need changes | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| from vortexdb import VortexDB | ||
| from vortexdb import DenseVector, Payload, Similarity # from vortexdb.models | ||
|
|
||
| # This file is written to show the user an example of how VortexDB can be used | ||
| # Further feature updates will be reflected in different *_usage.py files | ||
| # This one shows client initialization and insertion, fetching, search and deletion of vectors | ||
| # Along with that, it shows the usage of a few helper methods in the DenseVector and Point classes | ||
|
|
||
| def main(): | ||
| # Initialize client | ||
| db = VortexDB( | ||
| grpc_url="localhost:50051", | ||
| api_key="my-secret-password", | ||
| ) | ||
|
|
||
| # Insert a vector | ||
| point_id = db.insert( | ||
| vector=DenseVector([0.1, 0.2, 0.3]), | ||
| payload=Payload.text("hello world"), | ||
| ) | ||
| print("Inserted point:", point_id) | ||
|
|
||
| # Get the point | ||
| # Point.pretty() exists to provide a readable output of a Point | ||
| point = db.get(point_id=point_id) | ||
| print("Fetched point:", point.pretty()) | ||
| print("DenseVector as a list:", point.vector.to_list()) | ||
|
|
||
| # Search | ||
| results = db.search( | ||
| vector=DenseVector([0.1, 0.2, 0.3]), | ||
| similarity=Similarity.COSINE, | ||
| limit=3, | ||
| ) | ||
| print("Search results:", results) | ||
|
|
||
| # Delete | ||
| db.delete(point_id=point_id) | ||
| print("Deleted point") | ||
|
|
||
| # Close connection | ||
| db.close() | ||
|
|
||
| if __name__ == "__main__": | ||
| main() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| from vortexdb import VortexDB, DenseVector, Payload, Similarity | ||
|
|
||
| # This file exists to shows how to use VortexDB as a Context Manager | ||
| # The gRPC connection is automatically opened on entry and closed when the context exits | ||
|
|
||
| def main(): | ||
| with VortexDB( | ||
| grpc_url="localhost:50051", | ||
| api_key="my-secret-password", | ||
| ) as db: | ||
|
|
||
| # Insert a vector | ||
| point_id = db.insert( | ||
| vector=DenseVector([0.1, 0.2, 0.3]), | ||
| payload=Payload.text("hello world"), | ||
| ) | ||
| print("Inserted point:", point_id) | ||
|
|
||
| # Get the point | ||
| point = db.get(point_id=point_id) | ||
| print("Fetched point:", point.pretty()) | ||
|
|
||
| # Search | ||
| results = db.search( | ||
| vector=DenseVector([0.1, 0.2, 0.3]), | ||
| similarity=Similarity.COSINE, | ||
| limit=3, | ||
| ) | ||
| print("Search results:", results) | ||
|
|
||
| # Delete | ||
| db.delete(point_id=point_id) | ||
| print("Deleted point") | ||
|
|
||
| # At this point, the gRPC channel is closed automatically | ||
| print("Connection closed") | ||
|
|
||
| if __name__ == "__main__": | ||
| main() |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The spelling is incorrect. "thatt" should be "that".