Merge back Python CLI to main #35
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Python CLI Package for On-Prem Stack This PR adds: 1. Python CLI package for on-prem stack - Single-line installation via `pip install liquidai-cli` - YAML-based configuration management - Modular CLI design spanning stack, model, database, and infrastructure operations - Type-safe configuration and Docker container management - Interactive and automated workflows - Advanced model management integrating Hugging Face and vLLM 2. Development tooling: - black for code formatting (like prettier) - flake8 for linting (like eslint) - pre-commit hooks for automated checks (like husky) - Python-specific gitignore 3. Unit Tests and CI: - Tests for core helper functions (config and docker utilities) - GitHub Actions workflow for CI - Coverage reporting with Codecov - Multi-Python version testing (3.8-3.12) Link to Devin run: https://app.devin.ai/sessions/f75525e7bf3744dc9044bf4bab5d187b --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: liren@liquid.ai <liren@liquid.ai>
# Summary
A new command `liquidai model run-model-image` is added to launch a
model with our model images. Also refactor the stack
launch/test/shutdown to work with the new mode launch approach. The new
`docker-compose.yaml` file only describes the python-api, DB and web
containers. All model containers will be managed and launched by the
python `model` commands.
The end-to-end test flow are list here:
## Launch
```
(base) ubuntu@192-222-52-66:~/git/on-prem-stack/python_cli$ uv run liquidai stack launch
Setting JWT_SECRET
Setting API_SECRET
Setting AUTH_SECRET
Setting STACK_VERSION
Setting MODEL_IMAGE
Setting MODEL_NAME
Setting POSTGRES_DB
Setting POSTGRES_USER
Setting POSTGRES_PASSWORD
Setting POSTGRES_PORT
Setting POSTGRES_SCHEMA
Setting DATABASE_URL
WARN[0000] a network with name liquid_labs_network exists but was not created for project "python_cli".
Set `external: true` to use an existing network
[+] Running 3/3
✔ Container liquid-labs-postgres Healthy 12.0s
✔ Container liquid-labs-web Healthy 12.0s
✔ Container liquid-labs-python-api Healthy 17.0s
Creating volume for model data: lfm-7b-e
Loading model data from image: liquidai/lfm-7b-e:0.0.1
Launching model container: lfm-7b-e
Model 'lfm-7b-e' started successfully
Please wait 1-2 minutes for the model to load before making API calls
The on-prem stack is now running.
```
## Stop a model
```
(base) ubuntu@192-222-52-66:~/git/on-prem-stack/python_cli$ uv run liquidai model stop lfm-7b-e
Stopped and removed container: lfm-7b-e
```
## Launch a new model
```
(base) ubuntu@192-222-52-66:~/git/on-prem-stack/python_cli$ uv run liquidai model run-model-image --name lfm-3b-e --image "liquidai/lfm-3b-e:0.0.6"
Creating volume for model data: lfm-3b-e
Loading model data from image: liquidai/lfm-3b-e:0.0.6
Launching model container: lfm-3b-e
Model 'lfm-3b-e' started successfully
Please wait 1-2 minutes for the model to load before making API calls
```
## Test a model
```
(base) ubuntu@192-222-52-66:~/git/on-prem-stack/python_cli$ uv run liquidai stack test
Testing API call to get available models...
{'data': [{'id': 'lfm-3b-e', 'status': 'running'}]}
Testing model call...
Testing model: lfm-3b-e
{'id': 'chatcmpl-59437b10fb3a4a99aa7d279632d3e1f2', 'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'content': 'Silver melts at approximately 961.78 degrees Celsius (1,763.10 degrees Fahrenheit). This is a key property of the metal, which is used in various applications due to its high melting point and thermal conductivity.', 'role': 'assistant', 'tool_calls': []}}], 'created': 1745346397, 'model': 'lfm-3b-e', 'object': 'chat.completion', 'usage': {'completion_tokens': 51, 'prompt_tokens': 40, 'total_tokens': 91}}
```
## Shutdown the stack
```
(base) ubuntu@192-222-52-66:~/git/on-prem-stack/python_cli$ uv run liquidai stack shutdown
Stopped and removed model container: lfm-3b-e
[+] Running 3/3
✔ Container liquid-labs-python-api Removed 0.4s
✔ Container liquid-labs-web Removed 10.2s
✔ Container liquid-labs-postgres Removed 0.2s
Stack has been shut down.
```
Addressed the bugs in devin's original code. Now Python cli supports to
run a local checkpoint.
```
$ uv run liquidai model run-checkpoint --path ~/test_checkpoint/
Model 'lfm-3b-e-checkpoint' started successfully
Please wait 1-2 minutes for the model to load before making API calls
$ uv run liquidai stack test
Testing API call to get available models...
{'data': [{'id': 'lfm-3b-e-checkpoint', 'status': 'running'}]}
Testing model call...
Testing model: lfm-3b-e-checkpoint
{'id': 'chatcmpl-0e8778b43eef4a3da54b42c79f743995', 'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'content': 'Silver melts at approximately 961.78 degrees Celsius (1763.20 degrees Fahrenheit). This is a key property of the metal, which is used in various applications due to its high melting point and thermal conductivity.', 'role': 'assistant', 'tool_calls': []}}], 'created': 1745444813, 'model': 'lfm-3b-e-checkpoint', 'object': 'chat.completion', 'usage': {'completion_tokens': 50, 'prompt_tokens': 43, 'total_tokens': 93}}
```
This PR also brings back the docker health check command, which allows
`docker ps` to recognize the health status of the model containers.
1. `env_file` doesn't exist on the first run. Fixing this issue by touching the file when docker help is initialized. 2. Terminate the stack operations when `docker compose` command fails. 3. Remove unnecessary dependencies.
Following #28, db migration step is moved into the new docker container. Also moved the docker-compose file into the package to make it easier to be referenced as a package resource.
New docker images are distributed with different version codes. This PR enables Python CLI to use different version codes for different images to catch up the latest updates.
During my work, I found that model launch can easily fail due to multiple reasons (not enough memory, vLLM not compatible with the model, etc.) but it is confusing since the Python CLI only prints out the model has started and asks users to wait for its ready. This PR allows run model command to wait for the vLLM container to be ready based on the health check command. Python CLI will show the command to check container logs if the model container fails: ``` $ liquidai model run-model-image --name lfm-7b-e --image "liquidai/lfm-7b-e:0.0.1" Creating volume for model data: lfm-7b-e Loading model data from image: liquidai/lfm-7b-e:0.0.1 Launching model container: lfm-7b-e Model 'lfm-7b-e' started successfully Waiting for model 'lfm-7b-e' to be healthy. This may take a 1-2 minutes... Container lfm-7b-e is not healthy yet. Status: starting Container lfm-7b-e is not healthy yet. Status: starting Container lfm-7b-e is not healthy yet. Status: starting Error: Model 'lfm-7b-e' failed to start serving requests Use `docker logs 4d6a1c51978b` to obtain container loggings. ``` A successful launch looks like ``` $ liquidai model run-model-image --name "lfm-3b-e" --image "liquidai/lfm-3b-e:0.0.6" Creating volume for model data: lfm-3b-e Loading model data from image: liquidai/lfm-3b-e:0.0.6 Launching model container: lfm-3b-e Model 'lfm-3b-e' started successfully Waiting for model 'lfm-3b-e' to be healthy. This may take a 1-2 minutes... Container lfm-3b-e is not healthy yet. Status: starting Container lfm-3b-e is not healthy yet. Status: starting Container lfm-3b-e is not healthy yet. Status: starting Container lfm-3b-e is not healthy yet. Status: starting Container lfm-3b-e is not healthy yet. Status: starting Container lfm-3b-e is not healthy yet. Status: starting Container lfm-3b-e is not healthy yet. Status: starting Model 'lfm-3b-e' has started serving requests. ```
Before this PR, `--port` variable on running model commands will only change the port within Docker container. This PR will force vLLM to be launched on port 9000 and forward them to the port assigned by users. By default, we don't expose this port because users are able to access the model through the labs Python API stack.
# Add GitHub Action for PyPI Publishing This PR adds a GitHub Action workflow that automatically publishes the `liquidai-cli` Python package to PyPI when a new release is created. ## Changes: - Add workflow file that triggers on release creation - Set up testing with Python 3.10, 3.11, and 3.12 - Configure build and publish steps using PyPI credentials ## Required Secrets: Before this can work in production, the following secrets need to be added to the repository: - `PYPI_USERNAME`: PyPI username (or `__token__` if using API token) - `PYPI_API_TOKEN`: PyPI API token for authentication Link to Devin run: https://app.devin.ai/sessions/091656c5a70248bd94734475445a4b72 User: liren@liquid.ai --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: liren@liquid.ai <tuliren@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As title. Conflicts are resolved and the Python CLI vLLM version is updated.