Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 123 additions & 0 deletions .github/workflows/check-all-models.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
name: Check All Models

on:
workflow_dispatch:
inputs:
models:
description: 'Specific models to test (space-separated, leave empty for all)'
required: false
type: string
schedule:
# Run daily at 6:00 UTC
- cron: '0 6 * * *'
push:
branches:
- main
paths:
- 'check_models.py'
- 'check_models.js'
- 'requirements.txt'
- 'package.json'
- '.github/workflows/check-all-models.yml'

jobs:
check-models-python:
name: Python Model Check
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt

- name: Check all models
if: github.event.inputs.models == ''
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_API_BASE: ${{ vars.OPENAI_API_BASE || 'https://api.deep.assistant.run.place/v1' }}
run: python check_models.py

- name: Check specific models
if: github.event.inputs.models != ''
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_API_BASE: ${{ vars.OPENAI_API_BASE || 'https://api.deep.assistant.run.place/v1' }}
run: python check_models.py ${{ github.event.inputs.models }}

- name: Upload Python results
if: always()
uses: actions/upload-artifact@v4
with:
name: python-model-check-results
path: |
*.log
retention-days: 30

check-models-javascript:
name: JavaScript Model Check
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'

- name: Install dependencies
run: npm ci

- name: Check all models
if: github.event.inputs.models == ''
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_API_BASE: ${{ vars.OPENAI_API_BASE || 'https://api.deep.assistant.run.place/v1' }}
run: node check_models.js

- name: Check specific models
if: github.event.inputs.models != ''
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_API_BASE: ${{ vars.OPENAI_API_BASE || 'https://api.deep.assistant.run.place/v1' }}
run: node check_models.js ${{ github.event.inputs.models }}

- name: Upload JavaScript results
if: always()
uses: actions/upload-artifact@v4
with:
name: javascript-model-check-results
path: |
*.log
retention-days: 30

summary:
name: Test Summary
runs-on: ubuntu-latest
needs: [check-models-python, check-models-javascript]
if: always()

steps:
- name: Check job results
run: |
echo "Python Check: ${{ needs.check-models-python.result }}"
echo "JavaScript Check: ${{ needs.check-models-javascript.result }}"

if [ "${{ needs.check-models-python.result }}" != "success" ] || [ "${{ needs.check-models-javascript.result }}" != "success" ]; then
echo "⚠️ One or more model checks failed"
exit 1
else
echo "✅ All model checks passed"
fi
59 changes: 59 additions & 0 deletions .github/workflows/check-models-javascript.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: Check Models - JavaScript

on:
workflow_dispatch:
inputs:
models:
description: 'Specific models to test (space-separated, leave empty for all)'
required: false
type: string
schedule:
# Run daily at 6:00 UTC
- cron: '0 6 * * *'
push:
branches:
- main
paths:
- 'check_models.js'
- 'package.json'
- '.github/workflows/check-models-javascript.yml'

jobs:
check-models:
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'

- name: Install dependencies
run: npm ci

- name: Check all models
if: github.event.inputs.models == ''
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_API_BASE: ${{ vars.OPENAI_API_BASE || 'https://api.deep.assistant.run.place/v1' }}
run: node check_models.js

- name: Check specific models
if: github.event.inputs.models != ''
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_API_BASE: ${{ vars.OPENAI_API_BASE || 'https://api.deep.assistant.run.place/v1' }}
run: node check_models.js ${{ github.event.inputs.models }}

- name: Upload results
if: always()
uses: actions/upload-artifact@v4
with:
name: javascript-model-check-results
path: |
*.log
retention-days: 30
61 changes: 61 additions & 0 deletions .github/workflows/check-models-python.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: Check Models - Python

on:
workflow_dispatch:
inputs:
models:
description: 'Specific models to test (space-separated, leave empty for all)'
required: false
type: string
schedule:
# Run daily at 6:00 UTC
- cron: '0 6 * * *'
push:
branches:
- main
paths:
- 'check_models.py'
- 'requirements.txt'
- '.github/workflows/check-models-python.yml'

jobs:
check-models:
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt

- name: Check all models
if: github.event.inputs.models == ''
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_API_BASE: ${{ vars.OPENAI_API_BASE || 'https://api.deep.assistant.run.place/v1' }}
run: python check_models.py

- name: Check specific models
if: github.event.inputs.models != ''
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_API_BASE: ${{ vars.OPENAI_API_BASE || 'https://api.deep.assistant.run.place/v1' }}
run: python check_models.py ${{ github.event.inputs.models }}

- name: Upload results
if: always()
uses: actions/upload-artifact@v4
with:
name: python-model-check-results
path: |
*.log
retention-days: 30
126 changes: 125 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,128 @@ Each example directory contains:
## API Key

Get your API key from the Telegram bot: https://t.me/DeepGPTBot
Use the `/api` command to obtain your key.
Use the `/api` command to obtain your key.

## Model Availability Testing

This repository includes automated model availability checkers for both Python and JavaScript that can be run locally or via GitHub Actions.

### Supported Models

The following models are tested:
- `o3-mini`, `o1-preview`, `o1-mini`
- `gpt-4o`, `gpt-4o-mini`, `gpt-3.5-turbo`, `gpt-auto`
- `claude-3-opus`, `claude-3-5-sonnet`, `claude-3-5-haiku`, `claude-3-7-sonnet`
- `deepseek-chat`, `deepseek-reasoner`

### Local Testing

#### Python

```bash
# Install dependencies
pip install -r requirements.txt

# Set your API key
export OPENAI_API_KEY="your-api-key-here"

# Test all models
python check_models.py

# Test specific models
python check_models.py gpt-4o claude-3-5-sonnet
```

#### JavaScript

```bash
# Install dependencies
npm install

# Set your API key
export OPENAI_API_KEY="your-api-key-here"

# Test all models
node check_models.js

# Test specific models
node check_models.js gpt-4o claude-3-5-sonnet
```

### GitHub Actions

The repository includes three GitHub Actions workflows for automated model testing:

1. **Check Models - Python** (`.github/workflows/check-models-python.yml`)
- Tests models using Python
- Runs daily at 6:00 UTC
- Can be manually triggered with specific models

2. **Check Models - JavaScript** (`.github/workflows/check-models-javascript.yml`)
- Tests models using JavaScript
- Runs daily at 6:00 UTC
- Can be manually triggered with specific models

3. **Check All Models** (`.github/workflows/check-all-models.yml`)
- Tests models using both Python and JavaScript
- Runs daily at 6:00 UTC
- Provides a combined summary of results

#### Manual Workflow Execution

You can manually trigger any workflow from the GitHub Actions tab:

1. Go to the **Actions** tab in your repository
2. Select the workflow you want to run
3. Click **Run workflow**
4. Optionally specify models to test (e.g., `gpt-4o claude-3-5-sonnet`)
5. Click **Run workflow** to start

#### Required Secrets

To use GitHub Actions, you need to configure the following secret:

- `OPENAI_API_KEY` - Your Deep.Assistant API key (get from [@DeepGPTBot](https://t.me/DeepGPTBot) using `/api` command)

#### Optional Variables

- `OPENAI_API_BASE` - Custom API base URL (defaults to `https://api.deep.assistant.run.place/v1`)

#### Setting up Secrets

1. Go to your repository **Settings**
2. Navigate to **Secrets and variables** → **Actions**
3. Click **New repository secret**
4. Add `OPENAI_API_KEY` with your API key

### Output Format

The model checkers provide detailed output including:
- Total models tested
- Available models count
- Unavailable models count
- Detailed status for each model
- Error messages for failures

Example output:
```
============================================================
MODEL AVAILABILITY TEST RESULTS
============================================================

Total models tested: 13
Available models: 10
Unavailable models: 3

✓ AVAILABLE MODELS:
- gpt-4o
- gpt-4o-mini
- claude-3-5-sonnet
...

✗ UNAVAILABLE MODELS:
- model-name: HTTP 404: Model not found
Error: The model 'model-name' does not exist

============================================================
```
Loading