Add Robot Framework E2E Tests for University Agent

.github/workflows/check.yaml

@@ -106,3 +106,33 @@ jobs:
       - name: Build docs
         working-directory: ./src/sk-agents
         run: make build-docs
+
+  robot-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - name: checkout repository
+        uses: actions/checkout@v4
+      - name: setup uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+      - name: install dependencies
+        working-directory: ./src/orchestrators/assistant-orchestrator/orchestrator
+        run: make sync
+      - name: initialize browser library
+        working-directory: ./src/orchestrators/assistant-orchestrator/orchestrator
+        run: uv run rfbrowser init
+      - name: run robot framework tests
+        working-directory: ./src/orchestrators/assistant-orchestrator/orchestrator
+        env:
+          GEMINI_API_KEY: ${{ secrets.TA_API_KEY }}
+        run: uv run robot ../example/university/tests/e2e/robot/
+      - name: upload robot test results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: robot-test-results
+          path: |
+            src/orchestrators/assistant-orchestrator/example/university/log.html
+            src/orchestrators/assistant-orchestrator/example/university/report.html
+            src/orchestrators/assistant-orchestrator/example/university/output.xml

src/orchestrators/assistant-orchestrator/example/university/tests/e2e/robot/README.md

@@ -0,0 +1,138 @@
+# Robot Framework E2E Tests for University Agent
+
+This directory contains end-to-end tests for the University Agent using Robot Framework.
+
+## Test Structure
+
+- **`api_tests.robot`** - Tests for the University Agent REST API endpoints
+- **`ui_tests.robot`** - Tests for the Streamlit UI interface
+- **`resources/keywords.robot`** - Reusable custom keywords for service management and testing
+- **`resources/variables.robot`** - Common variables and configuration
+- **`fixtures/test_data.json`** - Test data fixtures
+
+## Prerequisites
+
+1. Install dependencies:
+```bash
+cd ~/repos/teal-agents/src/orchestrators/assistant-orchestrator/orchestrator
+uv sync --all-packages --group dev
+```
+
+2. Initialize Robot Framework Browser Library:
+```bash
+uv run rfbrowser init
+```
+
+3. Set required environment variable:
+```bash
+export GEMINI_API_KEY="your_google_gemini_api_key"
+```
+
+## Running Tests
+
+### Run All Tests
+```bash
+cd ~/repos/teal-agents/src/orchestrators/assistant-orchestrator/example/university
+uv run robot tests/e2e/robot/
+```
+
+### Run Specific Test Suite
+```bash
+uv run robot tests/e2e/robot/api_tests.robot
+uv run robot tests/e2e/robot/ui_tests.robot
+```
+
+### Run with Tags
+```bash
+uv run robot --include api tests/e2e/robot/
+```
+
+## Test Reports
+
+After running tests, Robot Framework generates three files in the current directory:
+
+- **`log.html`** - Detailed execution log with timestamps and keywords
+- **`report.html`** - High-level test results summary
+- **`output.xml`** - Machine-readable XML output for CI/CD integration
+
+## Test Coverage
+
+### API Tests (7 test cases)
+1. Agent Endpoint Health Check - Verifies `/UniversityAgent/0.1/docs` is accessible
+2. Search Universities By Country - Tests country-based search (Finland)
+3. Search Universities By Name - Tests name-based search (Aalto University)
+4. Invalid Query Handling - Tests empty query handling
+5. Response Structure - Verifies `output_raw` field exists
+6. Connection Error Handling - Tests behavior when agent is unreachable
+7. Timeout Error Handling - Verifies timeout is respected (<30s)
+
+### UI Tests (8 test cases)
+1. Streamlit UI Loads Successfully - Verifies page loads at http://localhost:8502
+2. Page Title Displays Correctly - Checks for "🎓 University Agent Chat"
+3. Check Agent Status Button Works - Tests status check functionality
+4. All Example Query Buttons Present - Verifies 5 example buttons exist
+5. Example Query Button Clickable - Tests button click functionality
+6. Chat Input Accepts User Queries - Tests text input and submission
+7. Agent Responses Display In Chat History - Verifies responses appear
+8. Clear Conversation Button Resets Chat - Tests chat clearing
+9. Agent URL Configuration Can Be Modified - Tests sidebar configuration
+
+## Service Management
+
+The tests automatically manage service lifecycle:
+
+- **Agent Service**: Started on port 8001 with 10-second startup wait
+- **Streamlit UI**: Started on port 8502 with 5-second startup wait
+- **Suite Teardown**: Both services are gracefully terminated after tests complete
+
+## Environment Variables
+
+Required environment variables (automatically configured by test keywords):
+
+- `GEMINI_API_KEY` - Your Google Gemini API key (must be set manually)
+- `TA_SERVICE_CONFIG` - Path to config.yaml
+- `TA_PLUGIN_MODULE` - Path to custom_plugins.py
+- `TA_CUSTOM_CHAT_COMPLETION_FACTORY_MODULE` - Gemini factory module path
+- `TA_CUSTOM_CHAT_COMPLETION_FACTORY_CLASS_NAME` - Factory class name
+
+## Troubleshooting
+
+### Port Already in Use
+If you see errors about ports 8001 or 8502 being in use:
+```bash
+lsof -ti:8001 | xargs kill -9
+lsof -ti:8502 | xargs kill -9
+```
+
+### Browser Library Not Initialized
+If Browser Library tests fail:
+```bash
+cd ~/repos/teal-agents/src/orchestrators/assistant-orchestrator/orchestrator
+uv run rfbrowser init
+```
+
+### Missing GEMINI_API_KEY
+Ensure the environment variable is set:
+```bash
+export GEMINI_API_KEY="your_api_key_here"
+```
+
+## Integration with CI/CD
+
+These tests are integrated into the GitHub Actions workflow in `.github/workflows/check.yaml`. The workflow:
+
+1. Installs dependencies via `uv sync`
+2. Initializes Browser Library with `rfbrowser init`
+3. Runs tests with environment variables from secrets
+4. Uploads test reports as artifacts
+
+## Development
+
+When adding new tests:
+
+1. Add test cases to the appropriate `.robot` file
+2. Create reusable keywords in `resources/keywords.robot` if needed
+3. Update variables in `resources/variables.robot` if needed
+4. Add test data to `fixtures/test_data.json` if needed
+5. Run tests locally before committing
+6. Ensure lint passes: `cd ~/repos/teal-agents/src/sk-agents && uv run ruff check .`

src/orchestrators/assistant-orchestrator/example/university/tests/e2e/robot/api_tests.robot

@@ -0,0 +1,70 @@
+*** Settings ***
+Documentation     API tests for University Agent endpoints
+Library           RequestsLibrary
+Library           Collections
+Library           OperatingSystem
+Resource          resources/keywords.robot
+Resource          resources/variables.robot
+
+Suite Setup       API Suite Setup
+Suite Teardown    Stop All Services
+Test Setup        Clear Request Session
+
+*** Variables ***
+${TEST_DATA_FILE}    ${CURDIR}/fixtures/test_data.json
+
+*** Test Cases ***
+Agent Endpoint Health Check
+    [Documentation]    Verify the agent endpoint is accessible
+    Create Session    agent    ${AGENT_URL}
+    ${response}=    GET On Session    agent    ${AGENT_ENDPOINT}/docs    expected_status=200
+    Should Be Equal As Numbers    ${response.status_code}    200
+
+Search Universities By Country Returns Valid JSON
+    [Documentation]    Test searching universities in Finland returns valid structure
+    ${response}=    Send University Query    Find universities in Finland
+    Validate University Response    ${response}
+    ${json}=    Set Variable    ${response.json()}
+    Should Contain    ${json}[output_raw]    Finland
+
+Search Universities By Name Returns Matching Results
+    [Documentation]    Test searching for Aalto University
+    ${response}=    Send University Query    Search for Aalto University
+    Validate University Response    ${response}
+    ${json}=    Set Variable    ${response.json()}
+    Should Contain    ${json}[output_raw]    Aalto
+
+Invalid Query Returns Appropriate Error
+    [Documentation]    Test that empty queries are handled
+    ${response}=    Send University Query    ${EMPTY}
+    Should Be Equal As Numbers    ${response.status_code}    200
+
+Response Includes Output Raw Field
+    [Documentation]    Verify all responses have output_raw field
+    ${response}=    Send University Query    Find universities in Japan
+    ${json}=    Set Variable    ${response.json()}
+    Dictionary Should Contain Key    ${json}    output_raw
+
+Connection Error Handling
+    [Documentation]    Test behavior when agent is not reachable
+    Stop All Services
+    Sleep    2
+    ${status}=    Run Keyword And Return Status    Send University Query    test
+    Should Be Equal    ${status}    ${False}
+    API Suite Setup
+
+Timeout Error Handling
+    [Documentation]    Verify timeout is respected
+    ${start_time}=    Get Time    epoch
+    ${response}=    Send University Query    Find universities in United States
+    ${end_time}=    Get Time    epoch
+    ${duration}=    Evaluate    ${end_time} - ${start_time}
+    Should Be True    ${duration} < ${TIMEOUT}
+
+*** Keywords ***
+API Suite Setup
+    Setup Environment Variables
+    Start University Agent Service
+
+Clear Request Session
+    Delete All Sessions

src/orchestrators/assistant-orchestrator/example/university/tests/e2e/robot/fixtures/test_data.json

@@ -0,0 +1,18 @@
+{
+  "valid_queries": {
+    "country_search": "Find universities in Finland",
+    "name_search": "Search for Aalto University",
+    "japan_search": "What universities are in Japan?",
+    "mit_search": "Tell me about MIT",
+    "germany_search": "Universities in Germany"
+  },
+  "invalid_queries": {
+    "empty": "",
+    "special_chars": "!@#$%^&*()",
+    "very_long": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+  },
+  "expected_response_keys": [
+    "output_raw"
+  ],
+  "timeout_threshold": 30
+}

src/orchestrators/assistant-orchestrator/example/university/tests/e2e/robot/resources/keywords.robot

@@ -0,0 +1,63 @@
+*** Settings ***
+Library           Process
+Library           OperatingSystem
+Library           RequestsLibrary
+Library           Browser
+
+*** Variables ***
+${AGENT_PROCESS}       None
+${STREAMLIT_PROCESS}   None
+
+*** Keywords ***
+Setup Environment Variables
+    Set Environment Variable    GEMINI_API_KEY                                    %{GEMINI_API_KEY}
+    Set Environment Variable    TA_SERVICE_CONFIG                                 ${CURDIR}/../../config.yaml
+    Set Environment Variable    TA_PLUGIN_MODULE                                  ${CURDIR}/../../custom_plugins.py
+    Set Environment Variable    TA_CUSTOM_CHAT_COMPLETION_FACTORY_MODULE          src.sk_agents.chat_completion.custom.gemini_chat_completion_factory
+    Set Environment Variable    TA_CUSTOM_CHAT_COMPLETION_FACTORY_CLASS_NAME      GeminiChatCompletionFactory
+
+Start University Agent Service
+    [Documentation]    Start the University Agent FastAPI service on port 8001
+    ${result}=    Start Process    uv    run    uvicorn    sk_agents.app:app    --host    0.0.0.0    --port    8001
+    ...    cwd=${EXECDIR}/../../../sk-agents
+    ...    alias=agent_service
+    Set Suite Variable    ${AGENT_PROCESS}    ${result}
+    Sleep    ${AGENT_STARTUP_WAIT}
+    Verify Agent Is Running
+
+Start Streamlit UI Service
+    [Documentation]    Start the Streamlit UI on port 8502
+    ${result}=    Start Process    uv    run    streamlit    run    streamlit_ui.py    --server.port    8502
+    ...    cwd=${CURDIR}/../..
+    ...    alias=streamlit_service
+    Set Suite Variable    ${STREAMLIT_PROCESS}    ${result}
+    Sleep    ${STREAMLIT_STARTUP_WAIT}
+
+Stop All Services
+    [Documentation]    Gracefully shutdown both services
+    Run Keyword If    '${AGENT_PROCESS}' != 'None'        Terminate Process    agent_service
+    Run Keyword If    '${STREAMLIT_PROCESS}' != 'None'    Terminate Process    streamlit_service
+
+Verify Agent Is Running
+    [Documentation]    Check that the agent is responding to health checks
+    Create Session    agent    ${AGENT_URL}
+    ${response}=    GET On Session    agent    ${AGENT_ENDPOINT}/docs    expected_status=200
+    Should Be Equal As Numbers    ${response.status_code}    200
+
+Send University Query
+    [Arguments]    ${query}
+    [Documentation]    Send a query to the University Agent
+    Create Session    agent    ${AGENT_URL}
+    ${chat_history}=    Create List
+    ${message}=    Create Dictionary    role=user    content=${query}
+    Append To List    ${chat_history}    ${message}
+    ${payload}=    Create Dictionary    chat_history=${chat_history}
+    ${response}=    POST On Session    agent    ${AGENT_ENDPOINT}    json=${payload}    timeout=${TIMEOUT}    expected_status=200
+    RETURN    ${response}
+
+Validate University Response
+    [Arguments]    ${response}
+    [Documentation]    Validate the structure of a university response
+    Should Be Equal As Numbers    ${response.status_code}    200
+    ${json}=    Set Variable    ${response.json()}
+    Dictionary Should Contain Key    ${json}    output_raw

src/orchestrators/assistant-orchestrator/example/university/tests/e2e/robot/resources/variables.robot

@@ -0,0 +1,8 @@
+*** Variables ***
+${AGENT_URL}              http://localhost:8001
+${AGENT_ENDPOINT}         /UniversityAgent/0.1
+${STREAMLIT_URL}          http://localhost:8502
+${TIMEOUT}                30
+${AGENT_STARTUP_WAIT}     10
+${STREAMLIT_STARTUP_WAIT}    5
+${UNIVERSITIES_API}       http://universities.hipolabs.com/search