Agentic AI update to the system

Agentic AI update to the system

Author: dt-nishant-chaturvedi

.env.example

@@ -8,20 +8,33 @@ TAG=latest
 
 # Frontend Configuration
 VITE_BACKEND_URL=
+
 # For production, this will be set by Azure App Service
 
-# Backend Configuration - Azure OpenAI
-AZURE_OPENAI_ENDPOINT= 
-AZURE_OPENAI_API_KEY= 
-OPENAI_API_VERSION= 
-AZURE_DEPLOYMENT_NAME= 
+# Backend Configuration - Azure OpenAI (Chat Model)
+AZURE_OPENAI_ENDPOINT = ""
+AZURE_OPENAI_API_KEY = ""
+AZURE_DEPLOYMENT_NAME = ""
+AZURE_OPENAI_API_VERSION = ""
+
+# Azure OpenAI Embedding Model Configuration
+AZURE_EMBEDDING_MODEL = ""
+AZURE_OPENAI_ENDPOINT_EMBED = ""
+AZURE_OPENAI_API_KEY_EMBED = ""
+AZURE_EMBEDDING_DEPLOYMENT_NAME = ""
+AZURE_OPENAI_API_VERSION_EMBED = ""
 
 # Backend Database Variables (used by the application)
-DB_USERNAME=
-DB_PASSWORD=
-DB_NAME=proposalgen
-DB_HOST=db
-DB_PORT=5432
+DB_USERNAME=""
+DB_PASSWORD=""
+DB_NAME=""
+DB_HOST=""
+DB_PORT=""
 
 # JWT Secret for authentication
-SECRET_KEY=
+SECRET_KEY=""
+
+# Azure Blob Container details
+AZURE_BLOB_CONNECTION_STRING=""
+AZURE_BLOB_KB=""
+AZURE_BLOB_UPLOAD=""

.github/workflows/azure-container-webapp.yml

@@ -33,6 +33,9 @@ env:
   DB_HOST: ${{ secrets.DB_HOST }}
   DB_PORT: ${{ secrets.DB_PORT }}
   SECRET_KEY: ${{ secrets.SECRET_KEY }}
+  AZURE_BLOB_CONNECTION_STRING: ${{ secrets.AZURE_BLOB_CONNECTION_STRING }}
+  AZURE_BLOB_KB: ${{ secrets.AZURE_BLOB_KB }}
+  AZURE_BLOB_UPLOAD: ${{ secrets.AZURE_BLOB_UPLOAD }}
 
 jobs:
   build-and-deploy:
@@ -93,6 +96,9 @@ jobs:
             -e "s|\${DB_HOST}|${DB_HOST}|g" \
             -e "s|\${DB_PORT}|${DB_PORT}|g" \
             -e "s|\${SECRET_KEY}|${SECRET_KEY}|g" \
+            -e "s|\${AZURE_BLOB_CONNECTION_STRING}|${AZURE_BLOB_CONNECTION_STRING}|g" \ 
+            -e "s|\${AZURE_BLOB_KB}|${AZURE_BLOB_KB}|g" \
+            -e "s|\${AZURE_BLOB_UPLOAD}|${AZURE_BLOB_UPLOAD}|g" \                                   
             "$DOCKER_COMPOSE_PATH"
     
           echo "→ Final compose file (for debugging):"
@@ -145,6 +151,9 @@ jobs:
               DB_NAME="${{ env.DB_NAME }}" \
               DB_HOST="${{ env.DB_HOST }}" \
               DB_PORT="${{ env.DB_PORT }}" \
+              AZURE_BLOB_CONNECTION_STRING="${{ env.AZURE_BLOB_CONNECTION_STRING }}" \
+              AZURE_BLOB_KB="${{ env.AZURE_BLOB_KB }}" \
+              AZURE_BLOB_UPLOAD="${{ env.AZURE_BLOB_UPLOAD }}" \
               SECRET_KEY="${{ env.SECRET_KEY }}"
 
       - name: Deploy Multi-Container App to Azure App Service
@@ -181,4 +190,4 @@ jobs:
             --resource-group ${{ env.RESOURCE_GROUP }} \
             --name ${{ env.AZURE_WEBAPP_NAME }} \
             --query "defaultHostName" -o tsv)
-          echo "App deployed at: https://$APP_URL"
+          echo "App deployed at: https://$APP_URL"

README.md

@@ -55,8 +55,8 @@ migration_pathways/
 ├── .github/workflows/      # GitHub Actions CI/CD pipelines
 ├── backend/                # Flask API, AI model integration, database operations
 ├── frontend/               # React UI
-├── docker-compose.yml      # Docker container orchestration
-├── init-db.sql             # Database initialization script
+├── migration-azure-docker-compose.yml      # Docker container orchestration
+├── database-setup.sql             # Database initialization script
 └── .env.example            # Environment variables template
 ```
 
@@ -66,81 +66,161 @@ migration_pathways/
 - Git
 - Azure OpenAI API access (for production)
 
-## Quick Start
+## Project Overview
 
-1. **Clone the repository:**
-   ```bash
-   git clone https://github.com/yourusername/migration_pathways.git
-   cd migration_pathways
-   ```
+This is an AI-powered migration assistance chatbot built for the International Organization for Migration (IOM). It provides verified migration information and personalized pathways to help migrants make safer, informed decisions.
 
-2. **Set up environment variables:**
-   ```bash
-   cp .env.example .env
-   # Edit .env file with your settings
-   ```
+**Architecture**: Flask backend with React frontend, PostgreSQL database, and enhanced AI system combining RAG (Retrieval-Augmented Generation) with CrewAI multi-agent architecture using Azure OpenAI and Chroma vector database.
 
-3. **Start the application with Docker Compose:**
-   ```bash
-   docker-compose up -d
-   ```
-
-4. **Access the application:**
-   - Frontend: http://localhost
-   - Backend API: http://localhost:8510
-
-## Development Environment
+## Development Commands
 
 ### Backend Development
-
 ```bash
 cd backend
 python -m venv env
 source env/bin/activate  # On Windows: env\Scripts\activate
 pip install -r requirements.txt
-
-# Set up PostgreSQL locally or use Docker:
-docker run --name postgres -e POSTGRES_PASSWORD=your_secure_password -e POSTGRES_USER=iom_uc2_user -e POSTGRES_DB=iom_uc2 -p 5432:5432 -d postgres:14-alpine
-
-# Run the backend server:
-python main.py
+python main.py  # Runs on port 80
 ```
 
 ### Frontend Development
-
 ```bash
 cd frontend
 npm install
-npm run dev
+npm run dev     # Runs on port 8509
+npm run build   # Production build
+npm run lint    # ESLint
+npm test        # Vitest tests
+npm run test:ui # Vitest UI
 ```
 
-## Database Setup
+### Testing
+```bash
+# Backend tests
+cd backend
+pytest
 
-The PostgreSQL database is automatically initialized with the required tables when using Docker Compose. For manual setup, see the [backend README](backend/README.md).
+# CrewAI integration test
+cd backend
+python test_crew_ai.py
 
-## CI/CD Pipeline
+# Frontend tests
+cd frontend
+npm test
+```
 
-This project utilizes GitHub Actions for Continuous Integration and Deployment:
+### Docker Development
+```bash
+# Full stack with Docker Compose
+docker-compose -f migration-azure-docker-compose.yml up -d
 
-- **CI Pipeline:** Runs on PR and pushes to main/dev branches
-  - Lints and tests frontend code
-  - Lints and tests backend code
-  - Builds Docker images
+# Individual services
+docker build -t migration-backend ./backend
+docker build -t migration-frontend ./frontend
+```
 
-- **CD Pipeline:** Runs on pushes to main branch and tags
-  - Builds and pushes Docker images to GitHub Container Registry
-  - Deploys to staging/production environments
+## Architecture Details
+
+### Backend (Flask)
+- **Main entry**: `backend/main.py` - Flask app with authentication, admin routes, and user management
+- **Chat systems**: 
+  - `backend/chat_rag.py` - Original RAG implementation with Azure OpenAI and Chroma
+  - `backend/chat_crew_ai.py` - Enhanced CrewAI multi-agent system
+- **Database**: PostgreSQL with SQLAlchemy ORM, connection via `utility/db_config.py`
+- **Authentication**: JWT tokens with httpOnly cookies, role-based access (users/admins)
+- **CrewAI Agent System**:
+  - **Agents**: `backend/crew_ai/agents.py` - 6 specialized migration agents
+    - Migration Advisor: Primary pathway guidance
+    - Risk Assessment: Safety and fraud detection
+    - Documentation Specialist: Legal requirements and paperwork
+    - Cultural Integration: Adaptation and community support
+    - Employment Advisor: Job markets and skill recognition
+    - Resource Coordinator: IOM contacts and NGO connections
+  - **Tasks**: `backend/crew_ai/tasks.py` - Structured task definitions for agent coordination
+  - **Tools**: `backend/crew_ai/tools.py` - Vector search, contact lookup, risk assessment tools
+  - **Orchestration**: `backend/crew_ai/crew.py` - Crew management and execution logic
+- **RAG Components**:
+  - Vector store: Chroma database in `backend/chroma_store/`
+  - Embeddings: Azure OpenAI embeddings for semantic search
+  - Knowledge sources: Web scraped content + JSON knowledge base
+  - Document processing: `utility/embedder.py`, `utility/scraper_and_embedder.py`
+
+### Frontend (React + Vite)
+- **Framework**: React 19 with React Router DOM for navigation
+- **Build tool**: Vite with SWC for fast compilation
+- **Testing**: Vitest + React Testing Library + MSW for API mocking
+- **Styling**: CSS modules with responsive design
+- **Proxy**: Development proxy `/api` → `http://localhost:8510` (backend)
+
+### Database Schema
+- **users**: User accounts with authentication, preferences, session management
+- **admins**: Admin accounts with elevated privileges
+- Key fields: email, password (hashed), request_count, active status, security_questions
+
+### Key Features
+- **Enhanced Chat Interface**: 
+  - `/api/chat` - Original RAG-based responses
+  - `/api/chat/crew` - CrewAI multi-agent specialized responses
+  - `/api/chat/hybrid` - Automatic fallback from CrewAI to RAG
+- **Multi-Agent AI System**: 6 specialized agents working collaboratively
+- **Intelligent Risk Assessment**: Automated detection of trafficking, fraud, and unsafe migration practices
+- **Comprehensive Guidance**: Legal documentation, cultural integration, employment advice
+- **Dynamic Resource Coordination**: Contextual IOM office and NGO recommendations
+- **Admin Dashboard**: Content management, user administration, URL scraping
+- **Authentication**: Secure login/logout with session management
+- **Content Management**: Web scraping, document upload, knowledge base updates
+- **Preferences**: User source/destination country settings for personalized responses
 
 ## Environment Variables
 
-See `.env.example` for the required environment variables.
-
-## Docker Deployment
-
-The application can be deployed using Docker Compose:
-
-```bash
-# Production deployment
-docker-compose -f docker-compose.yml up -d
-```
-
+Required environment variables (see `.env.example`):
+- `SECRET_KEY`: JWT signing key
+- `DB_*`: PostgreSQL connection details
+- `AZURE_OPENAI_*`: Azure OpenAI service credentials for chat model
+- `AZURE_*_EMBED`: Azure OpenAI embedding service credentials
+- `VITE_BACKEND_URL`: Frontend API endpoint
+
+**CrewAI-specific variables**:
+- `AZURE_EMBEDDING_MODEL`: Embedding model name (e.g., text-embedding-ada-002)
+- `AZURE_OPENAI_ENDPOINT_EMBED`: Separate endpoint for embeddings (if different)
+- `AZURE_OPENAI_API_KEY_EMBED`: API key for embedding service
+- `AZURE_EMBEDDING_DEPLOYMENT_NAME`: Deployment name for embedding model
+- `AZURE_OPENAI_API_VERSION_EMBED`: API version for embedding service
+
+## Development Workflow
+
+1. **Database Setup**: Run `database-setup.sql` to initialize PostgreSQL
+2. **Environment**: Copy `.env.example` to `.env` and configure all Azure OpenAI endpoints
+3. **Backend**: Install dependencies including CrewAI, start Flask server on port 80
+4. **Frontend**: Install dependencies, start Vite dev server on port 8509
+5. **CrewAI Testing**: Run `python test_crew_ai.py` to verify agent system
+6. **Testing**: Run pytest (backend) and vitest (frontend) before commits
+
+### CrewAI Development Notes
+- **Agent Modifications**: Edit `backend/crew_ai/agents.py` to adjust agent roles and capabilities
+- **Task Updates**: Modify `backend/crew_ai/tasks.py` to change agent coordination logic
+- **Tool Enhancement**: Add new tools in `backend/crew_ai/tools.py` for expanded functionality
+- **Configuration**: Adjust settings in `backend/crew_ai/config.py` for performance tuning
+- **Testing Endpoints**:
+  - GET `/api/chat/crew/status` - Check CrewAI system health
+  - POST `/api/chat/crew/reset` - Reset user conversation history
+  - POST `/api/chat/hybrid` - Use intelligent fallback system
+
+## Common Issues
+
+- **CORS**: Frontend dev server proxies API calls through Vite config
+- **Database**: Ensure PostgreSQL is running and credentials are correct
+- **Azure OpenAI**: Verify both chat and embedding endpoints are configured
+- **Chroma**: Vector database persists in `backend/chroma_store/`
+- **CrewAI Issues**:
+  - **Rate Limits**: Azure OpenAI rate limiting may affect agent execution
+  - **Memory Usage**: Multiple agents can consume significant memory
+  - **Timeout**: Complex queries may exceed execution time limits
+  - **Dependencies**: Ensure all CrewAI and LangChain packages are compatible versions
+  - **Tools Access**: Verify tools have access to vector store and JSON knowledge base
+
+## Deployment
+
+- **Production**: Uses `migration-azure-docker-compose.yml` with nginx proxy
+- **Services**: nginx-proxy (port 80) → frontend → backend → PostgreSQL
+- **CI/CD**: GitHub Actions workflow in `.github/workflows/`

backend/.env.example

@@ -1,8 +1,8 @@
 ## this env file is used to set environment variables for the backend application when running it locally without a Docker container
 AZURE_OPENAI_ENDPOINT = ""
-AZURE_OPENAI_API_VERSION = ""
 AZURE_OPENAI_API_KEY = ""
 AZURE_DEPLOYMENT_NAME = ""
+AZURE_OPENAI_API_VERSION = ""
 
 AZURE_EMBEDDING_MODEL = ""
 AZURE_OPENAI_ENDPOINT_EMBED = ""
@@ -12,9 +12,13 @@ AZURE_OPENAI_API_VERSION_EMBED = ""
 
 DB_USERNAME=""
 DB_PASSWORD=""
-DB_NAME="postgres"
+DB_NAME=""
 DB_HOST=""
-DB_PORT="5432"
+DB_PORT=""
 
 # JWT Secret for authentication
-SECRET_KEY=""
+SECRET_KEY=""
+
+AZURE_BLOB_CONNECTION_STRING=""
+AZURE_BLOB_KB=""
+AZURE_BLOB_UPLOAD=""