🧬 Copilot-LD

Copilot-LD is an intelligent agent leveraging GitHub Copilot, linked data and retrieval-augmented generation.

• 🎯 Amazingly Accurate: Chunking and similarity searches with linked data
• ⚡️ Incredibly Fast: Parallel vector retrieval using in-memory indices
• 🛡️ Robustly Secure: Network isolation, gRPC, HMAC auth, <10 MB containers
• ✨ Elegantly Simple: Plain JavaScript with no external dependencies

🏗️ Repository Structure

• /services/: gRPC microservices (agent, graph, llm, memory, tool, vector)
• /extensions/: Application adapters (teams, ui, web)
• /packages/: Reusable, framework-agnostic libraries
• /scripts/: Development and operational utilities
• /proto/: Authoritative Protocol Buffer source schemas (copied into packages/libtype/generated/proto/ during codegen)
• /tools/: Optional protobuf definitions for custom tools that extend the core system (not part of the base platform)
• /data/: Knowledge base, vectors, and resource data

🚀 Setup

1. Configuration

Set up environment variables and service configuration:

cp .env{.example,}
cp config/config{.example,}.json
cp config/assistants{.example,}.yml
cp config/tools{.example,}.yml

For detailed configuration options, see the Configuration Guide.

2. Install dependencies

npm install

3. Initialize data directories

Initialize the necessary data directories with empty indices:

npm run data:init

4. Generate code from Protocol Buffers

npm run codegen

This generates service interfaces and type definitions from the Protocol Buffer schemas. See Code Generation Details for more information.

5. Authentication and secrets

# Set GITHUB_CLIENT_ID in .env, then generate token
npx env-cmd -- node scripts/token.js

# Generate service authentication secret
node scripts/secret.js

6. Prepare knowledge base

A knowledge base with HTML microdata must be available at ./data/knowledge/. See the Processing Guide for complete knowledge base setup instructions including HTML structure examples and processing workflows.

7. Process all resources and create indices

npm run process

8. Start services

Option A: Local Development Environment

npm run dev

Access the services:

• UI Extension: http://localhost:3000/ui

Option B: Production-Like Environment

For a production-like environment with an Application Load Balancer (ALB) and S3-compatible storage, first generate SSL certificates and comment out host and port variables in .env (using GNU sed), then start all services including ALB and MinIO:

node scripts/cert.js
sed -i -E '/(HOST|PORT)=/s/^/# /' .env
docker compose up

This provides SSL termination, path-based routing, and S3-compatible storage. See the Configuration Guide for detailed setup options.

Access the services:

• UI Extension: https://localhost/ui
• MinIO Console: http://localhost:9001

📖 Detailed Guides

For comprehensive setup and deployment information:

• Development Setup: Complete development environment configuration with detailed troubleshooting
• Deployment Guide: Production deployment with Docker Compose or AWS CloudFormation

⚡ Usage

After starting services with npm run dev, you can interact with the system using available scripts.

Chat Script

Interactive mode:

npm run cli:chat
> Hello

Piping for scripted testing:

echo "Hello" | npm run cli:chat

Search Script

Interactive mode:

npm run cli:search
> What is Kanban?

Piping for scripted testing:

echo "What is Kanban?" | npm run cli:search

Command-line flags can be used for non-interactive runs to limit results and set a minimum similarity threshold:

echo "testing" | npm run cli:search -- --limit 10 --threshold 0.25
echo "find pipeline tasks" | npm run cli:search -- --index descriptor --limit 5

👨‍💻 Development

Code Quality

Run all quality checks or automatically fix linting and formatting issues:

npm run check
npm run check:fix

Testing

Run unit tests:

npm test

Manual integration testing by using scripts:

echo "test prompt" | npm run cli:chat
echo "search query" | npm run cli:search -- --limit 3 --threshold 0.2

Adding New Features

• Add new services in /services with schemas in /proto
• Add reusable logic in /packages
• Update root-level documentation for changes

📚 Documentation

Human Documentation

• Configuration Guide: Environment variables and YAML configuration
• Architecture Overview: System design and structure
• Development Setup: Local development and configuration
• Deployment Guide: Docker Compose and AWS CloudFormation deployment

AI Instructions

General AI instructions:

• GitHub Copilot: Summary of all instructions

AI instructions for specific domains:

• Architecture Instructions: Microservices-based, gRPC-enabled platform design with service organization patterns
• CLI Tools Instructions: Development utilities for interacting with platform capabilities through command-line interfaces
• Documentation Instructions: JSDoc standards for interface-based documentation contracts with IDE support
• Performance Instructions: Performance testing using @copilot-ld/libperf for consistent validation
• Security Instructions: Security architecture for defense-in-depth through network isolation and secure communication