OCR Evaluation Platform

A web-based evaluation platform for OCR (Optical Character Recognition) results, featuring real-time leaderboard and TEDS (Tree Edit Distance based Similarity) metric scoring.

📋 Overview

This platform allows users to:

Upload OCR prediction results in JSON format
Automatically evaluate predictions against ground truth using TEDS metrics
View real-time rankings on an interactive leaderboard with WebSocket progress updates
Analyze detailed evaluation results with filtering and statistics
Export results as CSV for further analysis
Switch between Traditional Chinese and English interfaces
Compare table recognition accuracy with other participants
Administrators can manage submissions and delete entries through a secure dashboard

🚀 Features

TEDS Metric: Industry-standard Tree Edit Distance based Similarity for table structure evaluation
Flexible Input: Supports both Markdown and HTML table formats
Real-time Leaderboard: Instant ranking updates after each submission
Detailed Score View: View individual table scores with filtering and statistics
WebSocket Progress: Real-time progress updates during evaluation
Multi-language Support: Switch between Traditional Chinese and English
Admin Dashboard: Manage submissions with authentication and delete capabilities
Format Validation: Automatic validation of uploaded JSON files
Modern UI: Clean and responsive web interface
Docker Support: Easy deployment with containerization
CSV Export: Download detailed scores as CSV files

🛠️ Tech Stack

Backend: FastAPI
Frontend: Jinja2 Templates, HTML/CSS/JavaScript
Real-time Communication: WebSocket
Internationalization: Custom i18n module (Chinese/English)
Metrics: TEDS (Tree Edit Distance), Levenshtein Distance, Edit Distance
Parsing: lxml, apted, distance, zss
Server: Uvicorn
Authentication: Cookie-based session management

📦 Installation

Prerequisites

Python 3.12 or higher
pip package manager

Local Setup

Clone the repository:

git clone https://github.com/wcks13589/ocr-eval-platform.git
cd ocr-eval-platform

Install dependencies:

pip install -r requirements.txt

Prepare your ground truth data:
- Place your ground truth JSON file at data/ground_truth.json
- Format: {"id": "<table>...</table>"} or markdown table format
Run the server:

uvicorn app.main:app --host 0.0.0.0 --port 8080

Access the platform:
- Open your browser and navigate to http://localhost:8080

Docker Deployment

Build the Docker image:

docker build -t ocr-eval-platform .

Run the container with volume mounting:

docker run -p 8080:8080 \
  -v $(pwd)/data:/app/data \
  ocr-eval-platform

Note: The -v flag mounts the local data/ directory to persist uploads and leaderboard data.

Access the platform at http://localhost:8080

Docker Compose (Recommended)

For easier management, use Docker Compose:

# Start the service
docker compose up -d

# View logs
docker compose logs -f

# Stop the service
docker compose down

The docker-compose.yml automatically handles volume mounting and configuration.

📝 Usage

Preparing Prediction Files

Your prediction file should be a JSON file with the following structure:

{
  "sample_id_1": "| Header1 | Header2 |\n|---------|----------|\n| Cell1   | Cell2    |",
  "sample_id_2": "<table><tr><td>Cell1</td><td>Cell2</td></tr></table>",
  "sample_id_3": "..."
}

Supported formats:

Markdown tables
HTML table strings
Mixed format (different IDs can use different formats)

Submitting Predictions

Navigate to the main page
Enter your participant name
Upload your JSON prediction file
Click "Start Evaluation" (🚀 開始評估)
Watch real-time progress updates via WebSocket
View your score and ranking on the leaderboard
Click "Details" to see individual table scores

Viewing Detailed Results

After submission, you can view detailed evaluation results:

Click the "🔍 詳細" (Details) button next to your name on the leaderboard
View statistics including:
- Overall TEDS score
- Valid data count
- Score distribution (Perfect/High/Medium/Low)
- Individual table scores
Use filters to show/hide:
- Normal data (✅)
- Missing data (❌)
- Error data (⚠️)
- Score range filtering
Download results as CSV for further analysis

Admin Functions

Administrators can manage submissions:

Navigate to /admin/login
Enter the admin password
Access the admin dashboard to:
- View all submissions
- Delete individual entries (removes all associated data)
- Monitor platform usage
Logout when finished to clear the session

Evaluation Metrics

The platform uses TEDS (Tree Edit Distance based Similarity) to evaluate table structure accuracy:

Range: 0.0 to 1.0 (higher is better)
Calculation: Measures structural and content similarity between predicted and ground truth tables
Normalization: Accounts for table size differences
Weighting: Considers both cell content and table structure

📊 API Endpoints

Public Endpoints

GET `/`

Main page with upload form and leaderboard

POST `/upload`

Upload prediction file without evaluation

Parameters:
- name (form field): Participant name
- file (file upload): JSON prediction file
Returns: JSON response with file path or error

POST `/evaluate`

Upload and evaluate prediction file (fallback for non-WebSocket)

Parameters:
- name (form field): Participant name
- file (file upload): JSON prediction file
Returns: Updated leaderboard with evaluation results

GET `/leaderboard`

View standalone leaderboard page

GET `/details/{name}`

View detailed evaluation results for a participant

Parameters:
- name (path): Participant name
Returns: HTML page with detailed scores, statistics, and filtering options

GET `/api/details/{name}`

Get detailed evaluation data in JSON format

Parameters:
- name (path): Participant name
Returns: JSON with detailed scores and statistics

GET `/set_language/{lang}`

Set interface language preference

Parameters:
- lang (path): Language code (zh-TW or en)
Returns: Redirect to previous page with language cookie set

WebSocket `/ws/{session_id}`

Real-time evaluation progress updates

Parameters:
- session_id (path): Unique session identifier
Messages:
- Receives: {name, file_path} to start evaluation
- Sends: Progress updates and completion status

Admin Endpoints

GET `/admin/login`

Admin login page

POST `/admin/login`

Admin authentication

Parameters:
- password (form field): Admin password
Returns: Redirect to dashboard on success

GET `/admin/dashboard`

Admin control panel (requires authentication)

Features: View all submissions, delete entries
Authentication: Cookie-based session token

POST `/admin/logout`

Admin logout and session cleanup

DELETE `/api/admin/delete/{name}`

Delete a participant's data (requires admin authentication)

Parameters:
- name (path): Participant name
- admin_token (cookie): Admin session token
Returns: JSON response with updated leaderboard

🗂️ Project Structure

ocr-eval-platform/
├── app/
│   ├── main.py              # FastAPI application and routes
│   ├── evaluation.py        # Evaluation logic and metrics
│   ├── TEDS_metric.py       # TEDS implementation
│   ├── parallel.py          # Parallel processing utilities
│   ├── i18n.py              # Internationalization (Chinese/English)
│   ├── static/
│   │   └── style.css        # Styling
│   └── templates/
│       ├── index.html       # Main page with upload form
│       ├── leaderboard.html # Standalone leaderboard page
│       ├── details.html     # Detailed score view page
│       ├── admin_login.html # Admin login page
│       ├── admin_dashboard.html # Admin control panel
│       └── result.html      # Results display (legacy)
├── data/                    # Data directory (separate from code)
│   ├── ground_truth.json    # Ground truth data
│   ├── leaderboard.json     # Leaderboard storage (auto-generated)
│   ├── details/             # Individual participant detailed scores
│   └── uploads/             # Uploaded prediction files
├── .gitignore              # Git ignore rules
├── Dockerfile              # Docker configuration
├── docker-compose.yml      # Docker Compose configuration
├── requirements.txt        # Python dependencies
└── README.md              # This file

🔧 Configuration

Data Management

The platform uses a dedicated data/ directory to separate data from code:

Benefits:

✅ Clean separation: Code and data are isolated
✅ Easy backup: Simply backup the data/ folder
✅ Docker persistence: Easy volume mounting for containers
✅ Version control: Data files can be gitignored separately
✅ Security: Sensitive data isolated from application code

Directory structure:

data/
├── ground_truth.json    # Your test dataset (required)
├── leaderboard.json     # Auto-generated rankings
├── details/             # Detailed scores for each participant
└── uploads/             # User-submitted predictions

Ground Truth Format

The data/ground_truth.json file should contain:

{
  "sample_id_1": "| Header1 | Header2 |\n|---------|----------|\n| Cell1   | Cell2    |",
  "sample_id_2": "<table><tr><td>Cell1</td><td>Cell2</td></tr></table>",
  "sample_id_3": "..."
}

Example with actual data:

{
  "table_001": "| Name | Age | City |\n|------|-----|------|\n| Alice | 30 | NYC |",
  "table_002": "<table><tr><td>Product</td><td>Price</td></tr><tr><td>Apple</td><td>$2</td></tr></table>"
}

Admin Configuration

Set the admin password using an environment variable:

# Linux/Mac
export ADMIN_PASSWORD="your_secure_password"

# Windows
set ADMIN_PASSWORD=your_secure_password

# Docker
docker run -p 8080:8080 -e ADMIN_PASSWORD=your_secure_password ocr-eval-platform

Default password (if not set): admin123

Security Note: Always change the default admin password in production environments.

Language Settings

The platform supports:

Traditional Chinese (zh-TW) - Default
English (en)

Users can switch languages using the language selector in the web interface. The preference is stored in a cookie for 1 year.

Modifying TEDS Parameters

In app/evaluation.py, you can adjust:

teds = TEDS(n_jobs=4)  # Number of parallel jobs

🐛 Error Handling

The platform handles various error cases:

Invalid JSON format: Returns error message with parsing details and removes uploaded file
Encoding errors: Detects non-UTF-8 files and provides helpful error messages
Duplicate names: Prevents overwriting existing submissions with clear warning
Missing fields: Gracefully handles incomplete predictions
WebSocket fallback: Automatically falls back to traditional POST if WebSocket is unavailable
Authentication errors: Redirects to login page for unauthorized admin access
File cleanup: Automatically removes uploaded files on evaluation failure

📄 License

🤝 Contributing

Contributions are welcome! Please feel free to submit issues or pull requests.

📞 Support

For questions or issues, please contact the project maintainer or open an issue on GitHub.

🙏 Acknowledgments

TEDS implementation based on PubTabNet by IBM Research
Tree edit distance using apted library
FastAPI framework for rapid web development

Version: 2.0.0
Last Updated: October 2025

🆕 What's New in v2.0.0

✨ Multi-language support (Traditional Chinese and English)
🔐 Admin dashboard with authentication
📊 Detailed score view with filtering and statistics
🌐 WebSocket real-time progress updates
📥 CSV export functionality
🗑️ Admin delete capabilities
🎨 Improved UI with better user experience
🔒 Cookie-based session management

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
app		app
data		data
.gitignore		.gitignore
Dockerfile		Dockerfile
README.md		README.md
docker-compose.yml		docker-compose.yml
requirements.txt		requirements.txt

wcks13589/ocr-eval-platform

Folders and files

Latest commit

History

Repository files navigation

OCR Evaluation Platform

📋 Overview

🚀 Features

🛠️ Tech Stack

📦 Installation

Prerequisites

Local Setup

Docker Deployment

Docker Compose (Recommended)

📝 Usage

Preparing Prediction Files

Submitting Predictions

Viewing Detailed Results

Admin Functions

Evaluation Metrics

📊 API Endpoints

Public Endpoints

GET /

POST /upload

POST /evaluate

GET /leaderboard

GET /details/{name}

GET /api/details/{name}

GET /set_language/{lang}

WebSocket /ws/{session_id}

Admin Endpoints

GET /admin/login

POST /admin/login

GET /admin/dashboard

POST /admin/logout

DELETE /api/admin/delete/{name}