Skip to content

docs: comprehensive documentation accuracy and compliance update #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cculb merged 1 commit into from
Dec 21, 2025
Merged

docs: comprehensive documentation accuracy and compliance update #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,8 @@ def test_my_new_button(self, streamlit_page: Page):
| `ui-only` | Changes to Streamlit UI |
| `scraper-only` | Changes to scraper code |
| `mcp-only` | Changes to MCP server |
| `alerts-only` | Changes to alerts system |
| `ground-truth` | Rebuild ground truth data |
| `full-validation` | Before completing a task |

## Test File Locations
Expand All @@ -121,6 +123,7 @@ def test_my_new_button(self, streamlit_page: Page):
| Unit tests | `tests/unit/*.py` |
| Integration | `tests/integration/*.py` |
| Scraper E2E | `tests/e2e/test_scraper.py` |
| Multi-student | `tests/e2e/test_multi_student_scraper.py` |
| MCP server | `tests/e2e/test_mcp_tools.py` |
| Alerts | `tests/e2e/test_alerts.py` |
| UI (Streamlit) | `tests/e2e/test_streamlit_ui.py` |
Expand All @@ -129,10 +132,12 @@ def test_my_new_button(self, streamlit_page: Page):

| Class | Purpose |
|-------|---------|
| TestLoginPage | Login page render, form elements, demo credentials |
| TestPageLoad | Initial render, title, no errors |
| TestSidebarSettings | API key, model dropdown, student name |
| TestSidebarSettings | Model dropdown, logout button, clear chat button |
| TestQuickActions | 4 quick action buttons |
| TestWelcomeInfoBox | Student summary display |
| TestDashboard | Dashboard metrics (courses, missing work, attendance) |
| TestWelcomeSection | Welcome message and tips when chat is empty |
| TestConversationStarters | Dynamic starter buttons |
| TestChatInterface | Chat input and messages |

Expand Down
22 changes: 16 additions & 6 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@ cat /tmp/results/test-summary.json
| `full-validation` | Final verification |
| `scraper-only` | Changes to scraper code |
| `mcp-only` | Changes to MCP server |
| `alerts-only` | Alert detection changes |
| `ground-truth` | Validate against known data |

## UI Testing

Expand Down Expand Up @@ -118,10 +120,12 @@ gh run watch $(gh run list --repo cculb/schoolconnect --limit 1 --json databaseI

| Class | Tests |
|-------|-------|
| TestLoginPage | Login form, credentials, loading indicator |
| TestPageLoad | App loads, title, subtitle |
| TestSidebarSettings | API key, model dropdown, student name |
| TestSidebarSettings | Settings, model dropdown, logout |
| TestQuickActions | 4 action buttons work |
| TestWelcomeInfoBox | Student summary display |
| TestDashboard | Dashboard metrics, courses, attendance |
| TestWelcomeSection | Welcome message, tips |
| TestConversationStarters | Dynamic starter buttons |
| TestChatInterface | Chat input functionality |

Expand All @@ -141,22 +145,28 @@ When adding a UI feature:
src/
scraper/ # PowerSchool web scraper
mcp_server/ # MCP server for Claude access
models/ # SQLAlchemy models
repositories/ # Data access layer
database/ # Database layer (schema, views, repository)
cli/ # Command line interface
logutils/ # Logging utilities
tests/
unit/ # Fast, isolated tests
integration/ # Component tests
e2e/ # Full E2E with live PowerSchool
scripts/
scrape_full.py # Main scraper entry point
load_data.py # Load scraped data into DB
generate_test_summary.py # Creates agent-readable JSON
generate_agent_report.py # Creates detailed reports
validate_ground_truth.py # Ground truth validation
```

## Key Files

- `src/scraper/powerschool_scraper.py` - Main scraper logic
- `scripts/scrape_full.py` - Main scraper entry point
- `src/scraper/auth.py` - PowerSchool authentication
- `src/mcp_server/server.py` - MCP server implementation
- `src/repositories/*.py` - Database queries
- `src/database/repository.py` - Database queries and access
- `src/database/schema.sql` - Database schema definition
- `tests/e2e/test_*.py` - E2E test suites

## Environment
Expand Down
82 changes: 48 additions & 34 deletions COMPREHENSIVE_DATA_REPORT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,10 @@ SchoolConnect is a PowerSchool parent portal scraper that aggregates student aca
| Grades (Q1-Q4, S1-S2) | Captured | 12 | 100% |
| Assignments | Captured | 36 | 95% |
| Teachers | Captured | 7 | 90% |
| Attendance Summary | Captured | 1 | 80% |
| Daily Attendance | NOT Captured | 0 | 0% |
| Teacher Comments | NOT Captured | 0 | 0% |
| Attendance Summary | Captured | 1 | 100% |
| Daily Attendance | Captured | Variable | 100% |
| Teacher Comments | Captured | Variable | 100% |
| Course Scores | Captured | Variable | 100% |
| School Calendar | NOT Captured | 0 | 0% |
| Announcements | NOT Captured | 0 | 0% |

Expand All @@ -33,8 +34,8 @@ SchoolConnect is a PowerSchool parent portal scraper that aggregates student aca

```
Students in System:
- Delilah (ID: 55260) - 6th Grade
- Sean (ID: 55259) - Grade TBD
- Student A (ID: XXXXX) - 6th Grade
- Student B (ID: XXXXX) - Grade TBD
```

**Fields Captured:**
Expand Down Expand Up @@ -74,7 +75,7 @@ Students in System:

### 1.3 Grade Data

**Current Grades (Delilah - Q1):**
**Current Grades (Student A - Q1):**

| Course | Grade | Term |
|--------|-------|------|
Expand Down Expand Up @@ -153,7 +154,7 @@ Students in System:

### 1.6 Attendance Data

**Year-to-Date Summary (Delilah):**
**Year-to-Date Summary (Student A):**

- Attendance Rate: 94.4%
- Total Days: 80
Expand All @@ -171,14 +172,14 @@ Students in System:

---

## Part 2: Additional Data Available (NOT Currently Scraped)
## Part 2: Additional Data Sources

Based on analysis of raw HTML and PowerSchool structure, the following data sources are available but not captured:
Based on analysis of PowerSchool structure, the following shows implementation status of additional data sources:

### 2.1 Teacher Comments (HIGH VALUE)
### 2.1 Teacher Comments - IMPLEMENTED ✅

**Location:** `/guardian/teachercomments.html`
**Raw HTML Files:** `comments_q1.html`, `comments_q2.html` (545KB each)
**Parser:** `src/scraper/parsers/teacher_comments.py`

**Structure:**

Expand All @@ -187,33 +188,37 @@ Headers: Exp., Course #, Course, Teacher, Comment
Rows: 15 per quarter
```

**Data Available:**
**Data Captured:**

- Per-course teacher comments
- Quarter-specific feedback
- Personalized observations
- Progress notes
- Course number and expression
- Teacher name and email

**Impact:** Teacher comments provide qualitative insights that grades alone cannot capture. Essential for understanding *why* a student is performing a certain way.

### 2.2 Daily Attendance Records (MEDIUM VALUE)
### 2.2 Daily Attendance Records - IMPLEMENTED ✅

**Location:** Home page attendance grid, `/guardian/mba_attendance_monitor/`
**Parser:** `src/scraper/parsers/attendance.py`

**Structure:**

```
Attendance By Day | Last Week | This Week | Absences | Tardies | M T W H F
```

**Data Available:**
**Data Captured:**

- Day-by-day attendance (M/T/W/H/F)
- Weekly patterns
- Per-course absence/tardy counts
- Attendance codes and reasons
- Period-level attendance records

**Impact:** Enables pattern detection (e.g., "always absent on Mondays") and early intervention.
**Impact:** Enables pattern detection (e.g., "always absent on Mondays") and early intervention. Database views available: `v_daily_attendance`, `v_attendance_patterns`, `v_weekly_attendance`.

### 2.3 Applications/External Links (LOW-MEDIUM VALUE)

Expand All @@ -235,17 +240,19 @@ Applications | Description

**Impact:** Could provide one-stop access to all educational tools.

### 2.4 Detailed Course Scores Page (MEDIUM VALUE)
### 2.4 Detailed Course Scores Page - IMPLEMENTED ✅

**Location:** `/guardian/scores.html?frn=...`
**Parser:** `src/scraper/parsers/course_scores.py`

**Data Available:**
**Data Captured:**

- Full assignment breakdown per course
- Score distributions
- Category weights
- Standards alignment
- Assignment descriptions
- Individual assignment details

**Impact:** More granular view of how grades are calculated.

Expand Down Expand Up @@ -290,11 +297,9 @@ Applications | Description

### Gaps

1. **Attendance detail missing** - Only summary, no daily records
2. **No teacher comments** - Qualitative feedback not captured
3. **Score parsing incomplete** - "/10" format not fully parsed to numerics
4. **Second student (Sean) not scraped** - Only Delilah has data
5. **No historical tracking** - Scrape history table empty
1. **Score parsing incomplete** - "/10" format not fully parsed to numerics
2. **Second student (Student B) not scraped** - Only Student A has data
3. **No historical tracking** - Scrape history table empty

### Data Freshness

Expand All @@ -306,7 +311,7 @@ Applications | Description

## Part 4: Database Analytics Views

The system includes 8 pre-built analytical views:
The system includes 13 pre-built analytical views:

| View | Purpose | Status |
|------|---------|--------|
Expand All @@ -318,12 +323,17 @@ The system includes 8 pre-built analytical views:
| v_assignment_completion_rate | Completion % by course | Working |
| v_student_summary | High-level summary per student | Working |
| v_action_items | Prioritized parent action items | Working |
| v_daily_attendance | Daily attendance records with student info | Working |
| v_attendance_patterns | Attendance patterns by day of week | Working |
| v_weekly_attendance | Weekly attendance summaries | Working |
| v_teacher_comments | All teacher comments with course info | Working |
| v_teacher_comments_by_term | Comments grouped by term with counts | Working |

---

## Part 5: MCP Server Capabilities

The MCP server exposes 22+ tools for AI agents:
The MCP server exposes 24 tools for AI agents:

### Student Tools

Expand Down Expand Up @@ -380,10 +390,10 @@ Based on the research document's vision for a bilingual parent engagement app:

### Short-Term Additions (PowerSchool Data)

1. **Scrape Teacher Comments** - Add parser for `teachercomments.html`
2. **Daily Attendance** - Extract from home page attendance grid
1. ~~**Scrape Teacher Comments**~~ - ✅ COMPLETED - Parser implemented at `src/scraper/parsers/teacher_comments.py`
2. ~~**Daily Attendance**~~ - ✅ COMPLETED - Parser implemented at `src/scraper/parsers/attendance.py`
3. **Both Students** - Add multi-student support to scraper
4. **Course Details** - Scrape individual course score pages
4. ~~**Course Details**~~ - ✅ COMPLETED - Parser implemented at `src/scraper/parsers/course_scores.py`

### Medium-Term Integrations (External Sources)

Expand Down Expand Up @@ -438,21 +448,24 @@ Based on the research document's vision for a bilingual parent engagement app:

## Conclusion

SchoolConnect has successfully built a robust foundation for parent engagement by capturing core academic data from PowerSchool. The system is production-ready for:
SchoolConnect has successfully built a robust foundation for parent engagement by capturing comprehensive academic data from PowerSchool. The system is production-ready for:

- Tracking grades and assignments
- Alerting parents to missing work
- Monitoring attendance
- Monitoring attendance (including daily records and patterns)
- Facilitating teacher communication
- Accessing teacher comments and qualitative feedback
- Analyzing detailed course scores

To fully realize the vision of a bilingual parent engagement app for Berry Creek Middle School, the following priorities should be addressed:

1. **Add teacher comments** - Rich qualitative data available
1. ~~**Add teacher comments**~~ - ✅ COMPLETED - Rich qualitative data now captured
2. **Integrate translation** - Spanish as primary secondary language
3. **Expand data sources** - Calendar, announcements, community resources
4. **Build conversational interface** - Natural language queries
5. **Multi-student support** - Extend scraper to handle multiple students

The technical foundation is solid. The next phase is expanding data coverage and building the parent-facing interface.
The technical foundation is solid with comprehensive data extraction complete. The next phase is building the bilingual parent-facing interface.

---

Expand Down Expand Up @@ -802,7 +815,7 @@ Reports generated from 75+ web sources, December 2025:

### Appendix B: Existing Codebase Assets

**Already Built (23 MCP Tools):**
**Already Built (24 MCP Tools):**

- `list_students` - List all students
- `get_student_summary` - Comprehensive overview
Expand Down Expand Up @@ -830,9 +843,10 @@ Reports generated from 75+ web sources, December 2025:
SchoolConnect has a **significant head start** over competitors:

1. ✅ PowerSchool data already extracted and structured
2. ✅ 23 MCP tools ready for Claude integration
2. ✅ 24 MCP tools ready for Claude integration
3. ✅ Streamlit chat interface exists
4. ✅ Database with 8 analytical views
4. ✅ Database with 13 analytical views
5. ✅ Teacher comments, daily attendance, and course scores implemented

**To reach MVP demo in 4 weeks:**

Expand Down
Loading
Loading