Design Generated Case Templates

[jharris1679]

Overview

Design templates that an LLM uses to generate repo-specific evaluation cases. The LLM analyzes the codebase and creates targeted challenges based on what it discovers.

Template Categories

1. Style Conformance Detection

template: style-conformance
prompt_to_llm: |
  Analyze this codebase and find code that doesn't match the project's style:
  - Naming conventions that differ from the norm
  - Formatting inconsistencies
  - Pattern violations
  
  For each finding, generate a case asking the agent to fix it.

2. Missing Coverage Detection

Find public functions/methods that lack test coverage.

3. Duplication Detection

Find duplicated code patterns in the codebase.

4. Type Safety Gaps

For typed languages, find code with weak/missing types.

5. Dependency Updates

Find deprecated or outdated API usage.

Generation Flow

1. sniff generate --template style-conformance
2. LLM analyzes codebase with template prompt
3. LLM outputs structured case definitions
4. Cases validated and stored in cases/generated/
5. User reviews (optional) before evaluation

Tasks

• Define 5 generation template types
• Design template schema
• Create LLM prompts for each template
• Design output validation
• Document template creation guide

Acceptance Criteria

• 5 template types defined with clear purposes
• Templates produce valid case definitions
• Output schema is consistent
• Documentation enables custom templates