Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
254 changes: 254 additions & 0 deletions src/cases/difficulty-ladder.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
/**
* Difficulty Ladder System
*
* Generates difficulty variants (easy/medium/hard) from a base case.
* Each variant provides different levels of scaffolding/context.
*/

import { Case, CaseFile, CaseDifficulty } from './types';

// =============================================================================
// Difficulty Levels
// =============================================================================

/**
* Scaffolding level for a difficulty variant
*/
export interface ScaffoldingLevel {
/** How much context is provided */
level: 'easy' | 'medium' | 'hard';

/** Additional files to include (beyond base case files) */
additionalFiles?: CaseFile[];

/** Hints or guidance to add to the prompt */
hints?: string[];

/** Whether to show the solution (for testing only) */
showSolution?: boolean;

/** Notes about this variant */
notes?: string;
}

/**
* Difficulty ladder configuration for a case
*/
export interface DifficultyLadder {
/** Base case (the original task) */
base: Case;

/** Scaffolding levels for each difficulty */
levels: Record<CaseDifficulty, ScaffoldingLevel>;

/** Whether to auto-generate variants on load */
autoGenerate?: boolean;
}

// =============================================================================
// Difficulty Ladder Generator
// =============================================================================

/**
* Generate difficulty variants from a base case
*/
export function generateDifficultyVariants(baseCase: Case): Case[] {
const variants: Case[] = [];

// Create easy variant
const easyVariant = createVariant(baseCase, 'easy');
if (easyVariant) variants.push(easyVariant);

// Create medium variant
const mediumVariant = createVariant(baseCase, 'medium');
if (mediumVariant) variants.push(mediumVariant);

// Create hard variant
const hardVariant = createVariant(baseCase, 'hard');
if (hardVariant) variants.push(hardVariant);

return variants;
}

/**
* Create a single difficulty variant
*/
function createVariant(baseCase: Case, difficulty: CaseDifficulty): Case | null {
const level = baseCase.difficultyLadder?.levels[difficulty];
if (!level) {
// No ladder defined, return null (use base case as-is)
return null;
}

// Combine base files with additional files
const allFiles = baseCase.files ? [...baseCase.files] : [];
if (level.additionalFiles) {
allFiles.push(...level.additionalFiles);
}

// Build prompt with hints
let prompt = baseCase.prompt;
if (level.hints && level.hints.length > 0) {
prompt += '\n\n' + level.hints.join('\n');
}

// Create variant case
const variant: Case = {
...baseCase,
id: `${baseCase.id}-${difficulty}`,
title: `${baseCase.title} (${difficulty})`,
prompt,
files: allFiles,
difficulty,
// Don't include solution in variants (unless explicitly requested)
solution: level.showSolution ? baseCase.solution : undefined,
notes: level.notes,
};
Comment on lines +96 to +106
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Variant inherits difficultyLadder from base and silently drops notes.

Two issues with the spread + overrides:

  1. ...baseCase copies the difficultyLadder property into every variant. Variants shouldn't carry the ladder configuration — it creates unnecessary circular nesting and bloats serialized output.
  2. notes: level.notes explicitly sets notes even when level.notes is undefined, which overwrites whatever baseCase.notes contained (the spread would have preserved it). Use a conditional or nullish-coalescing pattern instead.
Proposed fix
   const variant: Case = {
     ...baseCase,
     id: `${baseCase.id}-${difficulty}`,
     title: `${baseCase.title} (${difficulty})`,
     prompt,
     files: allFiles,
     difficulty,
-    // Don't include solution in variants (unless explicitly requested)
     solution: level.showSolution ? baseCase.solution : undefined,
-    notes: level.notes,
+    notes: level.notes ?? baseCase.notes,
+    difficultyLadder: undefined,  // variants shouldn't carry the ladder
   };
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const variant: Case = {
...baseCase,
id: `${baseCase.id}-${difficulty}`,
title: `${baseCase.title} (${difficulty})`,
prompt,
files: allFiles,
difficulty,
// Don't include solution in variants (unless explicitly requested)
solution: level.showSolution ? baseCase.solution : undefined,
notes: level.notes,
};
const variant: Case = {
...baseCase,
id: `${baseCase.id}-${difficulty}`,
title: `${baseCase.title} (${difficulty})`,
prompt,
files: allFiles,
difficulty,
solution: level.showSolution ? baseCase.solution : undefined,
notes: level.notes ?? baseCase.notes,
difficultyLadder: undefined, // variants shouldn't carry the ladder
};
🤖 Prompt for AI Agents
In `@src/cases/difficulty-ladder.ts` around lines 96 - 106, The variant currently
spreads ...baseCase which pulls in baseCase.difficultyLadder (causing
nested/bloated variants) and then unconditionally sets notes to level.notes
(overwriting baseCase.notes with undefined); to fix, exclude difficultyLadder
when building the variant (e.g., destructure baseCase to remove difficultyLadder
before spreading) and set notes using a nullish fallback like level.notes ??
baseCase.notes so existing notes are preserved; update the construction of
variant (the object built from baseCase and fields id, title, prompt, files,
difficulty, solution, notes) accordingly.


return variant;
}

/**
* Get the scaffolding level for a difficulty
*/
export function getScaffoldingLevel(baseCase: Case, difficulty: CaseDifficulty): ScaffoldingLevel | null {
return baseCase.difficultyLadder?.levels[difficulty] || null;
}

/**
* Check if a case has a difficulty ladder defined
*/
export function hasDifficultyLadder(baseCase: Case): boolean {
return !!baseCase.difficultyLadder;
}

// =============================================================================
// Difficulty Ladder Builder
// =============================================================================

/**
* Builder for creating difficulty ladders
*/
export class DifficultyLadderBuilder {
private base: Case;
private levels: Partial<Record<CaseDifficulty, ScaffoldingLevel>> = {};

constructor(baseCase: Case) {
this.base = baseCase;
}

/**
* Set the easy level scaffolding
*/
withEasy(hints?: string[], additionalFiles?: CaseFile[], notes?: string): this {
this.levels.easy = {
level: 'easy',
hints,
additionalFiles,
notes,
};
return this;
}

/**
* Set the medium level scaffolding
*/
withMedium(hints?: string[], additionalFiles?: CaseFile[], notes?: string): this {
this.levels.medium = {
level: 'medium',
hints,
additionalFiles,
notes,
};
return this;
}

/**
* Set the hard level scaffolding
*/
withHard(hints?: string[], additionalFiles?: CaseFile[], notes?: string): this {
this.levels.hard = {
level: 'hard',
hints,
additionalFiles,
notes,
};
return this;
}

/**
* Build the difficulty ladder
*/
build(): DifficultyLadder {
return {
base: this.base,
levels: this.levels as Record<CaseDifficulty, ScaffoldingLevel>,
autoGenerate: true,
};
}
Comment on lines +182 to +188
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Unsafe cast: Partial<Record<...>>Record<...> can produce undefined at runtime.

this.levels is Partial<Record<CaseDifficulty, ScaffoldingLevel>>, so any subset of keys may be set. The as Record<...> cast on line 185 silences the compiler, but consumers of DifficultyLadder.levels (e.g., levels['medium']) will get undefined at runtime when the key wasn't set — violating the non-optional Record contract.

Either validate that all three levels are present and throw if not, or change DifficultyLadder.levels to Partial<Record<...>> so the type is honest.

Option A: Validate at build time
   build(): DifficultyLadder {
+    const required: CaseDifficulty[] = ['easy', 'medium', 'hard'];
+    for (const d of required) {
+      if (!this.levels[d]) {
+        throw new Error(`Missing scaffolding level: ${d}`);
+      }
+    }
     return {
       base: this.base,
       levels: this.levels as Record<CaseDifficulty, ScaffoldingLevel>,
       autoGenerate: true,
     };
   }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
build(): DifficultyLadder {
return {
base: this.base,
levels: this.levels as Record<CaseDifficulty, ScaffoldingLevel>,
autoGenerate: true,
};
}
build(): DifficultyLadder {
const required: CaseDifficulty[] = ['easy', 'medium', 'hard'];
for (const d of required) {
if (!this.levels[d]) {
throw new Error(`Missing scaffolding level: ${d}`);
}
}
return {
base: this.base,
levels: this.levels as Record<CaseDifficulty, ScaffoldingLevel>,
autoGenerate: true,
};
}
🤖 Prompt for AI Agents
In `@src/cases/difficulty-ladder.ts` around lines 182 - 188, The build() method is
unsafely casting this.levels (Partial<Record<CaseDifficulty, ScaffoldingLevel>>)
to a full Record which can produce undefined at runtime; update build() in
DifficultyLadder to validate that all CaseDifficulty keys are present in
this.levels (e.g., iterate over Object.values(CaseDifficulty) or the explicit
['easy','medium','hard'] set) and throw a clear error if any are missing, then
return the object with levels typed as Record<CaseDifficulty, ScaffoldingLevel>;
alternatively, if you prefer not to enforce presence, change the
DifficultyLadder.levels type to Partial<Record<CaseDifficulty,
ScaffoldingLevel>> so the public API reflects optional entries—pick one of these
two fixes and apply it to the build() implementation and DifficultyLadder type
accordingly.

}

// =============================================================================
// Default Scaffolding Strategies
// =============================================================================

/**
* Default scaffolding for easy level
* - More context, hints, and guidance
*/
export function defaultEasyScaffolding(_baseCase: Case): ScaffoldingLevel {
return {
level: 'easy',
hints: [
'This is an easy task. Focus on correctness and following best practices.',
'You have all the context you need to complete this task.',
'Take your time to understand the codebase before making changes.',
],
notes: 'Easy: Maximum scaffolding provided',
};
}

/**
* Default scaffolding for medium level
* - Standard context, minimal hints
*/
export function defaultMediumScaffolding(_baseCase: Case): ScaffoldingLevel {
return {
level: 'medium',
hints: [
'Complete this task to the best of your ability.',
],
notes: 'Medium: Standard scaffolding',
};
}

/**
* Default scaffolding for hard level
* - Minimal context, agent must discover
*/
export function defaultHardScaffolding(_baseCase: Case): ScaffoldingLevel {
return {
level: 'hard',
hints: [
'You need to figure out the best approach for this task.',
'Explore the codebase to understand the context.',
'Make reasonable assumptions and document them.',
],
notes: 'Hard: Minimal scaffolding',
};
}

/**
* Create a difficulty ladder with default scaffolding
*/
export function createDefaultLadder(_baseCase: Case): DifficultyLadder {
return {
base: _baseCase,
levels: {
easy: defaultEasyScaffolding(_baseCase),
medium: defaultMediumScaffolding(_baseCase),
hard: defaultHardScaffolding(_baseCase),
},
autoGenerate: true,
};
}
18 changes: 18 additions & 0 deletions src/cases/types.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import type { ScaffoldingLevel } from './difficulty-ladder';
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

# First, let's check the imports structure
echo "=== types.ts imports from difficulty-ladder ==="
rg "from.*difficulty-ladder" src/cases/types.ts

echo ""
echo "=== difficulty-ladder.ts imports from types ==="
rg "from.*types" src/cases/difficulty-ladder.ts

Repository: AnswerLayer/sniffbench

Length of output: 281


🏁 Script executed:

# Check if both files exist and get their contents
echo "=== Checking src/cases/types.ts imports section ==="
head -20 src/cases/types.ts

echo ""
echo "=== Checking src/cases/difficulty-ladder.ts imports section ==="
head -20 src/cases/difficulty-ladder.ts

Repository: AnswerLayer/sniffbench

Length of output: 1460


🏁 Script executed:

# Search for DifficultyLadder definitions in both files
echo "=== DifficultyLadder in types.ts ==="
rg -A 10 "interface DifficultyLadder|type DifficultyLadder" src/cases/types.ts

echo ""
echo "=== DifficultyLadder in difficulty-ladder.ts ==="
rg -A 10 "interface DifficultyLadder|type DifficultyLadder" src/cases/difficulty-ladder.ts

Repository: AnswerLayer/sniffbench

Length of output: 697


🏁 Script executed:

# Check what ScaffoldingLevel is and where it's defined/used
echo "=== ScaffoldingLevel definition ==="
rg -B 2 -A 5 "ScaffoldingLevel" src/cases/difficulty-ladder.ts | head -30

echo ""
echo "=== Uses of ScaffoldingLevel in types.ts ==="
rg "ScaffoldingLevel" src/cases/types.ts

Repository: AnswerLayer/sniffbench

Length of output: 1121


Circular dependency and duplicate type definitions between types.ts and difficulty-ladder.ts.

types.ts imports ScaffoldingLevel from difficulty-ladder.ts, while difficulty-ladder.ts imports Case, CaseFile, and CaseDifficulty from types.ts. Additionally, DifficultyLadder is defined identically in both files, creating unnecessary duplication.

Move ScaffoldingLevel and DifficultyLadder into types.ts—both are pure type definitions that belong alongside the rest of the schema. This eliminates the circular dependency and consolidates the duplicate definitions. difficulty-ladder.ts would then depend only on types.ts with no reverse dependency.

🤖 Prompt for AI Agents
In `@src/cases/types.ts` at line 1, There’s a circular import and duplicated type
definitions: move the ScaffoldingLevel type and the DifficultyLadder type
definition into the shared types file (where Case, CaseFile, CaseDifficulty
live), remove the duplicate DifficultyLadder from difficulty-ladder.ts, and stop
importing ScaffoldingLevel from difficulty-ladder.ts; then update
difficulty-ladder.ts to import ScaffoldingLevel and DifficultyLadder from the
consolidated types module (and remove any reverse imports of
Case/CaseFile/CaseDifficulty), ensuring all usages reference the single
definitions ScaffoldingLevel and DifficultyLadder in the types file.

/**
* Case and Rubric Schema Types
*
Expand Down Expand Up @@ -39,6 +40,20 @@ export type CaseSource = 'bootstrap' | 'generated' | 'manual' | 'imported' | 'cl
*/
export type CaseDifficulty = 'easy' | 'medium' | 'hard';


/**
* Difficulty ladder configuration for a case
*/
export interface DifficultyLadder {
/** Base case (the original task) */
base: Case;

/** Scaffolding levels for each difficulty */
levels: Record<CaseDifficulty, ScaffoldingLevel>;

/** Whether to auto-generate variants on load */
autoGenerate?: boolean;
}
Comment on lines +47 to +56
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

DifficultyLadder is defined in both types.ts and difficulty-ladder.ts.

This interface is declared identically in both files (here at lines 47-56 and in difficulty-ladder.ts at lines 37-46). Having two exported definitions of the same interface will cause ambiguity for consumers — which one should they import? Consolidate it in one place (preferably types.ts since it's a type definition).

Additionally, DifficultyLadder.base: Case creates a self-referential structure since Case itself contains difficultyLadder?: DifficultyLadder. This means a Case embeds a ladder that re-embeds the whole Case. Consider using a lighter reference (e.g., baseId: string) instead of embedding the full Case to avoid circular nesting and serialization issues.

🤖 Prompt for AI Agents
In `@src/cases/types.ts` around lines 47 - 56, You have two identical exported
interfaces named DifficultyLadder (in types.ts and difficulty-ladder.ts) and the
current DifficultyLadder.base is a self-referential Case which causes circular
nesting; remove the duplicate export from difficulty-ladder.ts and keep the
single source of truth in types.ts, then change DifficultyLadder.base from type
Case to a lightweight identifier (e.g., baseId: string) and update all usages
that reference DifficultyLadder.base (including any constructors, serializers,
deserializers, and the Case type definition that currently has
difficultyLadder?: DifficultyLadder) to use baseId or resolve the Case by id
where needed; ensure imports/reference sites (functions/classes named
DifficultyLadder, Case, and any methods in difficulty-ladder.ts that manipulate
ladders) are updated to the consolidated type and that serialization no longer
nests full Case objects.

/**
* Agent behavior expectations for a case
*/
Expand Down Expand Up @@ -110,6 +125,9 @@ export interface Case {
/** Additional notes or hints (not shown to agent) */
notes?: string;

// Difficulty ladder configuration for generating variants
difficultyLadder?: DifficultyLadder;

// Metadata added by loader
/** Source file path (added by loader) */
_sourcePath?: string;
Expand Down
Loading
Loading