Use character positions for text slicing instead of reconstructing from sentences

[Copilot]

Feature extraction reconstructed text by joining split sentences (problem + " " + " ".join(sentences[:idx])), losing formatting and whitespace from the original CoT. This affects hidden state extraction accuracy.

Changes

• split_into_sentences(): Now returns (sentences, positions) tuple where positions track character boundaries in original text
• Data storage: Added sentence_positions field to preserve original text structure
• Feature extraction: Changed from sentence reconstruction to character slicing: cot[:end_pos]

Example

# Before: loses double spaces between sentences
cot = "First step.  Second step.  Third step."
sentences = split_into_sentences(cot)  # ['First step.', 'Second step.', ...]
text_before = " ".join(sentences[:2])  # "First step. Second step." (single space)

# After: preserves original formatting
sentences, positions = split_into_sentences(cot)
text_before = cot[:positions[1][1]]  # "First step.  Second step." (double space)

Position tracking uses unstripped boundaries for accurate slicing while sentences remain stripped for classification.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.