initial experiments by abirharrasse · Pull Request #2 · withmartian/mi-cot

abirharrasse · 2025-12-09T22:12:53Z

No description provided.

Copilot

Pull request overview

This PR introduces an experimental pipeline for analyzing causal relationships in chain-of-thought (CoT) reasoning. The code generates CoT reasoning traces from a language model, computes sentence-level causal influence matrices using attention masking, extracts high-importance "thought anchors," and trains classifiers to categorize reasoning steps into 8 semantic classes (e.g., problem setup, active computation, self-checking).

Implements causal tracing via attention masking to measure how masking source sentences affects target sentence predictions (KL divergence)
Classifies reasoning sentences into 8 anchor classes and selects high-importance sentences based on causal outgoing influence
Trains Logistic Regression and MLP classifiers on hidden states to predict anchor classes

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

exploration/rpc.py

Copilot · 2025-12-09T22:21:15Z

exploration/rpc.py

+            text_before = problem + " " + " ".join(sentences[:idx])
+            hidden_state = get_hidden_state(text_before)


The feature extraction concatenates problem + " " + " ".join(sentences[:idx]) to get context before each anchor sentence. This reconstructs text from split sentences, which may not match the original CoT text due to lost formatting, punctuation, or whitespace. This inconsistency could affect the hidden state extraction. Consider storing the original character positions of sentences and slicing the original CoT text instead of reconstructing from split sentences.

@copilot open a new pull request to apply changes based on this feedback

exploration/rpc.py

Copilot · 2025-12-09T22:21:16Z

exploration/rpc.py

+
+scaler = StandardScaler()
+X_scaled = scaler.fit_transform(X)
+X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42, stratify=y)


The train/test split uses stratify=y which is good practice, but with small class counts (min_samples=2), some classes may have only 2-3 samples total. Stratification with very small classes can fail or result in insufficient test samples for meaningful evaluation. Consider increasing min_samples to at least 5-10, or using cross-validation instead of a single train/test split for more robust evaluation with limited data.

Copilot · 2025-12-09T22:21:16Z

exploration/rpc.py

+# EXTRACT ANCHORS
+
+print("="*80)
+print("PHASE 2: EXTRACTING THOUGHT ANCHORS")


This is labeled as "PHASE 2" but Phase 2 was already used at line 207. The phases are misnumbered. This should be Phase 3, and subsequent phases should be renumbered (current Phase 3 → Phase 4, current Phase 4 → Phase 5).

Suggested change

print("PHASE 2: EXTRACTING THOUGHT ANCHORS")

print("PHASE 3: EXTRACTING THOUGHT ANCHORS")

Copilot · 2025-12-09T22:21:20Z

exploration/rpc.py

+                'outgoing': outgoing_feature
+            })
+
+    pickle.dump(all_features, open(ckpt_features, 'wb'))


File is opened but is not closed.

Suggested change

pickle.dump(all_features, open(ckpt_features, 'wb'))

with open(ckpt_features, 'wb') as f:

pickle.dump(all_features, f)

Copilot · 2025-12-09T22:21:21Z

exploration/rpc.py

+pickle.dump(clf_lr, open(f"{checkpoint_dir}/classifier_lr.pkl", 'wb'))
+pickle.dump(clf_mlp, open(f"{checkpoint_dir}/classifier_mlp.pkl", 'wb'))
+pickle.dump(scaler, open(f"{checkpoint_dir}/scaler.pkl", 'wb'))
+pickle.dump(class_to_idx, open(f"{checkpoint_dir}/class_to_idx.pkl", 'wb'))


File is opened but is not closed.

Suggested change

pickle.dump(clf_lr, open(f"{checkpoint_dir}/classifier_lr.pkl", 'wb'))

pickle.dump(clf_mlp, open(f"{checkpoint_dir}/classifier_mlp.pkl", 'wb'))

pickle.dump(scaler, open(f"{checkpoint_dir}/scaler.pkl", 'wb'))

pickle.dump(class_to_idx, open(f"{checkpoint_dir}/class_to_idx.pkl", 'wb'))

with open(f"{checkpoint_dir}/classifier_lr.pkl", 'wb') as f:

pickle.dump(clf_lr, f)

with open(f"{checkpoint_dir}/classifier_mlp.pkl", 'wb') as f:

pickle.dump(clf_mlp, f)

with open(f"{checkpoint_dir}/scaler.pkl", 'wb') as f:

pickle.dump(scaler, f)

with open(f"{checkpoint_dir}/class_to_idx.pkl", 'wb') as f:

pickle.dump(class_to_idx, f)

exploration/rpc.py

Copilot · 2025-12-09T22:21:21Z

exploration/rpc.py

+pickle.dump(clf_lr, open(f"{checkpoint_dir}/classifier_lr.pkl", 'wb'))
+pickle.dump(clf_mlp, open(f"{checkpoint_dir}/classifier_mlp.pkl", 'wb'))
+pickle.dump(scaler, open(f"{checkpoint_dir}/scaler.pkl", 'wb'))
+pickle.dump(class_to_idx, open(f"{checkpoint_dir}/class_to_idx.pkl", 'wb'))


File is opened but is not closed.

Suggested change

pickle.dump(clf_lr, open(f"{checkpoint_dir}/classifier_lr.pkl", 'wb'))

pickle.dump(clf_mlp, open(f"{checkpoint_dir}/classifier_mlp.pkl", 'wb'))

pickle.dump(scaler, open(f"{checkpoint_dir}/scaler.pkl", 'wb'))

pickle.dump(class_to_idx, open(f"{checkpoint_dir}/class_to_idx.pkl", 'wb'))

with open(f"{checkpoint_dir}/classifier_lr.pkl", 'wb') as f:

pickle.dump(clf_lr, f)

with open(f"{checkpoint_dir}/classifier_mlp.pkl", 'wb') as f:

pickle.dump(clf_mlp, f)

with open(f"{checkpoint_dir}/scaler.pkl", 'wb') as f:

pickle.dump(scaler, f)

with open(f"{checkpoint_dir}/class_to_idx.pkl", 'wb') as f:

pickle.dump(class_to_idx, f)

Copilot · 2025-12-09T22:21:22Z

exploration/rpc.py

+pickle.dump(clf_lr, open(f"{checkpoint_dir}/classifier_lr.pkl", 'wb'))
+pickle.dump(clf_mlp, open(f"{checkpoint_dir}/classifier_mlp.pkl", 'wb'))
+pickle.dump(scaler, open(f"{checkpoint_dir}/scaler.pkl", 'wb'))
+pickle.dump(class_to_idx, open(f"{checkpoint_dir}/class_to_idx.pkl", 'wb'))


File is opened but is not closed.

Suggested change

pickle.dump(clf_lr, open(f"{checkpoint_dir}/classifier_lr.pkl", 'wb'))

pickle.dump(clf_mlp, open(f"{checkpoint_dir}/classifier_mlp.pkl", 'wb'))

pickle.dump(scaler, open(f"{checkpoint_dir}/scaler.pkl", 'wb'))

pickle.dump(class_to_idx, open(f"{checkpoint_dir}/class_to_idx.pkl", 'wb'))

with open(f"{checkpoint_dir}/classifier_lr.pkl", 'wb') as f:

pickle.dump(clf_lr, f)

with open(f"{checkpoint_dir}/classifier_mlp.pkl", 'wb') as f:

pickle.dump(clf_mlp, f)

with open(f"{checkpoint_dir}/scaler.pkl", 'wb') as f:

pickle.dump(scaler, f)

with open(f"{checkpoint_dir}/class_to_idx.pkl", 'wb') as f:

pickle.dump(class_to_idx, f)

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot · 2025-12-10T09:03:54Z

@abirharrasse I've opened a new pull request, #3, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

initial experiments

2e2a941

Copilot AI review requested due to automatic review settings December 9, 2025 22:12

Copilot started reviewing on behalf of abirharrasse December 9, 2025 22:13 View session

Copilot AI reviewed Dec 9, 2025

View reviewed changes

wlg1 self-assigned this Dec 9, 2025

wlg1 approved these changes Dec 10, 2025

View reviewed changes

Update exploration/rpc.py

28e9b85

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI mentioned this pull request Dec 10, 2025

Use character positions for text slicing instead of reconstructing from sentences #3

Draft

abirharrasse and others added 3 commits December 10, 2025 10:15

Update exploration/rpc.py

9e49e7e

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update exploration/rpc.py

38392b5

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update exploration/rpc.py

2c27ccf

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

abirharrasse merged commit b49f87f into main Dec 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial experiments#2

initial experiments#2
abirharrasse merged 5 commits intomainfrom
abir-explorations

abirharrasse commented Dec 9, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Dec 9, 2025

Uh oh!

abirharrasse Dec 10, 2025

Uh oh!

Uh oh!

Copilot AI Dec 9, 2025

Uh oh!

Copilot AI Dec 9, 2025

Uh oh!

Copilot AI Dec 9, 2025

Uh oh!

Copilot AI Dec 9, 2025

Uh oh!

Uh oh!

Copilot AI Dec 9, 2025

Uh oh!

Copilot AI Dec 9, 2025

Uh oh!

Copilot AI commented Dec 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		text_before = problem + " " + " ".join(sentences[:idx])
		hidden_state = get_hidden_state(text_before)

	print("PHASE 2: EXTRACTING THOUGHT ANCHORS")
	print("PHASE 3: EXTRACTING THOUGHT ANCHORS")

	pickle.dump(all_features, open(ckpt_features, 'wb'))
	with open(ckpt_features, 'wb') as f:
	pickle.dump(all_features, f)

-pickle.dump(clf_lr, open(f"{checkpoint_dir}/classifier_lr.pkl", 'wb'))
-pickle.dump(clf_mlp, open(f"{checkpoint_dir}/classifier_mlp.pkl", 'wb'))
-pickle.dump(scaler, open(f"{checkpoint_dir}/scaler.pkl", 'wb'))
-pickle.dump(class_to_idx, open(f"{checkpoint_dir}/class_to_idx.pkl", 'wb'))
+with open(f"{checkpoint_dir}/classifier_lr.pkl", 'wb') as f:
+    pickle.dump(clf_lr, f)
+with open(f"{checkpoint_dir}/classifier_mlp.pkl", 'wb') as f:
+    pickle.dump(clf_mlp, f)
+with open(f"{checkpoint_dir}/scaler.pkl", 'wb') as f:
+    pickle.dump(scaler, f)
+with open(f"{checkpoint_dir}/class_to_idx.pkl", 'wb') as f:
+    pickle.dump(class_to_idx, f)

Conversation

abirharrasse commented Dec 9, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

abirharrasse Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Dec 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants