MisSynth

Datasets

• GPT-5

Baseline experiment

Environment setup

Experiment hardware: M1 MacBook Pro with 32 GB of RAM

git clone --recursive https://github.com/mxpoliakov/MisSynth.git && cd MisSynth

export PYTHONPATH=$(pwd):$(pwd)/missci
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Vector store

Create a JSON vector store based on scraped articles (web, pdf) from the MISSCI dev split. All 30 articles were scraped and vectorized using NeuML/pubmedbert-base-embeddings with a chunk size of 512 and chunk overlap of 64.

python create_vector_store.py

Synthetic fallacies (fallacious premise and context)

Generate synthetic fallacies using the single class prompt template. A vector store is used to retrieve relevant article excerpts to support the argument claim—essentially functioning as a lightweight RAG with metadata filtering. The OpenAI o4-mini model is used to generate 30 synthetic fallacies per sample from the Missci dev split. Each fallacy includes both a fallacious premise and context.

Additionally, 15 synthetic claim–accurate premise pairs with real fallacies are generated for each entry in the dev split, using the synthetic claim-premise template.

export OPENAI_API_KEY=...
python generate_synthetic_data.py --prompt-template single-class-synthetic-fallacy-context --n-synthetic-entries 30
python generate_synthetic_data.py --prompt-template synthetic-claim-premise --n-synthetic-entries 15

You can also create and analyze a unified jsonl dataset (stored in dataset folder) via:

python create_unified_dataset.py
python analyze_synthetic_dataset.py

Fine-tune LLM on synthetic fallacies

Create a dataset using raw data from the previous step. For the baseline experiment, we will classify fallacies with premise using classify with definition template. Given the synthetic fallacies generated, we can fill out the template and provide responses to fine-tune the LLM. Let's fine-tune Phi-4 from Microsoft with synthetic fallacies.

python create_fine_tuning_dataset.py

python -m mlx_lm lora --model mlx-community/phi-4-8bit --data output \
--train --fine-tune-type lora --batch-size 1 --num-layers 16 --iters 500 --adapter-path adapters

Benchmark vanilla model vs fine-tuned model

Benchmark on dev missci split to avoid data leakage:

python run_mlx_fallacy_classification.py --model-name phi-4-8bit
python run_mlx_fallacy_classification.py --model-name phi-4-8bit --adapter-path adapters

cd missci

python run-fallacy-classification-with-gold-premise.py parse-llm-output phi-4-8b
it_cls_with_premise_classify-D_test.jsonl

python run-fallacy-classification-with-gold-premise.py parse-llm-output phi-4-8b
it_cls_with_premise_classify-D_test_adapters.jsonl

Model	Vanilla acc	Vanilla F1	Finetune acc	Finetune F1	Lora layers	Params
LLaMA 2	0.577 (*)	0.464 (*)	-	-	-	70B
Phi-4 (8-bit)	0.667	0.550	0.762	0.690	16	15B

* Table 3 from MISSCI: Reconstructing Fallacies in Misrepresented Science