Fine-tune Question-Answering model on our own data

[FrancescoCasalegno]

Context

• Pre-trained QA models seem to give decent accuracy on our BBP internal QA samples (see Question-Answering: collect example questions + run first analysis with pre-trained QA models #612).
• Hopefully, we can get better accuracy by fine-tuning the best performing models on our own data.
• However, to do so, we need to have enough samples in our dataset to perform a train-valid split, and we also need to double-check the quality of our training data.

Actions

• Fine-tune the best performing QA model(s) on our own QA dataset, using k-fold cross-validation.
• Investigate also results when holdout (valid) splits are created by removing samples from one source (e.g. WvG, PS, HM, ...) and training on the others.
• If our results are better than the baseline, compute also training curves (i.e. increase training set size and check accuracy).

Dependencies

• Add Question-Answer samples to our dataset using data from KG #616
• Add more than one acceptable answer in our Question-Answering dataset #617
• In general, we should have ~200 samples at the very least to be able to run this experiment.