Code the Reproduce the Dataset

[stefanhgm]

Hello!

Thanks for sharing the code of your work. Could you point me to the code to reproduce the dataset used for the contrastive learning? I would be very interested in using this dataset.

Thank you
Stefan

[prajwal967]

Hey, thanks for your interest. Here are the main steps we did

• Step 1: Obtain access and download mimic-iii data

• Step 2: We first ran the following notebook (more details are on their repo): CAML - Data Load

• Step 3: We then ran (more details are on their repo): KEPT - Data Load (It doesn't do anything extra - but makes some changes for the "50" task). The final result is a json file for each of the splits (train, dev, test). We did an extra step where we saved it as jsonl instead. Given below.

• Step 4: Ran the following code to save as jsonl and some minor cleanup. Clean the data and store in a form that can be used for model training.

import re
import json
from collections import Counter, defaultdict
from pathlib import Path

def clean_text(text, long_char_max=8):
    text = re.sub(r'\[\*\*[^\]]*\*\*\]', '', text)
    text = text.replace('\n', ' ').replace("\r", " ").strip()
    text = re.sub('\s+', ' ', text)
    return re.sub(
        rf'(?P<specchar>([^a-zA-Z0-9_\s]|_)){{{long_char_max},}}', r'\g<specchar>' * long_char_max, text
    )

input_file = '<directory>/physionet.org/files/mimiciii/1.4/mimic3_train.json'

output_file = \
'<directory>phenotype_classification/data/multi_label_mimic_icd_9/full/train.jsonl'

data_list = json.load(open(input_file))

with open(output_file, 'w') as file:
    for data in data_list:
        data.pop('Addition')
        data['text'] = clean_text(data.pop('TEXT'))
        data['labels'] = data.pop('LABELS').replace(';', ' ')
        file.write(json.dumps(data) + '\n')

• Step 5: Ran this for the train, dev and test splits: https://github.com/obi-ml-public/NoteContrast/blob/main/note_pretraining/pre_train_datasets/setup/mimic/mimic_icd10_map.py

Let us know if you want anymore details or if you have any questions! Thanks.

[stefanhgm]

Hi @prajwal967

Thanks for your detailed reply. I stumbled over some sections in mimic_icd10_map.py:

1. Do you provide '/mnt/<path>/phi/ehr_projects/icd_embeddings/models/v0/no_sampling_trial_2/checkpoint-200000/'?
2. What exactly is vocab = AutoTokenizer.from_pretrained(tokenizer_name, add_prefix_space=True).vocab used for or can it be removed from the icd mapping?

Just to clarify: as an output you expect the discharge note and the resolved ICD codes, right?

[prajwal967]

Hi @stefanhgm

1. • We currently haven't uploaded the models, primarily because we were not sure if a model trained on diagnostic codes on patients can be released without IRB approval.

   • We're more than happy to discuss internally and see if we can release these models and what that process would look like.

2. • The mimic data contained ICD-9 labels and we wanted to use ICD-10 labels - the lines 17-33 and 59-64 is essentially used to map ICD-9 to ICD-10.

   • The vocabulary of the tokenizer contains this list of ICD codes that were used to train the model from "1" - we use a subset since not all codes appear in our internal dataset. This "vocab" object contains the ICD-10 codes that were used to train the model, and hence while mapping we only map the ICD-9 codes to ICD-10 codes that the model has seen.

   • The format of "vocab" is {'A30.0': 1, 'B05.9': 2 ...} - if you can create a new vocab object that looks like this - and it contains all the ICD-10 codes that you are interested in - the code should work

   • You can remove/replace that line as long as "codes_map_dict" is defined accordingly, i.e. contains a mapping between ICD-9 and ICD-10 codes.

Let us know if that still sounds a bit confusing and if you have any other questions, happy to provide more details!