Continual pretraining data

[xufana7]

Your research is highly valuable to the community, and I believe that having access to the continual pre-training data you used would greatly accelerate further research in this field. As you know, creating high-quality, large-scale continual pre-training datasets is a significant challenge.

I was hoping you might consider open-sourcing the dataset you used for your work. Sharing this data would be a tremendous contribution and would allow others to build upon your foundation, reproduce your results, and explore new research avenues more effectively.

[toooooodo]

Thank you for your kind words. We agree that high-quality datasets are essential for advancing research in this area.

• Molecular data: Our molecular pre-training data comes from PubChem, which can be downloaded directly from the official repository: https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/SDF/

• Protein data: For protein pre-training, we relied on UniRef, and a well-organized version is already available on Hugging Face: https://huggingface.co/datasets/agemagician/uniref50

• Natural language literature data: This dataset was collected and processed in-house. We are currently organizing it, and we plan to release it as an open resource in the future.

We hope this helps the community in the meantime, and we’ll share updates once the literature dataset becomes available.