Revokenization

[vilhil]

Hi,

Thank you for your great work.
I'm trying to train a RoBERTa-based VLM model on my own dataset. I plan to use your pre-trained vokenizer provided here. But, in my understanding, your pre-trained vokenizer or cross-modal matching model is trained based on bert-base-uncased tokenizer and thus, I need revokenization to obtain the vokenization result, which will be used in the RoBERTa-based VLM training.
While I found the code revokenize_corpus_mp.py, it seems that Revokenizer is not used there (it's imported though).

Could you update the code accordingly? or could you give me some instruction for how to use the Revokenizer class in the code?
Also, I would appreciate it if you could provide a sample bash script like mpvokenize_wiki.bash to apply revokenization to a given dataset.

Thanks.