This repository was archived by the owner on Jun 16, 2025. It is now read-only.
This repository was archived by the owner on Jun 16, 2025. It is now read-only.
How to apply the GeDi model on the models with different tokenizer? #20
Description
Hi, I want to apply the GeDi model on a model with BertTokenizer. However, a dimension problem will occur when combining the probability distributions of two models. In addition, the tokens tokenized by two Tokenizers are different, so I can't match the two vocabularies directly. Is there any approach to applying the GeDi model? Thanks!