include tokeniser config in download packages #2
Description
When loading a model in transformers, e.g. with
nlp = pipeline('fill-mask', model = 'TurkuNLP/wikibert-base-ga-cased')
a tokeniser that lowercases the input and removes accents is used. Users who do not notice this problem will get suboptimal results with your models as it still works to run cased models with a lowercased tokeniser. I am not sure whether the default tokeniser uses the supplied vocab or a default vocab. There were no error messages. We did not continue evaluation of model output when the problem was noticed but implemented the following workaround.
Workaround: Download to a local folder and manually add a suitable tokenizer_config.json
Suggestion / feature request: Add tokeniser config to the download folders on huggingface and http://dl.turkunlp.org