Incoherent repetitive data generated by teacher after clipping. #16
Description
Hi,
I'm attempting to reproduce the training steps for Llama-2, but generated incoherent data from the teacher, for example, the model begins repeating itself after a while.
Following the readme, the steps I took:
- Download https://huggingface.co/meta-llama/Llama-2-7b-hf locally and clip/quantise the weights with autoclip.py. I didn't change any of the flags/settings from the readme example when running autoclip.
- Run the generate.sh script on alpaca to generate 8 samples. Both torchrun and vllm have the same issues in the generated data.
Example of an entry in the json file generated by the teacher model.
[[" Critical reception towards Rachel has remained consistently positive throughout Friends ' decade @-@ long run , with The A. V. ", " Club @-@ writing that @-@ her \" character has remained the most complex and interesting among Friends ' main cast \" . The Guardian @-@ has described her as @-@ \" the most interesting character \" and wrote that \" Rachel Green is the closest to a truly multidimensional character on Friends .\" The New York Times @-@ has described her as @-@ \" the most complicated \" and \" the most intriguing \" character on the show . The Daily Telegraph @-@ wrote that Rachel is the \" most complex and interesting \" character on Friends . USA Today @-@ has written that @-@ \" Rachel is the most dimensional of the six friends .\" The Guardian @-@ described Rachel as @-@ \" the show's most sympathetic character \" and The Telegraph @-@ wrote that @-@ \" Rachel ( Aniston ) is the show's most sympathetic character \" . The New York Times @-@ has described Rachel as @-@ \" the most interesting \" and \" the most complex \" character on the show . IGN @-@ has written that @-@ \" It's surprising how much depth and complexity Rachel brought to the show .\"\n Rachel has been described as a \" feminist icon \" and a \" feminist heroine \" . The Guardian @-@ has written that @-@ \" Rachel has been a feminist heroine for a generation \" and @-@ \" Aniston's character remains a feminist icon .\" Rachel has been described as a feminist icon because of her success in the male-dominated world of fashion. Rachel's boss, Mr. Sacks, tells her that he is the reason that his employees are successful @-@ and that he \" knows everything \" and @-@ \" He's the reason we're here .\" Rachel's success @-@ is due to the fact that Mr. Sacks \" knows everything \" @-@ and that he is the reason that his employees are successful @-@ and Rachel is successful because of him. Rachel succeeds @-@ in the world of fashion because of her work ethic @-@ and her hard work @-@ and @-@ her \" drive and ambition \" @-@ and @-@ her \" determination \" and @-@ her \" dedication \" @-@ and @-@ her \" perseverance \" @-@ and @-@ her \" resilience \" @-@ and @-@ her \" tenacity \" @-@ and @-@ her \" ingenuity \" @-@ and @-@ her \" resourcefulness \" @-@ and @-@ her \" intelligence \" @-@ and @-@ her \" creativity \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" charisma \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@ her \" talent \" @-@ and @-@"]]
Is this an expected behaviour of the llama2-7B model and does this pose a significant issue for the KD-QAT training step? How was this addressed in the paper if encountered?