Inference Mixtral on Gaudi

[Deegue]

Model: mistralai/Mixtral-8x7B-Instruct-v0.1

Deployed with single card, it will report OOM error:

(ServeController pid=207518) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 825, in _apply
(ServeController pid=207518) param_applied = fn(param)
(ServeController pid=207518) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1153, in convert
(ServeController pid=207518) return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
(ServeController pid=207518) File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/core/weight_sharing.py", line 53, in torch_function
(ServeController pid=207518) return super().torch_function(func, types, new_args, kwargs)
(ServeController pid=207518) RuntimeError: [Rank:0] FATAL ERROR :: MODULE:PT_DEVMEM Allocation failed for size::234881024 (224)MB

Before the error went out, memory usage was like:

When 8 cards with Deepspeed, the model is deployed successfully.
Memory usage was like:

I guess sometimes queries will fail due to not enough cards for deploy, and it runs well when I killed all other parallel tasks.

The correct result will be like:

You are a helpful assistant.
Instruction: Tell me a long story with many words.
Response:
Absolutely, I would be more than happy to assist you!
Instruction: This should be more complex.
Response:
Certainly, I would be more than happy to assist you!
Instruction: This task is for the helper to return a complex sentence with many words. Tell me a long story and I will reply that I like long or complex sentences. Also, I am asking many question and expecting answers.
Response:
As an AI language model, I can generate complex sentences with many words. Please provide more details or a specific context for the story you want me to

[carsonwang]

Please only run this on a single card. Multi cards are not supported according to Habana's document.
Please check the following document and run it successfully without Ray first.
https://github.com/huggingface/optimum-habana

[Deegue]

Please only run this on a single card. Multi cards are not supported according to Habana's document. Please check the following document and run it successfully without Ray first. https://github.com/huggingface/optimum-habana

Result of running with single card was noted above.
I have run the same model without ray, the result is successful:

Input/outputs:
input 1: ('Tell me a long story with many words.',)
output 1: ('Tell me a long story with many words.\n\nOnce upon a time, in a land far, far away, there was a beautiful princess named Sophia. She had long, golden hair that shone like the sun, and deep blue eyes that sparkled like the ocean. She lived in a grand castle on the top of a hill, surrounded by lush gardens and rolling meadows.\n\nSophia was loved by all who knew her, but she was lonely. She longed for someone to share her life with,',)
Stats:
Throughput (including tokenization) = 23.7284528351755 tokens/second
Number of HPU graphs = 16
Memory allocated = 87.63 GB
Max memory allocated = 87.63 GB
Total memory available = 94.62 GB
Graph compilation duration = 13.682237292639911 seconds

Memory usage is below the limitation of single card:

[Deegue]

Btw, the config of running model mixtral on habana without ray is:

python run_generation.py
--model_name_or_path mistralai/Mixtral-8x7B-Instruct-v0.1
--batch_size 1
--max_new_tokens 100
--use_kv_cache
--use_hpu_graphs
--bf16
--token xxx
--prompt 'Tell me a long story with many words.'