Realtime performance

[gokturkDev]

I am trying to run MultiTalk in realtime (ideally 15+ fps) but having a hard time doing so. Is it possible? Is there a theoretical limit?

[yzhang2016]

It is possible. Please refer to this work by Character.ai, i.e., talking machine. For MultiTalk, we need a method to sample with only one or two steps. It would be better if the cfg is also distilled into the model. Then, the sparse attention and quantization methods can also be adopted. We are trying, but it might be not that easy.

[gokturkDev]

I see, is there an official roadmap for realtime performance planned for now?

[yzhang2016]

(1) We almost finish the validation of quantization.
(2) The sparse attention will be integrated after the quantization.
(3)We next will focus on the one or two-step distillation. It is something like a research problem that there is no satisfying solution yet. So we are not sure about the DDL for it.

[johndpope]

in wan2.1 paper - they do say they achieved distillation. so presumably they're just sitting on this.
i guess facebook comes to the party and attempts to scorch earth the world with video models - then maybe we see some more solutions.
have to go self forcing. https://github.com/guandeh17/Self-Forcing

[gokturkDev]

@johndpope The repo you have attached is very interesting. Sorry for the dumb question but is this something we can readily use with the released pretrained models, or does it require additional training with the original dataset?

@yzhang2016 Understood, thanks for your efforts, I will keep a close eye on this.

[johndpope]

We need to solve training code. I draft something here. I’m waiting on 5090 any day…
#27

If we can continue training from existing checkpoints or retrain from scratch
Then there’s question of gpu compute
. They have 1000 gpus at bytdedance

There’s lots of varriablrs - diffusion has been slow to converge.

https://github.com/johndpope/OmniHuman-1-hack

[gokturkDev]

I see, thanks for the explanation. I will close this issue now.

[StellarCheng]

(1) We almost finish the validation of quantization. (2) The sparse attention will be integrated after the quantization. (3)We next will focus on the one or two-step distillation. It is something like a research problem that there is no satisfying solution yet. So we are not sure about the DDL for it.

Excuse me, is there any progress on the acceleration?

Additionally, I have a few questions. Is it possible to distill the WAN model directly down to two steps by tuning a LoRA, without necessarily having to train it with audio data?"

[yzhang2016]

(1) We almost finish the validation of quantization. (2) The sparse attention will be integrated after the quantization. (3)We next will focus on the one or two-step distillation. It is something like a research problem that there is no satisfying solution yet. So we are not sure about the DDL for it.

Excuse me, is there any progress on the acceleration?

Additionally, I have a few questions. Is it possible to distill the WAN model directly down to two steps by tuning a LoRA, without necessarily having to train it with audio data?"

We trained a model for the 4-NFE infernece. It works well with face and lipsyn, but it has trouble on hands and arms. We are fixing it.