Skip to content

Minimal flux with fp8 freeze #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wkpark wants to merge 70 commits into
base: dev
Choose a base branch
from
Open

Minimal flux with fp8 freeze #6

wkpark wants to merge 70 commits into from

Conversation

@wkpark
Copy link
Owner

@wkpark wkpark commented Sep 13, 2024

Description

  • a simple description of what you're trying to accomplish
  • a summary of changes in code
  • which issues it fixes, if any

Screenshots/videos:

Checklist:

@wkpark wkpark force-pushed the minimal-flux-with-fp8-freeze branch from 015e20a to de9497a Compare September 17, 2024 03:04
@wkpark wkpark force-pushed the minimal-flux-with-fp8-freeze branch 4 times, most recently from c872b26 to e8a6c28 Compare September 28, 2024 14:53
@changeworld
Fix typo
d8ad364 
Github -> GitHub
@wkpark wkpark force-pushed the minimal-flux-with-fp8-freeze branch 3 times, most recently from a765b8f to 8f45d13 Compare October 22, 2024 10:32
w-e-w and others added 21 commits October 24, 2024 09:05
@w-e-w
Fix DAT models download (AUTOMATIC1111#16302)
984b952
@bluelovers
chore(js): avoid lots of Wake Lock is not supported.
ee0ad5c
@w-e-w
Merge pull request AUTOMATIC1111#16523 from changeworld/fix/typo
38c8043 
Fix typo: Github -> GitHub
@w-e-w
Merge pull request AUTOMATIC1111#16588 from bluelovers/patch-3
d88a3c1 
chore(js): avoid lots of `Wake Lock is not supported.`
@w-e-w
image embedding data cache (AUTOMATIC1111#16556)
deb3803
@w-e-w
Allow newline in Extra Network activation text (AUTOMATIC1111#16428)
820fe8d
@w-e-w
extra_only / main_ui_only ScriptPostprocessing (AUTOMATIC1111#16374)
0bf36cf
@w-e-w
Fix postprocessing_enable_in_main_ui ScriptPostprocessing elem_id (AU…
4ec10bc 
…TOMATIC1111#16373)
@w-e-w
Disable Hires checkpoint if same as First pass checkpoint (AUTOMATIC1…
14c6d6c 
…111#16269)
@viking1304
pyenv-win compatibility - another approach (AUTOMATIC1111#16287)
f31faf6
@w-e-w
addEventListener {passive: false} (AUTOMATIC1111#16575)
5948143
@w-e-w
InputAccordion duplicate elem_id handling (AUTOMATIC1111#16381)
5206b93
@catboxanon
Fix weighting config for SDXL v-pred
ac28cad 
Fixes a small oversight I made.
@w-e-w
Fix Default system None filter logic (AUTOMATIC1111#16309)
533c7b7
@w-e-w
XYZ option to disable grid (AUTOMATIC1111#16416)
28323cf
@catboxanon
sd_xl_v.yaml: use_checkpoint = False
e6f36d9 
In accordance with AUTOMATIC1111#15803
@w-e-w
Merge pull request AUTOMATIC1111#16606 from AUTOMATIC1111/fix/vweighting
aa52408 
Fix config for SDXL v-pred
@w-e-w
Warn if WebUI is installed under a dot directory (AUTOMATIC1111#16584)
91de919
@w-e-w
use shared.hf_endpoint (AUTOMATIC1111#16611)
1b16c62
@wkpark
import Flux from https://github.com/black-forest-labs/flux/
853551b 
License: Apache 2.0
original author: Tim Dockhorn @timudk
@wkpark
add flux model wrapper
d38732e
wkpark added 29 commits November 1, 2024 12:34
@wkpark
fix lora without backup
3cdc26a
@wkpark
revert to use without_autocast()
3b18b6f
@wkpark
fix position_ids
2ffdf01
@wkpark
fix for float8_e5m2 freeze model
1e73a28
@wkpark
fix load_vae() to check size mismatch
1318f61
@wkpark
add fix_unet_prefix() to support unet only checkpoints
eee7294
@wkpark
use assign=True for some cases
6675d1f
@wkpark
check lora_unet prefix to support Black Forest Labs's lora
1f77922
@wkpark
call lowvram.send_everything_to_cpu() for interrupted case
380e9a8
@wkpark
call torch_gc() to fix VRAM usage spike when call decode_first_stage()
71b430f
@wkpark
use text_encoders.t5xxl.transformer.shared.weight tokens weights
f569f6e 
 * some T5XXL do not have encoder.embed_tokens.weight. use shared.weight embed_tokens instead.
 * use float8 text encoder t5xxl_fp8_e4m3fn.safetensors
@wkpark
fix flux to use float8 t5xxl
28eca46
@wkpark
fixed typo in the flux lora map
4bea93b
@wkpark
fixed ai-toolkit flux lora support
30d0f95 
 * fixed some mistake
 * some ai-toolkit's lora do not have proj_mlp
@wkpark
make Sd3T5 shared.opts.sd3_enable_t5 independent
11c9bc7
@wkpark
support Flux schnell and cleanup
8c9c139
@wkpark
fix some nn.Embedding to set dtype=float32 for some float8 freeze model
2e65335
@wkpark
use isinstance()
03516f4
@wkpark
use shared.opts.lora_without_backup_weight option in the devices.auto…
ba499f9 
…cast()

 * add nn.Embedding in the devices.autocast()
 * do not cast forward args for some cases
 * add copy option in the devices.autocast()
@wkpark
do not use copy option for nn.Embedding
5f3314e
@wkpark
do not use assing=True for nn.LayerNorm
4ad5f22
@wkpark
flux: clean up some dead code
98cb284
@wkpark
task manager added
1d3dae1 
based on https://github.com/lllyasviel/stable-diffusion-webui-forge/blob/main/modules_forge/main_thread.py

 * classified
 * this way, gc.collect() will work as intended.
@wkpark
reduce backup_weight size for float8 freeze model
0ab4d79
@wkpark
extract backup/restore io-bound operations out of forward hooks to sp…
04f9084 
…eed up
@wkpark
call gc.collect() when wanted_names == ()
2a1988f
@wkpark
backup only for needed weights required by lora
412401b
@wkpark
fix for lazy backup
b783a96
@wkpark
restore org_dtype != compute dtype case
310d0e6
@wkpark wkpark force-pushed the minimal-flux-with-fp8-freeze branch from 8f45d13 to 310d0e6 Compare November 1, 2024 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@wkpark @changeworld @w-e-w @bluelovers @viking1304 @catboxanon