Skip to content

Comments

Experimental: ComfyUI Integration (Generate prompt)#5932

Open
jonoomph wants to merge 30 commits intodevelopfrom
comfy-ui
Open

Experimental: ComfyUI Integration (Generate prompt)#5932
jonoomph wants to merge 30 commits intodevelopfrom
comfy-ui

Conversation

@jonoomph
Copy link
Member

@jonoomph jonoomph commented Feb 14, 2026

This PR is experimental, and integrates ComfyUI (local AI node-based platform) for various AI integrations.

Current capabilities:

  • Text to image
  • Image + text to image
  • Text to video (technically text to image to video)
  • Text to audio
  • Upscale video (4x)
  • Video + text to video (style transfer)
  • Transcribe video/audio to SRT (subtitles)
  • Frame interpolation (2x FPS)
  • and more...
image

Integrated progress, queuing, and job canceling:
image

- Generate with no input file
- Generate with 1 input image
- Progress displayed in "Project Files"
- Cancel Job menu
- Full circle generations and editing proven
… work with remote servers (i.e. LoadFile -> Upload files). Updating file headers with correct header and copyright. Updating some base/default models for image generation and upscaling. Much more stable now at reconnecting, losing connection, and not giving up.
  - add txt2audio-stable-open, txt2video-svd, and video2video-basic
  - support SaveAudio plus image/video/audio output download/import
  - add model default selection for installed Comfy checkpoints/encoders
  - fix Stable Audio EmptyLatentAudio.batch_size validation
  - align txt→img→video flow with Comfy example and tune low-VRAM test defaults
@jonoomph
Copy link
Member Author

…, auto Caption effect when a file has caption data already.
…ion, with full workflow wiring.

 - Use remote Comfy node/model discovery (RIFE + flexible SVD checkpoint selection).
 - Show immediate generation progress in Project Files (including queued state) without mutating file names.
…ndling

  - Add video-segment-scenes-transnet Comfy workflow (LoadVideo + TransNetV2 + PreviewAny output)
  - Import all generated segment paths into Project Files, apply split-style scene naming, and add scene tags
  - Improve Comfy output extraction to handle list/string node outputs from custom nodes
…ve integration with many of the AI tasks. Also, new icons.
@github-actions
Copy link

Merge conflicts have been detected on this PR, please resolve.

@github-actions github-actions bot added the conflicts A PR with unresolved merge conflicts label Feb 16, 2026
# Conflicts:
#	src/windows/models/files_model.py
#	src/windows/views/files_listview.py
…t_qt/comfyui/ folder, and write a debug.json file (what we send to ComfyUI for debugging purposes). Also split out all our built-in workflows as simple JSON files, with a few overrides to help menus/categories. Also now use the category icon when generating new items (audio icon, image icon, etc...). Menu is now dynamic, and will load user-defined JSON workflows as well.
# Conflicts:
#	src/windows/views/files_treeview.py
      - src/comfyui/image-blur-anything-sam2.json
      - src/comfyui/video-blur-anything-sam2.json
- Added Comfy availability gating + URL validation:
      - Hide AI menus when Comfy is unreachable
      - Preferences now has Check for comfy-ui-url
- Added Comfy error truncation/sanitizing so failures don’t dump huge payloads in UI dialogs.
- Enabled proper maximize/min-max behavior for Region and Split dialogs.
- Moved Queued badge to the bottom of file thumbnails (list + tree views).
…connect with backoff (60s -> 300s), and fallback to /progress when WS is quiet.

- Fix template input rewriting for multi-loader workflows by matching both file/video (and audio/file) keys.
- Tune Blur Anything defaults: switch to sam2.1_hiera_tiny, keep model loaded, reduce batch size to 32.
…eaner WS event selection.

- Add queue progress metadata (progress_detail, sub_progress) and improve fallback behavior when /progress is unavailable.
- Switch blur-anything SAM2 workflow to OpenShot windowed/chunked nodes with masked blur and 96-frame batching.
…N-based graphs (UNET/CLIP/VAE loaders, ModelSamplingSD3, WAN model paths), and set video defaults to 832x480, 16 fps,

    length: 64 (~4s).
- Switched these template IDs (and img2video-svd) off legacy Python workflow generation so OpenShot now uses JSON templates directly at runtime.
- Added WAN node types to template classification, renamed the image-to-video menu label to Image to Video (WAN 2.2 TI2V), and added/used a new img2video-svd WAN 2.2 TI2V template in the
    Enhance flow.
…ckpointLoaderSimple, CLIPTextEncode, KSampler, VAEDecode,

    SaveImage).
- The image model/settings are now quality-oriented SDXL Base: ckpt_name = sd_xl_base_1.0.safetensors, steps = 28, and cfg = 6.5.
…es split-style file entries from it, reusing the original source path with start/end trims (no duplicate segment MP4

    imports).
- Scene split file names now match Split File dialog format exactly (base (start to end)), with only start and end attributes applied.
…ts), frame markers, and clear-all, plus new toolbar icons.

- Generate + SAM2 workflows now pass full tracking JSON (seed frame + per-frame points/rects/auto) instead of seed-only inputs.
- Tracker preprocess flow was fixed (dialog parenting + guards/imports) to avoid region selection crashes/regressions.
…inal video, Mask only imports mask; blocked fallback paths that were re-adding extra outputs.

- Fixed filename prefix handling so openshot_mask/openshot_gen suffixes are preserved (with payload appended), enabling reliable output filtering.
- Expanded Generate/Highlight UX: simplified Prompt + Tracking tabs, added Qt color pickers, plus new Mask Brightness and Background Brightness controls wired end-to-end.
…y/pause, and the play button resets correctly when end of video is reached.
…r windowed SAM2 state.

- Improved chunk-boundary stability using richer carries (point+bbox) plus 4-frame boundary replay.
- Added detailed DINO/SAM2 debug logging (currently always on).
- Renamed Audio... to Sound....
- Fixed music runs: random seeds, no stale output reuse, and prompt-to-tags/lyrics handling.
  - Track -> Blur
  - Track -> Highlight
  - Track -> Mask
…ze (so we minimize chance for OOM on higher resolution videos)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflicts A PR with unresolved merge conflicts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant