Skip to content

Add per-model stream: false option in my-models.yml for non-streaming custom endpoints #353

@Pratik-Nawale

Description

@Pratik-Nawale

Feature Request

Summary

Add a stream configuration option per model in my-models.yml so users can set stream: false for custom endpoints that do not support Server-Sent Events (SSE) streaming.

Current Behavior

AI Toolkit always sends "stream": true when calling custom model endpoints. If the endpoint does not support streaming, the request fails with a 500 error, making the model completely unusable in AI Toolkit even if the endpoint works correctly for non-streaming requests.

Expected Behavior

Allow users to opt out of streaming per model in my-models.yml:

- name: Claude Opus 4.6 v1
  chat_completion_url: https://my-gateway.example.com/v1/chat/completions
  stream: false   # <-- new option
  api_model_name: my-model # e.g. claude-opus-4.6

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs attentionThe issue needs contributor's attention

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions