Ollama alternatives

[zyfer9009-lgtm]

Hi there.😊 Really great work you are doing👌. Just a question from Denmark. I have a setup where I use AMD NPU via FastflowLM, not GPU. Is there a way to connect to FastFlowLM, instead of Ollama. I can connect via Open-AI Api (Dummy). I'm currently using Open webui, and it works fine, but not good at voice commands. Hope you can help, because I really want to try your cool CAAL..

[cmac86]

Hey @zyfer9009-lgtm, thanks for reaching out from Denmark! Appreciate the kind words.

Good news - this is definitely possible since FastFlowLM exposes an OpenAI-compatible API. CAAL currently supports Ollama and Groq, but the architecture is set up to add new providers.

I've created a feature request for this: #29

The Groq provider we added recently uses the same response format as OpenAI, so we can reuse most of that code - just need to point it at a configurable endpoint instead of Groq's cloud.

Question: Would you be willing to test a feature branch when we start implementing this? Having someone with actual FastFlowLM hardware to validate against would be really helpful. No ETA yet, but it's on the radar.

[zyfer9009-lgtm]

Hey @cmac86 😊. That sounds really good. And yes, I will of course test it. I used to run Ollama on a GPU. But I've become really happy with the NPU setup. I can run large models, without the fans going crazy.😂

[cmac86]

Hey @zyfer9009-lgtm — good news! We just shipped v1.6.0 which adds an OpenAI-compatible provider. This should work with FastFlowLM out of the box.

To set it up:

1. Pull the latest: git pull && docker compose up -d --build or git pull && docker compose --profile https up -d --build
2. In the setup wizard (or Settings > AI Provider), select OpenAI Compatible
3. Enter your FastFlowLM endpoint URL (e.g. http://localhost:8000/v1) and model name
4. API key is optional — leave it blank if FastFlowLM doesn't require one

You can also set it in .env if you prefer:

LLM_PROVIDER=openai_compatible
OPENAI_BASE_URL=http://your-fastflowlm-host:port/v1
OPENAI_MODEL=your-model-name

Let me know how it goes with your AMD NPU setup — would love to hear if it works well!