Testing: Groq Integration (No GPU Required) - PR #21

[cmac86]

Groq integration is ready for testing. This lets you run CAAL without a GPU using Groq's free tier for LLM and STT.

Looking for a few people to test before merging. Ideal if you:

• Don't have a dedicated GPU
• Want to try CAAL on a NAS, mini PC, or old laptop
• Can spare 15 minutes to report back

What's New in This PR

• GPU-Free Mode: Run CAAL with zero VRAM using Groq (LLM + STT) and Piper (TTS)
• First-Start Wizard: Browser-based setup - only one setting needed in the .env file
• Groq Provider: Free tier, fast inference, multiple LLM models, Whisper Large v3 Turbo for STT
• Piper TTS: CPU-friendly with automatic model downloads
• Redesigned Settings: Slide-out panel replaces the cramped modal

How to Test

1. Clone the PR branch:

git clone https://github.com/CoreWorxLab/CAAL.git
cd CAAL
git checkout feat/groq-llm-provider

2. Copy .env.example to .env and set your CAAL_HOST_IP and HTTPS_DOMAIN

3. Start without GPU containers:

docker compose -f docker-compose.cpu.yaml up -d

4. Get a free Groq API key from groq.com

5. Open the web UI (http://localhost:3000) - the setup wizard will launch automatically

Setup Wizard Steps

1. LLM Provider - Select Groq, enter your API key
2. TTS Provider - Select Piper (CPU-friendly, no GPU needed)
3. Integrations - Optionally connect Home Assistant or n8n

Hit save and start talking.

Feedback Needed

• Did the setup wizard make sense?
• Any errors in console or UI?
• Response latency - how does it feel?
• What hardware are you running on?

PR: #21

[NotAlreadyG]

Man you are a heroo!!

[NotAlreadyG]

But I am getting a path like something error

C:\Users\My-User\Desktop\CAAL>git checkout feat/groq-provider
error: pathspec 'feat/groq-provider' did not match any file(s) known to git

[cmac86]

@NotAlreadyG sorry a typo. Should be feat/groq-llm-provider.

[NotAlreadyG]

If you Don't Mind Can you send me a detailed step by step complete setup from starting(0) for Caal on Windows as I am facing lots and lots of errors!!!

[cmac86]

@NotAlreadyG I'll have to do testing on windows and get back to you. In the meantime, how far did you get and what errors did you see?

[NotAlreadyG]

I am mainly seeing docker errors right now!!

[cmac86]

Hi all, I made some changes to the PR. You now run the following for CPU only:

docker compose -f docker-compose.cpu.yaml up -d

and for https:

docker compose -f docker-compose.cpu.yaml --profile https up -d

This save about 17GB of images that you don't need if you're running CPU and not GPU.

cmac

[NotAlreadyG]

C:\Users\User-Name\CAAL>docker compose -f docker-compose.cpu.yaml up -d
unable to get image 'ghcr.io/speaches-ai/speaches:0.9.0-rc.3-cpu': error during connect: Get "http://%2F%2F.%2Fpipe%2FdockerDesktopLinuxEngine/v1.51/images/ghcr.io/speaches-ai/speaches:0.9.0-rc.3-cpu/json": open //./pipe/dockerDesktopLinuxEngine: The system cannot find the file specified.

Also Can you tell me in which directory the kokoro tts and all that installs by runnning the docker compose up -d command as it installed but not working so i wanna delete it to clean my storage too and if it is needed to be installed so ill install it again

[cmac86]

@NotAlreadyG,

This looks like it is a docker communication issue.

• Docker Desktop on Windows creates a named pipe (dockerDesktopLinuxEngine) for communication
• The Docker CLI talks to this pipe to reach the Linux VM running containers
• If the pipe doesn't exist, Docker Desktop either isn't running or failed to start properly

Try this:

1. Make sure Docker Desktop is running - Look for the whale icon in the system tray (bottom-right). If not there, launch Docker Desktop from the Start menu.

2. Wait for it to fully start - Docker Desktop takes a while to initialize. The whale icon will stop animating when ready.

3. If it's running but still failing, restart Docker Desktop:

   • Right-click the whale icon → Quit Docker Desktop
   • Wait 10 seconds
   • Launch Docker Desktop again
   • Wait for it to fully start

4. If still failing, check WSL2 is working:
   wsl --status

5. If WSL has issues, run:
   wsl --shutdown

6. Then restart Docker Desktop.

To remove the images you don't need for running CPU you can run this:

docker rmi ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.4-master ghcr.io/speaches-ai/speaches:latest-cuda-12.6.3

or find the images in Docker Desktop and delete them.

cmac

[HubertReX]

• Did the setup wizard make sense?

I like this new wizard a lot! All was clear and easy to setup. Most things worked from first try 💪

• Any errors in console or UI?

I used this command:
docker compose -f docker-compose.cpu.yaml --profile https up -d

No errors in the UI, but speech-to-text is not working. I found only this error in caal-speaches container logs:

INFO:     127.0.0.1:56344 - "GET /health HTTP/1.1" 200 OK
2026-01-11 21:19:30,534:DEBUG:speaches.executors.shared.base_model_manager:_load:72:Loading model speaches-ai/piper-en_US-lessac-medium
2026-01-11 21:19:30,535:DEBUG:speaches.executors.shared.base_model_manager:get_ort_providers_with_options:23:Available ONNX Runtime providers: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
2026-01-11 21:19:30,536:DEBUG:speaches.executors.shared.base_model_manager:get_ort_providers_with_options:33:Using ONNX Runtime providers: [('CUDAExecutionProvider', {}), ('CPUExecutionProvider', {})]
2026-01-11 21:19:30.727907970 [E:onnxruntime:Default, provider_bridge_ort.cc:2195 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1778 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory
2026-01-11 21:19:30.727987907 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:1055 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Require cuDNN 9.* and CUDA 12.*. Please install all dependencies as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.
2026-01-11 21:19:32,290:INFO:speaches.executors.shared.base_model_manager:_load:75:Model speaches-ai/piper-en_US-lessac-medium loaded in 1.76s
2026-01-11 21:19:32,290:DEBUG:speaches.executors.shared.base_model_manager:_increment_ref:83:Incremented ref count for speaches-ai/piper-en_US-lessac-medium, self.ref_count=1

• Response latency - how does it feel?

All works much smoother than on my mac-mini m4 24GB UM. Both pure llm responses and tool call conversations are visibly faster.

• What hardware are you running on?

VPS from RackNerd: KVM VPS 6 GB RAM, 5 vCore, Ubuntu 24.04 64 Bit

[HubertReX]

1. STT

A quick update - after switching off wake word, the stt is working like a charm 👌 Quick and reliable - also with interruptions 🎉

2. Multilangual

Since Piper supports the Polish language, I wanted to test if it would work. Caal behaved strangely:

• My text spoken in Polish, was transcribed correctly, but in the chat it showed as translated into English.
• Most responses from the model were in English, so the Polish speech model tried to read them with obviously bad results
• When I asked in Polish in chat (to avoid "automatic" translation) to switch the conversation to my language, I got one voice response in Polish - clear but with a slight accent. All the following responses were again in English

I understand that it would require more changes in the code, new system prompt etc. It was just a test for fun ;-)

[cmac86]

@HubertReX,

Thank you for the testing! I really appreciate this. I'm very curious about the wake word/STT issue. I think Speaches will try GPU then fallback to CPU if CUDA is not available. Can you confirm what Speaches image was pulled and being used? Did you try enabling wake word again? Will STT only work when its disabled? When wake word was enabled, did wake word work and just not STT? I did not have these issues when installing CPU only on my TrueNAS machine.

The multilingual behaviour: I had the same experience when trying to speak French to CAAL. Sometimes the transcription was English, sometimes French, sometimes the response was English, sometimes in French. Did you try editing the prompt to be in Polish? Maybe that would trigger the llm to be in that language. I have not tried this. I'm hoping @mmaudet's PR can help with this multilingual feature.

Thanks again.
cmac

[NotAlreadyG]

Yo man!
I managed to run Caal but there's are/a problem which is that it's listening to me but not responding.

1. Also the Wake Word is too hard to pronounce usually for me so I wanna change it, so how to?

2. Also Can you tell me How to setup .env cz I managed to setup everything but I didn't entered the Livekit API Key and Secret as I don't know how to add it like just remove the "devkey" and "secret" from .env and replace it with my key or any other way.

3. Also i WANNA know how to run Caal regularly like what I have to do if I shutdown my computer and then reopen it later to run CAAL?

And I think you should make a video on how to setup CAAL properly for every type of users like for linux(or mac), windows and also explaining each or important files and codes!!

[cmac86]

Good progress.

No response - is it responding with text or not at all (no text or voice) ?

Wake Word - you can switch to ‘hey Jarvis’ in the settings. Currently only two options. I plan on allowing custom wake word. Or you can disable for now and use the mute button.

.env - you only need to set CAAL_HOST_IP and HTTPS_DOMAIN. Don’t change Livekit API key. Keep as dev key and secret or whatever you want. For local livekit you don’t need a key. Only for cloud.

Restart - After a reboot, just run docker compose -f docker-compose.cpu.yaml --profile https up -d again from the CAAL folder. When I reboot on Linux the docker containers restart automatically, I’m not sure what the behaviour is on Windows.

Video Walkthrough - I have a video coming for installing from scratch on Linux for a CPU only install. Not windows yet. Good idea to do all platforms.

Let me know how it goes.
cmac

[NotAlreadyG]

So, I dont know how but I managed to run it properly. But it was difficult and maybe i cant do it again as it happened randomly to me
So I think you should create a fast paced youtube tutorial video on setting up this whole github repo for all types of users including cloud/local and also explain files and codes too. If you take time like could be 1 month and make a properly edited fast paced video then may be it become viral!!

[cmac86]

Well done! 💪 I'm glad you got it going. I have a video releasing Friday that is a walkthrough for Linux, specifically on a TrueNAS machine. I will look to film walkthroughs for Windows and Mac as well in the future.