encryptionEnabled not working

[xmlking]

encryption not working, still see data unencrypted in database. Am I doing something wrong?

import { Queen } from "queen-mq";

// Connect to Queen server
const queen = new Queen({
  url: process.env.QUEEN_BASE_URL,
  bearerToken: process.env.QUEEN_TOKEN,
});

// Queue with encryption
await queen
  .queue("sensitive-data")
  .config({
    encryptionEnabled: true,
    leaseTime: 300,
    retryLimit: 3,
  })
  .create();
console.log("✅ Queue created with encryptionEnabled=true");

// Push a message
await queen.queue("sensitive-data").push([
  {
    data: { message: "Hello, Queen MQ!" },
  },
]);

console.log("✅ Message pushed!");

// Pop and process the message
const messages = await queen.queue("sensitive-data").pop();

console.log("✅ Message received:", messages[0].data);

// Acknowledge the message
await queen.ack(messages[0], true);

console.log("✅ Message acknowledged!");

process.exit(0);

[alice-viola]

Are you running the server with the encryption key?
Like:
QUEEN_ENCRYPTION_KEY=XXXX NUM_WORKERS=2 bin/queen-server

You can generate it with: openssl rand -hex 32

I will document this better

[xmlking]

conformed working. I missed QUEEN_ENCRYPTION_KEY
suggestions:

1. Dashboard: Queue Detail page showcase encryption is enabled or not
2. Client error if QUEEN_ENCRYPTION_KEY is not set but user try to create queue with encryptionEnabled: true

[xmlking]

scaling question:
wonder how you are scaling database at Smartness? do you use database sharding? use PgBouncer?
use cases:
I’m developing a multi-tenant SaaS application and want to partition the database per tenant.
if I have many asymmetric queues, I want to isolate them to dedicated shards.

[alice-viola]

At Smartness we have a dedicated Postgres for Queen in GCP, with an active replica for failover, behind PgBouncer (but it's not necessary, Queen has its own pooling system). I've deliberately chosen to have one database per Queen deployment (we use 3 replicas of Queen for HA), without Postgres partition sharding. This is mainly to keep a simple design, and it's what allows Queen to guarantee ordered delivery and exactly-once processing within partitions.

All of our queues contain multi-tenant data, and some of them are asymmetric. We currently process on Queen around hundreds of thousands to millions of messages per day, on a small Postgres. We run 14 queues with a total of 50k partitions, some of them consumed by multiple consumer groups. We usually have a partition per single chat, sometimes a partition per single accommodation, and sometimes a partition per tenant, depending on the volume and isolation we need.

The benchmark I've made shows that with a decent Postgres size and properly tuned (32 vCPU and 64 GB RAM, with the right vacuum and memory settings) the practical limit is 10,000 push req/s, with batched pops, sustained for days (https://smartpricing.github.io/queen/server/benchmark-results.html . Of course in that case you also have to set completed retention time accordingly, otherwise the disk size goes to the moon. For multi-tenant data you can:

• use a queue per tenant
• use partitions per tenant inside a single queue
• deploy a Queen per tenant, with its own database

[xmlking]

Questions on Queen HA, clustering, and client deployment

Good to know Queen has a built‑in connection pool.

1. For HA PostgreSQL setups, do we need to explicitly provide primary + replica endpoints via PG_HOST, or can Queen auto‑discover primaries/replicas?
2. What is the role of UDP sync? Documentation seems limited—how does it work in a multi‑Queen cluster and what problem does it solve?
3. FILE_BUFFER_DIR:
   • Is persistent/mounted storage recommended?
   • Any guidance on storage size or type (local disk vs network, SSD, etc.)?
4. Client application deployment (queen‑js / Node):
   • Are clients typically browser‑based or server‑side (microservices)?
   • Do you containerize long‑running client scripts?
   • Any plans for serverless / FaaS support?
   • my tipical workflow:
     Ingest svc → Queen Q1 → Curate svc → Queen Q2 → Enrich svc (external APIs) → Queen Q3
     Looking for recommendations on push‑code‑and‑deploy platforms that fit this model.

[alice-viola]

1. For Postgres HA, we rely on cloud providers' HA, which keeps two Postgres instances in sync and switches to the replica if the primary goes down. I didn't want to build this into Queen, because it goes against Queen's philosophy of staying simple.
2. UDP sync is a way (that we use in production) to add a minimal layer of coordination between multiple instances of the same Queen deployment. If a push request to a queue arrives on replica 1, it notifies the other replicas of the event, so consumers attached to other replicas can start consuming. With this we can use a higher polling interval to the database, saving some queries. It's UDP, so it is best effort, not guaranteed. The fact is that I didn't want to create a distributed system that is hard to manage or that requires coordination.
3. We run all of our workloads in Kubernetes, and we use persistent storage for volumes. Writing to disk is faster than writing to Postgres (but we don't batch writes to disk), so it's usually not a big deal to have a fast volume. File buffer is mainly used for maintenance operations. You can set PUSH maintenance (so writes are buffered), wait for consumers to finish consuming the messages, then also enable POP maintenance, do migrations or operations on Postgres (we're migrating our Postgres to another instance next week), and then restart POP and PUSH. Keep in mind that in a multiple Queen replica setup, push order during maintenance is not guaranteed (the replicas don't coordinate when they read buffer files).
4. All backend services inside Kubernetes. Our current production environment is around 800 containers. Some of our services read from multiple queues and push to multiple, so inside a single container you can do whatever you want. You can do the same as you would with other queue/broker systems like Kafka, RabbitMQ, or NATS.

About FaaS, it's interesting and I thought about it some weeks ago. Can you tell me what you have in mind?

[xmlking]

Wishing you all the best for your PostgreSQL migration.
I’m trying to comprehend how to execute multiple daemon processes within a single Node.js container, as one subscription waiting may hinder the execution of others. It might be feasible using Java threads or GoLang goroutines. I’m curious to know what runtime backend services employ. Are they using JavaScript or Python?

In the past, I attempted to build a web application that enables users to submit kSQL data pipelines that execute on ksqlDB. However, after ksqlDB was discontinued, I explored arroyo, which offers a runtime for multiple data pipelines to run simultaneously on a set of workers within one or more containers.