Unable to launch managed postgres db in cluster

[shrutebattlestargalactica]

Hello @rooftopcellist! I am having some issues with deploying EDA into our EKS cluster:

eda-manager.log

it is stuck on

TASK [postgres : Wait for Database to initialize if managed DB]:

{"level":"info","ts":1697747113.740688,"logger":"proxy","msg":"cache miss: /v1, Kind=PodList err-Index with name field:status.phase does not exist"}

I deployed via recommended operator with ArgoCD/Customize

Are we not able to yet use an external db similar to what we do with AWX? thanks for your help!

[shrutebattlestargalactica]

@rooftopcellist I am wonder if this is related to this issue similar to the awx-operator: ansible/awx-operator#1022

I forgot to mention but we run this EKS cluster on k8s v1.28

[kurokobo]

Perhaps your PostgreSQL pod is not starting properly. Please check the status of your PostgreSQL pod, and solve the cause of the failure to start.

In most cases for this issue (and the issue that reported on AWX Operator repo), the cause is in your environment or your configuration, not in Operator.

[shrutebattlestargalactica]

Thats correct - the pod is crashing. I checked the pod logs and saw this:

mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied

eda-manager container shows this error after the DB task:

TASK [postgres : Wait for Database to initialize if managed DB]:

{"level":"info","ts":1697747113.740688,"logger":"proxy","msg":"cache miss: /v1, Kind=PodList err-Index with name field:status.phase does not exist"}

I have a very vanilla set up for the deployment:

eda.yaml

apiVersion: eda.ansible.com/v1alpha1
kind: EDA
metadata:
name: awxnonprod-eda
namespace: eda
spec:
automation_server_url: https://awxdev.mydomain.com
service_type: nodeport
ingress_type: none
service_labels: |
environment: nonprod
team: technology
hostname: edadev.mydomain.com

pv and pvc are successfully created but not sure why its throwing this permission error when creating the userdata directory

I'd ultimately like to see if we are able to use an external rds instance but I am unsure of how to set up the secret for EDA. I am already doing this with our AWX instance so I am curious if we can use the same settings

[shrutebattlestargalactica]

Alright I ended up getting the external RDS postgres working! I was mimicking my secret from the AWX configuration and had to remove the "type: unmanaged" from the secret below:

---

apiVersion: v1
kind: Secret
metadata:
name: eda-nonprod-postgres-configuration
namespace: eda-nonprod
stringData:
host: "mydatabase.rds.amazonaws.com"
port: "5432"
database: "mydatabase"
username: "supersecretusername"
password: "supersecretpassword"
sslmode: "prefer"
type: Opaque

[watsonb]

I'm running into the same permission issue:

mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied

PV/PVC are created successfully. This worked in 0.0.4 for me, but not working in 0.0.7 of the operator.

[watsonb]

OK, weird, after just letting it sit and 7 crash backoff loops later, it was able to create the directory. Must be some timing issue.

[goldyfruit]

Facing the same issue but no resolution after a while

NAME                                                     READY   STATUS    RESTARTS   AGE   IP              NODE           NOMINATED NODE   READINESS GATES
eda-postgres-13-0                                        0/1     Pending   0          10s   <none>          <none>         <none>           <none>
eda-redis-99c7f6c8f-9qr98                                1/1     Running   0          68s   10.233.66.56    k8s-worker-4   <none>           <none>
eda-server-operator-controller-manager-8f4cb5d7d-k785p   2/2     Running   0          79m   10.233.98.129   k8s-worker-2   <none>           <none>
eda-postgres-13-0                                        0/1     Pending   0          27s   <none>          k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     ContainerCreating   0          27s   <none>          k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     ContainerCreating   0          44s   <none>          k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Running             0          46s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Error               0          47s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Error               1 (2s ago)   48s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     CrashLoopBackOff    1 (2s ago)   49s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Error               2 (23s ago)   70s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     CrashLoopBackOff    2 (5s ago)    74s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Error               3 (32s ago)   101s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     CrashLoopBackOff    3 (2s ago)    103s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Running             4 (54s ago)   2m35s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Error               4 (55s ago)   2m36s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     CrashLoopBackOff    4 (9s ago)    2m44s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Running             5 (84s ago)   3m59s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Error               5 (85s ago)   4m      10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     CrashLoopBackOff    5 (2s ago)    4m1s    10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Error               6 (2m43s ago)   6m42s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     CrashLoopBackOff    6 (2s ago)      6m43s   10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Running             7 (5m5s ago)    11m     10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Error               7 (5m7s ago)    11m     10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     CrashLoopBackOff    7 (3s ago)      11m     10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Error               8 (5m12s ago)   16m     10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     CrashLoopBackOff    8 (7s ago)      17m     10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     Error               9 (5m4s ago)    22m     10.233.66.34    k8s-worker-4   <none>           <none>
eda-postgres-13-0                                        0/1     CrashLoopBackOff    9 (3s ago)      22m     10.233.66.34    k8s-worker-4   <none>           <none>

# kubectl logs -f --tail 50 -n eda eda-postgres-13-0
mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied

[shrutebattlestargalactica]

Hey I have a resolution that fixed this let me review and get back to you

…

On Wed, Nov 29, 2023 at 4:30 PM Gaëtan Trellu ***@***.***> wrote: Facing the same issue but no resolution after a while NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES eda-postgres-13-0 0/1 Pending 0 10s <none> <none> <none> <none> eda-redis-99c7f6c8f-9qr98 1/1 Running 0 68s 10.233.66.56 k8s-worker-4 <none> <none> eda-server-operator-controller-manager-8f4cb5d7d-k785p 2/2 Running 0 79m 10.233.98.129 k8s-worker-2 <none> <none> eda-postgres-13-0 0/1 Pending 0 27s <none> k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 ContainerCreating 0 27s <none> k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 ContainerCreating 0 44s <none> k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Running 0 46s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Error 0 47s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Error 1 (2s ago) 48s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 CrashLoopBackOff 1 (2s ago) 49s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Error 2 (23s ago) 70s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 CrashLoopBackOff 2 (5s ago) 74s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Error 3 (32s ago) 101s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 CrashLoopBackOff 3 (2s ago) 103s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Running 4 (54s ago) 2m35s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Error 4 (55s ago) 2m36s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 CrashLoopBackOff 4 (9s ago) 2m44s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Running 5 (84s ago) 3m59s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Error 5 (85s ago) 4m 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 CrashLoopBackOff 5 (2s ago) 4m1s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Error 6 (2m43s ago) 6m42s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 CrashLoopBackOff 6 (2s ago) 6m43s 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Running 7 (5m5s ago) 11m 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Error 7 (5m7s ago) 11m 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 CrashLoopBackOff 7 (3s ago) 11m 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Error 8 (5m12s ago) 16m 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 CrashLoopBackOff 8 (7s ago) 17m 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 Error 9 (5m4s ago) 22m 10.233.66.34 k8s-worker-4 <none> <none> eda-postgres-13-0 0/1 CrashLoopBackOff 9 (3s ago) 22m 10.233.66.34 k8s-worker-4 <none> <none> — Reply to this email directly, view it on GitHub <#138 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVNVYVTYWBMH7KTEPC4KQXLYG7ASHAVCNFSM6AAAAAA6HZORL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZSHA3DGMBXGY> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[shrutebattlestargalactica]

Oh sorry I was thinking of a separate issue. If you have access to an external Postgres db that would be what I use. If you need help with setting that up in your cluster let me know On Wed, Nov 29, 2023 at 5:07 PM Kerry Mason ***@***.***> wrote:

…

Hey I have a resolution that fixed this let me review and get back to you On Wed, Nov 29, 2023 at 4:30 PM Gaëtan Trellu ***@***.***> wrote: > Facing the same issue but no resolution after a while > > NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES > eda-postgres-13-0 0/1 Pending 0 10s <none> <none> <none> <none> > eda-redis-99c7f6c8f-9qr98 1/1 Running 0 68s 10.233.66.56 k8s-worker-4 <none> <none> > eda-server-operator-controller-manager-8f4cb5d7d-k785p 2/2 Running 0 79m 10.233.98.129 k8s-worker-2 <none> <none> > eda-postgres-13-0 0/1 Pending 0 27s <none> k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 ContainerCreating 0 27s <none> k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 ContainerCreating 0 44s <none> k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Running 0 46s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Error 0 47s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Error 1 (2s ago) 48s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 CrashLoopBackOff 1 (2s ago) 49s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Error 2 (23s ago) 70s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 CrashLoopBackOff 2 (5s ago) 74s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Error 3 (32s ago) 101s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 CrashLoopBackOff 3 (2s ago) 103s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Running 4 (54s ago) 2m35s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Error 4 (55s ago) 2m36s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 CrashLoopBackOff 4 (9s ago) 2m44s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Running 5 (84s ago) 3m59s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Error 5 (85s ago) 4m 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 CrashLoopBackOff 5 (2s ago) 4m1s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Error 6 (2m43s ago) 6m42s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 CrashLoopBackOff 6 (2s ago) 6m43s 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Running 7 (5m5s ago) 11m 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Error 7 (5m7s ago) 11m 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 CrashLoopBackOff 7 (3s ago) 11m 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Error 8 (5m12s ago) 16m 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 CrashLoopBackOff 8 (7s ago) 17m 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 Error 9 (5m4s ago) 22m 10.233.66.34 k8s-worker-4 <none> <none> > eda-postgres-13-0 0/1 CrashLoopBackOff 9 (3s ago) 22m 10.233.66.34 k8s-worker-4 <none> <none> > > — > Reply to this email directly, view it on GitHub > <#138 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AVNVYVTYWBMH7KTEPC4KQXLYG7ASHAVCNFSM6AAAAAA6HZORL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZSHA3DGMBXGY> > . > You are receiving this because you modified the open/close state.Message > ID: ***@***.***> >

[goldyfruit]

Seems to work with NFS but no iSCSI.

[rchaud]

I am having this issue in Operator 1.0.1. Why was this issue closed?

[rooftopcellist]

maybe this is the same issue that was resolved here...

• Add postgres init container to resolve permissions for some k3s deployments awx-operator#1805

The sclorg postgresql image uses UID 26, whereas the postgres:13 image from dockerhub uses root (UID 0). That seems to create permissions issues when running on k8s. No issues on Openshift, which is why I didn't notice it.

I haven't confirmed that this is the case here as well, but it's my best guess atm.

[msmagnanijr]

I'm creating some roles that could be quite handy for your test. It's a bit dated, so you'll need to go over the images, tags, and such. e.g: the redis image now is quay.io/fedora/redis-6 .

https://github.com/msmagnanijr/eda-server/tree/AAP-15757/automation