Docker images and AMIs

[hazmeister]

AMIs and other cloud images are based on Ubuntu 18 which is almost EOL. I looked at updating them, but essentially it's a image with docker compose and a couple of convenient shell scripts which I think could be documented instead and hosted in a reference repo.

I could write a terraform module for hosting with ECS and RDS since we sort of recommend using that anyway once you reach a certain size of deployment.

We're going to need hosting for docker images- a single account is free on docker hub if we can work with a single set of credentials, until we can join the Docker Open Source program.

I think GitHub and quay.io also have free tiers.

[justinclift]

Cool, good thinking. 😄

The EOL for Ubuntu 18 seems to be literally this month (!):

https://ubuntu.com/blog/ubuntu-18-04-eol-for-devices

Probably the easier update we can do in the short term is move the image to Ubuntu 20 (LTS). Should be less changes than a newer Ubuntu.

The "document it" thing sounds like a good idea too.

Do you want to keep on poking around options for the hosting side of things, and set up our Docker credentials?

[justinclift]

As a thought, the version numbers that show up in the Redash user interface are stored in package.json and redash/__init__.py.

I'm not sure if there's some process that automatically updates them prior to a release, but we can always update them manually before building the new docker image as we develop our own release processes. 😄

[hazmeister]

OK, pulled them down for the minute whilst we figure this out. I'll look at updating version numbers, although to migrate to our images, people will be changing tag anyway, so it's a deliberate change. If I can patch the images for v8 and v9 , should they be published as 8.1 and 9.1?

[justinclift]

Oh, that's a good point. 😄

I hadn't personally thought about the older releases, but it makes good sense to do them as well where possible.

For "what exact version numbers though"... good question.

Just pulling numbers out of thin air, maybe lets go with x.5 to indicate "some kind of big change, but not enough for a whole release number"?

So:

Old Redash	Updated Redash Community build
10.1	10.5
9.0	9.5
8.0	8.5

Thoughts?

[justinclift]

@hazmeister Potential beta testing volunteer: @joshuacronemeyer (from here).

[hazmeister]

Can do that. Looks like we ought to get some ami upgrade instructions written too. As the image is basically git source and docker compose it should be easy enough.

[rpmcoding]

This could do with updating also - https://marketplace.digitalocean.com/apps/redash

Not sure if we have access though.

[justinclift]

@hazmeister On the topic of building these security-fixing releases, you're doing so from the existing release/* branches yeah? eg release/10.x, release/9.x, etc

Asking because I haven't looked into the patches in those branches yet. No idea at all what state they're in (etc). 😇

[hazmeister]

Originally I was monkey patching- grabbing the original images and adding some fixes in new layers. If we have any other planned fixes in the pipeline it would make sense to build from source.

[justinclift]

If we can build from source, it's probably the way to go... though I'm somewhat suspecting the older versions of dependencies are going to create their own kind of hassles.

Oh well, if you hit issues with stuff then er... open Issues here? There are people around who are willing to help out, especially to get an updated working release out the door. 😄

[joshuacronemeyer]

Hmmm, forcing an update to v10 for places that haven't already done it, seems like it'd be a dick move on our part. I'd kind of expect that any place not yet running v10 is likely doing that for a reason.

Just FYI the only reason we're still on 8 is that the instructions on the website seemed to encourage using the AMI, which is on 8. IMO as long as there are decent instructions for moving to 10 I don't mind upgrading. Also, I don't require an AMI. My goal is to stay on a nice little documented path to maintaining our redash instance.

[justinclift]

Cool, that makes good sense. 😄

[tinder-tder]

I have a terraform module we can probably share (getting clearance) but its basically AWS based, ec2 (ubuntu+docker-compose) + rds + elasticache + alb + kms. We can adapt it for ecs too if needed. ubuntu 22 lts works fine

[justinclift]

Ahhhh, got it. That makes sense.

Yeah, that could be useful down the track a bit. 😄

[tinder-tder]

I had to do a bunch of cleanup/fixes on the setup repo to get things running the first time we deployed. So when I get time I will try and cleanup and update that repo if we want to keep it there. I am not sure the prev repo org was necessary and long term maintainable (ie one repo referencing another with stale links/info)

[justinclift]

Yeah, from rough memory I think the separate setup repo kind of grew out from its original home of being a "setup" directory in the main Redash repo.

But then it was kind of left to rot, which is likely the main problem. 😉

[tinder-tder]

Yea, I think I would prefer to keep it in the main repo but we should come up with a consensus. I don't want to rock the boat cause this isn't exactly greenfield.

[justinclift]

Yeah, I don't have a strong opinion either way, so "whatever works" from my point of view. 😄

[Avey777]

https://hub.docker.com/r/redashcommunity/redash
Looking forward to it working as soon as possible

[Avey777]

That's awesome. It's already having an impact.

[gaecoli]

@justinclift I build a pre version based on master code. But i don't have permission to push into our docker hub.

[justinclift]

@gaecoli Cool. I think @hazmeister was a bit faster, as he has added an automatic task to our GitHub repo now: 3f586da

This should automatically push to our Docker Hub repo from now on.

That being said, the username/password needed should be in our repo secrets somewhere, so you should be able to get them and do stuff. 😄

[gaecoli]

@gaecoli Cool. I think @hazmeister was a bit faster, as he has added an automatic task to our GitHub repo now. It should automatically push to our Docker Hub repo from now on.

That being said, the username/password needed should be in our repo secrets somewhere, so you should be able to get them and do stuff. 😄

ok, maybe docker hub username and password should be publiced to developers org.

[justinclift]

Good point. Are you ok to do that in the existing private area we have?

[gaecoli]

Good point. Are you ok to do that in the existing private area we have?

yes