Skip to main content

Distributed workers

Currently, we run packit service in Openshift Online, because we need our API (httpd/service) and dashboard to be publicly accessible. The most resources however consume workers, which on the other hand don't need to be publicly available. So the idea was to run (only) workers in an internal cluster which is not publicly accessible, but we don't have to pay for it.

Public broker & backend​

Because service and workers communicate via Celery we need the broker (Redis) and backend (PostgreSQL) to be publicly accessible.

Route​

If Redis/PostgreSQL were using HTTP(S) protocol, their exposing would be as simple as creating a route for them. Unfortunately (see here and here), routes are only for HTTP(S) or TLS+SNI.

Both, Redis (starting with latest version 6) and PostgreSQL can talk TLS, but they don't seem to have SNI support implemented, see here, here and here.

Exposing a service​

Other options (than route) for exposing a service are Load Balancer, NodePort and ExternalIP. Unfortunately, none of them I succeeded to use on Openshift Online.

Load Balancer​

Creating a Load Balancer is a privileged operation and we're not allowed to do that on OS Online.

NodePort​

Is a privileged operation and requires additional port resources.

ExternalIP​

Downside of using ExternalIP is that it consumes an IP address, so you already need to have one to assign to it.

oc port-forward​

Just for debugging/developing, needs to be done on client side.

Other Celery supported brokers​

Leaving aside that Redis can't do SNI, quoting: "Redis is designed to be accessed by trusted clients inside trusted environments. This means that usually it is not a good idea to expose the Redis instance directly to the internet or, in general, to an environment where untrusted clients can directly access the Redis TCP port or UNIX socket."

RabbitMQ​

Supports SSL/TLS and SNI as well.

AWS SQS​

Price is \$0.5 per million messages. Currently we're producing/consuming around 1200 messages per day so at this rate we'd be paying around 18 cents (\$0.018) a month for using SQS.

Other SQLAlchemy/Celery supported databases​

We use PostgreSQL via SQLAlchemy either directly or as a Celery result backend. If we can't use it because it doesn't support SNI, then we have to find other free/open source, SQLAlchemy supported database that supports TLS+SNI:

Or use cloud database, like AWS RDS for PostgreSQL whose pricing starts at $0.018/hour ($13/month) for db.t3.micro (1GiB mem).