AWX on Kubernetes
Hosting AWX on Kubernetes leverages the benefits of better availability and fault tolerance and brings improvements to deployment options, backups, and other parts.
AWX-operator
The AWX Operator is the preferred way to install AWX from release 18.0 onwards. The AWX operator is meant to provide a more Kubernetes-native installation method for AWX Tower. AWX Operator manages the AWX instance created using AWX Custom Resource Definition (CRD) and continuously watches the AWX instance deployment for any issues.
Installation of AWX operator and AWX instance has been already explained here. This article discusses the use of external Postgres DB in AWX installation and how to upgrade the AWX version.
Installing AWX instance with external Postgres DB
In order to use external Postgres DB, a k8s secret must be created containing the details for the connection endpoint and credentials for authentication to the Postgres database.
---
apiVersion: v1
kind: Secret
metadata:
name: external-postgres-configuration
namespace: awx
stringData:
host: postgresdb.example.com
port: 5432
database: awx
username: awx
password: myawxpassword
sslmode: prefer
type: unmanaged
type: Opaque
If you would like to also use your own AWX admin password, it should also be created in k8s secret beforehand.
---
apiVersion: v1
kind: Secret
metadata:
name: awx-admin-password
namespace: awx
stringData:
password: awxadminpassword
type: Opaque
AWX instance creation
Make sure that you have the AWX operator running before creating an AWX instance. AWX operator must be running in the same namespace where you are creating the AWX instance. Use the below manifest file to create an AWX instance. This manifest also includes the resource requirements and limits for AWX components.
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx
namespace: awx
spec:
service_type: loadbalancer
image: quay.io/ansible/awx
image_version: 21.0.0
admin_password_secret: awx-admin-password
postgres_configuration_secret: external-postgres-configuration
web_resource_requirements:
requests:
cpu: "1"
memory: 2Gi
limits:
cpu: "2"
memory: 4Gi
task_resource_requirements:
requests:
cpu: "1"
memory: 2Gi
limits:
cpu: "2"
memory: 4Gi
ee_resource_requirements:
requests:
cpu: "1"
memory: 2Gi
limits:
cpu: "2"
memory: 4Gi
This manifest creates the AWX tower with a load balancer service as defined service_type. The external IP of the AWX instance can be fetched via the following command in the EXTERNAL-IP column:
$ kubectl get svc -n awx
Upgrading AWX version
Upgrading the AWX is as simple as just changing the image_version to the desired release in the above manifest file and applying it again.
Creating backup for external Postgres DB
This method uses the pg_dump utility for taking the snapshot of Postgres database via Kubernetes cronjob. AWX operator also supports awxbackupCRD but it provides little flexibility on backup schedule, destination, and compression methods.
Creating backup cronjob:
Below is the example of Cronjob which runs every two hours to take the Postgres DB snapshot and stores it in a local PVC called awx-backup. The Postgres credential are provided to this cronjob via a k8s secret called awx-postgres-configuration.
apiVersion: batch/v1
kind: CronJob
metadata:
name: awx-postgres-backup
namespace: awx
spec:
concurrencyPolicy: Allow
failedJobsHistoryLimit: 3
spec:
backoffLimit: 3
template:
spec:
containers:
- command:
- bash
- -c
- "set -e -o pipefail \nDATE=$(date +%Y-%m-%d-%H-%M)\nBACKUP=/backups/awx-backup-$DATE.db\nPGPASSWORD=$password
pg_dump --clean --create -h $host -U $username -d $database -p $port
-F custom | gzip > $BACKUP.gz\necho \"backup succesfull\"\n"
envFrom:
- secretRef:
name: awx-postgres-configuration
image: awx-backup-image:latest
imagePullPolicy: Always
name: awx-backup
volumeMounts:
- mountPath: /backups
name: backup-volume
dnsPolicy: ClusterFirst
restartPolicy: OnFailure
volumes:
- name: backup-volume
persistentVolumeClaim:
claimName: awx-backup
schedule: "0 */2 * * *"
The awx-backup-image image used in the above cronjob is a custom docker image and can be built using the below Dockerfile.
FROM alpine:3.15
RUN apk --no-cache add postgresql12-client
ENTRYPOINT [ "psql" ]
Restoring AWX instance from database dump from the backup
Data Backups are only as good as the backup testing procedure. It’s of utmost importance to regularly test database backups by restoring them on a test instance. This section provided instructions on how to restore the AWX Postgres database snapshot.
On the fresh Postgres DB instance, create a new awx user and awx database and assign the required permissions. You would need a password for the Postgres DB user and a new password for awx user to be created. First access the Postgres DB instance using psql command below:
$ PGPASSWORD=<postgres_db_pass> psql -h <postgres_db_hostname_or_ip> -U postgres -p 5432
> create database awx;
> create user awx with encrypted password 'dbpassword';
> grant all privileges on database awx to awx;
Note: AWX db password should be the same as set in the original AWX instance. Now import the AWX database dump using pg_restore command as follow:
$ PGPASSWORD=<awx_db_pass> pg_restore --clean --if-exists -U awx -d awx -p 5432 -h <postgres_db_hostname_or_ip> <awx-db-dump>.db
Next, point AWX fronted to a new database instance by changing the host field in k8s secret named external-postgres-db-secret.
kubectl -n awx patch secrets external-postgres-db-secret --patch "{\"data\":{\"host\":\"$(echo <postgres_db_hostname_or_ip> | base64 -w 0)\"}}"
Summary
In this article, I have tried to explain the process of installing ansible tower aka AWX using AWX-operator with an external Postgres database. I have also listed the steps for taking frequent backups of AWX instances using k8s cronjobs and also the restore method as part of the disaster recovery process.
Writer // KUMORION BLOG //
Shankar Lal
Enthusiastic DevOps learner and sometimes like to write about his experiences for community awareness.