My setup for High Availability, Redundancy and Resilience
Coming from my homelab, Cloudflare, pfSense posts; to make my lab environment ready for production use I must have Resilience in my setup. In this post, I want to explore on how I configure and structure my production tools for High Availability and Redundancy of backend services for Resilience purpose.
High Availability
pfSense is the crucial part of my lab, it's sometime unstable and crash so I must have a backup instance for it, and there is a built-in High Availability feature for this purpose. The steps are straightforward:
- Clone my pfSense VM to another computer with the same setup, the original VM is the master and the clone is the the failover
- In the master, create a CARP Virtual IP for the WAN interface and another CARP Virtual IP for the LAN interface
- Enable High Availability in the master, put in the the failover's IP, after that Firewall and Virtual IP will be synced between master and failover
- Enable
Sync HAProxy configuration to backup CARP members via XMLRPC.settings to sync HAProxy configs as well.
The 2 tutorials helped me with my setup: [pfSense] Configuring High Availability and pfsense HA / High Availability Setup and Testing Using CARP, XMLRPC & pfsync. Also, my pfSense instances are Hyper-V VMs and they couldn't find each other, there is a promiscuous setting to enable in Hyper-V virtual switch to solve this (guide).
After this setup, if the master pfSense goes down the failover will becomes master instantly, this provides resilience for my lab's connectivity. Cloudflare Connector also need high availability, I cloned the Connector to 3 replicas so if anyone fails, Cloudflare still have connection to my lab.
Redundancy
Redundancy is also necessary for a multi computers setup. Kubernetes helps to clone my backend containers to as many instances as needed and PostgreSQL, MongoDB, Elasticsearch do have their built-in clusters feature.
Backend: with RoR as my backend, this guide Deploying a Rails application to Kubernetes helps with the steps to create the Docker Image and how to deploy to Kubernetes with multiple instances
PostgresSQL: Streaming replication is the key feature to provide redundancy for my SQL servers. I have the master database hosted on a resilient PC with UPS and battery, the 3 replicas are hosted on other machines with better performance and more optimized for query.
MongoDB: I had some confuses to set up a Replica Set for MongoDB, the documents are unclear about what steps to make, but it is quite stable after the correct setup with 3 nodes. Convert a Standalone Self-Managed mongod to a Replica Set and Add Members to a Self-Managed Replica Set
Elasticsearch: the cluster setup for Elasticsearch is straightforward, a 3 nodes setup will grant Elasticsearch green status. Creating an Elasticsearch Cluster: Getting Started and Mastering the Art of Elasticsearch Cluster Setup