Nginx Active-Passive Cluster

I have recently created my own nginx cluster. I use nginx as a reverse proxy and needed a high-availability solution. There’s already support for this in Nginx Plus, but I’m compiling my own version of the Open Source version of nginx, so I looked at their documentation as inspiration and created my own scripts. I have two servers a Primary (Node A - Active) and a Secondary (Node B - Passive), my Primary node is the where I edit/update my nginx configration and then it synchronizes the changes to my secondary node. Here’s what you need to get started

Prerequisites

3 IP addresses - Each server needs an IP address and the last is used as the floating IP address which is the one users should access
2 RHEL/CentOS Servers
Nginx - Webserver
Keepalived - VRRP software
Rsync - Synchronization software

Install software

Install the following packages on both servers

yum install -y nginx keepalived rsync

Firewall Configuration

This should be applied to both servers. If you are serving websites on non-standard ports, then remember to open them as well

firewall-cmd --add-service=http --permanent
firewall-cmd --add-service=https --permanent
firewall-cmd --add-rich-rule='rule protocol value="vrrp" accept' --permanent
firewall-cmd --reload

Tweak the Linux Kernel

The Nginx and Keepalived process needs the ability to bind to a non-local IP address. This is done by creating a file in /etc/sysctl.d/

echo "net.ipv4.ip_nonlocal_bind=1" > /etc/sysctl.d/90-keepalived.conf

NOTE. The above needs to be done on both servers

Node A - Primary Nginx Server - 192.168.1.11

/etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
	 enable_script_security					# Enable Script Security
	 script_user YOUR-USERNAME				# Run script as this user. For security reasons, don't use root
}

vrrp_script track_nginx {					# Tracking script to determine if the service is healthy
	script "/usr/sbin/pidof nginx"			# Checks if nginx is running
	interval 2								# Checks every 2 seconds
	timeout 1
}

vrrp_instance VI_1 {
    state MASTER							# MASTER/BACKUP
    interface ens192						# Name of interface to be used for VRRP
    virtual_router_id 51					# Router ID needs to match on all nodes
    priority 200							# A higher number has higher priority
    advert_int 1
    authentication {
      auth_type PASS
      auth_pass YOUR-PASSWORD				# Maximum 8 characters
    }
	
    unicast_peer {
		192.168.1.12						# IP address of our secondary node
	}
		
	virtual_ipaddress {
		192.168.1.10/24 dev ens192			# Floating/Virtual IP and the interface name to bind it to
    }

	track_script {
		track_nginx							# Tracking script defined above
	}
}

Start Keepalived and Nginx

systemctl enable --now keepalived.service
systemctl enable --now nginx.service

Sync Nginx files

Now we need to detect changes in /etc/nginx and synchronize them to our secondary server. This is a two step process, first we need a script to do the actual synchronization and then we need to run it when a change has been made.

Create a script called NginxSync.sh

#!/bin/bash
NodeB="192.168.1.12" # IP address of the Node B/Secondary nginx server

## Check if nginx configuration is valid before synchronization
if out=$(nginx -t 2>&1); then
	## Sync Files
	rsync -a --delete /etc/nginx/ $NodeB:/etc/nginx

	## Restart Nginx
	ssh $NodeB "systemctl restart nginx.service"

	echo "Success"
else
	echo "Failure, because $out"
fi

1a. Make the script executable

chmod +x NginxSync.sh

We need to create a ssh key and copy the public key to Node B

ssh-keygen -t rsa -b 4096 -C "Nginx Primary" -f ~/.ssh/id_NodeA_rsa -N ""

## Copy public key to Node B
ssh-copy-id -i ~./ssh/id_NodeA_rsa.pub root@192.168.1.12

Now we need to create the monitor script. It’s made of two files placed in /etc/systemd/system/

nginxFileChange.service

[Unit]
Description = Starts the synchronization job from Node A to Node B
Documentation = man:systemd.service

[Service]
Type=oneshot
ExecStart=/YOUR-PATH-TO/NginxSync.sh		# Remember to change this line to your needs

nginxFileChange.path

[Unit]
Description = Triggers the nginxFileChange.service which synchronizes changes
Documentation = man:systemd.path

[Path]
PathModified=/etc/nginx/			# Path to the nginx config folder
Unit=nginxFileChange.service

[Install]
WantedBy=multi-user.target		# Requires at least runlevel 3 otherwise our NginxSync.sh script wont work

Start the nginxFileChange.path service

systemctl daemon-reload
systemctl enable --now nginxFileChange.path		# Enables the file monitor check
systemctl status nginxFileChange.service		# Shows status of the sync service

Node B - Secondary Nginx Server - 192.168.1.12

/etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
	 enable_script_security
	 script_user root
}

vrrp_script track_nginx {
	script "/usr/bin/killall -0 nginx"
	interval 2
	timeout 1
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens192
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
      auth_type PASS
      auth_pass YOUR-PASSWORD
    }
	
    unicast_peer {
		192.168.1.11
	}
		
	virtual_ipaddress {
		192.168.1.10/24 dev ens192
    }

	track_script {
		track_nginx
	}
}

Start Keepalived and Nginx

systemctl enable --now keepalived.service
systemctl enable --now nginx.service

Troubleshooting

Here is a few useful commands to see if it’s working

ip address								# Shows network interfaces and IP

journalctl -r -u keepalived				# Shows nginx systemd log
journalctl -r -u nginx					# Shows keepalived systemd log

systemctl enable service-name.service		# Auto starts the service at boot
systemctl enable --now service-name.service	# Equal to systemctl enable + systemctl start 

systemctl status nginx.service			# Shows service status
systemctl status keepalived.service
systenctl status nginxFileChange.path

systemctl start nginx.service			# Start Service
systemctl start keepalived.service
systemctl start nginxFileChange.path

systemctl stop nginx.service			# Stops Service
systemctl stop keepalived.service
systemctl stop nginxFileChange.path