I’ve recently built a few apps, and have a few sites that have fairly high traffic. I swiftly discovered, that most shared hosts don’t really offer a great experiance, with timeouts and reource limits.
I have been using, and still use 20i, the platform behind Raleway Hosting, however again, i’m still hitting resource limits. I’ve tried to use large VPS’s on AWS and even a dedicated server, but wanted something that would be high availibility. So… what to do, over a few days I was very bored, wanting something to do I built my own high availibility, autoscaling and clustered webserver.
How i did it!
The first bit, was researching whether to use multiple small nodes, or fewer larger nodes. I decided, it would be best to use multiple smaller nodes, building a layer of redundancy. If one hardware node fails, then other nodes will still be available to handle the traffic.
All traffic going into the cluster, will first hit a load balancer, which is set to round robin each request, sending requests to nodes with the lowest resource usage. The cluster will auto-patch, where if one of the servers fails, all traffic will fail over to the remaining nodes, until the failed node comes back online.
When building a clustered hosting environment, one of the most important findings, was to ensure the site files sit off the host node. So, for this, i created a redundant RAID cluster, where files are stored in 2 load balanced RAID nodes. I used 2 seperate load balanced nodes, again for if one node needs to be taken offline, the hosted sites will not be affected.
For database’s, i chose to use the MySQL server, running in a cluster, with 3 nodes in the cluster.
The host nodes, run a simple LAMP stack, using virtual hosts to connect to the website files, stored within the storage RAID cluster. Each virtual host, connects to a dedicated file within the RAID server, that has permissions only for specific a V-Host. The protect the website files, the LAMP nodes, have ModSecurity and Fail2Ban.
While everything worked, there was an overarching problem of performance, running websites from a NFS Cluster. Removing the site hosted files from the LAMP node, was vital to ensure the cluster would work, it does have a significant performance side affect. PHPOpCache stores non volitile PHP files on the LAMP node’s ram. While this isn’t perfect, it does help performance significantly.
Performance was a secondary concern, over reliability. I wanted to create a service that would be able to handle a lot of traffic, autoscale to more nodes if required and did not have a single failure point.
In the future, i’m hoping to increase speed and performance by adding an Nginx proxy infront of the LAMP nodes and using faster storage, in replacement of NFS.
Why this setup?
I wanted to build something that i could infinatly scale, i have services that have fairly moderate traffic most of the time, but are fairly database heavy, using large SQL files. Instead of just dumping the sites on a single VPS server, i created this autoscaling cluster. If the sites were constantly busy, then i would obviously just use a single dedicated server or VPS, but i have multiple sites, that have peak traffic at different times.
What about DDOS and WAF
Each LAMP node is running ModSecurity and Fail2Ban, banned users are added to the Firewall (Infront of each node and the Load Balancer) to drop all connections. ModSecurity blocks are also shared in a Database across all the LAMP nodes, so they all block at the same time.
The whole cluster is protected by Layer 7 DDOS attacks, using Cloudflare, only Cloudflare can connect to the site, direct IP access to the cluster is blocked. While this is not a perfect solution, its aim is to protect the cluster, in a cost effective way.
Management
The whole cluster is managed by a very small management node, that is totally seperate to the host cluster. Changes to the config are made on the management node, prior to being pushed to the cluster.
Single points of failure
The load balancer, Database and NFS are all high availibility nodes, they will fail over to ensure the service is allways available.
Do i use a cloud service or is it on prem?
Its located in the cloud, i wanted to create an OnPrem setup, however having a single internet connection and no real backup power, putting it in the cloud seemed a good option. Also, having it cloud based, means i don’t need to worry about hardware failures and i can technically hyperscale (which it will do automatically). I do have some on-prem services, including Nextcloud and Home Assistant.
Why was this important for me?
Other than being a bit of a hobby, i’m building an exciting application, based on WordPress. While i already have some fairly busy application hosted on this cluster, the new application will increase traffic quite a lot, and will not be able to be hosted on any shared hosting. I wanted something that would be super reliable, and infinatly scailable.
Leave a Reply