How to Configure Varnish Cache as Load Balancer

Learn how to configure Varnish Cache as a load balancer with our detailed step-by-step guide. Improve your website’s performance and manage traffic efficiently. #centlinux #linux #loadbalancer

What is Varnish Cache?

Varnish Cache is an HTTP Accelerator designed for content-heavy dynamic websites and APIs. Varnish Cache is usually installed on the same web server, where Varnish Cache acts as the front-end and accelerator for the hosted websites. Varnish Cache is free and open source software which is distributed under two-clause BSD license.

Varnish Cache is an open-source HTTP reverse proxy and caching solution designed to improve the performance, scalability, and reliability of web applications. It sits between a web server and the internet, caching content to reduce load times and increase efficiency for frequently accessed web pages and resources.

Key Features

1. Caching Mechanism
   • HTTP Reverse Proxy: Varnish acts as an intermediary between clients and web servers, caching responses from the server and serving them directly to clients for subsequent requests.
   • Content Caching: Stores static content (like images, CSS, and JavaScript) and dynamic content (like HTML pages) to speed up response times.
2. Performance Optimization
   • High-Speed Caching: Uses an efficient, in-memory caching system to deliver content quickly and handle high traffic loads.
   • HTTP Acceleration: Reduces server load by caching responses and serving them directly to users, decreasing the number of requests to the backend server.
3. Load Balancing
   • Traffic Distribution: Can be configured as a load balancer to distribute incoming requests across multiple backend servers, improving application performance and availability.
   • Failover and Health Checks: Monitors backend servers, rerouting traffic in case of server failures and performing health checks.
4. Flexible Configuration
   • VCL (Varnish Configuration Language): A powerful scripting language that allows administrators to define caching policies, request handling, and response rules.
   • Custom Rules: Create complex rules for caching, such as setting expiration times, defining cache purging policies, and customizing responses.
5. Advanced Features
   • Content Invalidation: Supports mechanisms for purging or invalidating cached content based on changes to the underlying data.
   • Logging and Analytics: Provides detailed logs and analytics to monitor cache performance, traffic patterns, and server health.
6. Security Enhancements
   • Traffic Filtering: Control and filter incoming requests to prevent abuse and protect against malicious activities.
   • Secure Caching: Manage SSL/TLS termination and encryption for secure content delivery.

How Varnish Cache Works

1. Client Request
   • A client sends an HTTP request to the Varnish Cache server.
2. Cache Lookup
   • Varnish checks if the requested content is already cached.
   • Cache Hit: If the content is in the cache, Varnish serves it directly to the client.
   • Cache Miss: If the content is not in the cache, Varnish forwards the request to the backend web server.
3. Backend Response
   • The backend server processes the request and sends the response back to Varnish.
4. Content Caching
   • Varnish stores the response in the cache for future requests based on the caching rules defined in VCL.
5. Client Response
   • Varnish delivers the response to the client. For future requests, Varnish will serve the cached content if available.

Use Cases

• Website Performance Optimization
  • Speed up page load times for high-traffic websites and applications.
• Scalability Solutions
  • Improve the performance of web servers under heavy traffic loads.
• Load Balancing
  • Distribute requests across multiple backend servers to enhance reliability and performance.
• Content Delivery
  • Cache static and dynamic content for fast and efficient delivery.
• Security Enhancements
  • Implement filtering rules and manage SSL/TLS for secure web interactions.

Varnish Cache vs. Other Caching Solutions

Popular Alternatives to Varnish Cache

• Nginx: A popular web server and reverse proxy with caching and load-balancing capabilities.
• Apache Traffic Server: A high-performance caching proxy server from the Apache Software Foundation.
• Squid: A caching proxy for web content that supports HTTP, HTTPS, and FTP.

Conclusion

Varnish Cache is a powerful tool for optimizing web performance, managing high traffic loads, and balancing requests across servers. Its advanced caching capabilities, flexible configuration options, and robust performance make it a preferred choice for many organizations looking to enhance their web infrastructure.

Varnish cache supports multiple back-end hosts, therefore we can also configure Varnish Cache as the Reverse Proxy for load balancing of a cluster of web servers.

Recommended Online Training: Learn Bash Shell in Linux for Beginners

Linux Server Specification

We have configured a CentOS 7 virtual machine with following specifications:

• CPU – 3.4 Ghz (1 Core)
• Memory – 1 GB
• Storage – 20 GB
• Operating System – CentOS 7.7
• Hostname – varnish-cache-01.example.com
• IP Address – 192.168.116.213 /24

Install Apache on CentOS 7

Connect with varnish-cache-01.example.com using ssh as root user.

Build yum cache for standard CentOS 7 repositories.

# yum makecache fast
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.ges.net.pk
 * extras: mirrors.ges.net.pk
 * updates: mirrors.ges.net.pk
base                                                     | 3.6 kB     00:00
extras                                                   | 2.9 kB     00:00
updates                                                  | 2.9 kB     00:00
Metadata Cache Created

Update CentOS 7 server packages.

# yum update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.ges.net.pk
 * extras: mirrors.ges.net.pk
 * updates: mirrors.ges.net.pk
No packages marked for update

Our CentOS 7 server is already up-to-date.

Install Apache HTTP server using yum command.

# yum install -y httpd

Start and enable Apache web service.

# systemctl enable --now httpd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

Allow HTTP service in CentOS 7 firewall.

# firewall-cmd --permanent --add-service=http
success
# firewall-cmd --reload
success

Browse URL http://varnish-cache-01.example.com in a client’s browser.

Apache HTTP server is successfully installed and it is serving the default test page.

Configure Apache Virtual Hosts

We are configure two virtual hosts here, that will run on two different ports.

Create document root directories for virtual hosts.

# mkdir /var/www/html/{vhost1,vhost2}

Create default index page for Virtual Host 1.

# cat > /var/www/html/vhost1/index.html << EOF
> <html>
> <head><title>Virtual Host1</title></head>
> <body><h1>This is the default page of Virtual Host 1...</h1></body>
> </html>
> EOF

Similarly, create default index page for Virtual Host 2.

# cat > /var/www/html/vhost2/index.html << EOF
> <html>
> <head><title>Virtual Host2</title></head>
> <body><h1>This is the default page of Virtual Host 2...</h1></body>
> </html>
> EOF

Create configuration file for Virtual Host1.

# vi /etc/httpd/conf.d/vhost1.conf

add following directives therein.

Listen 8081
<VirtualHost *:8081>
    DocumentRoot "/var/www/html/vhost1"
    ServerName vhost1.example.com
</VirtualHost>

Similarly, create configuration file for Virtual Host2.

# vi /etc/httpd/conf.d/vhost2.conf

add following directives therein.

Listen 8082
<VirtualHost *:8082>
    DocumentRoot "/var/www/html/vhost2"
    ServerName vhost2.example.com
</VirtualHost>

Check Apache configurations for syntax errors.

# httpd -t
Syntax OK

Since, we are running Apache websites on non-default ports, therefore, we have to add these ports to SELinux port labeling.

Check, if these ports are already added in SELinux.

# semanage port -l | grep ^http_port_t
http_port_t                    tcp      80, 81, 443, 488, 8008, 8009, 8443, 9000

Add ports 8081 and 8082 to type http_port_t SELinux context.

# semanage port -m -t http_port_t -p tcp 8081
# semanage port -m -t http_port_t -p tcp 8082

Verify if these ports are added in SELinux port labeling.

# semanage port -l | grep ^http_port_t
http_port_t                    tcp      8082, 8081, 80, 81, 443, 488, 8008, 8009, 8443, 9000

Now, we can safely load our Apache configurations.

# systemctl reload httpd.service

Allow 8081/tcp and 8082/tcp service ports in CentOS 7 firewall.

# firewall-cmd --permanent --add-port={8081,8082}/tcp
success
# firewall-cmd --reload
success

Open URL http://varnish-cache-01.example.com:8081/ in a web browser.

Open URL http://varnish-cache-01.example.com:8082/ in a web browser.

Both of our Apache virtual hosts has been configured successfully.

Install Varnish Cache on CentOS 7

Varnish Cache software is available in EPEL (Extra Packages for Enterprise Linux) yum repository.

Therefore, first we have to enable EPEL yum repository as follows.

# yum install -y epel-release

Build cache for EPEL yum repository.

# yum makecache

Now, we can install Varnish Cache software using yum command.

# yum install -y varnish

We have installed the default version of Varnish Cache that is available in EPEL yum repository. However, you can always download and install a latest version of Varnish Cache from their Official Download Page.

Configure Varnish Cache as Load Balancer

To configure Varnish Cache, we are required to free the port 80 that is currently used by Apache HTTP server.

The directive that controls the service port 80 is defined in /etc/httpd/conf/httpd.conf file.

We can change it using a sed command.

# sed -i "s/Listen 80/Listen 8080/" /etc/httpd/conf/httpd.conf

Restart the Apache service to take changes into effect.

# systemctl restart httpd.service

Now, port 80 is available and we can use it for Vanish Cache service.

Edit Varnish Cache configuration file.

# vi /etc/varnish/varnish.params

Locate and set following directive therein.

VARNISH_LISTEN_PORT=80 #Default Port 6081

We have changed the Varnish Cache default port 6081 with 80.

It’s time to configure the backend for Varnish Cache server.

These settings are located in /etc/varnish/default.vcl file. We can easily replace this file with our custom configurations.

Rename the existing default.vcl file using mv command.

# mv /etc/varnish/default.vcl /etc/varnish/default.vcl.org

Create a custom backend configuration file.

# vi /etc/varnish/default.vcl

and add following lines of codes.

vcl 4.0;

import directors;    # Load the directors

backend vhost1 {
    .host = "192.168.116.213";
    .port = "8081";
    .probe = {
        .url = "/";
        .timeout = 1s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
    }
}

backend vhost2 {
    .host = "192.168.116.213";
    .port = "8082";
    .probe = {
        .url = "/";
        .timeout = 1s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
    }
}

sub vcl_init {
    new lb = directors.round_robin(); # Creating a Load Balancer
    lb.add_backend(vhost1); # Add Virtual Host 1
    lb.add_backend(vhost2); # Add Virtual Host 2
}

sub vcl_recv {
    # send all traffic to the lb director:
    set req.backend_hint = lb.backend();
}

Enable and start Varnish Cache service.

# systemctl enable --now varnish.service
Created symlink from /etc/systemd/system/multi-user.target.wants/varnish.service to /usr/lib/systemd/system/varnish.service.

Enable and start Varnish Cache logging service.

# systemctl enable --now varnishlog.service
Created symlink from /etc/systemd/system/multi-user.target.wants/varnishlog.service to /usr/lib/systemd/system/varnishlog.service.

Verify the backend list using following command.

# varnishadm backend.list
Backend name                   Refs   Admin      Probe
vhost1(192.168.116.213,,8081)  1      probe      Healthy 5/5
vhost2(192.168.116.213,,8082)  1      probe      Healthy 5/5

Check our website’s response header.

# curl -I http://varnish-cache-01.example.com

HTTP/1.1 200 OK Date: Sun, 13 Oct 2019 16:24:07 GMT Server: Apache/2.4.6 (CentOS) Last-Modified: Sun, 13 Oct 2019 09:35:58 GMT ETag: “7d-594c77a7e0839” Content-Length: 125 Content-Type: text/html; charset=UTF-8 X-Varnish: 32770 Age: 0 Via: 1.1 varnish-v4 Connection: keep-alive

Open URL http://varnish-cache-01.example.com in a web browser.

We have successfully configure Varnish Cache. The load balancer is now redirecting user requests to Virtual Host 1 and Virtual Host 2 in a round robin way.

If you are new to Linux and facing difficulty in working at Linux Bash prompt. We recommend that, you should read The Linux Command Line, 2nd Edition: A Complete Introduction by William Shotts.

Final Thoughts

Configuring Varnish Cache as a load balancer can significantly enhance your website’s performance and traffic management capabilities. By setting up Varnish Cache correctly, you can distribute incoming requests efficiently, reduce server load, and improve overall site responsiveness.

If you need professional assistance or a more detailed guide on configuring Varnish Cache as a load balancer, I offer expert services on Fiverr. Visit my Fiverr profile: Linux Performance Tuning Expert for personalized support and ensure a successful Varnish Cache setup for your web infrastructure.