Speed is an important factor for websites wanting faster growth. While Google’s cloud infrastructure is already tuned for faster speed, some small tweaks can sometimes help you get the fullest benefit from features you are using. I decided to check how much difference installing Varnish cache on a GCP VM instance can bring. While the speed was already satisfactory for a small website hosted on a GCP VM Instance with 2 GB RAM and 8 GB memory (e2-standard-2), installing varnish led to a noticeable and impressive improvement in page loading times.
Whether you are using wordpress or hosting a html website, in both cases, the speed improvement from installing Varnish Htttp accelerator on GCP VM instance (apache or NGINX server) can be great. If you are using wordpress, you can benefit from other type of caching systems and reverse proxies as well. For example, having Redis cache installed on your server can also help improve page load times. Opcode cache is already activated on GCP VM instances. However, the improvement that comes from Varnish is a lot better compared to using just Opcache with Redis or Memcached.
“Varnish Cache is a web application accelerator also known as a caching HTTP reverse proxy. You install it in front of any server that speaks HTTP and configure it to cache the contents. Varnish Cache is really, really fast. It typically speeds up delivery with a factor of 300 – 1000x, depending on your architecture.”
-Varnish Software.
I have also noticed that the results can differ based on the type of machine you are using. As in the case of GCP VM instances, the change was far more impressive compared to running varnish on a typical dedicated server with more than 50 GB in RAM. Any server can benefit from Varnish http accelerator but in the case of GCP, the change was more than satisfactory. You can also install Varnish to find out. Moreover, you will need to do a lot of optimization to find the right configuration for your particular server in GCP’s case.
How to install Varnish on a GCP VM instance (Apache or NGINX server):
To install Varnish on an instance, you need to use the official packages hosted on Package Cloud where you will find available for various distributions like Centos, Ubuntu, Fedora and Debian. We are installing Varnish http accelerator on an Apache server with Debian. First, you will need to choose the right Varnish version for your installation. The recommended version is Varnish Cache 6.0 LTS. It is the stable and supported version that is maintained by Varnish and receives frequent updates.
To make sure that Debian installs the right version of Varnish and not its own, we will first need to register the right package repository.
You can register the official Varnish Cache 6.0 LTS repository using the following commands:
First update the package list for information on the latest available packages. It is an essential step so that you can run the next command. Run the following command:-
sudo apt-get update
Dependencies:- Now, you will be able to install the dependencies required to configure the repository with the following command.
sudo apt-get install debian-archive-keyring curl gnupg apt-transport-https
GPG Key:- Import the GPG package key into the package manager confinguration:
curl -s -L https://packagecloud.io/varnishcache/varnish60lts/gpgkey | sudo apt-key add –
Now, we can register the package repository using the following command:-
. /etc/os-release
sudo tee /etc/apt/sources.list.d/varnishcache_varnish60lts.list > /dev/null <<-EOF
deb $VERSION_CODENAME main
EOF
sudo tee /etc/apt/preferences.d/varnishcache > /dev/null <<-EOF
Package: varnish varnish-*
Pin: release o=packagecloud.io/varnishcache/*
Pin-Priority: 1000
EOF
Update the package list once more to make sure that the Packagecloud repository is included. Run:
$ sudo apt-get update
Varnish Installation:
After getting the repositories registered and having the right repository configurations in place, it is time to install Varnish using the following command. This command will insatll the latest version of Varnish Cache 6.0 LTS.
sudo apt-get install varnish
The output will look like below. Sample output :–
Reading package lists...
DoneBuilding dependency tree...
DoneReading state information...
Done
The following additional packages will be installed:
binutils binutils-common binutils-x86-64-linux-gnu cpp cpp-10 fontconfig-config fonts-dejavu-core gcc gcc-10 libasan6 libatomic1 libbinutils libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdeflate0 libfontconfig1 libgcc-10-dev libgd3 libgomp1 libisl23 libitm1 libjbig0 libjemalloc2 libjpeg62-turbo liblsan0 libmpc3 libmpfr6 libnsl-dev libquadmath0 libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp6 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxpm4 linux-libc-dev manpages manpages-devSuggested packages: binutils-doc cpp-doc gcc-10-locales gcc-multilib make autoconf automake libtool flex bison gdb gcc-doc gcc-10-multilib gcc-10-doc glibc-doc libgd-tools varnish-dev
The following NEW packages will be installed:
binutils binutils-common binutils-x86-64-linux-gnu cpp cpp-10 fontconfig-config fonts-dejavu-core gcc gcc-10 libasan6 libatomic1 libbinutils libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdeflate0 libfontconfig1 libgcc-10-dev libgd3 libgomp1 libisl23 libitm1 libjbig0 libjemalloc2 libjpeg62-turbo liblsan0 libmpc3 libmpfr6 libnsl-dev libquadmath0 libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp6 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxpm4 linux-libc-dev manpages manpages-dev varnish
0 upgraded, 49 newly installed, 0 to remove and 5 not upgraded.Need to get 57.1 MB of archives.After this operation, 204 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://security.debian.org/debian-security bullseye-security/main amd64 libtiff5 amd64 4.2.0-1+deb11u3 [290 kB]
Get:2 http://deb.debian.org/debian bullseye/main amd64 manpages all 5.10-1 [1412 kB]
Get:3 http://security.debian.org/debian-security bullseye-security/main amd64 linux-libc-dev amd64 5.10.162-1 [1576 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 binutils-common amd64 2.35.2-2 [2220 kB]
Get:6 http://deb.debian.org/debian bullseye/main amd64 libbinutils amd64 2.35.2-2 [570 kB]
Get:7 http://deb.debian.org/debian bullseye/main amd64 libctf-nobfd0 amd64 2.35.2-2 [110 kB]
Get:8 http://deb.debian.org/debian bullseye/main amd64 libctf0 amd64 2.35.2-2 [53.2 kB]
Get:9 http://deb.debian.org/debian bullseye/main amd64 binutils-x86-64-linux-gnu amd64 2.35.2-2 [1809 kB]
Get:10 http://deb.debian.org/debian bullseye/main amd64 binutils amd64 2.35.2-2 [61.2 kB]Get:11 http://deb.debian.org/debian bullseye/main amd64 libisl23 amd64 0.23-1 [676 kB]
Once we have installed varnish, it is time to configure some varnishd runtime parameters. Systemd manages the varnishd processs and its unit file is located at: - /lib/systemd/system/varnish.serviceThe contents of this file look like below:-
[Unit]
Description=Varnish Cache, a high-performance HTTP accelerator
After=network-online.target nss-lookup.target[Service]
Type=forking
KillMode=process
# Maximum number of open files (for ulimit -n)LimitNOFILE=131072
# Locked shared memory - should suffice to lock the shared memory log# (varnishd -l argument)# Default log size is 80MB vsl + 1M vsm + header -> 82MB# unit is bytes
LimitMEMLOCK=85983232
# Enable this to avoid "fork failed" on reload.TasksMax=infinity
# Maximum size of the corefile.LimitCORE=infinity
ExecStart=/usr/sbin/varnishd \
-a :6081 \
-a localhost:8443,PROXY \
-p feature=+http2 \
-f /etc/varnish/default.vcl \
-s malloc,256m
ExecReload=/usr/sbin/varnishreload
[Install]
WantedBy=multi-user.target
You will need to make some modifications to the unit file and for that you will need to copy its contents to /etc/systemd/system/varnish.service. If you like you can make changes to the /lib/systemd/system/varnish.service file and save it as /etc/systemd/system/varnish.service, which is actually easier. Run the following command to do this:
$ sudo systemctl edit –full varnish
This will open the varnish.service unit file for editing. Find the ExecStart statement near the bottom of the file and change the port from 6081 to 80. The ExecStart statement will look like below after you have made the changes.
ExecStart=/usr/sbin/varnishd \
-a :80 \
-a localhost:8443,PROXY \
-p feature=+http2 \
-f /etc/varnish/default.vcl \
-s malloc,256mb
You can also increase the size of cache if you need by editing the value after -s malloc, . Suppose you want to change it from 256mb to 2g. After changing the last line of the ExecStart statement will look like :
-s malloc,2g
Once you have made the changes, save the file and exit the editor. It will have created the file /etc/system/system/varnish.service.
Whenever, you make changes manually to the unit file, you will need to reload the systemd daemon. Run the following command to do that:-
sudo systemctl daemon-reload
Now, we have configured Varnish to listen on port 80. However, the webserver is also listening on the same port and therefore to avoid a clash, we will need to change the port on which the web server is listening to 8080. You will need to make changes to the vhosts file and /etc/apache2/ports.conf file if you are running apache server on your instance. In the port.conf file, you need to change Listen 80 to Listen 8080. In the vhost file, you need to change <VirtualHost *:80> to <VirtualHost *:8080>. It looks much but can be accomplished by a single command. Run:
sudo find /etc/apache2 -name '*.conf' -exec sed -r -i 's/\bListen 80\b/Listen 8080/g; s/<VirtualHost ([^:]+):80>/<VirtualHost \1:8080>/g' {} ';'What the above command does is to replace Listen 80 with Listen 8080 in the vhost files on your server and also makes the same changes to all the .conf files found inside the /etc/nginx directory and its subdirectories.
To accomplish the above changes on NGINX server, you need a slightly different command:–
sudo find /etc/nginx -name '*.conf' -exec sed -r -i 's/\blisten ([^:]+:)?80\b([^;]*);/listen \18080\2;/g' {} ';'The change of port from 80 to 8080 also need to be reflected in the Varnish backend definition of the VCL file. The default VCL file that comes with the Varnish installation is located at /etc/varnish/default.vcl. This file already has a default backend definition pointing to 127.0.0.1 on port 8080. It looks something like this:–
vcl 4.0;backend default { .host = “127.0.0.1”; .port = “8080”;}
Now, since we have made changes to several configuration files, we will need to restart the essential services to bring the changes into effect.
Apache server:
$ sudo systemctl restart apache2 varnish
NGINX server:
$ sudo systemctl restart nginx varnish
Once you have restarted your server and varnish, you can check the status of Varnish using the following command.
sudo systemctl status varnish
The output will be like below showing Varnish is active and running on your server:–
$ sudo systemctl status varnish
● varnish.service - Varnish Cache, a high-performance HTTP accelerator Loaded: loaded (/etc/systemd/system/varnish.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2023-02-15 07:42:01 UTC; 41min ago
Process: 17507 ExecStart=/usr/sbin/varnishd -a :80 -a localhost:8443,PROXY -p feature=+http2 -f /etc/varnis> Main PID: 17509 (varnishd) Tasks: 217 Memory: 136.6M CPU: 2.598s CGroup: /system.slice/varnish.service ├─17509 /usr/sbin/varnishd -a :80 -a localhost:8443,PROXY -p feature=+http2 -f /etc/varnish/defaul> └─17521 /usr/sbin/varnishd -a :80 -a localhost:8443,PROXY -p feature=+http2 -f /etc/varnish/defaul>Feb 15 07:42:01 instance-2 varnishd[17509]: Warnings:Feb 15 07:42:01 instance-2 varnishd[17509]: VCL compiled.Feb 15 07:42:01 instance-2 varnishd[17509]: Debug: Version: varnish-6.0.11 revision a3bc025c2df28e4a76e10c2c412>Feb 15 07:42:01 instance-2 varnishd[17509]: Version: varnish-6.0.11 revision a3bc025c2df28e4a76e10c2c41217c9864>Feb 15 07:42:01 instance-2 varnishd[17509]: Debug: Platform: Linux,5.10.0-21-cloud-amd64,x86_64,-junix,-smalloc>Feb 15 07:42:01 instance-2 varnishd[17509]: Platform: Linux,5.10.0-21-cloud-amd64,x86_64,-junix,-smalloc,-sdefa>Feb 15 07:42:01 instance-2 varnishd[17509]:
Debug: Child (17521) StartedFeb 15 07:42:01 instance-2 varnishd[17509]: Child (17521)
StartedFeb 15 07:42:01 instance-2 varnishd[17509]: Child (17521) said Child startsFeb 15 07:42:01 instance-2 systemd[1]: Started Varnish Cache, a high-performance HTTP accelerator.
With some customizations you can achieve an even decent level of performance from Varnish. If you are running WordPress on a GCP VM instance, you can use a VCL file customized for the WordPress by Varnish software (developers of Varnish).
I have pasted the contents of the VCL file (custom VCL file for WordPress) below.
To open and edit the contents of your VCL file, you can use
sudo nano /etc/varnish/default.vcl.
After making the changes, hit ‘ctrl + X’ and ‘y’ and then ‘Enter’. The file below includes the necessary cache rules which can help you achieve decent performance on WordPress with Varnish. It also includes the default backend definition. Restart apache and Varnish after editing the VCL file.
vcl 4.1;
import std;
backend default {
.host = “127.0.0.1”;
.port = “8080”;
}
# Add hostnames, IP addresses and subnets that are allowed to purge content
acl purge {
“localhost”;
“127.0.0.1”;
“::1”;
}
sub vcl_recv {
# Remove empty query string parameters
# e.g.: www.example.com/index.html?
if (req.url ~ “\?$”) {
set req.url = regsub(req.url, “\?$”, “”);
}
# Remove port number from host header
set req.http.Host = regsub(req.http.Host, “:[0-9]+”, “”);
# Sorts query string parameters alphabetically for cache normalization purposes
set req.url = std.querysort(req.url);
# Remove the proxy header to mitigate the httpoxy vulnerability
# See https://httpoxy.org/
unset req.http.proxy;
# Purge logic to remove objects from the cache.
# Tailored to the Proxy Cache Purge WordPress plugin
# See https://wordpress.org/plugins/varnish-http-purge/
if(req.method == “PURGE”) {
if(!client.ip ~ purge) {
return(synth(405,”PURGE not allowed for this IP address”));
}
if (req.http.X-Purge-Method == “regex”) {
ban(“obj.http.x-url ~ ” + req.url + ” && obj.http.x-host == ” + req.http.host);
return(synth(200, “Purged”));
}
ban(“obj.http.x-url == ” + req.url + ” && obj.http.x-host == ” + req.http.host);
return(synth(200, “Purged”));
}
# Only handle relevant HTTP request methods
if (
req.method != “GET” &&
req.method != “HEAD” &&
req.method != “PUT” &&
req.method != “POST” &&
req.method != “PATCH” &&
req.method != “TRACE” &&
req.method != “OPTIONS” &&
req.method != “DELETE”
) {
return (pipe);
}
# Remove tracking query string parameters used by analytics tools
if (req.url ~ “(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=”) {
set req.url = regsuball(req.url, “&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)”, “”);
set req.url = regsuball(req.url, “\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)”, “?”);
set req.url = regsub(req.url, “\?&”, “?”);
set req.url = regsub(req.url, “\?$”, “”);
}
# Only cache GET and HEAD requests
if (req.method != “GET” && req.method != “HEAD”) {
set req.http.X-Cacheable = “NO:REQUEST-METHOD”;
return(pass);
}
# Mark static files with the X-Static-File header, and remove any cookies
# X-Static-File is also used in vcl_backend_response to identify static files
if (req.url ~ “^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|ogg|ogm|opus|otf|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$”) {
set req.http.X-Static-File = “true”;
unset req.http.Cookie;
return(hash);
}
# No caching of special URLs, logged in users and some plugins
if (
req.http.Cookie ~ “wordpress_(?!test_)[a-zA-Z0-9_]+|wp-postpass|comment_author_[a-zA-Z0-9_]+|woocommerce_cart_hash|woocommerce_items_in_cart|wp_woocommerce_session_[a-zA-Z0-9]+|wordpress_logged_in_|comment_author|PHPSESSID” ||
req.http.Authorization ||
req.url ~ “add_to_cart” ||
req.url ~ “edd_action” ||
req.url ~ “nocache” ||
req.url ~ “^/addons” ||
req.url ~ “^/bb-admin” ||
req.url ~ “^/bb-login.php” ||
req.url ~ “^/bb-reset-password.php” ||
req.url ~ “^/cart” ||
req.url ~ “^/checkout” ||
req.url ~ “^/control.php” ||
req.url ~ “^/login” ||
req.url ~ “^/logout” ||
req.url ~ “^/lost-password” ||
req.url ~ “^/my-account” ||
req.url ~ “^/product” ||
req.url ~ “^/register” ||
req.url ~ “^/register.php” ||
req.url ~ “^/server-status” ||
req.url ~ “^/signin” ||
req.url ~ “^/signup” ||
req.url ~ “^/stats” ||
req.url ~ “^/wc-api” ||
req.url ~ “^/wp-admin” ||
req.url ~ “^/wp-comments-post.php” ||
req.url ~ “^/wp-cron.php” ||
req.url ~ “^/wp-login.php” ||
req.url ~ “^/wp-activate.php” ||
req.url ~ “^/wp-mail.php” ||
req.url ~ “^/wp-login.php” ||
req.url ~ “^\?add-to-cart=” ||
req.url ~ “^\?wc-api=” ||
req.url ~ “^/preview=” ||
req.url ~ “^/\.well-known/acme-challenge/”
) {
set req.http.X-Cacheable = “NO:Logged in/Got Sessions”;
if(req.http.X-Requested-With == “XMLHttpRequest”) {
set req.http.X-Cacheable = “NO:Ajax”;
}
return(pass);
}
# Remove any cookies left
unset req.http.Cookie;
return(hash);
}
sub vcl_hash {
if(req.http.X-Forwarded-Proto) {
# Create cache variations depending on the request protocol
hash_data(req.http.X-Forwarded-Proto);
}
}
sub vcl_backend_response {
# Inject URL & Host header into the object for asynchronous banning purposes
set beresp.http.x-url = bereq.url;
set beresp.http.x-host = bereq.http.host;
# If we dont get a Cache-Control header from the backend
# we default to 1h cache for all objects
if (!beresp.http.Cache-Control) {
set beresp.ttl = 1h;
set beresp.http.X-Cacheable = “YES:Forced”;
}
# If the file is marked as static we cache it for 1 day
if (bereq.http.X-Static-File == “true”) {
unset beresp.http.Set-Cookie;
set beresp.http.X-Cacheable = “YES:Forced”;
set beresp.ttl = 1d;
}
# Remove the Set-Cookie header when a specific Wordfence cookie is set
if (beresp.http.Set-Cookie ~ “wfvt_|wordfence_verifiedHuman”) {
unset beresp.http.Set-Cookie;
}
if (beresp.http.Set-Cookie) {
set beresp.http.X-Cacheable = “NO:Got Cookies”;
} elseif(beresp.http.Cache-Control ~ “private”) {
set beresp.http.X-Cacheable = “NO:Cache-Control=private”;
}
}
sub vcl_deliver {
# Debug header
if(req.http.X-Cacheable) {
set resp.http.X-Cacheable = req.http.X-Cacheable;
} elseif(obj.uncacheable) {
if(!resp.http.X-Cacheable) {
set resp.http.X-Cacheable = “NO:UNCACHEABLE”;
}
} elseif(!resp.http.X-Cacheable) {
set resp.http.X-Cacheable = “YES”;
}
# Cleanup of headers
unset resp.http.x-url;
unset resp.http.x-host;
}