Clear Varnish cache via PHP: a Drupal 7 proof of concept

Using Varnish as reverse proxy or proxy is an useful approach to reduce the load of webservers like Apache.

In Drupal 7 I’ve to clear the varnish cache of a specific domain when Drupal caches are globally cleared. Drupal has the right hook invoked when cache are cleared:

function clearcachevarnish_flush_caches() {
  $filename = '/var/www/varnishdomains2cleardir/varnishdomains2clear';
  // each domain on a separate line: append to the end of the file
  $myfile = fopen($filename, "a");
  $h = $_SERVER['HTTP_HOST'];
  $txt = $h . "\n";
  fwrite($myfile, $txt);
  fclose($myfile);
  drupal_set_message('Varnish cache queued to be cleared. Please wait 1 minute before checking.');
  // no cache table should be cleared
  return array();
}

Now this piece of code simply adds the current domain to a ASCII text file on /var/www/varnishdomains2cleardir/varnishdomains2clear.

Preparing the file to the write

On CentOS you have to add /var/www/varnishdomains2cleardir to the httpd-writable directories list using:

mkdir /var/www/varnishdomains2cleardir;
chcon -v --type=httpd_sys_content_t /var/www/varnishdomains2cleardir;
chown myuser:mygroup /var/www/varnishdomains2cleardir;
chmod -R 777 /var/www/varnishdomains2cleardir;
touch /var/www/varnishdomains2cleardir/varnishdomains2clear;

Now the empty file is ready to be written by your hook_flush_caches() implementation. Now enable the clearvarnishcache module and clear the cache to write the current domain name to the file.

The clear varnish cache script

To clear the varnish cache you usually have to be logged as root using the command varnishadm. Here a script that will read the domains file written above, clear the varnish cache for that domain and then remove the domains lines.

#!/bin/bash
callinguser=`whoami`
if [ "root" != "$callinguser" ]
then
 echo "Only root can run this command."
 exit 1
fi
cd /path/to/clear/cache/command/

date=`date +%Y-%m-%d_%H:%M:%S`

# check lock
# prevent the script from being run more than once
if [ -f /tmp/clearcachevarnish-lock ]; then
echo "Script clearcachevarnish is already running. You can rm /tmp/clearcachevarnish-lock to break the lock manually."
exit 1
fi
touch /tmp/clearcachevarnish-lock
dominidapulire=`less /var/www/varnishdomains2cleardir/varnishdomains2clear`
while [[ ! -z $dominidapulire ]]
do
 dominio=$(echo "$dominidapulire" | sed -n '$p')
 echo $dominio
 dominidapulire=$(echo "$dominidapulire" | sed '$d')
 if [ "" != "$dominio" ]
 then
 varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret ban req.http.host == "$dominio"
 echo "varnish cleared on $dominio"
 fi
done
# remove all domains lines
truncate --size 0 /var/www/varnishdomains2cleardir/varnishdomains2clear

# remove lock
rm /tmp/clearcachevarnish-lock

Make this script as executable .sh file using chmod a+x on it. If you run the bash script, varnish cache for files on the domains list will be cleared. It’s not so useful when using the Drupal UI so we should schedule this task periodically, e.g. every minute.

Scheduling the varnish clear cache

Here the crontab entry for execute the script every minute:


* * * * * root /path/to/clear/cache/command/clearcachevarnish.sh

The steps

  1. User clear Drupal cache
  2. hook_flush_caches() is invoked: the domains list file is written
  3. clear varnish cache script is launched by root every minute
  4. for each domain in the list, varnish cache is cleared

This is the end of this proof of concept. The code wasn’t tested against attacks so please comment if you have any suggestion to improve it. I’m not very fond of the idea of a php script writing something read by a bash script but this is the less problematic solution I found for this case.

How to fix the Bash bug on CentOS 6

Recently a critical bash bug was discovered.

To fix your CentOS 6 you have to check if you have a vulnerable bash installed. From a non root user, type:

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

If you read “vulnerable” as output then you have to update bash. Type su- and then the password to log in as superuser, then type:

yum update bash

Type Y when asked. When the update process is completed, retype the test script:

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

And you shouldn’t read the “vulnerable” message anymore.

Read more:

Web fonts and dynamic height calculation issues on jQuery

Recently we’ve nice fonts on web pages like Google Fonts and other web fonts. Take this case, you have to set two divs to the same height. One (div.funny) has some text with Google Fonts, the other is empty.

On Chrome console you type something like:

jQuery(".very", ".myview").height(function () {
  jQuery(this).height(jQuery(this).parent(".myview").find('.funny').height());
});

Div.very and div.funny are now at the same height.

Now if you try to do the same on jquery document ready you got elements with different height. Why?

Because the calculation happens on document ready but before fonts are loaded. The solution is to wrap the code on $(window).load().

$(window).load(function () {
  $(".very", ".myview").height(function () {
    $(this).height($(this).parent(".myview").find('.funny').height());
  });
});

Now .very and .funny are at the same height.

See also:
Calculate Container’s Height After The Font File Loads

Web fonts and dynamic height calculation issues on jQuery

Recently we’ve nice fonts on web pages like Google Fonts and other web fonts. Take this case, you have to set two divs to the same height. One (div.funny) has some text with Google Fonts, the other is empty.

On Chrome console you type something like:

jQuery(".very", ".myview").height(function () {
  jQuery(this).height(jQuery(this).parent(".myview").find('.funny').height());
});

Div.very and div.funny are now at the same height.

Now if you try to do the same on jquery document ready you got elements with different height. Why?

Because the calculation happens on document ready but before fonts are loaded. The solution is to wrap the code on $(window).load().

$(window).load(function () {
  $(".very", ".myview").height(function () {
    $(this).height($(this).parent(".myview").find('.funny').height());
  });
});

Now .very and .funny are at the same height.

See also:
Calculate Container’s Height After The Font File Loads

Untitled document instead of Google Docs list: how to get your document list again

Until yesterday, the shortcut docs.google.com listed the files I’ve on Drive.

Today when I try to access the domain I can only get a blank, Untitled document.

In that case you have to move the cursor on the title and then click on the arrow.

back2gd

You’ll be back on drive.google.com.

How to enable gzip on proxy servers on Apache

I’m starting to use the gunicorn django app using supervisord. Here my configuration:

  • Varnish: port 80
  • Apache: port 8080
  • gunicorn: port 4180 (/path/to/my/manage.py run_gunicorn localhost:4180)

Only the port 80 is exposed to other clients than localhost. The Varnish default backend is Apache (localhost:8080). I have a Drupal installation and a django installation on the same machine: since I want to expose django on the same domain at a defined location, I add to Apache this location:

<IfModule mod_proxy.c>
ProxyRequests Off
ProxyPreserveHost On

<Proxy *>
Order deny,allow
Allow from all
</Proxy>

# on port 4180 gunicorn is running
# @see /etc/supervisor.conf
ProxyPass /foo http://localhost:4180/
ProxyPassReverse /foo http://localhost:4180/

<Location /mypath>
Order allow,deny
Allow from all
AddOutputFilterByType DEFLATE text/html
</Location>
</IfModule>

You can omit AddOutputFilterByType DEFLATE text/html: here I just take the response from gunicorn, compress and then serve to the client in this way:

(client) -> varnish -> apache -> gunicorn

(client) <- varnish <- apache (compress) <- gunicorn
                (X-Varnish-Cache: MISS) 

Here an example of what I get:
Image

It’s a big page, but using gzip from 2.2 MB of the uncompressed page I get 417 KB gzipped text/html, less than 1/4 of the original!

Autolaunch a command and restart it on quit on CentOS

I’ve a django-admin command running as a server thanks to gevent. I want this server to run on boot and autorestart on quit.  StackOverflow give me a hint: use Supervisor.

On a Centos 5 distro:

# find supervisor for your distro...
yum search supervisor
# ...and install it
yum install supervisor.noarch
nano /etc/supervisord.conf

At the end of the file, add a new program:

[program:myfunnydjangocommand]
command=/usr/bin/env python26 /usr/local/etc/django-apps/foo/manage.py tcpapi 4114
priority=999                ; the relative start priority (default 999)
autostart=true              ; start at supervisord start (default: true)
autorestart=true            ; retstart at unexpected quit (default: true)
; startsecs=-1                ; number of secs prog must stay running (def. 10)
; startretries=3              ; max # of serial start failures (default 3)
exitcodes=0,2               ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT             ; signal used to kill process (default TERM)
; stopwaitsecs=10             ; max num secs to wait before SIGKILL (default 10)
; user=root                   ; setuid to this UNIX account to run the program
log_stdout=true             ; if true, log program stdout (default true)
log_stderr=true             ; if true, log program stderr (def false)
logfile=/var/log/myfunnydjangocommand.log    ; child log path, use NONE for none; default AUTO
logfile_maxbytes=1MB        ; max # logfile bytes b4 rotation (default 50MB)
logfile_backups=10          ; # of logfile backups (default 10)

Then, start supervisord.

service supervisord start

Take a look to supervisord log file:

less +G /var/log/supervisor/supervisord.log

You’ll see something like this:

2013-06-07 11:54:16,559 CRIT Supervisor running as root (no user in config file)
2013-06-07 11:54:16,576 INFO /var/tmp/supervisor.sock:Medusa (V1.1.1.1) started at Fri Jun  7 11:54:16 2013
        Hostname: <unix domain socket>
        Port:/var/tmp/supervisor.sock
2013-06-07 11:54:16,645 CRIT Running without any HTTP authentication checking
2013-06-07 11:54:16,654 INFO daemonizing the process
2013-06-07 11:54:16,657 INFO supervisord started with pid 19316
2013-06-07 11:54:16,666 INFO spawned: 'myfunnydjangocommand' with pid 19318
2013-06-07 11:54:17,670 INFO success: myfunnydjangocommand entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Read documentation about the configuration options but keep in mind your Supervisor version. I don’t use supervisorctl because of this bug, if you get an error simply go with service supervisord… but if you have a newer version this should be already fixed.

Note: myfunnydjangocommand.log doesn’t contain anything useful in my experience but maybe it’s related how I write the output since I’ve written it to use interactively, outputting lines directly to the user. I’ll update this post if I find how to solve this issue.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: