High Fibre Programming

There is often a need when building a website to automate background processes, for example:

• Sending emails: many sites don’t immediately connect to an SMTP server when a user fires off an email request, but rather store the email in the database which is processed by a background process, thus decoupling the frontend from the SMTP server. If the SMTP server goes down, your users can still register
• Offline processing: let’s say your site has a social element to it, users have real-time news feeds that get updated whenever their friends do things on your site. By updating these feeds offline there’s no need for users to wait until all 1,000,000 of their friends’ feeds have been updated before completing a request to post a simple comment.

There are plenty more examples, in fact as a rule of thumb it’s best to do everything that doesn’t have to be done immediately on a HTTP request later in the background. It also helps manage peak traffic, when your site is overloaded, just don’t run the background tasks until the peak has gone. Ok, so news feeds aren’t bang up-to-date for a short period, but at least the site works. So how does one go about automating all these background tasks?

CRON

This is probably the first port-of-call; implement your background task in PHP, thereby enabling you to use the database abstraction layer and all the other libraries in your application, and set CRON to run the script every minute.

Great – everything works perfectly, that is until the data in your database grows so big that the tasks start taking longer than one minute and you end up with lots of tasks all running at the same time – and potentially all processing the same data! Ooops, the boss just got his registration email ten times.

So you could implement some sort of locking mechanism to prevent multiple instances but it’s all starting to get a bit fragile with lots of interdependencies – all you want to do is run a simple script!

Write a daemon in PHP

So it’s back to the drawing board – what is needed is a daemon that will sit in the background and run as script continually, restarting it immediately after it finishes. You could write a daemon in PHP – this would enable you to write a daemon which would interface well with your existing PHP scripts, there are even some good PHP packages which make writing a daemon in PHP easy.

So, full of enthusiasm you launch into writing your own daemon in PHP. A week or so later, after reading all about process control, (and probably pulling large chunks of hair out in the process as yo realise it wasn’t as simple as you thought), you finally have something that works. You deploy it to your website, sit back and feel smug.

After a week your boss rages into your office demanding to know why nobody has registered in the past week. Red faced, you go to check your PHP daemon that sends emails form the email queue is still running – and it isn’t. If you haven’t already been fired then your next step would probably be to find out what happened. Looking through your PHP error log you find a “FATAL ERROR” from your daemon script, dated exactly one week ago.

PHP can throw fatal errors for all sorts of reasons, and there’s no way to recover from them. Also, PHP is renowned for its rather flaky memory management – whilst this is fine for web scripts that render a page and then end, it makes it highly unsuitable to write applications.

Enter The Fat Controller

How can we solve the problem? We need to run PHP scripts as a daemon. I faced this problem one year ago and I solved it by writing The Fat Controller. It is a program written in C that runs a daemon and can continually run PHP scripts. As it is written in C, it is highly stable and can run for months or years without problem. As the daemon runs separately from the PHP scripts, no matter what happens in the PHP script, it does not affect the Fat Controller.

The Fat Controller is also very flexible, easy to install and configure and supports running multiple instances of a PHP at the same time to achieve parallel processing. You can even configure it to dynamically control the number of parallel processes dependent on how much work there is.

You can read more about The Fat Controller here   http://www.4pmp.com/fatcontroller/

I recently got this somewhat cryptic error message when trying to merge something in Subversion. In the root directory of my working copy, I ran svn merge, something along the lines of:

svn merge -c 1234 http://svn.example.com/myproject/trunk ./

Only to get:
svn: 'http://subversion.tigris.org/xmlns/dav/md5-checksum' was not present on the resource

I had absolutely no idea what it was on about, and a fair bit of Googling turned up nothing. Then, just for fun, I tried going one level up from my working copy and trying again:


cd ../
svn merge -c 1234 http://svn.example.com/myproject/trunk ./myworkingcopy/


And lo and behold – it worked. I haven’t the foggiest idea why, but it does. Any ideas anyone?

Using SVN v1.6.6

Redirecting STDOUT to a file is easy:

command > foo.log

Redirecting STDERR to a file is also easy:

command 2> foo.log

Redirecting both STDOUT and STDERR to a file is also fairly straightforward:

command &> foo.log

But what if we want to append to the log rather than truncating it each time we write? This is done for STDOUT by using the >> operator, and for STDERR by using 2>>. Unfortunately &>> does not exist, so how do we append both STDOUT and STDERR?

The solution is to redirect STDERR to STDOUT, and then redirect STDOUT to a file using the append operator:

command 1>> foo.txt 2>&1

I recently upgraded my Ubuntu install to the latest Karmic Koala, and to be honest I’ve not been totally impressed.   A range of little bugs that had been fixed before seem to have reappeared – just little things such as problems with icons in Gnome panel, nothing I couldn’t live with.

However, yesterday I got a message saying that there was less than 2GB left on my hard drive.   The last time I checked I had over 20GB free, so where did it all go?   After a bit of investigating I found two giant 10GB kernel log files /var/log/kern.log.

It seemed that the computer was constantly logging the same messages:

CPU0: Temperature above threshold, cpu clock throttled (total events = 208[ 8973.550089] CPU0: Temperature/speed normal

CPU0: Temperat cpu clock throttled (total events = 2080190)

I had a look on Google and it turns out that it is a bug with the new Karmic Koala update:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/453444

So far there’s no patch for it, so the best solution I came up with was just to turn off the logging.

Step 1:
Look in /etc/rsyslog.conf and it will either have the configuration for system logging, or in the case of my machine it points to include all files in /etc/rsyslog.d/   In my case there was only one file in here so it made things simpler.

Step 2:
Open the configuration file in an editor and find the line which specifies kernel logging and comment it, something like:

#kern.*                         -/var/log/kern.log

Step 3:
Now restart the system log daemon:   sudo restart rsyslog

It is the logging which hogs the processor so by turning it off the problem is at least not noticeable.

We decided it would be a good idea to be able to periodically log server load so we can identify potential problems So I wrote a little script that outputs the current server load along with a timestamp in CSV format which can easily be called via CRON and send the output to a log file which can then be imported into a spreadsheet.

getload.c

Just compile it and run it. The output should be something like:

"1251834476","0.21"

It turns out that Nagios logs server load anyway, but not quite as accurately as this.

I was looking for a way to get rsync to copy all the files in a particular directory and not recurse into the sub-directories.   Unfortunately I couldn’t find an appropriate parameter and a quick search on Google turned up nothing (apart from a lot of other people asking the same question!)

So as a last resort I decided to use my brain, and I came up with a simple solution that works.   It’s of the form:

rsync -avc --exclude "*/" ./source/* ./destination/


The above will copy all the files from the directory called “source” into the directory “destination” and will not recurse into any subdirectories of “source”.