High Fibre Programming

There is often a need when building a website to automate background processes, for example:

• Sending emails: many sites don’t immediately connect to an SMTP server when a user fires off an email request, but rather store the email in the database which is processed by a background process, thus decoupling the frontend from the SMTP server. If the SMTP server goes down, your users can still register
• Offline processing: let’s say your site has a social element to it, users have real-time news feeds that get updated whenever their friends do things on your site. By updating these feeds offline there’s no need for users to wait until all 1,000,000 of their friends’ feeds have been updated before completing a request to post a simple comment.

There are plenty more examples, in fact as a rule of thumb it’s best to do everything that doesn’t have to be done immediately on a HTTP request later in the background. It also helps manage peak traffic, when your site is overloaded, just don’t run the background tasks until the peak has gone. Ok, so news feeds aren’t bang up-to-date for a short period, but at least the site works. So how does one go about automating all these background tasks?

CRON

This is probably the first port-of-call; implement your background task in PHP, thereby enabling you to use the database abstraction layer and all the other libraries in your application, and set CRON to run the script every minute.

Great – everything works perfectly, that is until the data in your database grows so big that the tasks start taking longer than one minute and you end up with lots of tasks all running at the same time – and potentially all processing the same data! Ooops, the boss just got his registration email ten times.

So you could implement some sort of locking mechanism to prevent multiple instances but it’s all starting to get a bit fragile with lots of interdependencies – all you want to do is run a simple script!

Write a daemon in PHP

So it’s back to the drawing board – what is needed is a daemon that will sit in the background and run as script continually, restarting it immediately after it finishes. You could write a daemon in PHP – this would enable you to write a daemon which would interface well with your existing PHP scripts, there are even some good PHP packages which make writing a daemon in PHP easy.

So, full of enthusiasm you launch into writing your own daemon in PHP. A week or so later, after reading all about process control, (and probably pulling large chunks of hair out in the process as yo realise it wasn’t as simple as you thought), you finally have something that works. You deploy it to your website, sit back and feel smug.

After a week your boss rages into your office demanding to know why nobody has registered in the past week. Red faced, you go to check your PHP daemon that sends emails form the email queue is still running – and it isn’t. If you haven’t already been fired then your next step would probably be to find out what happened. Looking through your PHP error log you find a “FATAL ERROR” from your daemon script, dated exactly one week ago.

PHP can throw fatal errors for all sorts of reasons, and there’s no way to recover from them. Also, PHP is renowned for its rather flaky memory management – whilst this is fine for web scripts that render a page and then end, it makes it highly unsuitable to write applications.

Enter The Fat Controller

How can we solve the problem? We need to run PHP scripts as a daemon. I faced this problem one year ago and I solved it by writing The Fat Controller. It is a program written in C that runs a daemon and can continually run PHP scripts. As it is written in C, it is highly stable and can run for months or years without problem. As the daemon runs separately from the PHP scripts, no matter what happens in the PHP script, it does not affect the Fat Controller.

The Fat Controller is also very flexible, easy to install and configure and supports running multiple instances of a PHP at the same time to achieve parallel processing. You can even configure it to dynamically control the number of parallel processes dependent on how much work there is.

You can read more about The Fat Controller here   http://www.4pmp.com/fatcontroller/

A daemon is a process which runs in the background of your computer, periodically carrying out a specific task. The following is an example of a simple daemon written in C. It works by forking to create a child process, the parent then terminates but the child carries on in the background – entering a continuous loop of doing a task and then sleeping. The child process is of course an identical copy of the parent so care must be taken to close all file descriptors, thus detaching the child completely from the calling process. The deamonised process is controlled by sending it signals which it can catch and take action accordingly. In the example below, the process is terminated by sending SIGINT or SIGTERM, but you can of course add in your own handling – for example to re-read config data on SIGHUP.

#include <stdio.h>
#include <signal.h>
#include <syslog.h>
#include <errno.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>

#define DAEMON_NAME "simpledaemon"

    void daemonShutdown();
    void signal_handler(int sig);
    void daemonize(char *rundir, char *pidfile);

    int pidFilehandle;

    void signal_handler(int sig)
    {
        switch(sig)
        {
            case SIGHUP:
                syslog(LOG_WARNING, "Received SIGHUP signal.");
                break;
            case SIGINT:
            case SIGTERM:
                syslog(LOG_INFO, "Daemon exiting");
                daemonShutdown();
                exit(EXIT_SUCCESS);
                break;
            default:
                syslog(LOG_WARNING, "Unhandled signal %s", strsignal(sig));
                break;
        }
    }

    void daemonShutdown()
    {
        close(pidFilehandle);
    }

    void daemonize(char *rundir, char *pidfile)
    {
        int pid, sid, i;
        char str[10];
        struct sigaction newSigAction;
        sigset_t newSigSet;

        /* Check if parent process id is set */
        if (getppid() == 1)
        {
            /* PPID exists, therefore we are already a daemon */
            return;
        }

        /* Set signal mask - signals we want to block */
        sigemptyset(&newSigSet);
        sigaddset(&newSigSet, SIGCHLD);  /* ignore child - i.e. we don't need to wait for it */
        sigaddset(&newSigSet, SIGTSTP);  /* ignore Tty stop signals */
        sigaddset(&newSigSet, SIGTTOU);  /* ignore Tty background writes */
        sigaddset(&newSigSet, SIGTTIN);  /* ignore Tty background reads */
        sigprocmask(SIG_BLOCK, &newSigSet, NULL);   /* Block the above specified signals */

        /* Set up a signal handler */
        newSigAction.sa_handler = signal_handler;
        sigemptyset(&newSigAction.sa_mask);
        newSigAction.sa_flags = 0;

            /* Signals to handle */
            sigaction(SIGHUP, &newSigAction, NULL);     /* catch hangup signal */
            sigaction(SIGTERM, &newSigAction, NULL);    /* catch term signal */
            sigaction(SIGINT, &newSigAction, NULL);     /* catch interrupt signal */

        /* Fork*/
        pid = fork();

        if (pid < 0)
        {
            /* Could not fork */
            exit(EXIT_FAILURE);
        }

        if (pid > 0)
        {
            /* Child created ok, so exit parent process */
            printf("Child process created: %d\n", pid);
            exit(EXIT_SUCCESS);
        }

        /* Child continues */

        umask(027); /* Set file permissions 750 */

        /* Get a new process group */
        sid = setsid();

        if (sid < 0)
        {
            exit(EXIT_FAILURE);
        }

        /* close all descriptors */
        for (i = getdtablesize(); i >= 0; --i)
        {
            close(i);
        }

        /* Route I/O connections */
        close(STDIN_FILENO);
        close(STDOUT_FILENO);
        close(STDERR_FILENO);

        chdir(rundir); /* change running directory */

        /* Ensure only one copy */
        pidFilehandle = open(pidfile, O_RDWR|O_CREAT, 0600);

        if (pidFilehandle == -1 )
        {
            /* Couldn't open lock file */
            syslog(LOG_INFO, "Could not open PID lock file %s, exiting", pidfile);
            exit(EXIT_FAILURE);
        }

        /* Try to lock file */
        if (lockf(pidFilehandle,F_TLOCK,0) == -1)
        {
            /* Couldn't get lock on lock file */
            syslog(LOG_INFO, "Could not lock PID lock file %s, exiting", pidfile);
            exit(EXIT_FAILURE);
        }

        /* Get and format PID */
        sprintf(str,"%d\n",getpid());

        /* write pid to lockfile */
        write(pidFilehandle, str, strlen(str));
    }

    int main()
    {
        /* Debug logging
        setlogmask(LOG_UPTO(LOG_DEBUG));
        openlog(DAEMON_NAME, LOG_CONS, LOG_USER);
        */

        /* Logging */
        setlogmask(LOG_UPTO(LOG_INFO));
        openlog(DAEMON_NAME, LOG_CONS | LOG_PERROR, LOG_USER);

        syslog(LOG_INFO, "Daemon starting up");

        /* Deamonize */
        daemonize("/tmp/", "/tmp/daemon.pid");

        syslog(LOG_INFO, "Daemon running");

        while (1)
        {
            syslog(LOG_INFO, "daemon says hello");

            sleep(1);
        }
    }

The above example is only simple but it serves as a good starting point to create your own daemon. In my next article I will demonstrate how it can be extended to periodically call a PHP script.

If I find enough time I will also demonstrate how it can be extended from being a single-threaded daemon to a multi-threaded daemon, maintaining a continuous pool of x threads, each doing a given task.