Consider a car, generally speaking the efficiency (i.e. miles per gallon) increases with speed, up to a point, after which the efficiency drops off. The same is true for web applications, or indeed any applications supporting concurrent processing. Generally, the number of requests that can be served per second increases with the number of concurrent requests, up to a point, after which it decreases.

Problem

As more people use a site, the number of concurrent requests being handled by the server increases. The response time remains relatively stable up to a point, let’s say R_C, after which the response time goes up and eventually everything grinds to a halt.

Situation

A properly set-up server will hold all the information it needs to serve pages in memory and so the greatest factor affecting the number of concurrent requests it can serve is the processor. If a processor has, let’s say 16 cores, it will be able to effectively handle up to 16 concurrent requests, after which the throughput will decrease due to overheads caused by handling the threads.

For application servers this is easily resolved by buying more servers and distributing the requests between them using a load balancer. If you have a database then scaling becomes more complicated and the database generally will be the bottleneck in your system.

A benchmark published on mysqlperformanceblog.com clearly show how the throughput of a MySQL database varies with the number of concurrent requests on a 16-core server:

Reference: http://www.mysqlperformanceblog.com/2010/02/28/mysql-5-5-m2-scalability/

Solution

As the database is generally always the bottleneck, we need to ensure that the number of concurrent requests (R) does not exceed the critical number R_C. By looking at the above graph, we could satisfy the requirement by limiting the maximum number of concurrently processed HTTP requests to the number of CPU cores:

R_max = R_C

This basic form of controller is an open-loop control whereby it does not monitor the output to ensure the system is operating correctly.

An example of open-loop controller is a coffee machine. Providing the cups are all the same size, it will fill them with coffee. If the cup is too big then the cup will not be full, if the cup is too small then it will overflow.

Open-loop control is suitable for well-defined systems, operating under constant conditions or not required to adapt to change. Consider tuning a guitar; the tuning pegs are turned until then string tension is such that each string resonates at the required frequency. Providing nothing changes, the strings will remain in tune. If the strings heat up, the string tension is reduced and now resonate at lower frequencies.

Providing the graph holds under all circumstances then open-loop control would suffice. Unfortunately the test above is not indicative of the general case but rather an unrealistic ideal. Although the sysbench test includes read and write operations, it should be noted that the test was done using a solid-state HDD which of course can handle multiple concurrent IO operations. Normal HDDs support only one thread and so operation which need to use the HDD, write operations or read operations for data not cached in RAM, will have an adverse affect on concurrent performance.

The crux of the problem

Generally, most database operations are read operations and, providing the database is configured optimally, it will not touch the HDD. In this case we can get optimum concurrent performance, as shown in the graph. If the application is designed correctly, most HTTP requests will not require the database at all, thus removing the database bottleneck and so concurrent performance will be determined at the application layer. As mentioned earlier, the application layer can be distributed across multiple machines, achieving still greater concurrent performance.

As such, the maximum number of concurrent requests the system can optimally handle varies on the nature of the requests – just as the ability of the coffee machine to fill a cup depends on the size of the cup. Clearly, the open-loop control mechanism described earlier is not sufficient.

Feedback is required in order to regulate the number of concurrent requests by monitoring the system’s ability to process them. This is known as a closed-loop controller. The output of the system is fed back to a controller where it is compared with the required value, the result of which is used to regulate the input to the system in order to bring the output closer to the required value.

Proposal

It is proposed to create an HTTP proxy which regulates the rate at which it proxies requests to the web application. Feedback is to be used to restrict the the number of concurrent requests to ensure maximum throughput for the given load in terms of both volume and nature.

Requests which cannot be immediately processed are to be either held in a queue or returned immediately with a basic HTML page informing the user that the request cannot be processed at the moment and that they should try again soon.

Peaks, by their very nature are short lived affairs and by ensuring that the server is always operating at its maximum capacity, the effect of the peaks is managed and reduced. Furthermore, the proxy will prevent a surge of requests from crashing the system – all users who wait for a request to return are guaranteed that they will not wait in vain, only to see “ERROR 500” – or, to use a technical term, to prevent a blowout.

SELECT * FROM `myBigTable` LIMIT :OFFSET, :ROW_COUNT;

After each batch of rows is processed, the offset is increased by the size of the batch and the process is repeated until all rows have been processed.

Now that’s not exactly rocket science, but there is a problem – as the offset increases, the time taken for the query to execute progressively increases, which can mean processing very large tables will take an extremely long time. The reason is because offset works on the physical position of rows in the table which is not indexed. So to find a row at offset x, the database engine must iterate through all the rows from 0 to x.

The obvious solution would be to use the primary key instead of offset:

SELECT * FROM `myBigTable` WHERE `id` > :OFFSET LIMIT :BATCH_SIZE;

The problem here is that MySQL doesn’t guarantee that rows appear in the table in the same order as the primary key, so you could end up processing some rows twice, and worse still, some rows not at all. Luckily this can be easily solved by using ORDER BY:

SELECT * FROM `myBigTable` WHERE `id` > :OFFSET ORDER BY `id` ASC;

This might seem a bit counterintuitive using ORDER BY when we want to speed things up, but remember we’re ordering on a uniquely indexed column which will be fast anyway, and compared to the alternative of iterating through each row using an offset, it’s a huge improvement.

Conclusion

The general rule of thumb is “never use offset in a limit clause”. For small tables you probably won’t notice any difference, but with tables with over a million rows you’re going to see huge performance increases.

Remember, this doesn’t just apply to iterating through tables but whenever offset is used. For example implementing pagination of large tables, if the first few pages load quickly but subsequent pages load progressively slower then it’s likely this is the cause (or you are missing some indices!)

Lost connection to MySQL server during query

The MySQL manual (http://dev.mysql.com/doc/refman/5.0/en/gone-away.html) says that it can be caused by:

applications that fork child processes, all of which try to use the same connection to the MySQL server

Which was exactly our case as we were creating a benchmarking tool that created child processes to concurrently query the website.   So we checked the code to make sure that no database connections were shared between processes, and also that all connections were closed immediately after being used… but we still got the same error.

After a few minutes of scratching our heads, it dawned on us that we were using persistent connections, and of course when connection objects are destroyed the actual connections are not closed but are returned to a pool of connections, ready for re-use.

We changed the connections to non-persistent and Bob was our proverbial uncle!