Scaling up with GoGrid

My small GoGrid experiment the other day made me curios. Assume the blog I put online became really popular. How would my deployment cope, and how would I be able to increase capacity?

So I figured I'd script a small benchmark. Assume all visitors land on the Blog's front page, and that half of the visitors click on the first post to read it comments. Finally, assume a third of those add a comment of their own. Translated into a bash script that uses Apache ab, this scenario might look something like this:

#/bin/bash

AB="ab"
BASE=http://216.121.78.66/blog/index.esxx

function bench {
start=$(date +%s)

${AB} > /dev/null -n 600 -c 60 -e apa ${BASE} &
pid1=$!

${AB} > /dev/null -n 300 -c 30 -e apa ${BASE}/posts/1.html &
pid2=$!

${AB} > /dev/null -n 100 -c 10 -T application/x-www-form-urlencoded -p ab-blog.post ${BASE}/posts/1.html
pid3=$!

while [ -n "$(ps | grep -E "^ *(${pid1}|${pid2}|${pid3})")" ]; do
sleep 0.1
done

end=$(date +%s)

echo "Total time: $(expr ${end} - ${start})"
}

bench
bench
bench
bench
bench

As you can see, I assume 100 concurrent requests.

On my particular configuration (a 1 core, 0.5 GB RAM virtual GoGrid server in San Francisco and an OpenSolaris laptop in Sweden acting as the client), it turns out that the benchmark takes about 19 seconds per iteration, or 19 ms/request.

It's not too bad, but lets see if we can improve. First of all, we should switch from an embedded to a standalone SQL database. By watching 'top' on our single server, it appears that the ESXX Java process is using up all the CPU; however, since the DB is embedded, we don't know what is the limiting factor; it could be the XSLT transformation, the JavaScript code or the SQL queries.

Since the DB can't easily be scaled horizontally, we deploy a 3 core, 4 GB RAM virtual server and start H2 in server mode on that server, using the following command line:

java -cp h2/bin/h2-1.1.107.jar -Dh2.bindAddress=10.102.228.78 org.h2.tools.Server -tcp -tcpAllowOthers

We then modify the following line in /var/www/ajax-blog/blog.js, to make it connect to the external H2 server instead of using H2 in embedded mode:

var blog = new Blog("jdbc:h2:tcp://10.102.228.78/~/Blog;AUTO_RECONNECT=TRUE", "admin", "secret");

Re-running the benchmark reveals that the SQL database sits mostly idle. That's hardly surprising. After all, how much CPU can a few SELECTs from a 100-something row table use? Knowing this, I could have just started the H2 database as a process on the initial web node, but it's always easy to be wise after the event.

Anyway, now we know how to improve the performance. Let's deploy another web front-end, add a load balancer and rerun the benchmark. The result? Almost twice the performance, or 10 ms/request!


And all you need is a web browser, an SSH client and a credit card. The Internet is truly an amazing place.

PS. With four front-ends, we're no longer CPU limited and probably need to tweak the Apache configuration (it's forking like crazy) or ESXX's thread pool settings to obtain further performance improvements.