I’ve been receiving a lot of alerts from my monitoring system for this blog so I thought I’d do a bit of investigation. I checked my monitoring data to see if there was a trend in performance degradation and sure enough I found it in a graph that lays out the page load time and errors. Here’s what it looks like:
Performance Metrics
It doesn’t show any major overall performance increase (that is a left to right increase in the value of the dark green line) but I do notice that errors have started to increase with a culmination of 31 errors on Tuesday (11/4). Though I don’t have any analytics data to prove this I’m pretty sure it had to do with all the blogging (and blog reading) that was going on nationwide with the presidential election. All that traffic on WordPress certainly would have affected my blogs performance (or at least mine and every other blog that is on the same hardware). It’s not just WordPress but other sites containing news about the election were hammered with users looking for election results (here’s an article on Akamai’s traffic for yesterday).
When I drilled down into the monitoring results I found that the errors I was experiencing were due to timeout errors. In my monitoring software i’ve established a threshold for performance of 3 seconds. That means that my page and all its content must be completely downloaded by the browser (IE7) within 3 seconds. When I established this threshold I had been monitoring the performance of the page for a week and saw that the average performance was around 1.8 seconds. Two factors are making me reconsider that threshold and these are two factors that everyone providing a web service should think about and adapt to:
- Content (particularly in the world of blogging) will be added to a web page/application overtime. For example, this blog has nearly doubled in size (when I first started blogging the page download was 177951 bytes, now it’s 366514 bytes). This is going to affect the performance of the resource (another example, adding Ajax to a website will require more JavaScript that will need to be downloaded). Future growth needs to be considered when setting expectations for performance.
- Outside factors will always influence your performance, that’s why external monitoring is a must for web resources. I should have seen it coming a mile away, this is a classic example of current events or major news events drawing hordes of web users to servers that can barely contain them (ok, ok…too dramatic, in all reality it doesn’t look like the WordPress servers were in any danger of failing). Ultimately it brings up a very important issue. Know what shared resources in your application can make you vulnerable to other tenant’s traffic.
These considerations will make your SLA (or any agreement you may have with users of your application) stronger and more robust. It’s is definitely not recommended to continually adjust your SLA. The SLA is a pact with your users and it’s best to have it remain a strong/unchangeable core. Changing your SLA every week to keep up with performance degradations is counterproductive and at that point it doesn’t really make sense to even provide an SLA. So I have two options: first I could change my SLA/Timeout value from 3 seconds to 4 or 5 seconds, this would definitely reduce the number of alerts I’m receiving (it would also weaken my stance on what I promise to my readers). Second, I can keep my Timeout value at 3 seconds and see what happens with the performance of the site (and even proactively improve the performance of the site). At this point I’m not too concerned about the alerts and I think that as the election hype dies down I will see a reduction in alerting (which is the case over the last 24 hours). It may not even be necessary at this point to make any improvements to the site. Though it’s certainly something that will need to be considered in the future as I continue to post content.
