The Webmetrics API plug-in for Nagios is a Perl script that will access the realtime.getdata API from Webmetrics.  The script requires 3 parameters be passed to it; the API Key, Username, and Service ID.  The API Key can be found by logging into your Webmetrics account and then clicking on “My Account” in the top right hand corner.  You will then see an API tab.  This API Key should be kept secret as it will allow full access to your account.  The Service ID is able to be found by clicking on any service in your account.  The URL in the address bar will show account=XXXXXX.  These numbers are your Service ID.

Here are some examples of the integration with  Nagios XI:

Service in OK status:

To see a service in Warning or Error status please click on the following links:
Warning
Error

Service Performance Metrics:

Webmetrics Graph:

Below are some examples of the integration with Nagios:

There are 5 possible outputs from this script.

1. OK status showing the load time as well as the time the sample was taken.   Exit code = 0.
2. Warning status showing the time the sample was taken.  Exit code = 1.
3. Error status showing the time the sample was taken.  Exit code = 2.
4. Unknown sample status is shown for any unexpected results.  Exit code = 3.
5. Error message returned from the Webmetrics API.  Exit code = 3.

You will need to download the Nagios Perl script located at: http://download.webmetrics.com/nagios/nagios.pl.  The following Perl modules are also needed to ensure the script works properly:

LWP::Simple
LWP::UserAgent
HTTP::Request
HTTP::Response
XML::Simple
Getopt::Long
Digest::SHA1 qw(sha1_base64)
DateTime
DateTime::Format::Strptime

Once this is done, ensure you can run the Perl script without any errors:

/usr/bin/perl /usr/local/Nagios/libexec/nagios.pl –api_key=1234567890abcdefg –username=sample –serviceID=123456.  After running the command, you should see output like the following:

“SiteMonitor Load Time = 1.24, Sample Taken At: 15:37:46.01 US/Pacific | load_time=1.24”

If you see this type of output, the script is ready to be inputted as a plug-in into Nagios.

Setup Nagios XI:

First, log into the web interface.  Click Configure -> Enter Nagios Core Configuration Manager -> Enter Credentials -> Click the arrow for Commands -> Click Commands -> Click Add New.  You can give your command any name, but for the purposes of this article, we will call it Webmetrics-API.  For the Command Line option, you will want to give the full path to Perl and the full path to the Perl Script.  In this example, I used: /usr/bin/perl /usr/local/nagios/libexec/nagios.pl -api_key $ARG1$ -username $ARG2$ -serviceID $ARG3$.  Click Save.  Then click Apply Configuration.  Then under Monitoring, click Hosts -> Add New.  Provide a Host Name, an Address (127.0.0.1), and a Description.  Click on the Check Settings tab and fill out all fields with a *.  Click Save.  Under Monitoring, click Services -> Add New.  Type in the Webmetrics Service Name as the Config Name.  Select the Host Name that you just created in the previous step and provide a Service Description.  Under Check command, choose Webmetrics-API.  Under $ARG1$ enter the API Key.  Under $ARG2$ enter the Webmetrics Username.  Under $ARG3$ enter the Service ID.  Click on the Check Settings tab and fill out all items with a *.  Click Save.  Repeat this step for all services that you want to monitor.  Click Apply Configuration.

Now that the host and services are setup, click on the Views tab and then Service Detail.  Note: It may take a couple of minutes for the services to show up.

Setup Nagios:

The free version of Nagios is a little more complex to setup as you have to manually edit the configuration files on the server that you have it on.  If you are running the free version, I’m going to assume that you are aware of how to edit the configuration files and how to verify the configuration file has no errors, and how to restart the service.  If you are unaware, please refer to the Nagios help documentation.  You will need to create a Host Name, Service Name, and a Check Command for the Webmetrics API.  I wasn’t able to find a way to utilize the $ARG#$ variables, so I just created a check command for each service name and that worked as expected.  The command will be the same command that you tried earlier to verify the Perl script was working as expected.  Just change the Service ID for each service you want to monitor.

So why does Google think speed is of importance? Through the following tests:

Side by side testing of optimized vs Original versions of the sites:

Only difference is that the optimized version was faster but the content remained the same in both. Firefox and Shopzilla had some interesting findings.
Shopzilla had 7 – 12% increase in conversions and a 50% decrease in operating costs.
Firefox reported a 15.4% increase in downloads. Firefox estimates that this is about 60 million extra downloads.

User satisfaction test:

Google and MicroSoft conducted a user satisfaction test by returning delayed results to some users. Results showed that just about half a second delay caused 1% of the users to feel dissatisfied. As the delay increased to one second, the dissatisfaction rose to 4%. As the test continued, users were feeling so dissatisfied that Microsoft ended the test fairly quickly.

Google & Microsoft conducted another experiment. They found that a 400 ms delay in the rendering the search results reduced the query volume drastically and this trend continued for 7 weeks. In fact the query volume didn’t get back to normal until 11 weeks after the experiment was discontinued. So this speaks for itself how much a site’s speed plays a role in determining its conversions.

While content and relevance are still primary in determining a site’s rankings in SERPS, site speed plays a major role too now.

According to Steve Souders, a Google Employee and the author of ‘High performance Web sites blog’, the Performance Golden rule is that “80 – 90% of the end-user response time is spent on the front-end. Start there”.

How is that? Look at a Waterfall chart from webpagetest.org’s performance analysis of a web page.

It took under one second for the content to be returned by the web server. Close to one second, the browser started to render the content. But then it had to make all these requests to all the associated page components like the javascript, css, images. So the final page didn’t load until after 7 seconds. This proves conclusively that the front-end of a page needs a lot of attention and optimization.

Google’s webmaster tools’ site performance dashboard not only gives the average load time of a site’s pages but also provides a comprehensive comparison of the speed with other web sites on the web. So if your site performs better than 95% of the sites on the web, then turn your attention to the content. If not, make speed your priority.

‘Page Speed’ is yet another powerful tool to analyze a page’s bottlenecks. Page Speed is a firefox plugin that provides a score for each page based on its analysis and also provides its recommendations. People have found to decrease their load times by 5 seconds on an average following these recommendations.

What is a good response time to aim for?
According to Google, Following your competition is the best answer to this question. If your competition provides a better user experience then you would want to aim for a better and a reliable user experience for your audience. Akamais’ own case studies show that 2 seconds is the threshold for e-commerce acceptability. Google’s aim is however under half a second.

What other tips did Google provide?
Implement Progressive Rendering – Progressive rendering is when a browser can display content as it’s available incrementally rather than waiting for all the content to display at once. This provides users faster visual feedback and helps them feel more in control.
Bing found a 0.7% increase in customer satisfaction with progressive rendering.

How to implement it? Simple. Put stylesheets at the top of the page. This allows a browser to start displaying content ASAP.

1. Internal web monitoring – Provides monitoring of systems applications, and network connections.  Because there are no major players out there the resources do not exist to allow for monitoring from an end users perspective.  Performance monitoring is a very subjective process where an end user has to describe how the perceive the performance of a website/application.
2. External web monitoring – Starts to provide information about the performance of a website/application but it is rudimentary and usually the performance data is based off of a program that only hopes to emulate the performance of a browser against the site.  No actual browser is used for monitoring.
3. External RIA monitoring – Applications are becoming more dynamic and rely on the capabilities of the browser to display the page more like a desktop application.  This requires that external monitoring take these pieces of behavior into account when considering the performance of a web application.

That’s where we’re at right now.  I think a logical next step would be:

• Generic External RIA monitoring – This would be monitoring that would consider any (and every) platform possible for viewing the web application.  For simplicity just assume that it’s every browser on the market (Chrome, Safari, IE6, IE7, IEX, Firefox 2, etc.).

The browsers mentioned seem to be diverging and are trying to grab more market share by adding new functionality to their offerings.  So how does having performance data based on Chrome help you determine what your performance would be in IE7?    For the time being there really is no need for such distinctions…the important factor is that you have an actual browser that is generating your performance metrics.  Because of price and implementation I would suspect that monitoring solutions are going to become more decentralized.  Also, with all this new functionality there is going to be an increase in the different methods that are used for delivering functionality.

Generally this is a downloaded application that has two modes, record and playback (A co-worker of mine often compares it to the macro recording functionality in the Microsoft Office suite of products).  You essentially interact with the Script Recorder as though it is a browser and your actions are recorded in the background.  The playback mode will take the recorded script (or a previously recorded script) and process it, allowing the Script Recorder to playback the steps that were taken in a browser.  Playback is intended for verification.

Very rarely does the Script Recorder become a crucial need with performance monitoring.  The goal is to generate scripts, and if there is an easier way to do it without a Script Recorder then it’s irrelevant if a Script Recorder even exists.  Though some power users prefer to have a Script Recorder available if their scripts need constant updating, if they want to perform some low level QA, or if they just want to have more control over the process of the scripting.

To help in the process of researching Script Recorders I have identified some requirements that are based off of monitoring best practices.  Remember though, the Script Recorder is only a byproduct of the monitoring platform and should not be the primary deciding factor when looking for a monitoring solution.  Here are some desirable requirements of a Script Recorder:

1. Flexibility – Script Recorder should be easy to use and allow for the ability to enter custom code/steps/instructions by hand if necessary.  The creators of a Script Recorder generally do not account for all the problems that someone may encounter when scripting, therefore it is beneficial to not limit script generation to just what the browser itself can do.  Allowing the user to enter in custom instructions can be helpful particularly when trying to clean-up before or after a script is run.
2. Fidelity – Record functionality should be capable of recording asynchronous (and non-interactive) events such as Ajax/JavaScript.  This events are usually invisible to a user and therefore to monitoring.  If a Script Recorder is worth its salt then it should be able to track these events.
3. Completeness – Playback functionality should be capable of playing everything back.  The monitoring platform will have to play it back so if there is a major discrepancy in playback between the Script Recorder and the actual monitoring platform then that is a sign that things are all copacetic.
4. Openness – Ideally an open source platform is available.  The script generated by the Script Recorder should be easily portable (open-source) and easy to understand.  Immediately you have the support of an open source development community.  If your script is open source then you’re not tied down to the implementation of the Script Recorder vendor.  This will make it easier for you to up and leave if your needs change (or if you just don’t like your vendor!).

perf-a-lytics mashup

The graph shows the performance of the application (the dark blue line) graphed against the number of page views (the light green bars).  Normally, either of these data sets would be good data in their own right.  But each set of data alone leaves certain questions unanswered.

For example, if we consider only the analytics data we have answered the question How many users are on my site at any given time? But a very important follow up question that remains unanswered is What impact does that have on my site’s performance? The addition of the performance data answers that second question.  The performance data shows us that there is a considerable increase in page load time (roughly 4x more time to load the same page).

What about if we consider the performance monitoring data alone?  In this case we answer the question What is the performance of my application at any given time? We can tell what our performance is but there the question that goes unanswered is; What impact does that have on how people use/access my site? The answer that we get by adding the analytics data is that people tend to leave the site when the performance degrades, that’s exactly what we see in this graph, as the users leave the site (most likely upset) the site calms down because of fewer requests.  One other thing to note is that we also see what is the cause of a performance degradation.  We see that the increase in users has resulted in an increase in page load time (because the server has to use the same amount of resources to fulfill more requests).

This is just a small example of what can be done by combining data sets.  I would recommend that whenever you look at gathering data with a tool (whether it be analytics, performance data, etc.) you should make sure that an API for accessing the data is available.  This will allow you to get more value out of your tool and also will open new doors to you as you add tools in the future.

Sample mashup courtesy of Lenny.

• Alerting – This should always be first.  A good portion of the value in performance monitoring comes from being able to evaluate the data collected.  But, the most valuable functionality comes from being able to proactively alert yourself as problems are occurring.  This allows you to reduce downtime, meet SLAs, and investigate problems as they’re occurring.  Central to this is a monitoring frequency that will help accomplish this.  If you only monitor once an hour then there is going to be some noticable lag in how quickly you can respond to a problem.  If you monitor more frequently (ex: every 5 minutes) then you can react relatively quickly.  And even though every 5 minutes is the industry standard, monitoring once every 1 minute can of even more value for your mission critical applications.
• Reporting – This is key.  Outside of alerting, all that you have at the end of the data is a set of data and being able to draw conclusions from this data and make educated decisions about the performance of you site/application is key.  Your data set needs to be robust because you cannot go back after the fact and collect more data if you feel your data set is too sparse (you can adjust your settings to give you more data in the future but as for the past, that opportunity to collect data has come and gone).  Just like alerting, reporting requires a certain interval.  For example, you would not want to collect data once an hour and then use that data set to make adjustments to your site.
• Monitoring Client capabilities – This one is a bit more abstract but can be summarized fairly easily.  The monitoring client is what performs the monitoring, collects the data, and perceives.  Normally such a client is implemented as either an emulation of a browser (a program that makes HTTP requests) or is an actual browser that has a framework around it to automate interactions.  The actual browser implementation of this client is the most desired solution because it’s the closest representation of what your end users are experiencing (thus it indirectly makes your data set all the more accurate).  The emulated browser is generally better fit for the cost conscious monitoring consumer as it’s only an approximation of what your end user experiences (of the deficiencies of an emulated browser, lack of JavaScript support is probably the most egregious).

So there it is….when you’re shopping around for a monitoring solution you should consider the above three values.  You can then take those and sort them in order of importance for the objectives you plan to achieve with monitoring.  I generally consider the monitoring client to be the most important factor because it generates the data that is going to be by all other monitoring components (the data collected is more accurate and representative of your users which in turn makes your reports more valuable and makes your alerts more accurate).

Recently I had a conversation with a co-worker of mine and he asked about the importance of monitoring from an actual browser (that is, automating a transaction for the purposes of performance and availability measurements from an actual browser vs. an emulated browser), whether it was necessary, and if necessary, what type of browser should be used.

Well, the answer is that it depends.  It all depends on what your goals are.  I’ve talked about this before and the reality is that you only need to test from an actual browser if your goal is to get very accurate metrics on your end users experience.  Furthermore you want to match that end user perspective with your actual end users…therefore, if the majority of your users are accessing your application with IE then you you should consider a monitoring solution that uses IE.  If they access the site with FF more frequently then a solution using FF should be considered.  That’s not a hard and fast rule (unless you’ve developed your site for a specific type of browser – which happens quite often with IE) the important thing is that when you’re trying to get accurate user representation you should use an actual browser.  If you’re not trying to get actual user representations with your monitoring then don’t stay up at night worrying about the browser type used (in fact you’ll probably want to use an emulated browser solution which will be cheaper).

So why the blog post?  The conversation progressed on into a debate about browser neutrality.  In other words, what if someone wanted an accurate end user perspective that doesn’t favor one type of browser over the other?  Ultimately this is not a good situation to be in because you’re going to make so many trade offs that your initial objective will not be met.  Here are your objectives:

1. Primary: Get an accurate end user perspective.
2. Secondary: Use a browser independent monitoring solution.

The primary objective is easy to accomplish by using a monitoring platform that utilizes an actual browser (you’ll need to consider what type of browser is important to you: IE, FF, Safari, etc.).  The secondary objective is self defeating and undermines the primary objective.  There is no way to take the standard parts of FF, the standard parts of IE, the standard parts of Safari, and every other browser on the market and somehow create a super browser that will provide you with an accurate end user perspective.  In order to provide such a solution the browser would have to be re-built by hand and incorporate different (which would in turn move you further away from any real world user experience).  In other words, as the neutrality of the browser increases the accuracy of the end user perspective decreases.

It’s just one of those things, the two previously mentioned objectives are inversely correlated…addressing one of them will negatively impact the other.  It’s an important lesson, if you’re going to go after accurate end user perspectives with monitoring then you’ll have to abandon any notion of a browser neutral environment.  When monitoring, always define your objectives and give them weight, be ready to analyze dependencies between them and be willing to make concessions (or at least know what concessions you’re comfortable with).

It requires a similar concern because the more you monitor the more it costs.  However, in this case there is a bit more up front analysis that you have to do in order to determine the scope of what is to be monitored.  In order to monitor a web based application a script is required to tell the monitoring platform what steps to perform and how to interact with the application.  The steps that make up that script constitutes a Transaction and directly impact the cost of monitoring (most monitoring solutions will use a metered billing approach which uses a single credit per step of a transaction).  Now we can see that there is a direct correlation between how much a monitoring solution costs and the scope of that monitoring (number of steps goes up, so does the price).  It’s important to note that even if metered billing isn’t used, most monitoring platforms have a concept of increasing the price of the solution as the number of steps increases (ex: $100 for 1 to 3 steps, $200 for 4 to 6 steps, etc.).  It’s just a fact of life, those cost increases are sometimes to recoup processing power of performing the steps but mostly the increase is to offset the costs of storing, backing up, and reporting data.  So, how does one save money?  One defines a transaction as only the steps necessary to test functionality.  Often the following mistakes are made:

• Monitoring the mundane (or for the wrong reasons).
• Monitor duplicate functionality (too much).
• Monitor too little (taking these recommendations too far).

Let’s look at each of these situations and see how they impact costs as well as how they affect the bottom line (collecting actionable monitoring data).  In each case we will consider only monitoring of transactions:

Monitoring the mundane – This is generally the product of an organization that hasn’t thoroughly thought out the goals of monitoring.  A transaction that I would consider mundane is one that doesn’t really have an end goal and just meanders around the website.  For example, a transaction that clicks through each menu item in the left nav bar is probably mundane.  Sure there’s an argument for why to do that, maybe lots of revenue is generated from the left nav bar, or maybe that’s the only navigation available for the site.  But in actuality this is really a QA problem and should be addressed as such.  It’s common in the field of computer science that the later a problem is discovered the more it’s going to cost to fix it.  Which is definitely the case here: A QA process that occurs right after development could have caught any broken links or JavaScript funkyness more efficiently then a costly monitoring solution after code has been deployed.

Monitor duplicate functionality – Sometimes this is a hard one to get around.  But basically you need to make sure that your monitoring transactions are mutually exclusive.  Don’t monitor the updating of a web based calendar in two separate transactions when one will do.  Another case is when similar methods are invoked in a single transaction.  For example, if you have a tool that configures a product and does so in 20 steps it’s probably overkill to perform all configurations (since they all probably access the same front-end and back-end functionality).  Have the transaction perform a couple of configurations and then complete the transaction (i.e. purchase, or whatever the result of the transaction is).  In this last case the duplicate functionality is a bit obscure…on the front-end the functionality looks different (configure a tire vs configure a stereo) but on the back-end the functionality is more than likely the same (accessing the same database through the same web service) and therefore all you’re really doing is testing more client side code execution (which probably should have been done during the QA process again).

Monitor too little – If you start to get too carried away with the recommendations I’ve made you could end up shooting yourself in the foot.  For example, in the last section I gave the example of an application that configures a product, furthermore that application uses the same back-end technology for each step of the configuration.  But it very well could be the case that third party functionality is embedded in the configuration tool (one step could be hosted by you while another step makes a request to a third party).  In that case maybe it does make more sense to monitor additional steps (though it could probably be monitored more efficiently by breaking out that third party contents monitoring into it’s own monitoring service).  The end result is that you’re looking for efficiencies in monitoring that will help reduce cost while NOT altering the data set you expect to get from the monitoring.

To summarize, you want to focus your monitoring so that you can achive your goals in improving performance without creating convoluted and expensive data sets.  Also, you want to be aware of not getting to zealous with efficiency and stripping your dataset of all its value.

Performance Metrics

It doesn’t show any major overall performance increase (that is a left to right increase in the value of the dark green line) but I do notice that errors have started to increase with a culmination of 31 errors on Tuesday (11/4).  Though I don’t have any analytics data to prove this I’m pretty sure it had to do with all the blogging (and blog reading) that was going on nationwide with the presidential election.  All that traffic on WordPress certainly would have affected my blogs performance (or at least mine and every other blog that is on the same hardware).  It’s not just WordPress but other sites containing news about the election were hammered with users looking for election results (here’s an article on Akamai’s traffic for yesterday).

When I drilled down into the monitoring results I found that the errors I was experiencing were due to timeout errors.  In my monitoring software i’ve established a threshold for performance of 3 seconds.  That means that my page and all its content must be completely downloaded by the browser (IE7) within 3 seconds.  When I established this threshold I had been monitoring the performance of the page for a week and saw that the average performance was around 1.8 seconds.  Two factors are making me reconsider that threshold and these are two factors that everyone providing a web service should think about and adapt to:

1. Content (particularly in the world of blogging) will be added to a web page/application overtime.  For example, this blog has nearly doubled in size (when I first started blogging the page download was 177951 bytes, now it’s 366514 bytes).  This is going to affect the performance of the resource (another example, adding Ajax to a website will require more JavaScript that will need to be downloaded).  Future growth needs to be considered when setting expectations for performance.
2. Outside factors will always influence your performance, that’s why external monitoring is a must for web resources.  I should have seen it coming a mile away, this is a classic example of current events or major news events drawing hordes of web users to servers that can barely contain them (ok, ok…too dramatic, in all reality it doesn’t look like the WordPress servers were in any danger of failing).  Ultimately it brings up a very important issue.  Know what shared resources in your application can make you vulnerable to other tenant’s traffic.

These considerations will make your SLA (or any agreement you may have with users of your application) stronger and more robust.  It’s is definitely not recommended to continually adjust your SLA.  The SLA is a pact with your users and it’s best to have it remain a strong/unchangeable core.  Changing your SLA every week to keep up with performance degradations is counterproductive and at that point it doesn’t really make sense to even provide an SLA.  So I have two options: first I could change my SLA/Timeout value from 3 seconds to 4 or 5 seconds, this would definitely reduce the number of alerts I’m receiving (it would also weaken my stance on what I promise to my readers).  Second, I can keep my Timeout value at 3 seconds and see what happens with the performance of the site (and even proactively improve the performance of the site).  At this point I’m not too concerned about the alerts and I think that as the election hype dies down I will see a reduction in alerting (which is the case over the last 24 hours).  It may not even be necessary at this point to make any improvements to the site.  Though it’s certainly something that will need to be considered in the future as I continue to post content.