Mashing up data

As I was digging through some old documents I came across some screenshots from a mashup application that a co-worker of mine had created.  The mashup combined analytics data from Google Analytics with performance monitoring data from Webmetrics.  Here is a screenshot from the application:

perf-a-lytics mashup

The graph shows the performance of the application (the dark blue line) graphed against the number of page views (the light green bars).  Normally, either of these data sets would be good data in their own right.  But each set of data alone leaves certain questions unanswered.

For example, if we consider only the analytics data we have answered the question How many users are on my site at any given time? But a very important follow up question that remains unanswered is What impact does that have on my site’s performance? The addition of the performance data answers that second question.  The performance data shows us that there is a considerable increase in page load time (roughly 4x more time to load the same page).

What about if we consider the performance monitoring data alone?  In this case we answer the question What is the performance of my application at any given time? We can tell what our performance is but there the question that goes unanswered is; What impact does that have on how people use/access my site? The answer that we get by adding the analytics data is that people tend to leave the site when the performance degrades, that’s exactly what we see in this graph, as the users leave the site (most likely upset) the site calms down because of fewer requests.  One other thing to note is that we also see what is the cause of a performance degradation.  We see that the increase in users has resulted in an increase in page load time (because the server has to use the same amount of resources to fulfill more requests).

This is just a small example of what can be done by combining data sets.  I would recommend that whenever you look at gathering data with a tool (whether it be analytics, performance data, etc.) you should make sure that an API for accessing the data is available.  This will allow you to get more value out of your tool and also will open new doors to you as you add tools in the future.

Sample mashup courtesy of Lenny.

The Value of Monitoring

As I was sitting at my desk this morning processing email I was amazed at the number of questions I was getting about basic value propositions of external performance monitoring.  I thought maybe it’s not as clear as I suspect it is. So I took off my Engineering hat (which often causes me to be derisive and not understanding of the lack of knowledge) and instead put on my Sales hat (which makes me excited to share information).  So here it is, a quick summary of the value of external performance monitoring:

• Alerting – This should always be first.  A good portion of the value in performance monitoring comes from being able to evaluate the data collected.  But, the most valuable functionality comes from being able to proactively alert yourself as problems are occurring.  This allows you to reduce downtime, meet SLAs, and investigate problems as they’re occurring.  Central to this is a monitoring frequency that will help accomplish this.  If you only monitor once an hour then there is going to be some noticable lag in how quickly you can respond to a problem.  If you monitor more frequently (ex: every 5 minutes) then you can react relatively quickly.  And even though every 5 minutes is the industry standard, monitoring once every 1 minute can of even more value for your mission critical applications.
• Reporting – This is key.  Outside of alerting, all that you have at the end of the data is a set of data and being able to draw conclusions from this data and make educated decisions about the performance of you site/application is key.  Your data set needs to be robust because you cannot go back after the fact and collect more data if you feel your data set is too sparse (you can adjust your settings to give you more data in the future but as for the past, that opportunity to collect data has come and gone).  Just like alerting, reporting requires a certain interval.  For example, you would not want to collect data once an hour and then use that data set to make adjustments to your site.
• Monitoring Client capabilities – This one is a bit more abstract but can be summarized fairly easily.  The monitoring client is what performs the monitoring, collects the data, and perceives.  Normally such a client is implemented as either an emulation of a browser (a program that makes HTTP requests) or is an actual browser that has a framework around it to automate interactions.  The actual browser implementation of this client is the most desired solution because it’s the closest representation of what your end users are experiencing (thus it indirectly makes your data set all the more accurate).  The emulated browser is generally better fit for the cost conscious monitoring consumer as it’s only an approximation of what your end user experiences (of the deficiencies of an emulated browser, lack of JavaScript support is probably the most egregious).

So there it is….when you’re shopping around for a monitoring solution you should consider the above three values.  You can then take those and sort them in order of importance for the objectives you plan to achieve with monitoring.  I generally consider the monitoring client to be the most important factor because it generates the data that is going to be by all other monitoring components (the data collected is more accurate and representative of your users which in turn makes your reports more valuable and makes your alerts more accurate).