The Cost of Poor Performance

Phil Dixon recently gave a great talk at Velocity 2009 talking about the cost of poor performance. Through a series of rigorous tests, measurements and reports, Shopzilla was able to quantify the cost of performance to the bottom line of the business. Performance monitoring is often thought of as insurance, you pay a small premium every month so that you are notified when the site down. But it is increasingly becoming much more than this. With the variety of tools and APIs available now on multiple platforms, it’s possible (though this does require a good amount of effort) to tie performance and profitability directly together. There was another talk at Velocity that discussed the future of performance monitoring is really about being able to translate the performance and scalability costs into the direct impact to the business. In this model, the idea is to create a curve that plots costs of scalability and performance (hardware, engineering, etc.) against the impact to the bottom line (revenue, direct sales, etc.). Allistar Croll and Sean Power’s book talks a lot more about these topics in depth.

Evolution of the Monitoring Client

I’ve been exposed to the monitoring industry for almost 5 years now.  In that time the industry has actually made some pretty major changes to how monitoring is performed.  The ultimate goal of monitoring is to provide the most accurate availability and performance metrics.  For now availability metrics are only made more accurate by monitoring more frequently.  Frequency of monitoring is not a major hurdle and availability really only has one consistent definition (either you’re up or you’re down) so the collection/reporting of availability isn’t very dynamic and will not be subject to a series of evolutionary steps (though you could argue that availability could be different depending on the browser type you use – an argument I make below).  However, performance metrics will evolve as the industry strives to provide more accurate performance metrics and also, provide those metrics for any number of browsers currently on the market.  Here’s the evolution so far:

1. Internal web monitoring – Provides monitoring of systems applications, and network connections.  Because there are no major players out there the resources do not exist to allow for monitoring from an end users perspective.  Performance monitoring is a very subjective process where an end user has to describe how the perceive the performance of a website/application.
2. External web monitoring – Starts to provide information about the performance of a website/application but it is rudimentary and usually the performance data is based off of a program that only hopes to emulate the performance of a browser against the site.  No actual browser is used for monitoring.
3. External RIA monitoring – Applications are becoming more dynamic and rely on the capabilities of the browser to display the page more like a desktop application.  This requires that external monitoring take these pieces of behavior into account when considering the performance of a web application.

That’s where we’re at right now.  I think a logical next step would be:

• Generic External RIA monitoring – This would be monitoring that would consider any (and every) platform possible for viewing the web application.  For simplicity just assume that it’s every browser on the market (Chrome, Safari, IE6, IE7, IEX, Firefox 2, etc.).

The browsers mentioned seem to be diverging and are trying to grab more market share by adding new functionality to their offerings.  So how does having performance data based on Chrome help you determine what your performance would be in IE7?    For the time being there really is no need for such distinctions…the important factor is that you have an actual browser that is generating your performance metrics.  Because of price and implementation I would suspect that monitoring solutions are going to become more decentralized.  Also, with all this new functionality there is going to be an increase in the different methods that are used for delivering functionality.

Script Recorder Best Practices

Not a major component of performance monitoring but an important one none the less is the Script Recorder.  The Script Recorder is a utility that generates the script that will be used to perform the automation of a transaction on a monitoring platform.

Generally this is a downloaded application that has two modes, record and playback (A co-worker of mine often compares it to the macro recording functionality in the Microsoft Office suite of products).  You essentially interact with the Script Recorder as though it is a browser and your actions are recorded in the background.  The playback mode will take the recorded script (or a previously recorded script) and process it, allowing the Script Recorder to playback the steps that were taken in a browser.  Playback is intended for verification.

Very rarely does the Script Recorder become a crucial need with performance monitoring.  The goal is to generate scripts, and if there is an easier way to do it without a Script Recorder then it’s irrelevant if a Script Recorder even exists.  Though some power users prefer to have a Script Recorder available if their scripts need constant updating, if they want to perform some low level QA, or if they just want to have more control over the process of the scripting.

To help in the process of researching Script Recorders I have identified some requirements that are based off of monitoring best practices.  Remember though, the Script Recorder is only a byproduct of the monitoring platform and should not be the primary deciding factor when looking for a monitoring solution.  Here are some desirable requirements of a Script Recorder:

1. Flexibility – Script Recorder should be easy to use and allow for the ability to enter custom code/steps/instructions by hand if necessary.  The creators of a Script Recorder generally do not account for all the problems that someone may encounter when scripting, therefore it is beneficial to not limit script generation to just what the browser itself can do.  Allowing the user to enter in custom instructions can be helpful particularly when trying to clean-up before or after a script is run.
2. Fidelity – Record functionality should be capable of recording asynchronous (and non-interactive) events such as Ajax/JavaScript.  This events are usually invisible to a user and therefore to monitoring.  If a Script Recorder is worth its salt then it should be able to track these events.
3. Completeness – Playback functionality should be capable of playing everything back.  The monitoring platform will have to play it back so if there is a major discrepancy in playback between the Script Recorder and the actual monitoring platform then that is a sign that things are all copacetic.
4. Openness – Ideally an open source platform is available.  The script generated by the Script Recorder should be easily portable (open-source) and easy to understand.  Immediately you have the support of an open source development community.  If your script is open source then you’re not tied down to the implementation of the Script Recorder vendor.  This will make it easier for you to up and leave if your needs change (or if you just don’t like your vendor!).

Mashing up data

As I was digging through some old documents I came across some screenshots from a mashup application that a co-worker of mine had created.  The mashup combined analytics data from Google Analytics with performance monitoring data from Webmetrics.  Here is a screenshot from the application:

perf-a-lytics mashup

The graph shows the performance of the application (the dark blue line) graphed against the number of page views (the light green bars).  Normally, either of these data sets would be good data in their own right.  But each set of data alone leaves certain questions unanswered.

For example, if we consider only the analytics data we have answered the question How many users are on my site at any given time? But a very important follow up question that remains unanswered is What impact does that have on my site’s performance? The addition of the performance data answers that second question.  The performance data shows us that there is a considerable increase in page load time (roughly 4x more time to load the same page).

What about if we consider the performance monitoring data alone?  In this case we answer the question What is the performance of my application at any given time? We can tell what our performance is but there the question that goes unanswered is; What impact does that have on how people use/access my site? The answer that we get by adding the analytics data is that people tend to leave the site when the performance degrades, that’s exactly what we see in this graph, as the users leave the site (most likely upset) the site calms down because of fewer requests.  One other thing to note is that we also see what is the cause of a performance degradation.  We see that the increase in users has resulted in an increase in page load time (because the server has to use the same amount of resources to fulfill more requests).

This is just a small example of what can be done by combining data sets.  I would recommend that whenever you look at gathering data with a tool (whether it be analytics, performance data, etc.) you should make sure that an API for accessing the data is available.  This will allow you to get more value out of your tool and also will open new doors to you as you add tools in the future.

Sample mashup courtesy of Lenny.

The Value of Monitoring

As I was sitting at my desk this morning processing email I was amazed at the number of questions I was getting about basic value propositions of external performance monitoring.  I thought maybe it’s not as clear as I suspect it is. So I took off my Engineering hat (which often causes me to be derisive and not understanding of the lack of knowledge) and instead put on my Sales hat (which makes me excited to share information).  So here it is, a quick summary of the value of external performance monitoring:

• Alerting – This should always be first.  A good portion of the value in performance monitoring comes from being able to evaluate the data collected.  But, the most valuable functionality comes from being able to proactively alert yourself as problems are occurring.  This allows you to reduce downtime, meet SLAs, and investigate problems as they’re occurring.  Central to this is a monitoring frequency that will help accomplish this.  If you only monitor once an hour then there is going to be some noticable lag in how quickly you can respond to a problem.  If you monitor more frequently (ex: every 5 minutes) then you can react relatively quickly.  And even though every 5 minutes is the industry standard, monitoring once every 1 minute can of even more value for your mission critical applications.
• Reporting – This is key.  Outside of alerting, all that you have at the end of the data is a set of data and being able to draw conclusions from this data and make educated decisions about the performance of you site/application is key.  Your data set needs to be robust because you cannot go back after the fact and collect more data if you feel your data set is too sparse (you can adjust your settings to give you more data in the future but as for the past, that opportunity to collect data has come and gone).  Just like alerting, reporting requires a certain interval.  For example, you would not want to collect data once an hour and then use that data set to make adjustments to your site.
• Monitoring Client capabilities – This one is a bit more abstract but can be summarized fairly easily.  The monitoring client is what performs the monitoring, collects the data, and perceives.  Normally such a client is implemented as either an emulation of a browser (a program that makes HTTP requests) or is an actual browser that has a framework around it to automate interactions.  The actual browser implementation of this client is the most desired solution because it’s the closest representation of what your end users are experiencing (thus it indirectly makes your data set all the more accurate).  The emulated browser is generally better fit for the cost conscious monitoring consumer as it’s only an approximation of what your end user experiences (of the deficiencies of an emulated browser, lack of JavaScript support is probably the most egregious).

So there it is….when you’re shopping around for a monitoring solution you should consider the above three values.  You can then take those and sort them in order of importance for the objectives you plan to achieve with monitoring.  I generally consider the monitoring client to be the most important factor because it generates the data that is going to be by all other monitoring components (the data collected is more accurate and representative of your users which in turn makes your reports more valuable and makes your alerts more accurate).

Monitoring Browser Perspective

I’m alive!!  Sorry for being away so long…it’s been over a month since I’ve posted anything.  What with Christmas and Thanksgiving book ending a very busy month it’s no wonder I haven’t posted anything.  But I apologize, I will get back on track with blogging.

Recently I had a conversation with a co-worker of mine and he asked about the importance of monitoring from an actual browser (that is, automating a transaction for the purposes of performance and availability measurements from an actual browser vs. an emulated browser), whether it was necessary, and if necessary, what type of browser should be used.

Well, the answer is that it depends.  It all depends on what your goals are.  I’ve talked about this before and the reality is that you only need to test from an actual browser if your goal is to get very accurate metrics on your end users experience.  Furthermore you want to match that end user perspective with your actual end users…therefore, if the majority of your users are accessing your application with IE then you you should consider a monitoring solution that uses IE.  If they access the site with FF more frequently then a solution using FF should be considered.  That’s not a hard and fast rule (unless you’ve developed your site for a specific type of browser – which happens quite often with IE) the important thing is that when you’re trying to get accurate user representation you should use an actual browser.  If you’re not trying to get actual user representations with your monitoring then don’t stay up at night worrying about the browser type used (in fact you’ll probably want to use an emulated browser solution which will be cheaper).

So why the blog post?  The conversation progressed on into a debate about browser neutrality.  In other words, what if someone wanted an accurate end user perspective that doesn’t favor one type of browser over the other?  Ultimately this is not a good situation to be in because you’re going to make so many trade offs that your initial objective will not be met.  Here are your objectives:

1. Primary: Get an accurate end user perspective.
2. Secondary: Use a browser independent monitoring solution.

The primary objective is easy to accomplish by using a monitoring platform that utilizes an actual browser (you’ll need to consider what type of browser is important to you: IE, FF, Safari, etc.).  The secondary objective is self defeating and undermines the primary objective.  There is no way to take the standard parts of FF, the standard parts of IE, the standard parts of Safari, and every other browser on the market and somehow create a super browser that will provide you with an accurate end user perspective.  In order to provide such a solution the browser would have to be re-built by hand and incorporate different (which would in turn move you further away from any real world user experience).  In other words, as the neutrality of the browser increases the accuracy of the end user perspective decreases.

It’s just one of those things, the two previously mentioned objectives are inversely correlated…addressing one of them will negatively impact the other.  It’s an important lesson, if you’re going to go after accurate end user perspectives with monitoring then you’ll have to abandon any notion of a browser neutral environment.  When monitoring, always define your objectives and give them weight, be ready to analyze dependencies between them and be willing to make concessions (or at least know what concessions you’re comfortable with).

How to define a monitoring transaction

The first step in monitoring the performance of an on line asset is to determine what really needs to be monitored.  This is usually pretty easy if you’re just looking to monitor the availability and performance of a website.  In this case you just need to monitor a single URL (or a subset of the most important URLs) and not every URL on the site (since all web pages are most likely hosted on the same infrastructure).  The example above shows that focusing your attention on the essentials can help reduce costs.  For example, if it costs $10 to monitor a single URL and you 1000 pages on your site, that’s $10,000 you have to cough up to monitor every page.  But the return on each page over a certain threshold is less and less (i.e. The core value you’re getting from the monitoring is that the site is up and running and if 10 pages are not available it’s more than likely that all pages are not available).  What about in the case where you want to monitor a complex web application?

It requires a similar concern because the more you monitor the more it costs.  However, in this case there is a bit more up front analysis that you have to do in order to determine the scope of what is to be monitored.  In order to monitor a web based application a script is required to tell the monitoring platform what steps to perform and how to interact with the application.  The steps that make up that script constitutes a Transaction and directly impact the cost of monitoring (most monitoring solutions will use a metered billing approach which uses a single credit per step of a transaction).  Now we can see that there is a direct correlation between how much a monitoring solution costs and the scope of that monitoring (number of steps goes up, so does the price).  It’s important to note that even if metered billing isn’t used, most monitoring platforms have a concept of increasing the price of the solution as the number of steps increases (ex: $100 for 1 to 3 steps, $200 for 4 to 6 steps, etc.).  It’s just a fact of life, those cost increases are sometimes to recoup processing power of performing the steps but mostly the increase is to offset the costs of storing, backing up, and reporting data.  So, how does one save money?  One defines a transaction as only the steps necessary to test functionality.  Often the following mistakes are made:

• Monitoring the mundane (or for the wrong reasons).
• Monitor duplicate functionality (too much).
• Monitor too little (taking these recommendations too far).

Let’s look at each of these situations and see how they impact costs as well as how they affect the bottom line (collecting actionable monitoring data).  In each case we will consider only monitoring of transactions:

Monitoring the mundane – This is generally the product of an organization that hasn’t thoroughly thought out the goals of monitoring.  A transaction that I would consider mundane is one that doesn’t really have an end goal and just meanders around the website.  For example, a transaction that clicks through each menu item in the left nav bar is probably mundane.  Sure there’s an argument for why to do that, maybe lots of revenue is generated from the left nav bar, or maybe that’s the only navigation available for the site.  But in actuality this is really a QA problem and should be addressed as such.  It’s common in the field of computer science that the later a problem is discovered the more it’s going to cost to fix it.  Which is definitely the case here: A QA process that occurs right after development could have caught any broken links or JavaScript funkyness more efficiently then a costly monitoring solution after code has been deployed.

Monitor duplicate functionality – Sometimes this is a hard one to get around.  But basically you need to make sure that your monitoring transactions are mutually exclusive.  Don’t monitor the updating of a web based calendar in two separate transactions when one will do.  Another case is when similar methods are invoked in a single transaction.  For example, if you have a tool that configures a product and does so in 20 steps it’s probably overkill to perform all configurations (since they all probably access the same front-end and back-end functionality).  Have the transaction perform a couple of configurations and then complete the transaction (i.e. purchase, or whatever the result of the transaction is).  In this last case the duplicate functionality is a bit obscure…on the front-end the functionality looks different (configure a tire vs configure a stereo) but on the back-end the functionality is more than likely the same (accessing the same database through the same web service) and therefore all you’re really doing is testing more client side code execution (which probably should have been done during the QA process again).

Monitor too little – If you start to get too carried away with the recommendations I’ve made you could end up shooting yourself in the foot.  For example, in the last section I gave the example of an application that configures a product, furthermore that application uses the same back-end technology for each step of the configuration.  But it very well could be the case that third party functionality is embedded in the configuration tool (one step could be hosted by you while another step makes a request to a third party).  In that case maybe it does make more sense to monitor additional steps (though it could probably be monitored more efficiently by breaking out that third party contents monitoring into it’s own monitoring service).  The end result is that you’re looking for efficiencies in monitoring that will help reduce cost while NOT altering the data set you expect to get from the monitoring.

To summarize, you want to focus your monitoring so that you can achive your goals in improving performance without creating convoluted and expensive data sets.  Also, you want to be aware of not getting to zealous with efficiency and stripping your dataset of all its value.

Blog Performance

I’ve been receiving a lot of alerts from my monitoring system for this blog so I thought I’d do a bit of investigation.  I checked my monitoring data to see if there was a trend in performance degradation and sure enough I found it in a graph that lays out the page load time and errors.  Here’s what it looks like:

Performance Metrics

It doesn’t show any major overall performance increase (that is a left to right increase in the value of the dark green line) but I do notice that errors have started to increase with a culmination of 31 errors on Tuesday (11/4).  Though I don’t have any analytics data to prove this I’m pretty sure it had to do with all the blogging (and blog reading) that was going on nationwide with the presidential election.  All that traffic on WordPress certainly would have affected my blogs performance (or at least mine and every other blog that is on the same hardware).  It’s not just WordPress but other sites containing news about the election were hammered with users looking for election results (here’s an article on Akamai’s traffic for yesterday).

When I drilled down into the monitoring results I found that the errors I was experiencing were due to timeout errors.  In my monitoring software i’ve established a threshold for performance of 3 seconds.  That means that my page and all its content must be completely downloaded by the browser (IE7) within 3 seconds.  When I established this threshold I had been monitoring the performance of the page for a week and saw that the average performance was around 1.8 seconds.  Two factors are making me reconsider that threshold and these are two factors that everyone providing a web service should think about and adapt to:

1. Content (particularly in the world of blogging) will be added to a web page/application overtime.  For example, this blog has nearly doubled in size (when I first started blogging the page download was 177951 bytes, now it’s 366514 bytes).  This is going to affect the performance of the resource (another example, adding Ajax to a website will require more JavaScript that will need to be downloaded).  Future growth needs to be considered when setting expectations for performance.
2. Outside factors will always influence your performance, that’s why external monitoring is a must for web resources.  I should have seen it coming a mile away, this is a classic example of current events or major news events drawing hordes of web users to servers that can barely contain them (ok, ok…too dramatic, in all reality it doesn’t look like the WordPress servers were in any danger of failing).  Ultimately it brings up a very important issue.  Know what shared resources in your application can make you vulnerable to other tenant’s traffic.

These considerations will make your SLA (or any agreement you may have with users of your application) stronger and more robust.  It’s is definitely not recommended to continually adjust your SLA.  The SLA is a pact with your users and it’s best to have it remain a strong/unchangeable core.  Changing your SLA every week to keep up with performance degradations is counterproductive and at that point it doesn’t really make sense to even provide an SLA.  So I have two options: first I could change my SLA/Timeout value from 3 seconds to 4 or 5 seconds, this would definitely reduce the number of alerts I’m receiving (it would also weaken my stance on what I promise to my readers).  Second, I can keep my Timeout value at 3 seconds and see what happens with the performance of the site (and even proactively improve the performance of the site).  At this point I’m not too concerned about the alerts and I think that as the election hype dies down I will see a reduction in alerting (which is the case over the last 24 hours).  It may not even be necessary at this point to make any improvements to the site.  Though it’s certainly something that will need to be considered in the future as I continue to post content.

5 Key Graphs

Over the past week I’ve been developing a presentation/demonstration for training people on performance monitoring using the Webmetrics GlobalWatch platform for monitoring.  As part of that training I identified 5 key graphs that prove to be invaluable when analyzing performance data.  To present these graphs I developed a slide that would allow my audience to draw by hand the 5 graphs and list the value that each graph provides.  So here is a copy of the slide filled out by me:

5 Key Graphs

To clarify, these graphs are specific to a monitoring service that monitors a multiple step transaction (ex: purchase on Amazon.com, user log-in, or user registration) however similar graphs should be available for other types of monitoring (stream performance, web service monitor, URL monitor, DNS monitoring).  Basically, I recommend that whenever you look to implement performance monitoring you should make sure that the presentation layer of that monitoring allows you (at least) to view your data in fashions similar to these 5 graphs:

1. Transaction Average Load Time – A graph that shows you the high level view of the data you are collecting, in this case a view of the average load time throughout the day of the performing the transaction (or viewing a single URL).  This acts as an executive summary as well that helps to show basic trends in the performance of the transaction over time.  Optionally it is of value to display any errors that occurred in the graph.  Also, it helps if the graph can easily be drilled-down on so that it does not lock you into only a high level view (the drill-down will allow you to see the individual sample values that make up the displayed average).  Again, the primary benefit of this graph is a high level view that allows you to look for trends in performance.
2. Transaction Step Averages – Another view at average data, but this time we’re drilling down to the individual steps that make up the transaction.  The example drawing shows a 6 step transaction with errors on steps 1 and 4 (and a performance bottleneck at step 4 as well).  The benefit of this graph is that you can now breakdown the performance of the steps that make up the transaction being monitored.  However, it’s still an average.  So it’s going to give us a high level view that allows us to identify what steps in a transaction can use improvements in performance as well as breakdown a complex set of data.  While errors on a per step basis should be an option to the graph drill-down capabilities would probably be overkill and presentation would be clunky at best.
3. Transaction Steps Over Time – A graph that shows the average performance over time for each step in the transaction relative to the other steps.  This graph is similar to the first graph discussed but it breaks down the data so that we can look at trending for each individual step (as well as see how performance degradation affects individual steps in the transaction – as opposed to the transaction as a whole).  Errors should again be an optional parameter to the graph but errors should be distinguished by what step it occurred on since the primary data plotted is per step performance.  This graph would only add value for a service that monitors multiple steps (either a transaction or a number of URLs).
4. Full-Page Breakdown – Arguably the most important graph that external performance monitoring can generate.  If you’re looking at a monitor solution that doesn’t provide it (or…gasp: doesn’t even collect full-page data) then you are not getting the true value of external performance monitoring.  The full-page breakdown is a waterfall style graph that shows the download/rendering characteristic of a web page (or pages).  The full-page displays performance data for every item that makes up a web page (images, CSS, JavaScript files, etc.).  Whenever you record performance of these items you should consider at least the basic performance metrics: HTML download, redirection, network latency, and transfer time).  The full-page breakdown is a great reflection of browser fidelity (which is a representation of how accurately a monitoring solution emulates an actual client – such as a browser).  This is generally the lowest level of granularity you can get with external monitoring and it allows you to see what impact components (JavaScript, images, etc.) have on the overall performance of your web applications and pages.
5. Uptime & Average Load Time -  This graph is central to external monitoring solutions (and would only exist on massively deployed internal solutions).  The focus of this graph is on providing performance metrics on a per location basis.  Since external monitoring is done from global locations outside your firewall you will see different performance for different regions (samples originating further away from your servers will take longer to traverse the Internet).  Monitoring solutions that are deployed in house suffer from proximity between the resource being monitored and the tool that is monitoring…this graph will show you what is the performance from different locations (the line drawn across the graph) as well as the uptime from each location (the bars in the background of the graph).  A common usage of this graph is to evaluate the benefits of a CDN.  If a site is not using a CDN you would expect to see a rightward trend in performance improvement (that is the line representing performance would descend to the right for locations that are closer to the server being monitored).  When a CDN is used you would expect to see a very consistent performance line because accessing a site/resources provided by a CDN will reduce overall performance no matter where the client is accessing the site/resource from (CDN = content delivery/distribution network.  This is a network of servers that push content to the edges of the networks around the globe so that requests for the content don’t have to travel far…thus reducing latency times).

One important note on the last graph.  I mentioned that the Uptime & Average Load Time graph is good for evaluating CDNs, this is true if the external monitoring solution is a impartial third party.  Some monitoring services may be partnered with CDNs such that the CDN refers customers to the monitoring company in exchange for performance metrics that are skewed to show better results then what are actually achieved.  There isn’t anything tricky going on here…it simply has to do with where the monitoring agent is located in comparison to a CDN server that is providing content.  If they’re in the same data center then the performance data collected is somewhat biased and will show better performance improvements than most people will experience in using the CDN.  Definitely check and find out what the context is behind data that you’re using to evaluate CDNs or any other web technology that promises to improve your performance.

Three Perspectives of Load Testing

In the world of Load Testing there are three potential perspectives you can test from: Internal, external, and Last Mile. Which perspective you choose to monitor from really depends on your goals (just as it does with performance monitoring).  Here is a breakdown of the three perspectives and the PROS and CONS of each.

Internal – This type of load test is performed from inside the network that is being tested.  It provides the best flexibility (because you manage the whole process internally) but is also the most complex and involved.  Here are the PROS and CONS:

• PRO: You have total control over the process of testing (you write the scripts, you setup the infrastructure).
• PRO: Most likely you will be able to generate the load you need without having that level of load be disrupted (though you’re not 100% secure from disruption in the level of load you generate).
• CON: This is normally a very expensive approach since you have to pay for licenses, maintenance, infrastructure, training, support, etc.
• CON: Complexity in deploying/managing infrastructure.
• CON: Doesn’t stress bandwidth issues and immediate ISP issues.

External – This type of load test is performed from outside the network that is being tested.  A slight reduction in flexibility (if you’re not deploying the infrastructure yourself) but an increase in scope of the test (you can test more infrastructure) as well as a decrease in setup time.  Here are the PROS and CONS:

• PRO: Emulates traffic as your network (and ISP) will see it.
• PRO: You test under known conditions and in a consistent fashion.
• PRO: Amount of load to apply during the test is essentially unbounded.
• CON: For cost effective tests the load is generally an emulation of real world users.

Last Mile – This is similar to external load testing but it is a bit more representative of the end users that will actually use the site.  There is a bit more reduction in flexibility (and security) but you gain a bit more perspective into what users will experience when they access your site under load.  Here are the PROS and CONS:

• PRO: Emulates traffic as your network (and ISP) will see it.
• CON: You will generally collect information that will not be of help in making decisions on improvements to your infrastructure.
• CON: Problems with public networks and computers being used to generate load are more likely to disrupt the generated load during the test.

As always the right infrastructure to use depends on your needs and what you want to accomplish with the load test.  Probably the three most important considerations of a load test are:

1. Generating consistent levels of load during the test – First, you want to make sure the level is correct.  It is a waste of money to apply 400,000 concurrent users during a test if you know from preliminary tests that the server is probably only capable of handling 40,000 concurrent users.  This is where it really helps to have support or use a third party for load testing because they have experience with load tests and scoping their parameters (the amount of load being the most critical).  Generating too little load is also a problem (for similar reasons).  In addition, you do not want to have a disruption in the load during the test.  A consistent ramp up in a load test is critical for a successful test.  If you load generation network fails halfway during the test you will lose valuable time and resources re-running the test.
2. The correct emulation of traffic – This is more a matter of taste but it has to do with how accurately you want your generated load to reflect your actual end users.  In load testing this is generally not as critical as it is in performance monitoring.  For example, during a load test it is overkill to require that the emulated users actually reflect a real browser (say IE 7) because you’re really only concerned with generating the load (not how it’s displayed in a browser).  If your needs to require that the end user perspective is accurately represented then you will want to consider more expensive solutions that do that.  Remember, the primary objective of a load test is to test the server (not the client).
3. Define what data is important – Make sure that you know before the test what data you expect to get and how you will utilize it.  For example, having information about your CPU, bandwidth, and memory utilization can help you tune your systems for improved performance.  However, knowing that an ISP for clients in China is slow or unavailable isn’t going to help you improve your performance (and it could potentially disrupt the flow of concurrent users during the test).