Three Perspectives of Load Testing

In the world of Load Testing there are three potential perspectives you can test from: Internal, external, and Last Mile. Which perspective you choose to monitor from really depends on your goals (just as it does with performance monitoring).  Here is a breakdown of the three perspectives and the PROS and CONS of each.

Internal – This type of load test is performed from inside the network that is being tested.  It provides the best flexibility (because you manage the whole process internally) but is also the most complex and involved.  Here are the PROS and CONS:

• PRO: You have total control over the process of testing (you write the scripts, you setup the infrastructure).
• PRO: Most likely you will be able to generate the load you need without having that level of load be disrupted (though you’re not 100% secure from disruption in the level of load you generate).
• CON: This is normally a very expensive approach since you have to pay for licenses, maintenance, infrastructure, training, support, etc.
• CON: Complexity in deploying/managing infrastructure.
• CON: Doesn’t stress bandwidth issues and immediate ISP issues.

External – This type of load test is performed from outside the network that is being tested.  A slight reduction in flexibility (if you’re not deploying the infrastructure yourself) but an increase in scope of the test (you can test more infrastructure) as well as a decrease in setup time.  Here are the PROS and CONS:

• PRO: Emulates traffic as your network (and ISP) will see it.
• PRO: You test under known conditions and in a consistent fashion.
• PRO: Amount of load to apply during the test is essentially unbounded.
• CON: For cost effective tests the load is generally an emulation of real world users.

Last Mile – This is similar to external load testing but it is a bit more representative of the end users that will actually use the site.  There is a bit more reduction in flexibility (and security) but you gain a bit more perspective into what users will experience when they access your site under load.  Here are the PROS and CONS:

• PRO: Emulates traffic as your network (and ISP) will see it.
• CON: You will generally collect information that will not be of help in making decisions on improvements to your infrastructure.
• CON: Problems with public networks and computers being used to generate load are more likely to disrupt the generated load during the test.

As always the right infrastructure to use depends on your needs and what you want to accomplish with the load test.  Probably the three most important considerations of a load test are:

1. Generating consistent levels of load during the test – First, you want to make sure the level is correct.  It is a waste of money to apply 400,000 concurrent users during a test if you know from preliminary tests that the server is probably only capable of handling 40,000 concurrent users.  This is where it really helps to have support or use a third party for load testing because they have experience with load tests and scoping their parameters (the amount of load being the most critical).  Generating too little load is also a problem (for similar reasons).  In addition, you do not want to have a disruption in the load during the test.  A consistent ramp up in a load test is critical for a successful test.  If you load generation network fails halfway during the test you will lose valuable time and resources re-running the test.
2. The correct emulation of traffic – This is more a matter of taste but it has to do with how accurately you want your generated load to reflect your actual end users.  In load testing this is generally not as critical as it is in performance monitoring.  For example, during a load test it is overkill to require that the emulated users actually reflect a real browser (say IE 7) because you’re really only concerned with generating the load (not how it’s displayed in a browser).  If your needs to require that the end user perspective is accurately represented then you will want to consider more expensive solutions that do that.  Remember, the primary objective of a load test is to test the server (not the client).
3. Define what data is important – Make sure that you know before the test what data you expect to get and how you will utilize it.  For example, having information about your CPU, bandwidth, and memory utilization can help you tune your systems for improved performance.  However, knowing that an ISP for clients in China is slow or unavailable isn’t going to help you improve your performance (and it could potentially disrupt the flow of concurrent users during the test).

More MobileMe

The other day Ohm Malik wrote an interesting post about the availability issues with Apple’s MobileMe site. He made quite a few key observations related to monitoring and load testing that I wanted to reiterate here. They are:

• There is no-unified IT plan vis-a-vis applications; each has their own set of servers, IT practices and release scenarios. This is becoming more and more of a problem with adoption SOA architectures, SaaS, and mashup models (see yesterday’s post for more information on monitoring these types of architectures). You now need to work closer with your partners when performing load testing to ensure that all components of your application are thoroughly tested. A load test is of little value if it fails to test 25% of your application/infrastructure. This requires more up-front time planning a load test and if you’re using a vendor for load testing then ensure that they provide services that are consultative and strive to understand your environment and infrastructure.
• There’s no unified monitoring system. Monitoring data from all sources (server performance, external application performance, analytics, etc.) needs to be considered and in an ideal environment would be mashed up to gain new insight into the data. Consistent and fine-grained monitoring intervals are two properties that are important in effectively monitoring a web application.

Even the giants like Apple will feel the pain if they fail to ensure performance and availability of their applications through pre-release load testing of infrastructure and on-going monitoring for performance and continued availability.

The performance for this blog (today’s average load time and uptime):

• Average load time is 1.81seconds.
• Availability (uptime) is 100%.

MobileMe or: How I Learned to Stop Worrying and Love the Load Test

So, you’ve got your new iPhone 3G, you’ve waited in line at the Apple or AT&T store for hours and it was worth it. You probably bought the 8 Gig…but maybe you bought the 16 Gig (and like a fellow co-worker you bought the white version so people knew you had the 16 Gig iPhone). You’ve now got the cool new iPint application, or you’ve finally solved an age old problem with the BigTipper application. Furthermore, you’re extremely happy that you can use MobileMe to store your contacts, calendar appointments, tasks, etc. on “The Cloud” so that you can easily keep your iPhone and MacBook synchronized. Life is good, Windows is bad, and Apple rules….but, er, wait. What’s this? Why can’t I manage my contacts and why are the changes I made not propagated to my devices? And where are my emails?

I don’t care how good Apple products are. “The Cloud” is bad (not that it means to be). With all this hype that the new iPhone 3G is getting some unexpected issues arose on the Mac run me.com (MobileMe) site when it was flooded with traffic (i.e. users). These issues caused downtime, loss of functionality, and even loss of mail messages for some MobileMe users.

MobileMe is a web application that allows you to keep all your information on the web so that all your devices can be synced from one convenient location, that conveniently isn’t under your control. I don’t know what exactly Apple did to prepare for the launch of this application but it would seem that Load Testing was not performed (or not properly performed) since there was no concept of how much load the servers could handle.

A load test would have helped the IT folks at Apple understand what capacity they were currently operating at, where their likely points of failure are at (and at what levels of usage would those points fail), and what they would need (infrastructure) to surpass those limits and prevent failure. Load testing is something that should be done before any major release and ideally would be done by someone with a proficient background in load testing and the review of load test results (either someone internal or external to the company, as long as they have some experience with load testing).

Generally load tests are launched before a major release of a web application or service. A load test will consist of a number of iterations in which virtual users (the load) are applied to the server for a duration of time (usually 1 hour). during the iteration multiple types of scenarios will be run. in the case of the MobileMe application scenarios could have been:

1. Login to MobileMe.
2. Sync contacts on MobileMe.
3. Update calendar event.

The first iteration of a load test would have consisted of 1000′s of virtual users (each one performing one of the above scenarios) accessing the MobileMe application. During the test, the virtual users would track performance metrics (availability and load time) for the various scenarios mentioned above. For example, as the load increases on the server we could see how the performance would degrade for someone who was trying to login to the MobileMe application while 1000′s of other requests were being served by the MobileMe server. Once this initial iteration is complete, the Apple team would make revisions to their application and architecture based on the results of the load test. After those changes are made another load test iteration could be conducted to ensure that the changes have improved performance and not caused any unattended issues. Rinse and repeat.

Here’s a link to the MobileMe status blog (http://www.apple.com/mobileme/status/) which outlines more specifics about the outage. This is a problem that all companies providing web based applications and services need to consider; it’s great when you launch an application that’s popular, it’s not so great when that popularity damages your brand and prevents your service from being usable. The iPhone is popular enough that most users will probably ignore the outage (and the lost emails between July 16th and 18th), heck, those users may even endure a couple more outages. But the majority of applications on the web do not have the brand backing of Apple/iPhone.

More on load testing offered by Webmetrics: http://www.webmetrics.com/loadtesting.html.

And here’s the performance for this blog:

• Average load time is 1.70 seconds.
• Availability (uptime) is 100%.