Three Perspectives of Load Testing

In the world of Load Testing there are three potential perspectives you can test from: Internal, external, and Last Mile. Which perspective you choose to monitor from really depends on your goals (just as it does with performance monitoring).  Here is a breakdown of the three perspectives and the PROS and CONS of each.

Internal – This type of load test is performed from inside the network that is being tested.  It provides the best flexibility (because you manage the whole process internally) but is also the most complex and involved.  Here are the PROS and CONS:

• PRO: You have total control over the process of testing (you write the scripts, you setup the infrastructure).
• PRO: Most likely you will be able to generate the load you need without having that level of load be disrupted (though you’re not 100% secure from disruption in the level of load you generate).
• CON: This is normally a very expensive approach since you have to pay for licenses, maintenance, infrastructure, training, support, etc.
• CON: Complexity in deploying/managing infrastructure.
• CON: Doesn’t stress bandwidth issues and immediate ISP issues.

External – This type of load test is performed from outside the network that is being tested.  A slight reduction in flexibility (if you’re not deploying the infrastructure yourself) but an increase in scope of the test (you can test more infrastructure) as well as a decrease in setup time.  Here are the PROS and CONS:

• PRO: Emulates traffic as your network (and ISP) will see it.
• PRO: You test under known conditions and in a consistent fashion.
• PRO: Amount of load to apply during the test is essentially unbounded.
• CON: For cost effective tests the load is generally an emulation of real world users.

Last Mile – This is similar to external load testing but it is a bit more representative of the end users that will actually use the site.  There is a bit more reduction in flexibility (and security) but you gain a bit more perspective into what users will experience when they access your site under load.  Here are the PROS and CONS:

• PRO: Emulates traffic as your network (and ISP) will see it.
• CON: You will generally collect information that will not be of help in making decisions on improvements to your infrastructure.
• CON: Problems with public networks and computers being used to generate load are more likely to disrupt the generated load during the test.

As always the right infrastructure to use depends on your needs and what you want to accomplish with the load test.  Probably the three most important considerations of a load test are:

1. Generating consistent levels of load during the test – First, you want to make sure the level is correct.  It is a waste of money to apply 400,000 concurrent users during a test if you know from preliminary tests that the server is probably only capable of handling 40,000 concurrent users.  This is where it really helps to have support or use a third party for load testing because they have experience with load tests and scoping their parameters (the amount of load being the most critical).  Generating too little load is also a problem (for similar reasons).  In addition, you do not want to have a disruption in the load during the test.  A consistent ramp up in a load test is critical for a successful test.  If you load generation network fails halfway during the test you will lose valuable time and resources re-running the test.
2. The correct emulation of traffic – This is more a matter of taste but it has to do with how accurately you want your generated load to reflect your actual end users.  In load testing this is generally not as critical as it is in performance monitoring.  For example, during a load test it is overkill to require that the emulated users actually reflect a real browser (say IE 7) because you’re really only concerned with generating the load (not how it’s displayed in a browser).  If your needs to require that the end user perspective is accurately represented then you will want to consider more expensive solutions that do that.  Remember, the primary objective of a load test is to test the server (not the client).
3. Define what data is important – Make sure that you know before the test what data you expect to get and how you will utilize it.  For example, having information about your CPU, bandwidth, and memory utilization can help you tune your systems for improved performance.  However, knowing that an ISP for clients in China is slow or unavailable isn’t going to help you improve your performance (and it could potentially disrupt the flow of concurrent users during the test).