| Systems that work well during development, | | | | 1. Research and quantify the data volumes and |
| deployed on a small scale, can fail to meet | | | | transaction volumes the target market |
| performance goals when the deployment is | | | | implies. Some of these figures can be eye |
| scaled up to support real levels of use. | | | | openers and help the business users realise |
| | | | the full scale of the system. This alone can |
| An apposite example of this comes from a | | | | lead to reassessment of the priority of many |
| major blue chip company that recently | | | | features. |
| outsourced the development of an innovative | | | | |
| high technology platform. Though development | | | | 2. Determine the way features could be |
| was behind schedule this was deemed | | | | presented to users and the system structured |
| acceptable. The system gradually passed | | | | in order to make scaling of the system |
| through functional elements of the user | | | | easier. Do not try and have the same |
| acceptance testing and eventually it looked | | | | functionality you would have for a single |
| like a deployment date could be set. But then | | | | user desktop solution provide an appropriate |
| the supplier started load testing and | | | | scalable alternative. |
| scalability testing. There followed a | | | | |
| prolonged and costly period of architectural | | | | 3. Recognise that an intrinsic part of the |
| changes and changes to the system | | | | development process is load testing at |
| requirements. The supplier battled heroically | | | | representative scale on each incremental |
| to provide an acceptable system, until | | | | software release. This is continual testing, |
| finally the project was mothballed. | | | | focusing on the biggest risk to the project: |
| | | | the ability to operate at full scale. |
| This is not an isolated case. IT folklore | | | | |
| abounds with similar tales. From ambulance | | | | 4. Ensure load testing is adequate both in |
| dispatch systems to web-sites for the | | | | scope and rigour. Load testing is not just |
| electronic submission of tax returns, systems | | | | about measuring response times with a |
| fail as they scale and experience peak | | | | performance test. The load testing programme |
| demands. All of these projects appear not to | | | | needs to include other types of load testing |
| have identified and ordered the major risks | | | | including stress testing, reliability |
| they faced. This is a fundamental stage of | | | | testing, and endurance testing. |
| risk based testing, and applies equally to | | | | |
| scalability testing or load testing as it | | | | 5. Don?t forget that failures will occur. |
| does to functionality testing or business | | | | Large scale systems generally include server |
| continuity testing. With no risk assessment | | | | clusters with fail-over behaviour. Failure |
| they did not recognise that scaling was | | | | testing, fail-over testing and recovery |
| amongst the biggest risks, far more so that | | | | testing carried out on representative scale |
| delivering all the functionality | | | | systems operating under load should be |
| | | | included. |
| Recent trends towards Service Oriented | | | | |
| Architecture (SOA) attempt to address the | | | | 6. Don?t forget catastrophic failure could |
| issue of scalability but also introduce new | | | | occur. For large scale problems, disaster |
| issues. Incorporating externally provided | | | | testing and disaster recovery testing should |
| services into your overall solution means | | | | be carried out at representative scale and |
| that your ability to scale now depends upon | | | | loads. These activities can be considered the |
| these external system operate under load. | | | | technical layers of business continuity |
| Assuring this is a demanding task and sadly | | | | testing. |
| the load testing and stress testing here is | | | | |
| often overlooked. | | | | 7. Recognise external services if you use |
| | | | them. Where you are adopting an SOA approach |
| Better practice is to start the development | | | | and are dependent on external services you |
| of a large scale software system with its | | | | need to be certain that the throughput and |
| performance clearly in mind, particularly | | | | turnaround time on these services will remain |
| scalability testing, volume testing and load | | | | acceptable as your system scales and its |
| testing. To create this performance testing | | | | demands increase. A smart system architecture |
| focus: | | | | will include a graceful response and |
| | | | fall-back operation should the external |
| | | | service behaviour deteriorate or fail. |
| | | | |