In our discussions with clients of all sizes that launch a new application, the following question comes up quite often, “So, we are good to launch now and have enough infrastructures to run our application today. How do I know when it’s time to change/upgrade? I don’t want to wait for upset customers to call my support team and tell me our application performance is deteriorating. How can I proactively predict applications performance and manage the degrade in performance?” If the same question has crossed your mind too then do read on…
The first question that we need to answer is, does the performance of an application actually degrade over time and if yes then what are the causes of this degradation? In contrast to the other man made things, a piece of code does not decay with time. A for loop that you wrote 5 years ago will perform with same efficiency as it performed on the day you wrote it. So if the code does not degrade itself, then what causes the performance degradation? Let us take an example of a Bank that has set up a new branch with 4 executives serving the customers. In the beginning, the performance of the branch is great, there is no queue in the branch and customers can get in and out within a specified time.
Now, let us say that this Bank decides to incentivize the new account opening by giving higher interest rates. As the word of this offer spreads around, you see queues at the branch with long waiting periods. The customers start complaining about the time they have to wait in order to complete their transactions. Has the performance of the executives degraded? Not really. The branch was never designed or staffed to handle the load that it is experiencing. This impacts the overall performance from the customer’s point of view.
This branch’s executives are similar to your application servers. If the user base of the application has increased – due to some seasonal traffic or due to increase in popularity of the application itself then the servers will not be able to do their work as efficiently as they did before. If you see a surge in your user base then you should think of increasing the horsepower available to your application servers – the CPU cycles, memory, storage and the network speed. There is an another side to this story, some part of this issue can be addressed before the application goes live if enough thought is giving to performance engineering in the development phase, but this a story that needs to be discussed separately.
As the number of customers increase in the branch, the number of requests that the executives need to handle also increase. The same staff strength, the seating space and number of open counters will not be able to handle the increased customer load. Similarly, on the technology side too, the number of servers, the memory and disk space available to each server and the network bandwidth will not suffice to handle the increased number of transactions.
Now coming back to the question that we started with, “When is the time to upgrade?” There is no specific time frame that can be applied to all applications. The approach to get to an answer is two fold – benchmarking your application’s vital statistics and periodically matching them against the actual numbers.
Before you launch your application and after it is ready with close to the final code base, you should conduct a performance benchmarking to give you the following numbers:
- The average number of transactions that a server can handle concurrently without deteriorating the performance
- The acceptable response time for users for the top 20% transactions.
Once the application is live, there are a few factors that you need to continuously monitor to look for signals that will warn you about the upcoming upgrade:
- End User Response Time
- Highest number of transactions processed per minute – Daily, Weekly and Monthly.
- Vital Signs of the hardware – CPU, storage, network and memory utilization.
The combined knowledge of above five factors will help you predict the performance of your application for some time to come. For example, if you figure out from the benchmarking exercise that on an average, a single application server can handle one thousand transactions per minute and your monitoring of the application tells you that you are currently serving two hundred transactions per minute then you know that there is no reason to worry yet. When that number starts coming up to seven hundred, you need to start planning for an infrastructure upgrade. If you start worrying when the number is close to nine hundred then you will not have enough time to plan and change. Your application is most likely to end up in trouble in that case.
Besides the above-mentioned factors, there are two additional things that will determine when an upgrade is on the cards:
Effect of Releases
If your application gives you a throughput of one thousand transactions per minute today, it does not imply that you will continue to get the same number by adding new features or changing the code. The benchmark is tied to a code base of the application. Whenever you make a significant functional change, your benchmarks need to be re-calibrated. Doing this will give you a more accurate picture of how long of a runway you have before you start worrying about the infrastructure.
Rate of Growth
The second important thing that you need to keep an eye is your rate of growth of the application’s transactions, users and data. If you have grown from 10 transactions per minute to 100 transactions per minute in 1 month, then it does not automatically imply that you will have ten more months before you reach 1000 transactions per minute. This is governed by the rate of growth of your users and transactions. For a young, growing business these numbers change very frequently. So, it is important to know them for better planning. For a steady business that has been running for a few years, the rate of growth is relatively easy to predict. For the mathematics of ‘how long do I have?’ it is also important for you to observe, “how fast am I going?”
Performance management of large and complex systems involves a lot many factors and activities. But these five factors will give you a good sense of when you need to start worrying about the application’s performance. If you need any further help in getting these questions answered then shoot them across to firstname.lastname@example.org and we will be happy to help.