The secret performance tuning button

secret button

Over the years, I have felt at several occasions that people think there is secret performance tuning button hidden somewhere in the application/environment that a performance tuning expert can locate with his magic wand and turn ON to boost the performance of their application. This is especially evident when they reach out to you indicating how soon they want the problem fixed and how much improvement they expect to see after you turn the button on.

Please note that I do not intend to disrespect their expectations. When someone is responsible for a serious application and he/she is hit with a performance problem, there is certainly an emergency and the rules of emergency are completely different. A performance expert can show some value additions only if he/she can address the emergency in a timely and satisfactory manner. So the expectation of a magic button may be well acceptable in this context.

The fact is that there is no such button or short cut available to turn a performance starving application to performance rich application (in most cases). I have seen such buttons in video games (especially in car racing) but have not seen in real world applications yet.

A performance turning expert usually achieve the desired goals by performing either one or both of the following:

  1. Cut down any unnecessary operations that adds overhead and slows down the application.
  2. Improve the efficiency of the operations by a combination of hardware, software and architecture rework.

Most of the activities involved in a tuning project can be broadly classified into one of the two categories above (usually). But there can always be exceptions which varies from case to case. So I don’t want draw concrete lines here. The goal of this post is to touch this from a very high altitude.

Do-it-yourself options

You don’t need to be a performance tuning expert to get started with some basic troubleshooting. You can start a performance tuning exercise by checking the best practices, check lists and “don’t-do’s”. This is a check list that anyone can use to identify possible traps or shortfalls. Each organization or team may have its own check lists such as database configuration best practices, database development best practices, application programming best practices etc. The best practices may be evolved over period of time gaining knowledge from mistakes done in the past as well as influence from the gradual learning the team is going through. A new hire may bring in a bunch of additional check lists and best practices in to the team which may be the result of his/her experience with the previous projects.

So, in most cases, before an expert can step in and help you (except for emergencies), you can help yourself by quickly checking your code, application, environment, configuration against the known/available best practices and check lists. Most products/platforms publish basic guidelines and check lists to get the best out of their products/platforms.

In many cases, you may be able to resolve some of the performance problems by following the best practices. Keep in mind that there is no single ‘best’ approach for all problems or environments. An approach that works for John may not work for Michael. Each environment, use case, architecture and challenge may be different and the best practices may be used as a set of basic guide lines.

Discovering tuning opportunities

When a performance tuning expert starts looking into your problem, usually he/she will be focusing on identifying possible ‘tuning opportunities’. I believe this is half an art and half a science. So different people can come up with solutions providing different levels of performance results, primarily due to the ‘art’ part of it. I believe, to be able do a good job in performance tuning, one may need to develop an attitude and way of thinking that is performance oriented. It is important to realize that even milliseconds matter. A tuning opportunity that reduces an operation by 1 millisecond may result in huge performance improvement if that operation is used heavily. I was hardly lucky to find single tuning opportunity that saved many seconds with a single change. Instead most of the tuning opportunity that I dealt with were within milliseconds.

Examining the Hardware and Software Layers

I wrote my first program in early 90’s with DBase, which was single user, single threaded, single tier and ran on a single computer. DBase managed the data and user interface of my application. Everything was tightly bound into DBase engine. When a user performs an application function, the command goes directly into the DBase engine and the route was pretty short between the user and the core application code/data.

This is not the scenario today because most applications are accessed by millions of users over the internet simultaneously. The performance of your application depends on a number of hardware, software and network components within and outside your application. On the far end outside your reach exists the end user’s computer, its configuration, the capabilities of the browser running your application, internet speed of the user etc. Within your data centre, a number of hardware components affect the performance of your application such as the data centre bandwidth, capabilities of the router and switches, intranet speed, capabilities of the load balancers, server hardware etc. Within your application, the various components and subsystems such as the presentation layer, business layer, caching layer, database layer, external APIs and any additional layers you may have affects the overall performance of the application.

A tuning expert usually will look at the different layers and subsystems to identify the area where he/she can find the most tuning opportunity. Based on that he/she may plan and execute a tuning plan to achieve the desired performance goals. Sometimes the rework or changes to a single layer may help to achieve the expected performance goals. Many times it may also need rework on multiple layers, introduction or removal of one or more layers as well as major rework on the overall architecture.

Zooming into the layers

In the next several posts, I would like to drill down into each of those layers and share my thoughts and comments on various check lists, best practices, tuning opportunities, common mistakes etc. None of the ideas and approaches I am presenting through these series of posts are claimed to be the ‘best’ or ‘the only’ of its kind.  These ideas and thoughts are shaped through my experience, what I have seen, heard and done over the years and what I have learned from reading, listening or speaking to other experts in the industry.

To be able to serve as ‘Quick Reads’ and avoid boredom, I will try to limit the posts to be around 1000 words. Whoops, It is already 1128 words. Bye, see you soon.