Size Does Matter! (for baselines and sub-process control)

Here is a small brain-teaser.

Let us take the example of a examination/ test centers, that run an exam throughout the year, every day of the year. Analysis of the past one-year data shows that 30% of the candidates pass the exam and 70% fail the exam, all over India.

The Bangalore test center handles around 1000 candidates per month, whereas the Mysore center handles around 100 per month. Over the last one year, both centers have shown the same 30 pass: 70 fail ratio.

For the month of June 2010, one center has reported 38% pass and another has reported 29% pass. Which center (Bangalore or Mysore) is more likely (has a higher probability) to have reported 38%? Why do you think so?

See my post dated August 3, 2010 for the answer and implications.


I am Rajesh Naik. I am an author, management consultant and trainer, helping IT and other tech companies improve their processes and performance. I also specialize in CMMI® (DEV and SVC), People CMM® and Balanced Scorecard. I am a CMMI Institute certified/ authorized Instructor and Lead Appraiser for CMMI® and People CMM®. I am available on LinkedIn and I will be glad to accept your invite. For more information please click here. To get email alerts for new posts, click here to subscribe.

Why Can’t Metrics be Used for Performance Appraisals?

While discussing collection and usage of metrics, one often hears an emphatic “We should not use metrics for individual performance management!”. The statement is made as if it is an unquestionable tenant of the religion called process management.

“And pray, why not?” Why should the performance management process be deprived of metrics? A process oriented organization would definitely not like to boast that their performance management system is completely subjective.

Here are some reasons why metrics should be used for individual performance management.

*    An individual performance management (including the appraisal part) needs to be SMART – the “M” stands for measurable.

*    Most individual performance parameters are the similar to and derived from the project, product and process objectives, they typically relate to cycle time, quality (defects), meeting commitments (schedule) and productivity (cost, effort and usage of resources).

*    A strong metrics system, that provides accurate, precise and valid data can support the project, process and individual performance management requirements.

*    Using the same sources of data, we can create a more aligned organization – the individual objectives are aligned to the project, product and process objectives. In this manner, individuals know that meeting their individual goals helps in meeting the other goals (and vice versa); conflict of interest is minimized.

The situations where we may not want to use process/ project metrics for managing individual performance are:

*    The metrics collection system is not stable, and there questions on the credibility of the data. In such a case, the use of the data for managing the project/ process is also diluted.

*    Usage of the data for individual performance management may make the individuals sabotage the process and the accuracy of the metrics. In which case, we need to strengthen the process and make it sabotage proof.

In the old SW-CMM® days, most metrics collection systems were unstable, and hence many experts of that time were pretty insistent on the metrics not being used for performance appraisals – some organizations even have policy level statements for the same!

We have now moved on from the SW-CMM® days for process management, so we need to move on in other aspects too.

Your comments?


I am Rajesh Naik. I am an author, management consultant and trainer, helping IT and other tech companies improve their processes and performance. I also specialize in CMMI® (DEV and SVC), People CMM® and Balanced Scorecard. I am a CMMI Institute certified/ authorized Instructor and Lead Appraiser for CMMI® and People CMM®. I am available on LinkedIn and I will be glad to accept your invite. For more information please click here. To get email alerts for new posts, click here to subscribe.

Generating Lots of Data through Monte Carlo (a misuse?!?)

I have seen the metrics groups of organizations generating “enough” data for creating process performance baselines, from very few available data points, using Monte Carlo simulation.

Here is the method they use: Ten data points are available; using the pattern of the ten data points, they generate a thousand (or maybe a million) data points using Monte Carlo simulation. Now they feel that they have enough data points to generate a baseline.

But in reality the baseline has been generated using 10 data points. The 1000 data points only give a feeling of having lots of data and this is clearly a misuse of Monte Carlo simulation.


I am Rajesh Naik. I am an author, management consultant and trainer, helping IT and other tech companies improve their processes and performance. I also specialize in CMMI® (DEV and SVC), People CMM® and Balanced Scorecard. I am a CMMI Institute certified/ authorized Instructor and Lead Appraiser for CMMI® and People CMM®. I am available on LinkedIn and I will be glad to accept your invite. For more information please click here. To get email alerts for new posts, click here to subscribe.

Normal Distribution is Actually Rare

When we often use statistical analysis tools and techniques, the underlying assumption is that process/ sub-process displays a “normal” behavior. Even if the limited data that we have shows non-normal behavior, we assume that the reason is the lack of data, and we approximate the distribution to normal.

This assumption and subsequent analysis, conclusions and decisions are therefore inaccurate, especially if we are combining “assumed” normal behavior across multiple processes, viz Process Performance Modeling.

“Normal” behavior is very rare in real life. For example, you travel from your home to office, let us say usually in 1 hour. The least time you have ever done the trip is in 30 mins. If the distribution was normal, the worst time should have been 1 hour 30 mins (symmetrical on both sides). You will find that on some days that you were delayed, the time could have been 2 or even 3 hours!

Another way of saying that real life does not behave in a “normal” way, is “there is a limit on how well you can do, but no limit on how badly you can screw up!”

There is more on this in the books “Fooled by Randomness” and “Black Swan” by Nassim Taleb — must-reads for anyone involved in high maturity CMMI® implementation.

Also see:


I am Rajesh Naik. I am an author, management consultant and trainer, helping IT and other tech companies improve their processes and performance. I also specialize in CMMI® (DEV and SVC), People CMM® and Balanced Scorecard. I am a CMMI Institute certified/ authorized Instructor and Lead Appraiser for CMMI® and People CMM®. I am available on LinkedIn and I will be glad to accept your invite. For more information please click here. To get email alerts for new posts, click here to subscribe.