# Size Does Matter! (for baselines and sub-process control) -Continued

Let us take the example of  examination/ test centers, that run an exam throughout the year, every day. Past one-year data shows – 30% of the candidates pass the exam and 70% fail the exam, all over India.

The Bangalore test center handles around 1000 candidates per month, whereas the Mysore center handles around 100 per month. Over the last one year, both centers have shown the same 30 pass: 70 fail ratio.

For the month of June 2010, one center has reported 38% pass and another has reported 29% pass. Which center (Bangalore or Mysore) is more likely (has a higher probability) to have reported 38%?

Well, Mysore is more likely to have the higher deviation from the average (+8%) than Bangalore (-1%), because Mysore, handling lesser candidates, has a lesser number of opportunities to “average out”. An easy way to figure this out is to take the case of a center that handles only 1 candidate. This center can have either 0% or 100%  pass percentage; a -30% to +70% deviation from the average.

Let us now get back to the process performance baselines that we create and the way we do sub-process control. Here are some things that we need to keep in mind while creating, publishing and using baselines:

1) Baseline (mean and standard deviation) for a sub-process parameter (like coding productivity) will be different depending on whether we consider each the coding phase of each project as a data point, or we consider each program coded in each project as a data point. The standard deviation in the first case (large base) is likely to be smaller than the second case (small base).

2) When we publish performance baseline data, we need to qualify it with the level of detail at which it applies.

3) When we use the baseline data to do sub-process control, it needs to be applied to the same level of detail. So, to do sub-process control on program level coding productivity, we need to use the baseline that was created using programs as data points (not each project as a data point).

4) Baselines need to be created using similar situations of the base data. For example, we cannot combine the coding productivity on large programs with the productivity on small programs. Even if the average/ mean remains the same, the standard deviation will be higher when we take data from a smaller base as against a larger base.

The above points are not just “nits” but have an impact of the usefulness of baselines and sub-process control. Incorrect usage of baselines leads to incorrect displays of process instability / stability.

I am Rajesh Naik. I am an author, management consultant and trainer, helping IT and other tech companies improve their processes and performance. I also specialize in CMMI® (DEV and SVC), People CMM® and Balanced Scorecard. I am a CMMI Institute certified/ authorized Instructor and Lead Appraiser for CMMI® and People CMM®. I am available on LinkedIn and I will be glad to accept your invite. For more information please click here.

# Size Does Matter! (for baselines and sub-process control)

Here is a small brain-teaser.

Let us take the example of a examination/ test centers, that run an exam throughout the year, every day of the year. Analysis of the past one-year data shows that 30% of the candidates pass the exam and 70% fail the exam, all over India.

The Bangalore test center handles around 1000 candidates per month, whereas the Mysore center handles around 100 per month. Over the last one year, both centers have shown the same 30 pass: 70 fail ratio.

For the month of June 2010, one center has reported 38% pass and another has reported 29% pass. Which center (Bangalore or Mysore) is more likely (has a higher probability) to have reported 38%? Why do you think so?

See my post dated August 3, 2010 for the answer and implications.

I am Rajesh Naik. I am an author, management consultant and trainer, helping IT and other tech companies improve their processes and performance. I also specialize in CMMI® (DEV and SVC), People CMM® and Balanced Scorecard. I am a CMMI Institute certified/ authorized Instructor and Lead Appraiser for CMMI® and People CMM®. I am available on LinkedIn and I will be glad to accept your invite. For more information please click here.

# Why Can’t Metrics be Used for Performance Appraisals?

While discussing collection and usage of metrics, one often hears an emphatic “We should not use metrics for individual performance management!”. The statement is made as if it is an unquestionable tenant of the religion called process management.

“And pray, why not?” Why should the performance management process be deprived of metrics? A process oriented organization would definitely not like to boast that their performance management system is completely subjective.

Here are some reasons why metrics should be used for individual performance management.

*    An individual performance management (including the appraisal part) needs to be SMART – the “M” stands for measurable.

*    Most individual performance parameters are the similar to and derived from the project, product and process objectives, they typically relate to cycle time, quality (defects), meeting commitments (schedule) and productivity (cost, effort and usage of resources).

*    A strong metrics system, that provides accurate, precise and valid data can support the project, process and individual performance management requirements.

*    Using the same sources of data, we can create a more aligned organization – the individual objectives are aligned to the project, product and process objectives. In this manner, individuals know that meeting their individual goals helps in meeting the other goals (and vice versa); conflict of interest is minimized.

The situations where we may not want to use process/ project metrics for managing individual performance are:

*    The metrics collection system is not stable, and there questions on the credibility of the data. In such a case, the use of the data for managing the project/ process is also diluted.

*    Usage of the data for individual performance management may make the individuals sabotage the process and the accuracy of the metrics. In which case, we need to strengthen the process and make it sabotage proof.

In the old SW-CMM® days, most metrics collection systems were unstable, and hence many experts of that time were pretty insistent on the metrics not being used for performance appraisals – some organizations even have policy level statements for the same!

We have now moved on from the SW-CMM® days for process management, so we need to move on in other aspects too.

I am Rajesh Naik. I am an author, management consultant and trainer, helping IT and other tech companies improve their processes and performance. I also specialize in CMMI® (DEV and SVC), People CMM® and Balanced Scorecard. I am a CMMI Institute certified/ authorized Instructor and Lead Appraiser for CMMI® and People CMM®. I am available on LinkedIn and I will be glad to accept your invite. For more information please click here.

# Generating Lots of Data through Monte Carlo (a misuse?!?)

I have seen the metrics groups of organizations generating “enough” data for creating process performance baselines, from very few available data points, using Monte Carlo simulation.

Here is the method they use: Ten data points are available; using the pattern of the ten data points, they generate a thousand (or maybe a million) data points using Monte Carlo simulation. Now they feel that they have enough data points to generate a baseline.

But in reality the baseline has been generated using 10 data points. The 1000 data points only give a feeling of having lots of data and this is clearly a misuse of Monte Carlo simulation.

I am Rajesh Naik. I am an author, management consultant and trainer, helping IT and other tech companies improve their processes and performance. I also specialize in CMMI® (DEV and SVC), People CMM® and Balanced Scorecard. I am a CMMI Institute certified/ authorized Instructor and Lead Appraiser for CMMI® and People CMM®. I am available on LinkedIn and I will be glad to accept your invite. For more information please click here.

# Normal Distribution is Actually Rare

When we often use statistical analysis tools and techniques, the underlying assumption is that process/ sub-process displays a “normal” behavior. Even if the limited data that we have shows non-normal behavior, we assume that the reason is the lack of data, and we approximate the distribution to normal.

This assumption and subsequent analysis, conclusions and decisions are therefore inaccurate, especially if we are combining “assumed” normal behavior across multiple processes, viz Process Performance Modeling.

“Normal” behavior is very rare in real life. For example, you travel from your home to office, let us say usually in 1 hour. The least time you have ever done the trip is in 30 mins. If the distribution was normal, the worst time should have been 1 hour 30 mins (symmetrical on both sides). You will find that on some days that you were delayed, the time could have been 2 or even 3 hours!

Another way of saying that real life does not behave in a “normal” way, is “there is a limit on how well you can do, but no limit on how badly you can screw up!”

There is more on this in the books “Fooled by Randomness” and “Black Swan” by Nassim Taleb — must-reads for anyone involved in high maturity CMMI® implementation.

Also see:

I am Rajesh Naik. I am an author, management consultant and trainer, helping IT and other tech companies improve their processes and performance. I also specialize in CMMI® (DEV and SVC), People CMM® and Balanced Scorecard. I am a CMMI Institute certified/ authorized Instructor and Lead Appraiser for CMMI® and People CMM®. I am available on LinkedIn and I will be glad to accept your invite. For more information please click here.

# People CMM® Appraisal Results now on CMMI Institute Website

People CMM® appraisal results are now published on CMMI Institute (earlier SEI’s website) https://sas.cmmiinstitute.com/pars/pars.aspx

Many organizations have benefited from the implementation of People CMM(R). However, till now, there was no easy way to communicate successful appraisal result to all relevant stakeholders.

Starting 2010, all People CMM® SCAMPISM Class A appraisal results will be published on the PARS website (after appropriate permissions and quality checks).

So, now you have one more reason to pursue and validate your HR related improvements using the People CMM®, the de facto standard for world-class people related processes.

SM – SCAMPI is a service mark of Carnegie Mellon University.

I am Rajesh Naik. I am an author, management consultant and trainer, helping IT and other tech companies improve their processes and performance. I also specialize in CMMI® (DEV and SVC), People CMM® and Balanced Scorecard. I am a CMMI Institute certified/ authorized Instructor and Lead Appraiser for CMMI® and People CMM®. I am available on LinkedIn and I will be glad to accept your invite. For more information please click here.

# Green IT

“What is Green IT? Should I invest my time learning about it?”, these are some questions typically asked by participants in conferences and workshops involving process and quality folks.

Here are some pointers:

• The British Computer Society (BCS) has established a 3 day Foundation Certificate in Green IT in association with Information Systems Examination Board.
• The 3 day course ends with a 1 hour examination, which contains multiple-choice questions
• The course (and the exam) focuses on (1) How IT can help greeni-fy its other operations, and (2) how IT operations can be made more green
• Topics include disposal of hazardous waste, resource conservation and sustainable working practices
• The participants are expected to become familiar with (1) various protocols, panels, summits, and international frameworks on Climate Change, and  (2) legislation and standards like ISO 14001, Energy Star, EPEAT and WEEE
• Internal assessment of an organization’s Green IT status and action planning are other topics covered in the course

The syllabus has been published here.