In the last few months, I have been frequently asked the question, “Should we do our DEV and SVC appraisal as a single multi-model appraisal?” This question is posed usually by large IT organizations in India. These organizations have already been appraised at ML5 of the DEV model (maybe more than once). And they now are on the verge of their first SVC appraisal in 2012. I guess the issue of multi-model appraisals will become more important in another 1-2 years, when the next round DEV and SVC appraisals are due for many organizations.
Well, the answer is “it depends”:-).
In this note we will try to understand the factors to consider (elaboration of what “it depends” on), so that you can take them into account when you face the situation. This note has been put together with a large dose of inputs from D Sankararaman, Mukul Madan, and V Seshadri. These were validated by Channaveer Patil and Dan He. However, they are not responsible for any errors that may have crept into this note.
Multi-model appraisals are covered in detail in Appendix G of the SCAMPISM A v1.3 Method Definition Document (MDD) downloadable from here.
One appraisal team’s experience on a multi-model appraisal (SCAMPISM v1.2 completed in 2010) at TCS is shared in a SEPGSM 2011 presentation by Ron Radice, et al, is available here.
Disclaimer – this note is not definitive, nor is it an “official” position paper of any organization or lead appraiser. However, it may be considered as one of the inputs while evaluating the option of a multi-model appraisal.
The current queries for multi-model appraisals are typically arising from organizations wanting to do DEV+SVC together, and hence we will use that situation as an example in this note. However, multi-model appraisals could comprise any combination of two or more of DEV, SVC, ACQ and People CMM®, and the factors discussed in this note apply to the other situations as well.
Here are the factors to consider.
Organizational Disruption. If you are a big organization, you could have either two (or more) long organizational disruptions, or one mega-ultra-long disruption. The choice is yours :-).
Number of ATMs. In multi-model appraisal you are likely to need lesser number of ATMs trained on the models. Assuming that you will try to keep a gap of a few months between the two appraisals (if done separately), the number of ATMs trained on the models may need to be higher, if you are doing the appraisals separately. During the interval between the two appraisals, the ATMs may resign, retire, go on leave, be allocated to some other useful work (assuming that they are still capable of doing some other useful work :-)), or just refuse to be ATMs again (“not another appraisal as ATM!”). So instead of training a bunch of 10-12 people on the models, you may have to train a higher number if you are doing the appraisals separately.
LA/ ATL requirements. For a multi-model appraisal, you will need to engage a lead appraiser appropriately certified as SCAMPISM-A LA for all the models (constellations, actually) covered in the appraisal. Therefore, the choice of LAs on multi-model appraisals may be significantly lower, especially if your appraisal is “high-maturity” (ML 4 or ML 5).
LA Willingness. The calendar time for the on-site activities for a multi-model appraisal is definitely going to be much higher than for a single model appraisal (this is also discussed as a separate factor later on). LAs may not be willing stay away from their families, pets and home city for such a long time. Or they may demand a fat sum as hardship allowance :-).
Sampling of Projects (or Workgroups). This does not change whether you are doing a single multi-model appraisal or two separate appraisals. If there were X projects selected for DEV and Y workgroups selected for SVC, then in the multi-model appraisal, the number of instances would be X+Y. Sampling will be done as if they were different appraisals.
Overall Effort. This is one area where there is a lot of misunderstanding. Note that the sample size remains the same (multi or otherwise). Hence, the effort for artefact collection remains similar, the effort for artefact review by the appraisal team is also similar and so is the effort for interviews and discussions. There could be some (a tiny bit) effort reduction in a multi-model appraisal due to the following:
- Single batch ATM training (instead of possible two batches). However, one batch of ATM training can have a max of 12 participants, so with backup ATMs you may have to run two batches anyways, even for a multi-model appraisal.
- Sponsor meeting (assuming the same sponsor for both the appraisals)
- Opening meeting can be a single one instead of two
- Some economies of scale (not a lot) on artefact collection, artefact review, interviews and preliminary findings for “Oh” areas – organizational PAs like OPD, OPF, etc. However, organizational PAs will have to be investigated from both (DEV and SVC) the contexts explicitly. So the saving would be more in terms of being familiar with the terminology, document architecture and names/ faces of people running the “Oh” processes, assuming that the people are the same in the DEV and SVC contexts.
- General effort saved for the LA and ATMs due to familiarity with the layout of the office, the security procedure, the parking lot, the cafeteria food, the washrooms, the office furniture, the room freshener, the air-conditioning, etc. (this factor may be invalid, if the appraisal team has to constantly move across buildings and cities anyway).
The project-level (or work-group level) process areas will have to be investigated for each instance separately (either in the DEV or the SVC context). Since the sample size is going to be determined the same way (whether it is a two separate appraisals or a multi-model one), the effort to investigate instance level data is going to be same. This includes the effort for the preliminary findings (or equivalent).
With the above micro-savings, the overall appraisal effort savings (LA + ATMs) is likely to be in the range of 15%-20% (i.e., the total effort for a multi-model appraisal is likely to be around 15-20% less than the total effort for separate appraisals).
Calendar Time. With the large one-time effort for the multi-model appraisal, the calendar time for the onsite period is also likely to be higher (than that of a single model appraisal), because there is a limit to the number of ATMs that an LA can handle. Hence the ATMs will need to be out of their day-job for a longer period. The long drawn absence of the ATMs from their day job can be disruptive. A reported multi-model appraisal done had an onsite period of close to the upper limit of 90 days (see here).
ATM / LA Fatigue. This is where the multi-model (for high maturity, large organizational scope) becomes untenable. As the on-site period start crossing three weeks, the fatigue becomes obvious. In organizations that use standard processes, and have done this over many years, one can expect the documents to be similar. The responses during the interview sessions will also be similar.
For the ATMs, after the glamour of being ATMs, the novelty of PIIDs and Process Area Worksheets, and the thrill of FI-LI-PI-NI-(and NY, of course) wears out, it is an extremely boring, mind-numbing and dull exercise.
(Digression: This may be one of the reasons that LAs have become good storytellers and general entertainers. I know of a LA who sings to keep the ATMs entertained, another one tells jokes on a non-stop basis. Some LAs have started blogging. SEI may have to initiate a study to understand whether LAs have a higher tendency to ….whatever 🙂 End of Digression).
After around three weeks, the productivity, alertness, and eye for detail falls down steeply for the ATMs as well as the LA. The other issue is the stress on the ATMs of maintaining confidentiality. They cannot talk to their friends and colleagues, or smile at passing acquaintances, because they may be asked that dreaded question “and how are we today?” Those who have been ATMs know about this, others readers may ask their friends/ colleagues who have been ATMs to confirm this :-).
Target Levels/ Results. For the purpose of the results (ML/ CL), the multi-model appraisal will deliver two results, two different sets of ratings. Your target rating could be different for the two models and the appraisal result rating will also be different for the two models. This is the same as doing two separate appraisals.
Novelty / Publicity Value. “Will we be the first to do a multi-model appraisal?”; “No?”; “Okay, can we be the first do to it in this country?”, “How about this city?” and so on….
Well, if your organization is looking to announce itself as the first in something, we can surely work out some combination of conditions that you will be the first in. Not just the first, but maybe the only one. Ever.
Having said all that, are there any conditions where it may be worth considering a multi-model appraisal? Yes, if you have the following it may definitely be worth considering:
- Low number of Process Areas (ML2 kind of stuff) in both the models, and
- Small organization (number of sampled instances are likely to be low)
Under these circumstances, the number of ATMs for each separate appraisal would be low (say 4-5), so one can increase the ATM team size to 8-10 and run the multi-model appraisal in the same number of days as a single separate one. So the organizational disruption time and LA cost can be much lower. Also, ATM training can also be done in a single batch (max batch size is 12).
Finally, the issue is a complex one, and let us conclude by saying once again that “it depends” and that you should consult multiple LAs and take their opinions before coming to any firm conclusion.
Also refer to:
Thanks a lot to D Sankararaman, Mukul Madan, V Seshadri, Channaveer Patil, and Dan He.
Please feel feel to share your views, experiences or queries, using the “comments” feature available at the top of this article/ post.
Nothing Official About It! – The views presented above are in no manner reflective of the official views of any organization, community, group, or association.
SM-SCAMPI and SEPG are service marks of Carnegie Mellon University.
LA- is a short form of SCAMPISM Lead Appraiser. (It is not a term of endearment like “da”, “pa”, “ma”, or “po” used in different parts of the world :-)).
ATM – Appraisal Team Member (not an Automated Teller Machine :-))
You may also be interested in the following posts uploaded on the same blog: