Multi-Model CMMI® Appraisals – Factors to Consider

In the last few months, I have been frequently asked the question, “Should we do our DEV and SVC appraisal as a single multi-model appraisal?” This question is posed usually by large IT organizations in India. These organizations have already been appraised at ML5 of the DEV model (maybe more than once). And they now are on the verge of their first SVC appraisal in 2012. I guess the issue of multi-model appraisals will become more important in another 1-2 years, when the next round DEV and SVC appraisals are due for many organizations.

Well, the answer is “it depends”:-).

In this note we will try to understand the factors to consider (elaboration of what “it depends” on), so that you can take them into account when you face the situation. This note has been put together with a large dose of inputs from D Sankararaman, Mukul Madan, and V Seshadri. These were validated by Channaveer Patil and Dan He. However, they are not responsible for any errors that may have crept into this note.

Multi-model appraisals are covered in detail in Appendix G of the SCAMPISM A v1.3 Method Definition Document (MDD) downloadable from here.

One appraisal team’s experience on a multi-model appraisal (SCAMPISM v1.2 completed in 2010) at TCS is shared in a SEPGSM 2011 presentation by Ron Radice, et al, is available here.

Disclaimer – this note is not definitive, nor is it an “official” position paper of any organization or lead appraiser. However, it may be considered as one of the inputs while evaluating the option of a multi-model appraisal.

The current queries for multi-model appraisals are typically arising from organizations wanting to do DEV+SVC together, and hence we will use that situation as an example in this note. However, multi-model appraisals could comprise any combination of two or more of DEV, SVC, ACQ and People CMM®, and the factors discussed in this note apply to the other situations as well.

Here are the factors to consider.

Organizational Disruption. If you are a big organization, you could have either two (or more) long organizational disruptions, or one mega-ultra-long disruption. The choice is yours :-).

Number of ATMs. In multi-model appraisal you are likely to need lesser number of ATMs trained on the models. Assuming that you will try to keep a gap of a few months between the two appraisals (if done separately), the number of ATMs trained on the models may need to be higher, if you are doing the appraisals separately. During the interval between the two appraisals, the ATMs may resign, retire, go on leave, be allocated to some other useful work (assuming that they are still capable of doing some other useful work :-)), or just refuse to be ATMs again (“not another appraisal as ATM!”). So instead of training a bunch of 10-12 people on the models, you may have to train a higher number if you are doing the appraisals separately.

LA/ ATL requirements. For a multi-model appraisal, you will need to engage a lead appraiser appropriately certified as SCAMPISM-A LA for all the models (constellations, actually) covered in the appraisal. Therefore, the choice of LAs on multi-model appraisals may be significantly lower, especially if your appraisal is “high-maturity” (ML 4 or ML 5).

LA Willingness. The calendar time for the on-site activities for a multi-model appraisal is definitely going to be much higher than for a single model appraisal (this is also discussed as a separate factor later on). LAs may not be willing stay away from their families, pets and home city for such a long time. Or they may demand a fat sum as hardship allowance :-).

Sampling of Projects (or Workgroups). This does not change whether you are doing a single multi-model appraisal or two separate appraisals. If there were X projects selected for DEV and Y workgroups selected for SVC, then in the multi-model appraisal, the number of instances would be X+Y. Sampling will be done as if they were different appraisals.

Overall Effort. This is one area where there is a lot of misunderstanding. Note that the sample size remains the same (multi or otherwise). Hence, the effort for artefact collection remains similar, the effort for artefact review by the appraisal team is also similar and so is the effort for interviews and discussions. There could be some (a tiny bit) effort reduction in a multi-model appraisal due to the following:

  • Single batch ATM training (instead of possible two batches). However, one batch of ATM training can have a max of 12 participants, so with backup ATMs you may have to run two batches anyways, even for a multi-model appraisal.
  • Sponsor meeting (assuming the same sponsor for both the appraisals)
  • Opening meeting can be a single one instead of two
  • Some economies of scale (not a lot) on artefact collection, artefact review, interviews and preliminary findings for “Oh” areas – organizational PAs like OPD, OPF, etc. However, organizational PAs will have to be investigated from both (DEV and SVC) the contexts explicitly. So the saving would be more in terms of being familiar with the terminology, document architecture and names/ faces of people running the “Oh” processes, assuming that the people are the same in the DEV and SVC contexts.
  • General effort saved for the LA and ATMs due to familiarity with the layout of the office, the security procedure, the parking lot, the cafeteria food, the washrooms, the office furniture, the room freshener, the air-conditioning, etc. (this factor may be invalid, if the appraisal team has to constantly move across buildings and cities anyway).

The project-level (or work-group level) process areas will have to be investigated for each instance separately (either in the DEV or the SVC context). Since the sample size is going to be determined the same way (whether it is a two separate appraisals or a multi-model one), the effort to investigate instance level data is going to be same. This includes the effort for the preliminary findings (or equivalent).

With the above micro-savings, the overall appraisal effort savings (LA + ATMs) is likely to be in the range of 15%-20% (i.e., the total effort for a multi-model appraisal is likely to be around 15-20% less than the total effort for separate appraisals).

Calendar Time. With the large one-time effort for the multi-model appraisal, the calendar time for the onsite period is also likely to be higher (than that of a single model appraisal), because there is a limit to the number of ATMs that an LA can handle. Hence the ATMs will need to be out of their day-job for a longer period. The long drawn absence of the ATMs from their day job can be disruptive. A reported multi-model appraisal done had an onsite period of close to the upper limit of 90 days (see here).

ATM / LA Fatigue. This is where the multi-model (for high maturity, large organizational scope) becomes untenable. As the on-site period start crossing three weeks, the fatigue becomes obvious. In organizations that use standard processes, and have done this over many years, one can expect the documents to be similar. The responses during the interview sessions will also be similar.

For the ATMs, after the glamour of being ATMs, the novelty of PIIDs and Process Area Worksheets, and the thrill of FI-LI-PI-NI-(and NY, of course) wears out, it is an extremely boring, mind-numbing and dull exercise.

(Digression: This may be one of the reasons that LAs have become good storytellers and general entertainers. I know of a LA who sings to keep the ATMs entertained, another one tells jokes on a non-stop basis. Some LAs have started blogging. SEI may have to initiate a study to understand whether LAs have a higher tendency to ….whatever 🙂 End of Digression).

After around three weeks, the productivity, alertness, and eye for detail falls down steeply for the ATMs as well as the LA. The other issue is the stress on the ATMs of maintaining confidentiality. They cannot talk to their friends and colleagues, or smile at passing acquaintances, because they may be asked that dreaded question “and how are we today?” Those who have been ATMs know about this, others readers may ask their friends/ colleagues who have been ATMs to confirm this :-).

Target Levels/ Results. For the purpose of the results (ML/ CL), the multi-model appraisal will deliver two results, two different sets of ratings. Your target rating could be different for the two models and the appraisal result rating will also be different for the two models. This is the same as doing two separate appraisals.

Novelty / Publicity Value. “Will we be the first to do a multi-model appraisal?”; “No?”; “Okay, can we be the first do to it in this country?”, “How about this city?” and so on….

Well, if your organization is looking to announce itself as the first in something, we can surely work out some combination of conditions that you will be the first in. Not just the first, but maybe the only one. Ever.

===========================================================================

Having said all that, are there any conditions where it may be worth considering a multi-model appraisal? Yes, if you have the following it may definitely be worth considering:

  • Low number of Process Areas (ML2 kind of stuff) in both the models, and
  • Small organization (number of sampled instances are likely to be low)

Under these circumstances, the number of ATMs for each separate appraisal would be low (say 4-5), so one can increase the ATM team size to 8-10 and run the multi-model appraisal in the same number of days as a single separate one. So the organizational disruption time and LA cost can be much lower.  Also, ATM training can also be done in a single batch (max batch size is 12).

Finally, the issue is a complex one, and let us conclude by saying once again that “it depends” and that you should consult multiple LAs and take their opinions before coming to any firm conclusion.

Also refer to:

Thanks a lot to D Sankararaman, Mukul Madan, V Seshadri, Channaveer Patil, and Dan He.

Please feel feel to share your views, experiences or queries, using the “comments” feature available at the top of this article/ post.

Notes:

Nothing Official About It! – The views presented above are in no manner reflective of the official views of any organization, community, group, or association.

SM-SCAMPI and SEPG are service marks of Carnegie Mellon University.
LA- is a short form of SCAMPISM Lead Appraiser. (It is not a term of endearment like “da”, “pa”, “ma”, or “po” used in different parts of the world :-)).
ATM – Appraisal Team Member (not an Automated Teller Machine :-))

You may also be interested in the following posts uploaded on the same blog:


I am Rajesh Naik. I am an author, management consultant and trainer, helping IT and other tech companies improve their processes and performance. I also specialize in CMMI® (DEV and SVC), People CMM® and Balanced Scorecard. I am a CMMI Institute certified/ authorized Instructor and Lead Appraiser for CMMI® and People CMM®. I am available on LinkedIn and I will be glad to accept your invite. For more information please click here.

20 thoughts on “Multi-Model CMMI® Appraisals – Factors to Consider”

  1. SEI LAs not only have a standard appraisal method, but tend to have a standard set of jokes and sense of humor – the side effect of spending ample time together..

  2. An additional aspect that could be useful to know when planning a multi-model appraisal is that the scoping needs to be defined for each of the models covered, separately i.e. for each of the models included, the part(s) of the organization covered and the target maturity/ capability level(s) should be identified separately.

  3. Many organizations ask the question on “can the appraisals(for multi-models) be done together” during the intial discussions because:

    a) it sounds like a logical question to ask 🙂
    b) has a perceived appeal of “do one appraisal, get the other free” for cost conscious decision makers.

    The blog post provides insights on how though “logically” possible, this could pose a challenge “logistically” and can tend to distract the organization from its business rhythm for a substantive and continuous period.

    The other aspect to this, in my view, is that, since different models address implementation practices in different parts of the organizations and to a large extent, a different set of stakeholders, with the attendant change management issues, it may be highly unlikely for the different parts of the organization to be “appraisal ready” at the same point in time. And the business exigencies related to appraisals (or re-appraisals) could also be different (for the different work types).

    And to top it all, if the financial savings are also not significant, the probability of a large number of organizations actually undertaking a “multi-model appraisal” does seem to be less.

    1. Yes Prakash, you are right. Thanks for highlighting that different parts of the organization may not be ready for an appraisal at the same time, and that will become another factor against multi-model appraisals.

  4. Good consolidation of various factors.

    One of the factors that might save time in multi-model appraisal is the type of activities covered. For example, if it is about software development, maintenance, testing type of projects and CMMI Dev is being used for development projects and SVC for maintenance and testing projects, there can be considerable savings in appraisal time, cost in multi-model appraisal since the participants, domain etc. would be common. However, if the type of activity differs totally – like BPO practice or totally production support tickets, the time saved will be negligible as posted in the blog.

    Any way, the experts and organziation need to sit together and do a DAR (systematic Decision Analysis and Resolution, a process area covered by CMMI) for the same!

    Keep up the posting …. Good to see quite a few topics on the blog now.

    Gururaj Managuli

  5. Great topic and very nice compilation of points to consider in Multi Model situations!

    I have some thoughts to share:

    First and the foremost, organisations must perform a careful examination to consider the most suitable model and the most suitable approach regardless of marketing advantage [as, many perceive that being a CMMI appraised organisation will automatically get the attention of their buyers].

    Coming to combining the models, say in a situation that is considered critical for the company, LAs and consultants must collaborate to help make the most of the integration of the core PAs through correct interpretation as suitable to the organisation’s business conditions.

    Also, it is good to keep in mind any on-going initiatives that have relevance – such as, Portfolio Management Framework, Project Management Framework as is applied in IT Organisation before suggesting an approach for integration. Examining dependencies will help the organisations save costs significantly.

    Lastly, there should be a way to make the best from one single appraisal.

    Just my 2 cents.

  6. Sounds like writing exams for two or more different subjects concurrently. Like wanting a combined question paper. Who is saving what? You’ve answered that question brilliantly. QED.

  7. In case there is a need to align an organization’s processes with CMMi-DEV as well as CMMi-SVC, the most suitable route would probably be ITIL (ISO 20000) and not CMMi.

    ‘Not CMMi’ because 2 models will definitely be an overkill.

    The problem occurs when there’s a Biz imperative of going thru a CMMi Assessment for certain Maturity level.

    With more and more ITES organizations coming to the fore, the focus might shift to a model that is more aligned with the Service Industry.

    There is a need to bring about more awareness to the industry on ITIL and how the right implementation would benefit everyone.

    The paradigm shift is to consider Software Development as a Service.

    Trying to align the 2 giants, the following Process Areas of CMMi-DEV might just look something like this:

    ITIL: Application Life Cycle Management, Access Management, and Request Fulfillment Management
    • Requirements Management
    • Requirement Development
    • Project Planning
    • Project Monitoring & Control
    • Software Quality
    • Integrated Project Management
    • Technical Solution
    • Product Integration
    • Decision Analysis & Resolution
    • Software Product Engineering
    • Peer Reviews

    ITIL: Service Asset & Configuration Management
    • Software Configuration Management

    ITIL: Knowledge Management
    • Training Program
    • Inter-group Coordination
    • Organization Process Focus
    • Organization Process Definition

    ITIL: Event Management, Incident & Problem Management
    • Defect Prevention
    • Causal Analysis
    • Risk Management (because it would means proactive Problem management)

    ITIL: Supplier Management
    • Software Subcontract Management

    ITIL: 7-Step Improvement Process, Service Measurement, Service Reporting, Service Improvement
    • Measurement & Analysis
    • Quantitative Process Management
    • Software Quality Management

    ITIL: Change Management
    • Technology Change Management
    • Process Change Management
    • Organizational Innovation & Deployment

    These 2 changes must be handled like any other change to the organizational Asset base

    The rest of the CMMi-SVC Process Areas are in any case very well aligned with ITIL already.

    ITIL: Service Design
    · Service Continuity
    · Service Definition
    · Service Transition
    · Service System Development
    · Capacity Management
    · Availability Management
    · Supplier Management

    ITIL: Service Strategy
    · Strategic Service Management

    I may have missed out some process areas from either framework.
    However, my intent is to demonstrate how ITIL could help such organizations.

Leave a Reply

Your email address will not be published. Required fields are marked *