Multi-Criteria Decision Analysis (MCDA/MCDM)

Points systems

A common and equivalent representation of the linear equations at the heart of weighted-sum models is a schedule of point values (sometimes referred to as preference values) for each criterion. These schedules are also known as points systems, and also as additive, linear, scoring, point-count and points models or systems.

The points-system equivalent of the weighted-sum model in Table 2 appears in Table 3. In the table, the point value for each level on a criterion represents the combined effect of the criterion’s relative importance (weight) and its degree of achievement as reflected by the level.

In other words, consistent with the linear equations described above, a level’s point value can be obtained by multiplying the criterion weight by the criterion score from the equation. This relationship can be easily confirmed by multiplying the weights by the scores in Table 2, as reported in the fourth column below (for illustrative purposes). Likewise, a point value can be decomposed – in the opposite direction – into the criterion’s weight and score.

A points-system representation is sometimes preferred because, arguably, a simple schedule of point values is easier to implement than an equation (a set of weights and scores that need to be multiplied together). Also, in tabular form, a points system takes up less space (e.g. compare Table 2 and Table 3).

Applying the points system in Table 3 to Romeo’s and Juliet’s ratings on the three criteria (Table 1) generates their total scores:

• Romeo’s total score (for poor, > 3 years, good): 0 + 32 + 18 = 50 points
• Juliet’s total score (good, ≤ 2 years, good): 40 + 0 + 18 = 58 points

Notice that these total scores are the same as before, thereby confirming the equivalence of Tables 2 and 3. Again, based on their total scores, Juliet is ranked ahead of Romeo for the job.

Notice, again, that the hypothetically best candidate, with the highest possible rating on each criterion, would get: 50 + 32 + 18 = 100 points.

Simplicity and accuracy

In practical terms, a major attraction of weighted-sum models – in either their linear equation (Table 2) or points-system (Table 3) forms – is the simplicity by which each alternative’s performance on each criterion is aggregated to produce the alternative’s total score. (Non-linear functions for aggregating the criteria, including multiplicative functions, are also possible but very rare.)

More importantly, such simple models (sometimes referred to as algorithms or formulas) have been found nearly universally in very many studies to be more accurate than the intuitive or holistic judgments of decision-makers (Kahneman 2011). They are recognized as being a simple and powerful solution to the enormous (and undesirable) variability – noise – observed in most experts’ judgments (Kahneman, Sibony & Sunstein 2021).

According to Hastie and Dawes (2010, p. 52), weighted-sum models are:

surprisingly successful in many applications. We say “surprisingly”, because many judges claim that their mental processes are much more complex than the linear summary equations would suggest – although empirically, the equation does a remarkably good job of “capturing” their judgment habits.

Hastie and Dawes (p. 52) also explained:

The mind is in many essential respects a linear weighting and adding device. In fact, much of what we know about the neural networks in the physical brain suggests that a natural computation for such a “machine” is weighting and adding, exactly the fundamental processes that are well described by linear equations.

Also, according to Hadorn et al. (2003, p. 49):

[Weighted-sum models / points systems] are, as a rule, more accurate than human predictors. This is not surprising, as it has been known for decades that human cognitive performance is limited with respect to complex multi-variable judgment and prediction tasks (Meehl 1954).

Performance matrices

The most basic alternative approach is a simple table, sometimes referred to as a performance matrix, for reporting the alternatives’ performance on the criteria. In the table/matrix, each alternative is a row and each criterion is a column.

If one alternative dominates the others on all criteria, or where the trade-offs involved in selecting an alternative are clear and uncontroversial, then decision-makers can use such a table to reach their decision quickly.

However, most decision problems are more complicated than this!

Most MCDA / MCDM applications involve confronting non-trivial trade-offs between criteria. Therefore, merely tabulating alternatives’ performance on the criteria is insufficient for most applications. Nonetheless, such a table may be useful at step 3 in the MCDA process outlined in the next section.

Outranking methods

A MCDA / MCDM approach capable of evaluating trade-offs between criteria, but that is not based on aggregative functions for combining alternatives’ performance on criteria (via weights), is the group of outranking methods.

Methods in this tradition include:

• PROMETHEE (Vincke & Brans 1985) – an acronym for “Preference Ranking Organization METHod for Enrichment of Evaluations”
• GAIA (Brans & Mareschal 1994) – for “Geometrical Analysis for Interactive Aid”
• ELECTRE family of methods (Roy 1991) – for “ELimination Et Choix Traduisant la REalité” (which translates as “elimination and choice expressing reality”)

In essence, outranking methods involve decision-makers pairwise ranking all the alternatives relative to each other on each criterion in turn and then combining these pairwise-ranking results but not via weights. The objective is to obtain measures of support for judging each alternative to be the top-ranked alternative overall.

Relative to weighted-sum models, outranking methods are rarely used. Their unpopularity is probably mostly due to their complexities and the non-intuitive nature of their inputs and algorithms relative to weighted-sum models.

The steps for creating and applying weighted-sum models, including a survey of methods for determining weights, is explained next.

1. Structuring the decision problem

The first step of the MCDA / MCDM process involves structuring and framing the decision problem being addressed (and see the first stage in the earlier diagram again).

In particular, it’s important to clarify the over-arching objectives of the decision-making exercise, and to think about the overall outcomes that are desired.

Related issues include:

• Identifying the alternatives under consideration
• Whether the decision is a one-off or repeated application (as discussed earlier)
• The type of output required from the MCDM, e.g. a ranking or a selection

These elements should be validated with stakeholders.

2. Specifying criteria

The criteria specified for the decision should be valid, reliable and relevant to decision-makers and other stakeholders. They should also be preferentially independent, as explained below.

The criteria should be structured to avoid double-counting and minimize overlap between them. For example, having criteria for attractiveness and also beauty when choosing a car would invalidate the weighting methods discussed below and their results.

Depending on the application, the criteria can be identified from: similar decisions, reviews of the literature, facilitated discussions, expert opinions, focus groups and surveys. As in the previous step, stakeholders should be involved in identifying and validating the criteria.

For example, 1000minds offers noise audits whereby participants use their intuition to rank real or imaginary alternatives to inspire a discussion of the criteria (as well as to demonstrate how inconsistent decision-makers are and hence the need for a better approach than relying on intuition).

3. Measuring alternatives’ performance

Data about the alternatives’ performance in terms of each of the criteria can be gathered in a variety of ways, ranging from expert opinions to rapid literature reviews, to full systematic reviews and modeling exercises. These data can be presented in a performance matrix or table (as discussed earlier), if useful.

The sophistication and intensity of the data-gathering activity depend on the availability of relevant evidence, the decision problem, and also other practical factors such as the resources available for the job.

MCDA / MCDM is capable of including both quantitative and qualitative data. It is also capable of combining subjective judgments (in the absence of “harder” data) with more traditional scientific evidence in the same application. For example,1000minds has surveying tools for collecting decision-makers’ ratings of the alternatives on the criteria, and for aggregating results.

4. Scoring alternatives and weighting the criteria

Scoring alternatives and weighting criteria are sometimes considered separate steps. They are treated together here because they are intrinsically linked. They can be performed sequentially, simultaneously or iteratively, depending on the application and methods (or software) used.

Scoring alternatives on the criteria

Scoring the alternatives on the criteria involves converting each alternative’s performance on each criterion into a numerical score. The scores are usually normalized so the worst performance on the criterion gets a score of zero and the highest performance gets 100.

Scores on the criteria can be implemented using a continuous scale (0-100). Or, alternatively, as in the example at the beginning of the article, two or more mutually-exclusive and exhaustive levels (e.g. low, medium, high) can be employed, each with a point value (e.g. medium = 60).

Weighting the criteria

Weighting the criteria involves determining their weights, representing their relative importance to decision-makers. The weights are usually normalized to sum across the criteria to one (100%).

Overall, it’s very important that the alternative scores and criterion weights are as valid and reliable as possible.

Otherwise, if the scores or weights are “wrong” (inaccurate), then the “wrong” decision, as determined by the ranking of the alternatives’ total scores, is almost certain – even if valid and reliable criteria have been specified (step 2) and the alternatives’ performances have been accurately measured (step 3).

A variety of methods for scoring alternatives on the criteria and weighting the criteria are available, as discussed in the next section.

5. Applying scores and weights to rank alternatives

Having scored the alternatives on the criteria and weighted the criteria, it’s easy to calculate their total scores (usually performed automatically by MCDA software).

With reference to the scoring and weighting methods explained below:

• For the direct methods and, in effect, the PAPRIKA (choice-based) method, each alternative’s scores on the criteria are multiplied by the weights, and then the weighted scores are summed across the criteria to get each alternative’s total score (e.g. see Table 2 and Table 3 above).
• For the conjoint analysis (or DCE) method, the regression technique estimates each alternative’s value (or utility) or its probability of being preferred by decision-makers.

6. Supporting decision-making

The final step of the MCDA / MCDM process is to use the results, usually presented in tables like the one above, or charts, to support your decision-making – i.e. depending on your application, for ranking or selecting alternatives.

Where appropriate, the MCDM inputs, process and results can be used to communicate and explain the final decision to stakeholders.

As for any analysis based on uncertain inputs, it’s important to perform sensitivity analysis to check the robustness of your results to plausible variations, e.g. in the alternatives’ performance (step 3) and their scores, or on the criterion levels, or in the criterion weights (step 4).

Overall, decision-makers need to understand the MCDM results, including any significant limitations of the analysis. These results can be used to support decision-makers in reaching and, if need be, defending their decision.

It is important to remember, as emphasized at the beginning of the article, that MCDM is intended as a tool to help people, individually or in groups, to reach a decision – their decision (the humans’ decision), not the tool’s decision.

Also, in some applications, other considerations in addition to the MCDM results may also be relevant for the decision being made.

For example, decisions with budgetary implications, such as investment or project decision-making (e.g. CAPEX or OPEX), involve comparing the benefits of each alternative, as represented by their MCDM total score, relative to their costs. Other factors, such as strategic or legal factors, might also be worthwhile considering.

Value-for-money assessments

The MCDA / MCDM process explained above, resulting in a decision-making tool – a weighted-sum model, or points system, comprising criteria and weights – can be complemented by a framework for making resource-allocation decisions based explicitly on value for money and efficiency, as summarized here.

For the simple example of project decision-making subject to a budget, consider the value-for-money (VfM) chart in Figure 2.

The vertical axis of the VfM chart represents each project’s benefit, expressed in terms of a total score, i.e. based on the criteria and weights derived earlier. The horizontal axis displays each project’s estimated cost.

As an illustration, the chart can be used to think about the following questions.

Suppose you had a budget of $8 million.

Which of the 15 projects in the chart – costing more than $16 million, in total – represent high quality spending, e.g. with high benefit-cost ratios?

Which projects would you reject because they represent low quality spending, e.g. that are dominated by the efficient projects?

Thinking about the first question should draw your focus of attention to the chart’s top-left quadrant, i.e. projects with relatively high Benefit (total score) and low Cost.

Possible trade-offs between Cost and Benefit are along the gold line, known as the efficiency or Pareto frontier: higher Cost can be compensated for by higher Benefit.

The size and color of the bubbles in the chart can also be used to represent other potentially important considerations, such as confidence in cost estimates (e.g. high, medium, low), synergies, political support, etc.

In general, applying this value for money framework involves decision-makers considering alternative affordable combinations to arrive at their “optimal portfolio” of projects.

In operations research, this combinatorial optimization problem is an example of the famous “knapsack problem” (Dantzig & Mazur 2007).

Direct versus choice-based methods

Direct methods for scoring alternatives on the criteria involve decision-makers directly expressing how they feel about the relative performance of each alternative on a criterion relative to the other alternatives.

These expressions of a decision-maker’s preferences are usually measured using either an interval scale (e.g. alternatives are rated on a 0-100 scale on a criterion) or a ratio scale (e.g. “alternative A is three times as important as alternative B on a criterion”).

Similarly, direct methods for weighting the criteria involve decision-makers (directly) expressing how they feel about the relative importance of the criteria, again usually represented in terms of either an interval scale (e.g. criteria are rated on a 0-100 scale) or a ratio scale measurement (e.g. “criterion A is three times as important as criterion B”).

Direct methods have the superficial appeal of directly asking the decision-maker about how they feel about the relative importance of the criteria or alternatives, from which scores for the alternatives on each criterion or criterion weights are derived (i.e. directly).

Also, when you read the direct methods’ explanations below, hopefully, you’ll to be struck by their algorithmic and computational simplicity, such that they can be easily implemented “by hand” (e.g. supported by spreadsheets), whereas the choice-based methods require specialized software (which come with other useful features, of course).

Unfortunately, however, these direct methods have the major disadvantage of being based on preference-elicitation tasks that are cognitively difficult for most people, and so the validity and reliability of their results is questionable. For example, expressing how strongly you feel about one criterion (or alternative) relative to another one on a scale ranging from “equally preferred” (ratio = 1) to “extreme importance” (ratio = 9) is not a natural human mental activity that many of us have much experience of.

As an example, suppose you’re an employer looking to hire someone for a job. How easy would you find thinking about a question like this:

On a nine-point scale ranging from “equally preferred” (ratio = 1) to “extreme importance” (ratio = 9), how much more important to you is a person’s qualifications than their experience?

How confident would you be in the accuracy (validity and reliability) of your answer?

In contrast, the choice-based methods explained below involve decision-makers expressing their preferences usually by answering relatively simple questions based on choosing between two or more alternatives defined on some or all of the criteria – an example of such a question appears in Figure 3 below. From these rankings or choices, scores and weights are derived (indirectly) using regression-based techniques or other quantitative methods.

The advantage of these choice-based methods is that choosing between things is a natural type of decision activity experienced by all of us in our daily lives. We make such judgments dozens, if not hundreds, of times every day (e.g. “would you like scrambled or poached eggs for breakfast?”). Therefore, choice-based methods are likely to be more valid and reliable than direct methods, as well as being easier for decision-makers.

In the words of Drummond et al. (2015):

The advantage of choice-based methods is that choosing, unlike scaling, is a natural human task at which we all have considerable experience, and furthermore it is observable and verifiable.

The eight direct and two choice-based methods are briefly explained in turn below (and more detailed information is available from their references above). Given the advantages of choice-based methods discussed above, you might like to skip to that sub-section.

Direct methods

Each of the following eight methods is explained in terms of scoring alternatives on the criteria, and weighting the criteria.

Choice-based methods

In contrast to the direct methods above, the following choice-based methods are based on scoring alternatives on the criteria and weighting the criteria together in the same process.

These methods are algorithmically more sophisticated than the direct methods above, and so they cannot be implemented “by hand” (like most direct methods), and the explanations below are relatively “broad brush”.

Which method is best?

The eight direct and two direct methods surveyed above all have strengths and weaknesses – such that choosing the “best” method is itself a multi-criteria decision problem!

As discussed earlier, the choice-based methods – the PAPRIKA method and conjoint analysis (or discrete choice experiments, DCE) – are likely to be more valid and reliable than the direct methods, as well as being easier for decision-makers.

Because they tend to be algorithmically more sophisticated than direct methods, choice-based methods need to be implemented using specialized software instead of “by hand” (as is possible for most direct methods). This is not necessarily a disadvantage of course.

In brief, De Montis et al. (2004) recommends considering the following issues when thinking about which method to use.

• How well do the methods elicit trade-offs between the criteria?
• How much time and other resources are required to implement the methods?
• How high is the cognitive burden imposed on participants?
• Is a skilled facilitator required to implement the methods?
• Is additional data processing and statistical analysis required?
• Are the underlying assumptions valid with respect to decision-makers’ preferences?
• Will the outputs produced satisfy decision-makers’ objectives?

Try 1000minds!

Step up your decision-making with 1000minds, which uses MCDA to make decisions consistently, fairly and transparently. Try a free 15-day trial today, or book a free demo to see how 1000minds can help you make better decisions.

Groups and websites

• Decision Analysis Society of INFORMS
• Decision Sciences Institute
• International Society on Multiple Criteria Decision Making
• EURO Working Group on Multicriteria Decision Aiding
• Society for Medical Decision Making
• Multiple Criteria Decision Analysis eLearning Site

Wikipedia articles

• Decision-making
• Multi-criteria decision analysis
• Decision-making software
• PAPRIKA

Journals

• Journal of Multi-Criteria Decision Analysis
• Decision Analysis
• Decision Sciences
• Operations Research
• Management Science
• European Journal of Operational Research
• International Journal of Information Technology & Decision Making