Criteria: the Citizens of the MCDA Realm

At the end of our recent discussion on Even Swaps, we talked about several limitations of this simple decision-making method: it is more suited to smaller problems, it can't be used to communicate our decisions to others, and its result is only a single alternative. We get no ranking of alternatives, nor more insight into the problem.

These issues are a good incentive to look at other decision-making methods. Before we can do that though, we first need to learn a little more about the concept of criteria: the foundation of the whole Multiple-Criteria Decision Analysis business. Equipped with this knowledge, we can then dive into more interesting decision methods.

What is a Criterion (again)

We already know what a criterion is. Let's use a definition from Denis Bouyssou to refresh our memory:

.. we will call criterion a "tool" allowing to compare alternatives according to a particular "significance axis" or a "point of view"

So a criterion has two important attributes: (1) it represents a certain point of view and (2) it allows us to compare different alternatives. Those with some technical background can think of a criterion as a function. Its input is an alternative, and its output is (in most cases) a number that we can use to make meaningful comparisons between different alternatives.

Just as there are different groups of people living in a country, there are also different types of criteria we can consider when dealing with a decision-making problem:

Criteria by Value Types

Perhaps the most important classification is based on the type of value we assign to alternatives during evaluation. We have:

• Numeric or real-valued criteria: These criteria are the most straightforward. We have already mentioned examples such as monthly rent, apartment size, phone screen size, or storage capacity. In all of these cases, we can assign a number to each alternative. For example: "Apartment A costs 580 EUR per month and its total area is 55 square meters".

• Binary criteria: Here we have only two possible values to assign to each alternative: yes/true or no/false. These criteria allow us to express the presence or absence of a feature or characteristic. When looking for an apartment we can have a binary criterion such as "Has balcony?". Similarly, when buying a laptop, we might be curious about the presence of an SD card slot.

• Ordinal scale criteria: Ordinal criteria allow us to rank alternatives on a predefined scale. These scales can be numerical, such as a rating from 1 to 5, or linguistic, such as a scale ranging from very good to very bad. Our world is full of features that can't be fully measured and quantified. Ordinal scales give us an approximation. We have already seen some examples in the Even Swaps example: neighborhood quality and apartment condition.

• Nominal criteria: Values of nominal criteria are words or phrases with no objective order. Color or material are good examples. We can't automatically say that "blue" is preferred to "red" or that "magnesium" is preferred to "aluminum". If you ask multiple people about their favorite color, you'll get many different answers. This does not happen with ordinal scales. Everyone will agree that a "good" neighborhood is better than an "average" neighborhood, which is better than a "bad" one.

Sidenote: There is a field at the intersection of mathematics and computer science called computing with words. As the name suggests, the idea is to develop methods for performing "calculations" on natural language expressions instead of numbers, while embracing all their vagueness and ambiguity.

Low or High?

Another important classification of criteria is based on whether we prefer lower or higher values. If a higher value is preferred, we talk about beneficial or maximizing criteria. Everything else being equal, most of us probably prefer a larger apartment or a higher salary. On the opposite side, we have non-beneficial or minimizing criteria whose value should be as low as possible. We had two such criteria in our apartment problem: commute time and monthly rent.

There is also a kind of middle ground between the two extremes. We may encounter criteria whose value should be as close as possible to some target value. Consider team size, for example. Too few people on a team and not enough work gets done. Too many, and communication overhead and bureaucracy become an issue.

Such "ideal value" criteria can be transformed into minimizing criteria. Instead of working directly with team size, we can use the difference between the actual value and our ideal. If we consider 7 to be an ideal team size, then a project who has 9 (or 5) team members misses the target by 2 people. After this transformation, we will prefer projects with a smaller difference:

As we'll see, the distinction between minimizing and maximizing criteria is very important. Many methods only work if all criteria have the same "direction". Fortunately, these methods usually come with ways to invert this direction.

Words to Numbers

Many, perhaps most, MCDA methods can only work with criteria that have numerical values. However, as we just saw, there are many criteria types whose values are words, not numbers. What can we do about them? We could simply ignore such criteria, but that would mean a significant loss of information resulting in a wrong decision. Finding a way to convert words into numbers might be a better approach.

Binary criteria

Binary criteria are the easiest to deal with. We can represent yes/true as 1 and no/false as 0. Or, if we prefer the absence of a given feature to its presence, we can invert this mapping.

Linguistic scales

We can deal with linguistic scales such as very good - good - average - bad - very bad by transforming them into a numerical scale. The simplest approach is a scale based on the order of each linguistic value:

This representation assumes, that the difference between, say, very bad and bad is equal to the difference between, say, average and good. This may not be true in reality. Economists often talk about something called diminishing marginal utility. If your salary went up from 1000 EUR a month to 1500 EUR a month, it would probably have a much greater impact than an increase from 2500 EUR to 3000 EUR. One way to express this diminishing marginal utility is with a logarithm:

Now the difference between good and very good is much smaller than the difference between very bad and bad. Note that the logarithmic transformation can be also applied to numeric criteria.

Nominal criteria

Let's move on to the most difficult situation: nominal criteria. The first approach we can use is to rank the values of a nominal criterion according to our subjective preference. For example, I can say that my ranking of car colors is:

1. black (most preferred)

2. green

3. blue

4. white

5. red (least preferred)

Now that we have an order of preference, we can transform it into a numerical scale, just as if we were working with an ordinal criterion.

Determining a preference order directly can sometimes be difficult. We have to think about all the possible values of a given criterion at once. This is a hard task. We can simplify it by working with only two values at a time. We will make a series of pairwise comparisons between all possible pairs of values.

The simplest pairwise comparison method is Fuller's triangle. Let's use this method for our car color problem. First, we create an empty table with each row and column representing a different color. Then, we focus on the upper right triangle of this table. In each pair of colors, we pick the one we prefer:

Now we simply count the "number of points" given to each color. Green and black got 3 points, blue and white got 2. Red got no points at all. These scores can now serve as a numerical representation of our colors.

There is an even more advanced method of pairwise comparison, first proposed by Thomas Saaty. It extends the concept by allowing you to also express how much you prefer one option over the other. We will discuss this method in depth once we get to Analytic Hierarchy Process.

Let me briefly mention two other ideas for quantifying nominal criteria:

• Instead of scoring different values with pairwise comparisons, you can directly assign scores from a scale, say 0 to 10.

• It is also possible to transform a nominal criterion into a set of binary criteria. We could split our car color criterion, into five binary criteria: "Is green?", "Is blue?", "Is red?", "Is white?", and "Is black?". Ones and zeros are all the numbers we now need. This approach isn't used much in decision-making. However, it is still worth mentioning because of its wide utilization in machine learning.

Selecting the right criteria

We have already talked about two principles for selecting criteria: completeness and non-redundancy. Completeness refers to the requirement to include all relevant criteria. On the other hand, non-redundancy tells us to avoid criteria that provide information similar to another criterion.

Bouyssou gives us more advice:

• We should make sure that every person involved in the process understands what each criterion means and how it is evaluated.

• We should also consider the quality of information we can get on a given criterion. Caution is in place if the values of a criterion are more of a guess than a fact.

• Finally, given the limitations of our brains, it does not make sense to consider more than 10 to 12 criteria when solving a decision problem. This issue can be somewhat mitigated by methods such as Analytic Hierarchy Process which, as the name suggests, allows us to build hierarchies of criteria.

---

We have explored many aspects of the concept of criteria. Let's summarize what we have learned:

Criteria are tools that allow us to compare different alternatives from different points of view. We can categorize criteria according to values they can take: binary, numeric, ordinal, and nominal. The direction of is preference also important. We distinguish between maximizing/beneficial and minimizing/non-beneficial criteria. Finally, many decision-making methods cannot be applied to criteria whose values are words. There are many methods to solve this problem: transforming binary criteria into 0's and 1's, ordering criteria according to our preference, or pairwise comparison.

As you might already be guessing, when solving a decision problem, not all criteria should be equally important. Some should affect the outcome more than others. In the next article, we will discuss the concept of criteria weights which is a very common approach to deal with this requirement.