The Dutch Institute for Human Rights has published an evaluation framework for risk profiling intending to prevent discrimination based on race or nationality.
The framework can be downloaded in its full version or in a shorter version.
The evaluation is in two parts. First, the criteria are evaluated. Does the risk profile make a distinction between groups and persons? And is this distinction “direct” on the basis of race or nationality? If yes, then you shouldn’t use the profile. Is there an indirect distinction made based on race or nationality? If yes, then you go to the second part.
The second part evaluates the legitimacy of the risk profile. Is the goal legitimate? Is the risk profile fitting for the goal? Is it necessary? Is there a reasonable alternative (checking the subsidiarity)? And looking at the whole picture, is the use of the profile proportionate?
For this proportionality assessment, you check the importance of the benefits of the profiling and weigh those against the harmful effects of the profiling, for example, the risk of stigmatisation or the burden of being checked.
The Institute then tells you to keep monitoring the situation, reactie appropriately to complaints, and following up on signals of discrimination. All in all, the (Dutch) flowchart looks like this:
The evaluation framework is a classic legal – and therefore relatively narrow – perspective on the matter. It is a good start and important to follow, but it won’t address all potential adverse effects of algorithmic risk profiling.
For example, the framework seems to think that there are risk profiling tools that are not indirectly discriminatory. This misunderstands the true nature of variables in the wild: any variable is usually a proxy for any other variable.
The framework also doesn’t tackle one of the most critical problems of machine learning-based risk profiling algorithms: they create a fundamental opacity for the people subjected to the profiling as to what rules are applied to them. This, in turn, prevents the ability of these people to contest these profiling decisions meaningfully.
See: Nieuw toetsingskader tegen discriminatie door risicoprofilering at College voor de Rechten van de Mens.