A multi-stakeholder working group, with the support of the Dutch standards authority (NEN), has created a non-binding technical agreement for profiling algorithms. Unfortunately, the standard does not prohibit discriminatory practices.
The DUO scandal, where students with a non-European migration background had a higher chance of receiving an unfounded visit by the Dutch students grant’s fraud department, is an example of the fact that many (government) organisations have no idea what they are doing when it comes to using profiling algorithms. This is why, in 2025, a working group with representatives from many different organisations started working on creating a technical standard to improve this matter.
Amnesty International, Bits of Freedom, Controle Alt Delete, and Expertisecentrum Data-Ethiek were part of the conversation, but couldn’t put their name under the final result. Their dissenting minority opinion has also become part of the standard. In human rights law, these organisations see no space for allowing indirect discrimination through profiling. Yet, other organisations in the working group, the Dutch Institute of Human Rights included, were unwilling to prohibit this form of discrimination.
There are some advantages if organisations are going to use and properly follow the standard. Firstly, the standard argues quite strongly for being explicit about the goals that you want to accomplish as an organisation, and making sure that you can assess whether those goals are met. The standard requires you to look at other ways of accomplishing the same goal, ways that don’t require profiling. And it forces you to write up your reasons for choosing profiling over other solutions. If correctly applied, the standard should move a lot of algorithmic racism from implicit to explicit. Hopefully, that puts somewhat of a brake on potentially discriminatory algorithms coming into this world.
The standard defines ways of measuring whether the profiling algorithm is discriminatory. But there is a fundamental problem with verifying technical performance, namely that it presupposes a correct/accurate decision, a ‘ground truth’ against which measurements can be made. But what if that ground truth is itself discriminatory in nature? In that case, you end up building a predictive model that approximates that discriminatory reality as closely as possible. To explain this issue, we created the following thought experiment:
Suppose that at present there is a decision-making process involving two decision officers. The first selects all cases that need to be additionally checked for fraud, and does so with reasonable precision. The second reviews the selected cases and, after investigation, decides whether fraud has occurred. Suppose further that there are two groups, A and B, and that for the individuals concerned in both groups the real probability that they have committed fraud is 50%. But suppose that the second decision officer is strongly biased and always labels all individuals from group A as fraudsters and none of the individuals from group B. Suppose we now want to replace the first decision officer: we let a data-driven algorithm determine whether someone should be checked. If the algorithm is trained to match the second decision officer’s final decision as closely as possible, then, under this technical standard, this will lead to more discrimination than in the previous situation, but without us necessarily knowing it.
This extreme case will not occur in such an extreme form in real life. Still, the problem is nonetheless a real risk, particularly in a society with all kinds of structural and problematic forms of discrimination. Moreover, it is a risk that is in fact almost impossible to measure. Our comment about this problem led to an extra sentence being added to the standard, in paragraph 8.7, where the user of the standard is asked to take another look at whether their plans can lead to discrimination. The sentence reads:
When making this assessment, take into account that residual risks of discriminatory effects may remain, given all the processes surrounding the profiling algorithm that can lead to discriminatory effects.
Every government organisation that uses profiling algorithms now has homework to do. “We didn’t know it was discriminatory” has now shifted to “We were too incompetent (not up to standard) to know it was discriminatory” or “We knew it was discriminatory, but we didn’t care”.
See: Waarom weigert de overheid te stoppen met discriminerende algoritmen? at the Volkskrant, and NTA 8047:2026 nl at NEN.
(The fact that this standard is copyright-protected and requires an account and agreeing to a terms of service agreement to be able to read it, rather than being published in a truly open manner, is a fight for another day).
Image: Kathryn Conrad / Datafication / Licenced by CC-BY 4.0.