City of Amsterdam’s Algorithm Register(algoritmeregister.amsterdam.nl)
algoritmeregister.amsterdam.nl
City of Amsterdam’s Algorithm Register
https://algoritmeregister.amsterdam.nl/en/ai-register/
41 comments
Since it's just copied (or bought) from Helsinki (https://ai.hel.fi/en/ai-register/) / Saidot (Saidot.ai), I wonder if it's just something they bought but don't really care/understand what it's about or what they're putting there.
On the Saidot website it sais:
> Saidot develops and operates the City of Amsterdam's Algorithm Register
> Saidot develops and operates the City of Helsinki's AI Register
I don't know how much they've poured into this, but I've worked on a project managed by Amsterdam Data [0] before and they're pretty serious about data.
[0]: https://data.amsterdam.nl
> Saidot develops and operates the City of Amsterdam's Algorithm Register
> Saidot develops and operates the City of Helsinki's AI Register
I don't know how much they've poured into this, but I've worked on a project managed by Amsterdam Data [0] before and they're pretty serious about data.
[0]: https://data.amsterdam.nl
Amsterdam and Helsinki BOTH have developed this register together with Saidot actually. We've build on our work on procurement conditions for AI, defining and operationalizing concepts like technical and procedural transparency (see https://www.amsterdam.nl/wonen-leefomgeving/innovatie/de-dig... at the bottom of the page)
The development of these two registers has been a very deliberate process for both cities, to come up with a register that has technical detail as well as makes the algorithms understandable for the masses. If we can improve in any way, please leave your feedback in the survey, it's most welcome!
The development of these two registers has been a very deliberate process for both cities, to come up with a register that has technical detail as well as makes the algorithms understandable for the masses. If we can improve in any way, please leave your feedback in the survey, it's most welcome!
What is the intent of an algorithm register? And why Amsterdam? Isn't this more something that should be on an international level?
Amsterdam is trying to set a precedent. By developing this register, they're showing national/international policy makers that transparency can be done.
Aside from the fact that any level of gov should be open about how technology affects their citizens.
This kind of question gives me strong flashbacks of corporate meetings (:
Transparency. It’s Amsterdam because the city of Amsterdam decided to publish their algorithms in use. They do the same for all kinds of data and systems already, see data.amsterdam.nl
Transparency. It’s Amsterdam because the city of Amsterdam decided to publish their algorithms in use. They do the same for all kinds of data and systems already, see data.amsterdam.nl
Wow cool to see my country is doing this. is actually pretty awesome. Do other governments also do this? There is even an architecture breakdown on the Api that does the scanning of the license plate for ticketing.
https://algoritmeregister.amsterdam.nl/en/automated-parking-...
https://algoritmeregister.amsterdam.nl/en/automated-parking-...
Same service in Helsinki: https://ai.hel.fi/en
The sites look the same. Both are developed by https://www.saidot.ai/.
I wonder what are the implications and liabilities to leave such a task to a private company. Let's say Sadiot overlooks some model behavior that leads to racial discrimination, who'd be liable? The government's data science team who set up the model or Saidot?
Yeah, happens to be a company from the Helsinki area.
Interesting some good quality details into the service and Algo construction.
Quite interesting to see how the word "algorithm" seems to be changing over time from a straightforward mathematical or CS "series of steps to solve a problem" to a general byword for any sort of automated reasoning, especially ones with a more malevolent undertone.
I'm actually kind of struggling to come up with a concise definition in the new popular usage.
I'm actually kind of struggling to come up with a concise definition in the new popular usage.
Isn't it still just "a series of steps to solve a problem", but just in a little broader sense?
E.g. if your problem would be how to rate a person's need for social housing, you probably would apply some objective parameters to that. You are trying to get a binary answer that ideally selects all people who would need such a thing and leaves out all the others, without having much false negatives or false positives.
Unlike in the traditional algorithm definition this involves bureaucratic and legal steps, some of which can be a bit handwavey and have their outcome depend on world image and daily form of the individual "processor" who was tasked to carry them out (better not have your task run right before the lunch break by grumpy Edwald). In other cases the step might be more of the nature "insert numbers in that one Excel sheet some wizard built aeons ago and now nobody understands".
An algorithm however doesn't need to be a good algorithm or entirely deterministic to fit the definition of "algorithm". It is a series of steps carried out to solve a problem. A cooking recipe.
E.g. if your problem would be how to rate a person's need for social housing, you probably would apply some objective parameters to that. You are trying to get a binary answer that ideally selects all people who would need such a thing and leaves out all the others, without having much false negatives or false positives.
Unlike in the traditional algorithm definition this involves bureaucratic and legal steps, some of which can be a bit handwavey and have their outcome depend on world image and daily form of the individual "processor" who was tasked to carry them out (better not have your task run right before the lunch break by grumpy Edwald). In other cases the step might be more of the nature "insert numbers in that one Excel sheet some wizard built aeons ago and now nobody understands".
An algorithm however doesn't need to be a good algorithm or entirely deterministic to fit the definition of "algorithm". It is a series of steps carried out to solve a problem. A cooking recipe.
Dutch public prosecutors used (until 2015) a software system called 'BOS/Polaris' for sentencing guidelines. This was an 'algorithmic' system in its most 'pure', CS/math meanings. It was a fairly simple deterministic software package that used a point-based system to go from objective (type of crime, first time offender or not etc) and some subjective (attenuating circumstances) criteria to a specific charge for a given case. This software was free to download (no source though), the 'rules' were based on a number of guidelines for various types of crimes that were published separately.
It was discontinued because apparently it was felt it didn't allow for enough of the 'subjective' aspect of it. The problem was also a bit that the software had slowly become less 'guiding' and more 'the norm', where judges and prosecutors felt they had to really justify deviating from the outcomes of the software. I imagine that could have been solved with a more tunable parameter for how much to weigh the 'subjective' part, but that would open another can of worms of course; and it's also a philosophical issue, so I imagine that a lot more went on behind the scenes that the general public will never know.
Anyway, that system would never be put under the umbrella 'algorithms' nowadays. That has shifted to ML/stats systems where the 'morally problematic' part of the algorithm are really the 'neural network weights' (or regression coefficients, depending on how impressed you are by raw processing power over system understanding... but that, again, is a philosophical issue).
To rephrase: my position is that the problem people have with what they call 'algorithms' isn't really the algorithms at all (nobody who talks about this problem, at this abstraction level, cares abour gradient descent vs simulated annealing), but the parameters to the resultant models and how those parameters were estimated; specifically what data was used to make those estimates.
I personally really despise this, in my view, dumbing down of the matter and muddying of the terminology and hence issue, but then again nobody asked me when 'hacker' was rebranded from 'a specific type of technical intellectual curiosity' to 'computer criminal', so...
It was discontinued because apparently it was felt it didn't allow for enough of the 'subjective' aspect of it. The problem was also a bit that the software had slowly become less 'guiding' and more 'the norm', where judges and prosecutors felt they had to really justify deviating from the outcomes of the software. I imagine that could have been solved with a more tunable parameter for how much to weigh the 'subjective' part, but that would open another can of worms of course; and it's also a philosophical issue, so I imagine that a lot more went on behind the scenes that the general public will never know.
Anyway, that system would never be put under the umbrella 'algorithms' nowadays. That has shifted to ML/stats systems where the 'morally problematic' part of the algorithm are really the 'neural network weights' (or regression coefficients, depending on how impressed you are by raw processing power over system understanding... but that, again, is a philosophical issue).
To rephrase: my position is that the problem people have with what they call 'algorithms' isn't really the algorithms at all (nobody who talks about this problem, at this abstraction level, cares abour gradient descent vs simulated annealing), but the parameters to the resultant models and how those parameters were estimated; specifically what data was used to make those estimates.
I personally really despise this, in my view, dumbing down of the matter and muddying of the terminology and hence issue, but then again nobody asked me when 'hacker' was rebranded from 'a specific type of technical intellectual curiosity' to 'computer criminal', so...
Oh, and how that is related to current practice (because I lost my train of thought somewhere) is that 'modern' (=ML-based) experimental sentencing guideline systems try to overcome this 'subjective bias' by injecting an 'objective subjective bias' by using ML methods. So basically by looking at many more circumstances and training the former 'points' system using ML, so in a way automatically deriving what used to be set by manually devised guidelines (as crude as they were) from past data - under the assumption that our past data is somewhat objective, and/or because we know it wasn't, additionally trying to correct for past biases.
It's easy to see the problems with this approach and I don't have to rehash them here; and much of the interesting (academic) work here is in rebalancing and fixing the biases; but that is research, not something that is ready for actual use in democratic societies (not that that has stopped some places from using them anyway...). I'm not convinced it's even possible and any such system will invariably be very opaque. Imagine, if judges and prosecutors already find what was essentially a decision tree too inflexible and incomprehensible, how they would experience an ML model! Imagine being the person who has to explain how such a new model works to a classroom of lawyers! Strong men have wept for less.
Anyway so the Dutch judicial system is now back to a system of spelled out sentencing guidelines in textual form, systematised by criminal law area, like there is one for cybercrimes, etc. With some tables that probably used to be encoded in a DBase file (that's how old the former software based tool was). But it seems that the fact that this information is now consumed from their original texts, gives the impression that there is more room for 'customization' - although there isn't actually that much more room, and in another 15 years we'll go full circle I suspect and get someone to code up the guidelines again...
So these texts are still just algorithms but spelled out; and the former method was to have software to help with 'interpreting' those textual algorithms; yet nobody called them out for the things modern ML-based systems are called out. But still I think this is a very clear example of how the word 'algorithm' has become overloaded in its meaning - not just become broader, but has actually shifted in meaning depending on the context in which it's used.
It's easy to see the problems with this approach and I don't have to rehash them here; and much of the interesting (academic) work here is in rebalancing and fixing the biases; but that is research, not something that is ready for actual use in democratic societies (not that that has stopped some places from using them anyway...). I'm not convinced it's even possible and any such system will invariably be very opaque. Imagine, if judges and prosecutors already find what was essentially a decision tree too inflexible and incomprehensible, how they would experience an ML model! Imagine being the person who has to explain how such a new model works to a classroom of lawyers! Strong men have wept for less.
Anyway so the Dutch judicial system is now back to a system of spelled out sentencing guidelines in textual form, systematised by criminal law area, like there is one for cybercrimes, etc. With some tables that probably used to be encoded in a DBase file (that's how old the former software based tool was). But it seems that the fact that this information is now consumed from their original texts, gives the impression that there is more room for 'customization' - although there isn't actually that much more room, and in another 15 years we'll go full circle I suspect and get someone to code up the guidelines again...
So these texts are still just algorithms but spelled out; and the former method was to have software to help with 'interpreting' those textual algorithms; yet nobody called them out for the things modern ML-based systems are called out. But still I think this is a very clear example of how the word 'algorithm' has become overloaded in its meaning - not just become broader, but has actually shifted in meaning depending on the context in which it's used.
Modern media use of the word "algorithm" is almost exclusively about AI type of stuff. Recommenders, likelihood detectors, image recognizers etc. That's very much not a series of steps: it's primarily a function of the data that it had been fed. You can't summarize a training body of 50000 cat pictures as "a series of steps". And the steps are worthless without the training data.
No but the algorithms used by AI to do recommendation etc. are generally referred to as algorithms, hence https://www.geeksforgeeks.org/top-10-algorithms-every-machin...
Here's a white paper on 'Public AI Registers' by the CEO of the company (Saidot) that runs the service for both Amsterdam and Helsinki (co-authored with representatives of both cities):
https://uploads-ssl.webflow.com/5c8abedb10ed656ecfb65fd9/5f6...
https://uploads-ssl.webflow.com/5c8abedb10ed656ecfb65fd9/5f6...
Really interesting and useful way of displaying information in a very non government-y way
I tend to believe this is a valid approach for ML application in public institutions. Being ML a black box or a potential source of discrimination, setting up systems to ensure credibility within the public should be addressed early on in the design process (perhaps while designing the model itself).
I wouldn't be surprised to see a future where all ML applications affecting citizens and the general public interest (excluding defense, military, etc.) will be obliged to be open-source and subject to public scrutiny.
I wouldn't be surprised to see a future where all ML applications affecting citizens and the general public interest (excluding defense, military, etc.) will be obliged to be open-source and subject to public scrutiny.
> I wouldn't be surprised to see a future where all ML applications affecting citizens and the general public interest (excluding defense, military, etc.) will be obliged to be open-source and subject to public scrutiny.
I would be positively surprised.
I would be positively surprised.
Sorry if it didn't come across in the post, it'd be no doubt a fair step towards more ethically sound governments.
In Australia, any Federal Government algorithm can be FOI’d by a member of the public.
It’s one of the reasons the Aus Federal Government statisticians tend away from black box AI and towards more traditional methods.
> [...] Government statisticians tend away from black box AI and towards more traditional methods.
I was indeed wondering how constraints on model interpretation forces governments to stay away from NN and more complex approaches.
I have worked for a fraud-prevention company for a few years, and the financial nature of the decisions involved led to reward model interpretation (sometime at accuracy's expense) in order not to create friction with clients. Definitely not the best not being able to explain why your model performs poorly or what actions are needed to solve a problem.
I was indeed wondering how constraints on model interpretation forces governments to stay away from NN and more complex approaches.
I have worked for a fraud-prevention company for a few years, and the financial nature of the decisions involved led to reward model interpretation (sometime at accuracy's expense) in order not to create friction with clients. Definitely not the best not being able to explain why your model performs poorly or what actions are needed to solve a problem.
In theory for NL one could argue this is already the case. Government internals are public (WOB / transparent government). When I was a civil servant we once shared data and code with a citizen, although more regularly shared are internal documents. Our government has to make informed and documented decisions (AWB - general public law) and no decision that has legal ramifications can be made without manual intervention (an example: even automatical traffic fines have a name / number of an attested civil servant that finalized the fine in the paper trail). So all ingredients are there. Suppose they use external models or tools, how could they ever substantiate their decision in court without giving quite a lot transparency to the court?
Now in practice .... Our IRS has had a discriminatory way of working in place where no-one involved had access to their own files (GDPR?!) and even judges were reluctant to order transparency in the following proceedings giving the IRS the benefit of the doubt. Ultimately this became a politican scandal and will take years to resolve. And in the paragraph above I would have argued that the basics considering any case should be easily available in as much as 8 weeks for every citizen...
Now in practice .... Our IRS has had a discriminatory way of working in place where no-one involved had access to their own files (GDPR?!) and even judges were reluctant to order transparency in the following proceedings giving the IRS the benefit of the doubt. Ultimately this became a politican scandal and will take years to resolve. And in the paragraph above I would have argued that the basics considering any case should be easily available in as much as 8 weeks for every citizen...
> no decision that has legal ramifications can be made without manual intervention
I think there's no other way around this: ultimately, everything that has legal implication will be potentially subject to dispute. We're not yet in the dystopian environment where disputes can be handled by a model :)
I think there's no other way around this: ultimately, everything that has legal implication will be potentially subject to dispute. We're not yet in the dystopian environment where disputes can be handled by a model :)
Regarding the automatic parking enforcement they claim, on the topic of non-discrimination:
> The service works the same way for all license plates regardless of the car model, age, or the owner’s profile.
This whole thing seems to be designed to give the appearance of government transparency without any of the transparency.
I bet if they released some "raw" (but anonymized) data from their collection it would be easy to find evidence of discrimination.
E.g. some of the richest parts of the city have some of the smallest parking spots. Surely cars bunched up against each other lowers the accuracy rate. Same with certain car models with recessed areas holding the number plate, which again will have socio-economic correlations. I don't see how you could plausibly design a system like this without any discrimination built into it.
But instead they just say there's no discrimination, without providing any of the source code, models or other data for the system. If they actually found that it was discriminatory does anyone think Amsterdam's first task would be to change its website to say "we're discriminating!", or would they just quietly try to fix it?
That's the issue with these sorts of systems. At least if one poor neighborhood is full of parking attendants with none in the adjacent rich neighborhood people will notice sooner than later. You have almost none of that basic sorts of transparency with automated scanning.
> The service works the same way for all license plates regardless of the car model, age, or the owner’s profile.
This whole thing seems to be designed to give the appearance of government transparency without any of the transparency.
I bet if they released some "raw" (but anonymized) data from their collection it would be easy to find evidence of discrimination.
E.g. some of the richest parts of the city have some of the smallest parking spots. Surely cars bunched up against each other lowers the accuracy rate. Same with certain car models with recessed areas holding the number plate, which again will have socio-economic correlations. I don't see how you could plausibly design a system like this without any discrimination built into it.
But instead they just say there's no discrimination, without providing any of the source code, models or other data for the system. If they actually found that it was discriminatory does anyone think Amsterdam's first task would be to change its website to say "we're discriminating!", or would they just quietly try to fix it?
That's the issue with these sorts of systems. At least if one poor neighborhood is full of parking attendants with none in the adjacent rich neighborhood people will notice sooner than later. You have almost none of that basic sorts of transparency with automated scanning.
If you look at the other projects, there's also a subsection on Non-discrimination.
This specific example of the license plate registration doesn't have anything to do with discrimination because it simply checks the output of an OCR (Optical Character Recognition) algorithm with a database which consists of the location and time a certain license plate has paid for.
The parking attendants just drive their vehicles and the OCR does its work. The only reason I see for the city of Amsterdam to include this subsection into the article is to tell the citizens that the part of license registration doesn't come into play until after the algorithm (or in case of a failure a human check) has decided that the license plate wasn't in the database on that point in time.
The parking attendants just drive their vehicles and the OCR does its work. The only reason I see for the city of Amsterdam to include this subsection into the article is to tell the citizens that the part of license registration doesn't come into play until after the algorithm (or in case of a failure a human check) has decided that the license plate wasn't in the database on that point in time.
I see your point but as OP mentioned:
1) Where is the data that shows each street is equally/proportionally scanned by parking attendant vehicles? There is potential discriminatory policy there.
2) Where is the data that shows that scan-success rate (i.e., percentage of detected cars, for which a plate number could be made out) is distributed normally, rather than e.g. influenced by parking infrastructure which is correlated with socioeconomic factors? Again, potential discriminatory practice.
Other possibilities exist, too.
These may sound far fetched, and you may be right to say that, but the data would show it. It'd be nice to see the data. It cannot be said a priori that 'algorithm just checks the image, so there's no discrimination possible', if there's no word about ensuring that regardless of various factors (e.g. socioeconomic, racial etc), your non-compliance with parking rules has an equal chance of your plate being scanned and fed into the algorithm.
It's the same issue why racial profiling is controversial. You can say 'this computer/algorithm is perfect at assessing non-compliance with a rule or regulation by individual X regardless of e.g. ethnicity', and you could still have discrimination because ethnic group A is fed into this system by way of racial profiling 10x more than ethnic group B. By the same token, we'd want to see where/how-often the scanning vehicles operate, and how their scans may have lower or higher success rates in various situations.
In short, the OCR might be alright, but the entire process/model/architecture (from scanning to issuing a fine, in which OCR is just one of the middle-steps) may not be. That entire model including scanning (photographing) by a computer is described on the page, so saying its non-discriminatory implies they've assessed more than just the OCR step. And if that is the case, they cannot say it's a priori non-discriminatory quite yet without providing more info/data.
1) Where is the data that shows each street is equally/proportionally scanned by parking attendant vehicles? There is potential discriminatory policy there.
2) Where is the data that shows that scan-success rate (i.e., percentage of detected cars, for which a plate number could be made out) is distributed normally, rather than e.g. influenced by parking infrastructure which is correlated with socioeconomic factors? Again, potential discriminatory practice.
Other possibilities exist, too.
These may sound far fetched, and you may be right to say that, but the data would show it. It'd be nice to see the data. It cannot be said a priori that 'algorithm just checks the image, so there's no discrimination possible', if there's no word about ensuring that regardless of various factors (e.g. socioeconomic, racial etc), your non-compliance with parking rules has an equal chance of your plate being scanned and fed into the algorithm.
It's the same issue why racial profiling is controversial. You can say 'this computer/algorithm is perfect at assessing non-compliance with a rule or regulation by individual X regardless of e.g. ethnicity', and you could still have discrimination because ethnic group A is fed into this system by way of racial profiling 10x more than ethnic group B. By the same token, we'd want to see where/how-often the scanning vehicles operate, and how their scans may have lower or higher success rates in various situations.
In short, the OCR might be alright, but the entire process/model/architecture (from scanning to issuing a fine, in which OCR is just one of the middle-steps) may not be. That entire model including scanning (photographing) by a computer is described on the page, so saying its non-discriminatory implies they've assessed more than just the OCR step. And if that is the case, they cannot say it's a priori non-discriminatory quite yet without providing more info/data.
I think we can be pretty certain that the city of Amsterdam will do everything it can to discover and issue fines for more illegally parked cars. They're heavily incentivized to do so given that it's a not insignificant source of revenue.
Is there discrimination? Probably. But over time as the system gets better and more fines get issued less illegal parkers will get away with it. That's the goal right, to fine everyone illegally parking. It's not as if right now you can park a car illegally in Amsterdam and expect to get away with it. That's never been the case regardless of where you parked. As we get closer to catching everyone discrimination becomes irrelevant, because as enforcement approaches 100% it becomes less and less selective.
I live in Rotterdam and I don't own a car. So my priority is to see less space given to cars and for people who park cars pay more for parking. I don't actually care about discrimination because the general trend is for parking to get more expensive, and for urban areas to become more pedestrian friendly. I want more parking enforcement, fewer cars, and more revenue from car owners.
My friends who own cars are constantly complaining about parking fees and I have little to no sympathy for them. For many people in The Netherlands cars are a luxury. Most people don't need them, and if they claim to need them they usually actually don't. This is especially true in Amsterdam.
All of this leads me to have very little sympathy for any discrimination argument.
Is there discrimination? Probably. But over time as the system gets better and more fines get issued less illegal parkers will get away with it. That's the goal right, to fine everyone illegally parking. It's not as if right now you can park a car illegally in Amsterdam and expect to get away with it. That's never been the case regardless of where you parked. As we get closer to catching everyone discrimination becomes irrelevant, because as enforcement approaches 100% it becomes less and less selective.
I live in Rotterdam and I don't own a car. So my priority is to see less space given to cars and for people who park cars pay more for parking. I don't actually care about discrimination because the general trend is for parking to get more expensive, and for urban areas to become more pedestrian friendly. I want more parking enforcement, fewer cars, and more revenue from car owners.
My friends who own cars are constantly complaining about parking fees and I have little to no sympathy for them. For many people in The Netherlands cars are a luxury. Most people don't need them, and if they claim to need them they usually actually don't. This is especially true in Amsterdam.
All of this leads me to have very little sympathy for any discrimination argument.
It's about government transparency and the precedent that's being set. You may not care about parking enforcement (neither do I, really), but in 10 years time this won't be the last thing ML is being used for.
We're replacing systems that are readily obvious to anyone (uniformed human parking attendants) with algorithmic enforcement.
The government has decided that we don't need to know how it works (source code and models) or what the flaws and externalities are (anonymized data on how it's used). Thus something we're paying for whose function was pretty obvious to anyone becomes a black box.
This lack of transparency will become the norm unless it's opposed, and eventually you or your group will become the disproportionate target of some overzealous algorithm.
We're replacing systems that are readily obvious to anyone (uniformed human parking attendants) with algorithmic enforcement.
The government has decided that we don't need to know how it works (source code and models) or what the flaws and externalities are (anonymized data on how it's used). Thus something we're paying for whose function was pretty obvious to anyone becomes a black box.
This lack of transparency will become the norm unless it's opposed, and eventually you or your group will become the disproportionate target of some overzealous algorithm.
[deleted]
I don’t see if one neighborhood is full of parking attendants and rich one is less proving discrimination.
This is an optimization problem, you have limited capability of enforcement, and you need to use it with most efficient way.
This is an optimization problem, you have limited capability of enforcement, and you need to use it with most efficient way.
> I don’t see if one neighborhood is full of parking attendants and rich one is less proving discrimination.
Suppose only a poor neighbourhood has laws enforced, while in another there is no enforcement. Would that be discriminatory? I think in effect, absolutely. In terms of intentions, it remains to be seen what the reasons were, it could've been a mistake or a matter of optimising resources. But there are clearly limits to this argument, and it is also clearly discriminatory in practice.
For example, racial profiling is very much a form of discrimination. Suppose two ethnicities live in a city, one (A) has a 0.5% serious crime rate, the other (B) has a 2% serious crime rate, on average, and say equal chance of minor offences. Profiling citizens of group B more often, e.g. stop & frisk, would be a more optimal use of police resources than random selection, to stop serious crimes.
Yet, 98% of group B is innocent of serious crimes and is being subjected to more enforcement, which lead to a higher than proportionate conviction rate of minor offences, simply for belonging to a different ethnic group. It also leads to more run-ins with the police, which disturbs your day, risks escalation, makes you feel like a second-class citizen etc.
That's discrimination, because the 98% innocent individuals from group B are being treated more harshly without cause, purely on the basis of their belonging to a group. Discrimination is when an individual is maltreated on the basis of belonging to a group, rather than as an individual.
Yet also 'more optimal/efficient use of resources'. They're not mutually exclusive.
And that's true not just within policing, by the way. If you know that 80% of ethnic group A have empirically been good hires, and 70% of ethnic group B, in the past, and suppose you had many people to choose from to invite for an interview, and then interview. Suppose you have limited resources and selection & interviewing is time-consuming. Does that allow you to 'optimize use of scarce resources and efficiently reject all applicants from group B'? No. It'd be perhaps efficient, but also very much discriminatory.
Of course if you have two neighbourhoods and one has rampant parking abuse, the other not, then indeed it does make sense to allocate enforcement capacity accordingly. But only to a point. And that point must be evaluated. And whether you've reached that point, will be indicated by the data. That data isn't available, yet it's a priori stated that the parking enforcement cannot be discriminatory. OP noted a few possibilities that could indicate it is discriminatory, but we'd need to see the data to draw any conclusions. I think he's right in saying that.
Suppose only a poor neighbourhood has laws enforced, while in another there is no enforcement. Would that be discriminatory? I think in effect, absolutely. In terms of intentions, it remains to be seen what the reasons were, it could've been a mistake or a matter of optimising resources. But there are clearly limits to this argument, and it is also clearly discriminatory in practice.
For example, racial profiling is very much a form of discrimination. Suppose two ethnicities live in a city, one (A) has a 0.5% serious crime rate, the other (B) has a 2% serious crime rate, on average, and say equal chance of minor offences. Profiling citizens of group B more often, e.g. stop & frisk, would be a more optimal use of police resources than random selection, to stop serious crimes.
Yet, 98% of group B is innocent of serious crimes and is being subjected to more enforcement, which lead to a higher than proportionate conviction rate of minor offences, simply for belonging to a different ethnic group. It also leads to more run-ins with the police, which disturbs your day, risks escalation, makes you feel like a second-class citizen etc.
That's discrimination, because the 98% innocent individuals from group B are being treated more harshly without cause, purely on the basis of their belonging to a group. Discrimination is when an individual is maltreated on the basis of belonging to a group, rather than as an individual.
Yet also 'more optimal/efficient use of resources'. They're not mutually exclusive.
And that's true not just within policing, by the way. If you know that 80% of ethnic group A have empirically been good hires, and 70% of ethnic group B, in the past, and suppose you had many people to choose from to invite for an interview, and then interview. Suppose you have limited resources and selection & interviewing is time-consuming. Does that allow you to 'optimize use of scarce resources and efficiently reject all applicants from group B'? No. It'd be perhaps efficient, but also very much discriminatory.
Of course if you have two neighbourhoods and one has rampant parking abuse, the other not, then indeed it does make sense to allocate enforcement capacity accordingly. But only to a point. And that point must be evaluated. And whether you've reached that point, will be indicated by the data. That data isn't available, yet it's a priori stated that the parking enforcement cannot be discriminatory. OP noted a few possibilities that could indicate it is discriminatory, but we'd need to see the data to draw any conclusions. I think he's right in saying that.
Yeah but you are mixing people in equation with your examples with ethnicity.
Imagine two same ethnicity neighborhoods, with same economic wealth, one has crime rate 2% other 0.5%, would you think it would be discrimination to allocate more police force to 2% one?
For the interview part, imagine I am saying I am only hiring CS graduates, and A ethnicity has 10% CS graduates, B has 1%, am I discriminating?
Imagine two same ethnicity neighborhoods, with same economic wealth, one has crime rate 2% other 0.5%, would you think it would be discrimination to allocate more police force to 2% one?
For the interview part, imagine I am saying I am only hiring CS graduates, and A ethnicity has 10% CS graduates, B has 1%, am I discriminating?
Nice try, I like the gesture.
Count me extremely skeptical though. You can get decent to really good tabular learning right off the shelf and it‘s so good and ubiquitous by now that interestingly enough, a few things start to happen that imho might make this effort close to impossible:
- The differences between different algorithms get ever smaller, leaving more and more weight of any machine driven decision to the actual input data itself, including proper labeling, preparation, feature engineering, QA and debiasing. You can‘t properly separate the data from the algorithm itself anymore these days.
- Even IF you added all this data together with the algorithm in a register (which brings its own privacy and practicality concerns), with the massive breadth and width of modern data sources, most data scientists don‘t even understand model decisions by themselves anymore. You‘ll need to dig deep into the explainability toolbox to be able to at least partially explain them.
- There are dozens of documented cases where algorithms just mirror and regurgitate human biases through data. Algorithms by themselves are (mostly) innocuous, but can get to make racist, sexist and populist decisions anyway, purely based on what they have trained on.
So if you can‘t even trust the people who build the algorithms to understand their reasoning, what benefit could a publication possibly bring to any outsider?
Count me extremely skeptical though. You can get decent to really good tabular learning right off the shelf and it‘s so good and ubiquitous by now that interestingly enough, a few things start to happen that imho might make this effort close to impossible:
- The differences between different algorithms get ever smaller, leaving more and more weight of any machine driven decision to the actual input data itself, including proper labeling, preparation, feature engineering, QA and debiasing. You can‘t properly separate the data from the algorithm itself anymore these days.
- Even IF you added all this data together with the algorithm in a register (which brings its own privacy and practicality concerns), with the massive breadth and width of modern data sources, most data scientists don‘t even understand model decisions by themselves anymore. You‘ll need to dig deep into the explainability toolbox to be able to at least partially explain them.
- There are dozens of documented cases where algorithms just mirror and regurgitate human biases through data. Algorithms by themselves are (mostly) innocuous, but can get to make racist, sexist and populist decisions anyway, purely based on what they have trained on.
So if you can‘t even trust the people who build the algorithms to understand their reasoning, what benefit could a publication possibly bring to any outsider?
An outsider can throw test data points at the model and see how it behaves. For example, whether changing name, race, sex, or variables correlating with such critical variables, alter a decision. Pointing out these flaws could then lead to better systems, or not using machine learning on some case where input data is unsuitable.
Nice to see my employer getting posted for this kind of stuff. In case you are interested we try to open-source as much as possible so check out our Github: https://github.com/amsterdam
Nice to see this picked up! We're hiring technical people at the municipality, reach out to me via my profile if you're interested.
So great to see this discussion. When developing these registers, the target has been to enable citizen participation and public oversight of city algorithms. And to help cities and other government organizations to systematically govern their algorithmic systems. With that target in mind, conversations like this, anchored on proper data on the systems themselves, really count. Thank you! I'd love to invite everyone to give feedback also via our surveys at both sites.
Meeri / Saidot
Meeri / Saidot