I'm sympathetic to the questions being asked, but I think the author is too quick to assume the worst here.
The key assumption is shown in the diagram that "shows a hypothetical pneumonia detection model that, during optimization, has learned to recognize racial identity." -- the author asserts that this is obviously incorrect and dangerous.
Look at the HPF 50 results, and the fact that the location being imaged is nearly irrelevant -- it's clear that image shape features are irrelevant to the results. It seems like the signal itself is picking up (is modified by) racial indicators. If that's true, it only makes sense that being good at detecting pneumonia is going to require correcting for that signal. That makes (this use of) AI anti-racist in a very important way.
The key assumption is shown in the diagram that "shows a hypothetical pneumonia detection model that, during optimization, has learned to recognize racial identity." -- the author asserts that this is obviously incorrect and dangerous.
Look at the HPF 50 results, and the fact that the location being imaged is nearly irrelevant -- it's clear that image shape features are irrelevant to the results. It seems like the signal itself is picking up (is modified by) racial indicators. If that's true, it only makes sense that being good at detecting pneumonia is going to require correcting for that signal. That makes (this use of) AI anti-racist in a very important way.
This is nonetheless a fascinating result!