Many companies have been doing this for years. Merely flagging my data for hiding and eventual deletion instead of deleting it, when I wanted it deleted as per GDPR :)
I have once seen an API that gave me a token, and that token needed to be fed back in a new API call as an "yes, I am sure"-type guardrail. However, since it's an API, and the "P" stands for PROGRAMMING, that is just adding network overhead.
Guy gives non-deterministic software root access, desaster happens. Movie at eleven.
Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to know: total and absolute failure of guard rails, but these guard rails can not be expected to be in an LLM.
My amazing experience with DICOM, an open standard with viewer software on SourceForge, is that I once took pictures in the format to a locally renowned neurologist.
She plugged my USB stick into her Windows XP workstation, ca. 2019, without the slightest hesitation, then proceeded to claim not to know what DICOM is, and to ask me why I didn't bring her either the proprietary viewer program, or JPEGs.
Had I brought the viewer, she would have probably launched the .exe with equally no hesitation.
At least in these parts of the woods, this once again confirmed to me that medicine and IT usually exist at opposite ends of a spectrum.
Many of us have, and painfully if I might add, finally convinced people to abandon insecure messengers and move to something like Signal. Now the solution is to tell them to abandon Signal in favor of Matrix? I'll pass.
p.s. I am color vision handicapped or whatever that is called.