Presidio: Customizable data protection and PII data anonymization service(github.com)
github.com
Presidio: Customizable data protection and PII data anonymization service
https://github.com/microsoft/presidio
21 comments
I can see this being used to detect accidentally leaked sensitive content (for example scanning repos, outgoing emails, shared folders).
However using this to redact material sounds a bit risky. Are there some use case where you could accept the potential mistakes (missing something that should have been redacted)?
However using this to redact material sounds a bit risky. Are there some use case where you could accept the potential mistakes (missing something that should have been redacted)?
License: MIT
Demo: https://presidio-demo.westeurope.cloudapp.azure.com/
Documentation: https://github.com/microsoft/presidio/blob/master/docs/index...
Demo: https://presidio-demo.westeurope.cloudapp.azure.com/
Documentation: https://github.com/microsoft/presidio/blob/master/docs/index...
This is good.. just need a hashbytes formula for Excel to make anonymization accessible to the majority of MS customers who are fumbling around pii haphazardly.
My first thought was it was related to the cloud security company Presidio not MS.
Although Presidio would like you to think they are cloud security (higher margins) they are really just a VAR/reseller. Weak services but they will sell you stuff. So will I!
More importantly, this is proximate enough that they may sue for the name, OP.
More importantly, this is proximate enough that they may sue for the name, OP.
Still some work to do ; on their demo page [1]:
"Simply follow the instructions" -> "Simply follow the <US_DRIVER_LICENSE>" ; same for "contribution". So I'm guessing some overly eager regex is to blame, which doesn't make you super confident about using this for something sensitive.
[1] https://presidio-demo.westeurope.cloudapp.azure.com
"Simply follow the instructions" -> "Simply follow the <US_DRIVER_LICENSE>" ; same for "contribution". So I'm guessing some overly eager regex is to blame, which doesn't make you super confident about using this for something sensitive.
[1] https://presidio-demo.westeurope.cloudapp.azure.com
To be fair, there are different levels of detection [1], the demo is probably using the weakest one.
[1] https://github.com/microsoft/presidio/blob/74ea983cc50ff76d7...
[1] https://github.com/microsoft/presidio/blob/74ea983cc50ff76d7...
Thanks, this is indeed the case. the US_DRIVER_LICENSE confidence is 0.01 and the demo doesn't put any threshold on the response. We're working on fixing the demo.
The core of the product is a bunch of regex or something smarter?
Odd that a product that want you to base your legal safety is not more transparent about the actual implementation on the initial page, instead it reads like an advertisement for closed source SaaS.
Odd that a product that want you to base your legal safety is not more transparent about the actual implementation on the initial page, instead it reads like an advertisement for closed source SaaS.
answering myself, yes. It is a pile of regex
https://github.com/microsoft/presidio/blob/master/presidio-a...
and i can't find the body of test data easily... that's not good.
https://github.com/microsoft/presidio/blob/master/presidio-a...
and i can't find the body of test data easily... that's not good.
They also have a NLP engine, the regexes are just predefined recognizers for known formats.
https://github.com/microsoft/presidio/tree/master/presidio-a...
https://github.com/microsoft/presidio/tree/master/presidio-a...
Serious question - is there anything fundamentally wrong with using regex here?
For the specific example cited (credit card numbers) the Luhn check is very standard. So, no there isn't :)
So you might have more fun with this email one https://github.com/microsoft/presidio/blob/master/presidio-a...
:)
... seriously, they should have broken that down, or used the multi-line with comments pattern, if they really want people to contribute to this.
:)
... seriously, they should have broken that down, or used the multi-line with comments pattern, if they really want people to contribute to this.
Yeah, seriously. Are email addresses even regular to the extent that a regex would work 100% of the time?
[deleted]
Please everyone:
Before you brand your software, google the name! It's not difficult!
https://www.presidio.com/
Before you brand your software, google the name! It's not difficult!
https://www.presidio.com/
Is anyone using presidio in production?
The engine has two main advantages: It's easily expandable and customizable, and it works well at scale. We do know of organizations who are very close to production with it.
Every organization has their own requirements for PII entities, many of them specific to the org itself, so the engine allows a developer to easily add support for new PII entities using code, regex or black-lists.
See hree: https://github.com/microsoft/presidio/blob/master/docs/custo...
As for the productization aspect, we did some performance tests and are confident with taking it to production. Single instance cluster with a medium size machine has a ~65ms response time for a 100 word sentence. Using better machines lowers the response time to ~24ms for 100 words and 150ms for a 1,000 words input.
The current service uses regex for known patterns and Spacy for named entity recognition (person names, places etc.). Users often built custom ML models to detect new types of entities.
Presidio is free, completely transparent, and fully customizable. Feel free to use it and let us know what you think.