jostkolega

5 tháng trướcdiscuss

Nice! Working in a similar space - will be publishing here also soon. How are you handling finding issues beyond pattern matching where deep code understanding is required?

jostkolega

5 tháng trướcdiscuss

Running security audits on open source repos with a tool we've built and reporting what I find to maintainers. Mostly infrastructure stuff — vector databases, LLM tooling, secrets managers. Been doing responsible disclosure and submitting fixes which are all autogenerated. Surprisingly high acceptance rate so far, which is encouraging. Working on automating more of the test process...

jostkolega

5 tháng trướcdiscuss

Semgrep's a solid choice for this. The stuff you're catching - secrets, SQLi, weak hashing- is where pattern matching shines.

The tricky part with LLM-generated code is when it's syntactically fine but semantically broken. Stuff like:

- auth logic that checks user.role but also has a `req.query.admin === 'true'` fallback because the model left debug code in

- JWT validation that calls verify() but passes `algorithms: ['none']` or disables signature checking — the code looks right

- async TOCTOU bugs where permission check and action aren't atomic because the LLM wrote it like sync code

None of these match simple patterns. You need dataflow analysis to catch them reliably.

Curious if you're thinking about that for the paid tier, or keeping it pattern-based to stay fast?

jostkolega

5 tháng trướcdiscuss

+1 on triage being the real problem. Question, when Semgrep surfaces something ambiguous, lets say a SQL query that looks parameterized but the ORDER BY is built elsewhere, what does reviewing that actually look like? I'm wondering how much context you get before needing to jump out to the codebase.

comments