Ask HN: How do you handle audit logs in your systems?
8 comments
Relatable recent thread https://news.ycombinator.com/item?id=44602532 "When root meets immutable: OpenBSD chflags vs. log tampering" (153 points, 45 comments).
Not to necessarily focus on the operating system used, but think of the attacker model and risk appetite of the organisation. What are the required integrity goals? What retention do you (legally) require? Who should be able to access those logs; on their own, or n-eye principle? Do such accesses need to be logged as well? What are the requirements from the users of the audit log?
The things you'll need to log will become clear after answering such questions. How you structure them depends on the required access patterns. Tamper evidence can be achieved in many ways, but that depends on the integrity requirements; is an empty log line where one should have been enough "tamper evident"? Do you need some sort of verification mechanism that confirms the log lines you see were stored in that order?
If your log only persists on the machine where it originated, does that satisfy integrity requirements?
Not to necessarily focus on the operating system used, but think of the attacker model and risk appetite of the organisation. What are the required integrity goals? What retention do you (legally) require? Who should be able to access those logs; on their own, or n-eye principle? Do such accesses need to be logged as well? What are the requirements from the users of the audit log?
The things you'll need to log will become clear after answering such questions. How you structure them depends on the required access patterns. Tamper evidence can be achieved in many ways, but that depends on the integrity requirements; is an empty log line where one should have been enough "tamper evident"? Do you need some sort of verification mechanism that confirms the log lines you see were stored in that order?
If your log only persists on the machine where it originated, does that satisfy integrity requirements?
I can’t help with all those queries, but a nice pattern I was introduced to a while back for integrity checks is to have the logging service embed a hash of message N in message N+1. This creates a simple Merkle tree that you can use to detect any omissions or manipulation of and flag. It’s minimal overhead and trivial to implement in any logging backend.
Here's a redacted example interface in TypeScript for something important. It's added to a PostgreSQL database. The "extra" field is a common pattern I use, with JSON, for fields that are frequently added later that don't require indexes or as much data integrity as the other fields, so a migration isn't necessary:
https://gist.github.com/BrunoBernardino/df806e3e902017308abf...
Hope it helps!
https://gist.github.com/BrunoBernardino/df806e3e902017308abf...
Hope it helps!
We store in mongodb, we just store api calls as:
RequestId, Date, UserId, UserIP, UserRole, HandlerMachineIP, TheRequestBody, TheResponseBody/Error(if anything happened)
First hypothetical: If your "audit" logging suggests a different reality than your database, which one should you trust?
I personally would not assume that there is a question of ‘which one to trust’. In this case I would know that something has gone wrong and use both of these two as clues only to find the root cause which made them different.
log minimum most important first. timestamp, path, user, and exception messages. don't implement update or delete. prefer storing logs outside the system. are you logging for debugging or audit purposes? can you do both in one place? and log the response code.
I use event sourcing so that itself serves as audit log.
I'm curious how others handle this in production:
- What data do you log?
- How do you structure audit logs (JSON, text, DB records)?
- How do you ensure logs are immutable/tamper-evident?
- Do you store them separately from application logs?
- What tooling or patterns have worked well (or poorly) for you?
Any war stories, best practices, or pitfalls to avoid would be really appreciated.