Track Errors First(bugsink.com)
bugsink.com
Track Errors First
https://www.bugsink.com/blog/track-errors-first/
52 comments
Semi-relatedly, this is why I'm quite liberal about throwing errors while developing programs. During development it makes sure I'm reminded of my slipups when I inevitably forget some invariant that needed to be upheld, and then towards the end of the process any throws remaining are easily found and vetted. And in production, the closer to the source the error occurs the easier it is to fix; if you're especially unlucky, the error doesn't occur until after state has been persisted and read back, in which case you not only need to fix the bug but also have to figure out how to handle and/or remediate the bogus state written to your database.
I used to code this way, then I started to write tests, as I was writing my code. I could ensure the behavior was expected with the test. If the code wasn't easily testable, I would refactor it right way and that would generally leave me with easier to read and follow code as well. The end benefit being that if I wanted to change the behavior later, now I knew if I was breaking anything. With everything tested, very little made it to production that wasn't working.
It's not about being easily testable, it's about being in an invalid state for an unexpected reason, you can't test that trivially.
You can try to avoid it by architecture or abstractions, but it only works to some extent.
You can try to avoid it by architecture or abstractions, but it only works to some extent.
Testing isn't trivial, it can be downright difficult. But if you test for those invalid states, the reasons are expected.
No, this is a pipe dream. The only valid state for a sorted list is sorted. You can not possibly "test for invalid state" here because there's no way you can handle a sorted list that is not sorted in any meaningful way.
Mind you this example is the one of the most trivial data structures in existence. Real software has insanely more complicated valid/invalid state, the size of the state space growing exponentially with the number of components.
Mind you this example is the one of the most trivial data structures in existence. Real software has insanely more complicated valid/invalid state, the size of the state space growing exponentially with the number of components.
If I have a method wrapped around adding items to a list, then I'd call that method with random data. Then, I'd call the method that gets the items from the list and I'd check that the order is consistent. Now, you're testing that the underlying implementation is a sorted list.
I'm sure you can object in many specific cases, but the point is there's no general way to do that. Even your description could be unable to find the famous Java bug with integer overflow in binary search had it only happen on extreme cases. I'm not even talking about race conditions.
And these are still single components, so the exponential growth of state combinations point is left unaddressed. In fact, it directly supersedes your point in this comment, as the exponentiality eats your "random data" idea before you even wrote it.
And these are still single components, so the exponential growth of state combinations point is left unaddressed. In fact, it directly supersedes your point in this comment, as the exponentiality eats your "random data" idea before you even wrote it.
We're not testing Java, we're testing your code which is a method to add items to a list. If it passes the tests, great. If there is a bug in the JVM, maybe it picks it up or doesn't. That doesn't negate the need for the test.
The way I look at testing is less about the current point in time. It is about changes over time. If someone else comes along and changes the underlying implementation to a list where sort order is not guaranteed (maybe thinking their version is faster or something), then the tests will fail and that's exactly what you want. You're testing the expected behavior over time and right now, the expected behavior is a sorted list.
Regardless, I don't understand what you're trying to argue or prove here. Yes, testing is hard and not easily won, I already said that.
The way I look at testing is less about the current point in time. It is about changes over time. If someone else comes along and changes the underlying implementation to a list where sort order is not guaranteed (maybe thinking their version is faster or something), then the tests will fail and that's exactly what you want. You're testing the expected behavior over time and right now, the expected behavior is a sorted list.
Regardless, I don't understand what you're trying to argue or prove here. Yes, testing is hard and not easily won, I already said that.
> But if you test for those invalid states, the reasons are expected
I am arguing against your claim that this particular suggestion is feasible for anything but toy projects.
I am arguing against your claim that this particular suggestion is feasible for anything but toy projects.
Exactly what I'm talking about:
https://erikras.com/blog/final-form-to-typescript
"Of course, it helped that long ago I had put in all of the legwork to have near 100% test coverage, so I could be certain that the migration did not introduce any bugs."
https://erikras.com/blog/final-form-to-typescript
"Of course, it helped that long ago I had put in all of the legwork to have near 100% test coverage, so I could be certain that the migration did not introduce any bugs."
Suggesting that tests are only for toy projects is some weird kind of ego troll thing you've got going. Look inward.
> An exception isn’t just another event. It’s your code telling you it failed, in a specific place, with a specific context.
No! The other way around! All the other events should have context as well. I want to cry every time when I get a random warning-severity log that may or may not indicated something I need to look into.. without collecting the context which is collected on error-severity logging.
So the text was truncated? What? Where? Why? (don't truncate passwords before comparison, please!?) So the kernel logged some WARN dump? In response to what operation? Does only the 6.12 do that, or am I receiving this one from 6.14 machines as well? So the library says the "update()" method is deprecated? Which one of 10 update() methods in that library? Called from where?
No! The other way around! All the other events should have context as well. I want to cry every time when I get a random warning-severity log that may or may not indicated something I need to look into.. without collecting the context which is collected on error-severity logging.
So the text was truncated? What? Where? Why? (don't truncate passwords before comparison, please!?) So the kernel logged some WARN dump? In response to what operation? Does only the 6.12 do that, or am I receiving this one from 6.14 machines as well? So the library says the "update()" method is deprecated? Which one of 10 update() methods in that library? Called from where?
Maybe... I even wrote about how to do this in the case of Python and the Sentry SDK [0], so I don't fully disagree... but: in the general case this might get very expensive data-wise. So you have to cut off somewhere/start somewhere and/or budget how much info you send per type of log
[0] https://www.bugsink.com/blog/capture-stacktrace-no-exception...
[0] https://www.bugsink.com/blog/capture-stacktrace-no-exception...
I love Sentry but it has become a bit over-the-top complicated as of late (for my taste).
Glad to see one more on the market. I only knew about GlitchTip prior to today.
Glad to see one more on the market. I only knew about GlitchTip prior to today.
Agreed. We use both Sentry and Bugsnag (or whatever terrible name it has now) on different projects, and over time Sentry's interface has gotten considerably worse while Bugsnags got slightly better. I don't know if Sentry is aimed at significantly different users to us (an agency with many projects, mostly Django), but the sheer information overload when trying to understand an error is so bad that I ended up building a tool that receives Sentry webhooks and displays the information in a logical, sparse and easily digestible way.
Bugsink looks nice and straight forward, and if anything other than email - the absolute worst medium for notifications - lands in there I'll probably make the switch.
Bugsink looks nice and straight forward, and if anything other than email - the absolute worst medium for notifications - lands in there I'll probably make the switch.
> notifications
it's done: https://github.com/bugsink/bugsink/issues/3
it's done: https://github.com/bugsink/bugsink/issues/3
We use Datadog and Sentry because their mechanism of interaction and retention are different - if Datadog changed their process to be closer to the way Sentry aggregates errors over time I can see us never needing Sentry again.
Tried Sentry, then went back to BugSnag again. For me Sentry is just too much for a single developer.
For a single developer? What do you mean?
Not the parent but I imagine something like "the more software caters to big orgs, the worse fit it becomes for individuals"
Sorry, what I meant was the features on BugSnag’s free plan is enough for me. I feel like Sentry caters more on their business customer.
Don't get your hopes up.
Hey, that's me!
Dicussed on r/programming this morning[0] (on my side of the world) where someone probably picked it up
I'll be in the thread for a while I guess
[0] https://www.reddit.com/r/programming/comments/1l3sj8g/track_...
Dicussed on r/programming this morning[0] (on my side of the world) where someone probably picked it up
I'll be in the thread for a while I guess
[0] https://www.reddit.com/r/programming/comments/1l3sj8g/track_...
Is this available only for Python? or can I send stacktraces from JS/Rust/etc?
It's Sentry-SDK-compatible [0] which means that there's like a 100 integrations available. The amount of work one needs to do server-side to enrich events is language-dependent and driven by demand, which means that JS support is "pretty good" and for Rust you'll probably be looking at something that's not very "rich" (since symbol info will be missing[1]).
[0] https://www.bugsink.com/sentry-sdk-compatible/
[1] https://github.com/bugsink/bugsink/issues/20
[0] https://www.bugsink.com/sentry-sdk-compatible/
[1] https://github.com/bugsink/bugsink/issues/20
I’ve used Sentry to track bugs from client-side JavaScript, server-side Python, native iOS apps, and native Android apps.
You say you’re compatible with the Sentry SDKs, but it’s not at all clear to me which of these platforms will do useful things with Bugsink and which will not.
You say you’re compatible with the Sentry SDKs, but it’s not at all clear to me which of these platforms will do useful things with Bugsink and which will not.
Fair enough; I'll update the documentation to reflect the state of the art a bit more clearly. In the meantime, I'd say the best way is simply to try the 30 second-install and send over a test-event (a matter of switching out the DSN) to see how it shows up.
Thanks for creating this! I would love to contribute to this a little bit, maybe if I can get into python I will try and make a useful contribution, but for now I am using it to make all my go services alert me faster and more easily when shit hits the fan.
I have rarely found logs useful to debug or track down any production issue. Errors on the other hand are almost always useful. I never understood why some people focus so strongly on logs, which tend to always just get ignored, and errors never get noticed.
Maybe one reason is that in many organisations I've seen a lot of "noise" in the error logs, logging things as errors that really aren't errors, which hugely reduces the usefulness. It requires a bit of discipline to keep the error logs clean, but honestly, it's not that hard.
Maybe one reason is that in many organisations I've seen a lot of "noise" in the error logs, logging things as errors that really aren't errors, which hugely reduces the usefulness. It requires a bit of discipline to keep the error logs clean, but honestly, it's not that hard.
Looking at the article, I'm not convinced logs and errors are different enough to change the taxonomy. I agree that a stack trace linked to source code is a useful tool, but don't understand why e.g. a warning can't have a stack trace too, or why your logging system is the wrong place to put this tool.
I've found logs of successes, rather than errors, to be useful for understanding attacks or gnarly performance issues. Seeing that you have a lot of long-running requests for a particular user or tenancy when you're running out of database connections is an example that springs to mind.
I've found logs of successes, rather than errors, to be useful for understanding attacks or gnarly performance issues. Seeing that you have a lot of long-running requests for a particular user or tenancy when you're running out of database connections is an example that springs to mind.
I had a section "Why" in one of the drafts but scrapped it because I thought it was maybe a bit too harsh. But the gist of it is (IMHO):
* For vendors logging is _awesome_. There's basically no upper limit, you can log all kinds of things, and you can bill them by usage. And because of the scale it's hard to do locally, which is also good news for vendors.
* On the developer-side, there's always the "let's not think about it too much right now, let's just collect a bunch of info" argument for logging. Which is great if you don't want to think (more kindly: if you're facing deadlines)
I'm sure there's good cases for logging (I read logs occasionally too) but it wouldn't be my _first_ thing to heavily invest in (hence the article)
* For vendors logging is _awesome_. There's basically no upper limit, you can log all kinds of things, and you can bill them by usage. And because of the scale it's hard to do locally, which is also good news for vendors.
* On the developer-side, there's always the "let's not think about it too much right now, let's just collect a bunch of info" argument for logging. Which is great if you don't want to think (more kindly: if you're facing deadlines)
I'm sure there's good cases for logging (I read logs occasionally too) but it wouldn't be my _first_ thing to heavily invest in (hence the article)
So, the observability community is finally so full of ungrounded astronauts that "log" isn't the place you go to look for errors anymore and it's so full of noise that if you push the errors there, you will lose them.
Well, they've going that way for ages. But officially entering the stage where it's better to always ignore whatever they have to say is an important landmark anyway.
Well, they've going that way for ages. But officially entering the stage where it's better to always ignore whatever they have to say is an important landmark anyway.
I'm doing both. A line of logging that looks like "[date without timezone] Foo: bar additional stuff, almost no context" so, almost all lines found in /var/log/*/*.log, are mostly useless.
Right now, I'm playing with the idea to log json in those files and send this also to Signoz, Sentry, graylog, or whatever. This way, my logfiles are still tail-able and grep-able. Very important for development and "the customer is furious, WTF is going on in production" situations.
And also some other software can email me, according to some rules. This helps to have less furious customers. And maybe it can draw some cute graphs, so I can tell management that prod runs great!
Right now, I'm playing with the idea to log json in those files and send this also to Signoz, Sentry, graylog, or whatever. This way, my logfiles are still tail-able and grep-able. Very important for development and "the customer is furious, WTF is going on in production" situations.
And also some other software can email me, according to some rules. This helps to have less furious customers. And maybe it can draw some cute graphs, so I can tell management that prod runs great!
If you need an activity log as complementary information, you can send it into another stream. You can have more than 1.
Logs should be readable. What that means you should get the relevant information by reading the file after maybe searching for something a few times. That requires that the information there should be about a single topic. If that's too much, it should be tagged so that your tooling can make it readable, but this is for huge services.
Logs should be readable. What that means you should get the relevant information by reading the file after maybe searching for something a few times. That requires that the information there should be about a single topic. If that's too much, it should be tagged so that your tooling can make it readable, but this is for huge services.
We are testing Bugsink for a few (Django) projects now and it works great. Implementation is very easy, in two steps, if you are already using the Sentry SDK (there is no third step, as the documentation states :) We value the privacy of our end users and therefore it is important to us that the data stays within the EU.
I agree, I was trying to set up self-hosted centry recently and while it is possible, it has a lot of moving parts which might be overkill if your priority is error tracking. With bugsink I was up and running in a few minutes and the compatability with the sentry sdk is great!
Gotta admit these kinda overly-enthousiastic endorsements (parent & grandparent) make me blush slightly but I guess I shouldn't and should just stick 'm on the website under the testimonials section :-)
Can't speak for grandparent but you can happily use mine
Pretty sure you can get EU data residency with sentry
https://docs.sentry.io/organization/data-storage-location/
https://docs.sentry.io/organization/data-storage-location/
Awesome to see a self-hosted alternative to Sentry. Can't wait to try it out.
In practice most companies are always sending some PII (or even PHI....) to Sentry because filtering is always one step behind. Having it self-hosted would be my strong preference, but self hosting that is a PITA.
In practice most companies are always sending some PII (or even PHI....) to Sentry because filtering is always one step behind. Having it self-hosted would be my strong preference, but self hosting that is a PITA.
You can self host sentry. Although I hear it's a bit of a pain.
You might have even heard that right here on hacker news: https://news.ycombinator.com/item?id=43725815
I try and lean on Sentry and error tracking for as long as possible. And I'm happy to see some competition (and a simpler looking option).
What's everyone using for error reporting these days?
We're currently moving multiple projects from Sentry to Bugsink, and so far it's looking great, thanks for making this @vanschelven :)
No hate on Sentry by the way, but we (and some of our clients in the Netherlands) also like to have a European alternative, and it's a big bonus that this has a very low switching cost.
No hate on Sentry by the way, but we (and some of our clients in the Netherlands) also like to have a European alternative, and it's a big bonus that this has a very low switching cost.
I didn't want to put a "now that you're here" banner over the screen or at the bottom (I hate those myself) but as the OP I can heartily recommend https://www.bugsink.com/
User complaints.
> All of those are useful. But none of them tell you where the code broke.
Good logs do
Good logs do
For some definition of "where" :-)
it's my experience that a full stacktrace is needed (or at least super-helpful) when debugging, since the innermost (triggering) level is not always the level where the problem was introduced. Getting full context (including the values of local variables) all the way up the stack makes that puzzle a whole lot easier.
it's my experience that a full stacktrace is needed (or at least super-helpful) when debugging, since the innermost (triggering) level is not always the level where the problem was introduced. Getting full context (including the values of local variables) all the way up the stack makes that puzzle a whole lot easier.
I like the acronym MELT for Metrics, Errors, Logs, and Traces.
Passing the torch
your username made me do a double-take as it's a portmanteau of my current and previous startup (both in founder role)... but I see your actual story is basically the real-life version of the OP:
> Bugsense is a crash reporter [..] In September 2013, Bugsense was acquired by Splunk
> Bugsense is a crash reporter [..] In September 2013, Bugsense was acquired by Splunk