Yeah that sounds about right. Of course if you had 200 connections and 50 queues you'd more likely be seeing 100000 msg/s. The number of connections and queues has a big effect on total throughput.
I like the concept of error budgets. Start off by knowing what kind of quality and resiliency a system requires and design your test strategy around that. Means talking to the client about that.
I'm not going to invest a load of time in various types of automated test for an internal site with a form over a database that 2 users use for low priority work. The idea of 80%-100% code coverage for basic work like that seems like waste to me.
But for critical path of the eCommerce shopping experience I'm going to going to write all kinds of automated tests at multiple layers of the stack, right up to chaos/stress testing it, so that we know when black friday comes we can handle it.
I don't like dogma and TDD seems too dogmatic for me. I am very pro testing, having been both a QA, Developer and Ops engineer. I want the freedom to exercise my own expert judgement. The problem with dogma is that it makes Thinking take a back seat. Suddenly we have 80% code coverage enforced on a page that loads a grid from a table, going through a three layered monstrosity of code.
It all depends. For example with Apache Pulsar, tailing readers are served from an in-memory cache in the serving layer (the Pulsar brokers) and only catch-up readers end up having to be served from the storage layer (Apache BookKeeper). This is a little different from DistributedLog which always required going to BookKeeper for reads.
Apache BookKeeper can add additional latency to catch-up readers, on top of the extra hop, because the data of multiple topics are combined into each ledger. This means that we lose some performance from sequential reads. This is mitigated in BookKeeper by writing to disk in batches and sorting a ledger by topic so messages of the same topic are found together, but it still involves more jumping around on disk.
Also, BookKeeper allows the nice separation of disk IO. The read and write path are separate and can be served by different disks so you can scale your reads and writes separately to a certain extent.
For all those reasons, I would have loved to have seen Twitter look at Apache Pulsar and compare performance profiles with Apache Kafka.
To get some perspective on the difficulty of building this model, imagine you can see fine, but your screen can only display one word (token) at a time. You can navigate with arrow keys and a list of keyboard short-cuts.
This restriction applies to whether you are coding, browsing the web (on Stack Overflow), reading XML (shudder).
So the importance of a mental model is critical, because without maintaining context of where you are, you can get lost and spend too much time rediscovering your surroundings. The better your model the faster you can navigate and the less effort required in general.
I used a screen reader for a few years due to sight issues, these days I use a screen again with magnification. When you work without a screen you end up having to build up a mental model of the code, which you keep inside your head. When you navigate the code, you are doing it mentally, inside this cathedral you maintain in your mind.
So given that, the main challenge for me was code navigation. I used Visual Studio at the time and it allowed me to jump to method definitions, call references, jump to the start/end of a method etc. So the worst thing was long methods as 1) I had no efficient way of navigating them except to read each line 2) it was hard to keep track of all the things the method did. Breaking up code into smaller pieces with good naming of each method helped speed up my understanding and navigation of the code a lot. It also simplified the mental model in my mind.
My wife had HSCT to treat MS a year and a half ago. So far some symptoms went away, others stayed, but most importantly she hasn't got worse. Fingers crossed it will stay that way.
This is analysis is for people who want to understand the internals of Apache Pulsar, rather than a high-level overview of how to use the technology and how it compares to Apache Kafka.
This might just be the best, well-balanced talk on how agile has gone wrong, and ways to combat the decay.
I've seen a lot great teams and poor teams operate. A common set of components in great teams: Technical excellence, freedom coupled with accountability and giving a shit about the quality of their work.
Poor teams: lack of discipline, not caring and poor technical skills.
I've also seen great teams fall apart due to external influence. That influence was agile gone wrong, imposed from the top. Freedom: gone, but accountability remains. Technical practices valued by the team: deprioritized by the process. Giving a shit: demotivation followed by quitting or being fired.
I work at a boring company but I manage to get interesting projects to work on. I left CRUD behind about 5 years ago and I couldn't go back now. I just don't find most business logic problems interesting enough.
The transition for me came when I started a new job and my first project was to fix a slow batch job. Then I managed to get a second performance improvement project and soon I was the performance guy focusing entirely on performance projects. I really loved it.
Then through side projects targeted at problems I saw at the company, I built valuable tools using new open source tech and I got made tech lead of a new platforms team, solely focused on resolving technical problems and improving the system. Since then I have been an R+D engineer and now starting as a big data developer at the same company.
I am no genius, failed my interviews to get into big tech companies. I am a pretty big fish in a pretty small pond, I am known as one of the best technical guys in the company, which opens many doors and gives me a lot of freedom to move about inside the company and get some of the most interesting technically focused projects.
Never ascribe to malice, that which can be explained by incompetence.
I don't mean to be harsh on anyone, but after working for over a decade in enterprises, mediocrity seems to be the norm. There are of course bright and insightful people, but mediocrity is pretty much the standard. Take any good process, technical practice or whatever and see it be abused and misused.
With Agile what happens is a mediocre implementation that usually leads to the usual complaints we see about Agile. The number of times I've been told about how Scrum enables self-organising teams only to be told I have no control over the process because it has been standardised across the company.
Mediocrity demands cookie cutter solutions that can be standardised without need to engage thinking. If you work in an incompetent organisation, department or team, then there's no hope that agile or anything else can help you. In fact, the religious nature of Agile often works in favour of mediocrity as critics, even well-meaning ones, get labelled as waterfall sympathisers.
My antidote to this has been to craft a reputation for doing high quality work with good documentation. I don't let myself get pushed around now with this crap. If I want to spend more time on a feature, because of refactoring, writing some tests or whatever, I just do it. I don't pay attention to the sprint deadline but focus on quality work. I often miss the sprint deadline and make no effort to rush like the rest seem to do.
The result has worked out really well. Managers have quit hassling me about time and I get given the more complex and interesting work. People know I work hard and care about what I produce, if they have to wait they have to wait. I don't ask permission to write the tests I want, or spend a couple of days refactoring if that needs done.
Basically I am the professional coder, not the project managers, so I don't let them dictate how the software gets made. You just need to make sure you don't take it too far and too gratuitous refactoring work. Your reputation is critical for this approach to work.
I take the opposite view, the microservice anti-pattern is the API explosion. When you start chaining together HTTP REST APIs into wide and/or deep call graphs then you get all kinds of problems:
- need complex circuit breaker patterns to deal with API call failures
- overall uptime is affected by direct dependencies on a graph of synchronous calls (you are at the mercy of the other APIs for your own uptime)
- latency can increase as what could be done asynchronously is done synchronously
- developing and testing microservices becomes difficult because of so many dependencies. You start having to develop your own service stubs or evening running the other services.
Microservices are about independently deployable units. When you isolate services via message queues you can create services that have many fewer runtime dependencies which can lead to greater decoupling, greater reliability and easier development/testing. You are less likely to end up with a distributed monolith also.
Content marketing for SEO is everywhere. It's quite easy to spot, the author will explain why the practice of XYZ is great, in a pretty generic way, and then sprinkle in a couple of links to the sponsor that provides a product or service around that XYZ.
My wife had HSCT treatment for MS a year ago. She was deteriorating fast. Since the treatment the progression has stopped and some symptoms have gone away.
HSCT is real and it works. It stops disease progression. Usually there is lasting damage which means that even when the disease is stopped there is remaining disability. This is why the word Cure is contentious as many symptoms persist. But my wife can live with a certain level of disability. The most important thing is that her decline has stopped. Now she has a future.