thomaslee

https://en.wikipedia.org/wiki/Quoted-printable

Note that 0x3D is the "=" character in ASCII, so "=3D" in QP is "=" in ASCII. :)

This email has probably been through a few conversions to QP and back again between different email clients. Perhaps some buggy client got confused between an ASCII "=" and a QP escape sequence or something like that.

thomaslee

7 lat temudiscuss

Another FYI type comment I guess. :) Some of my more gripey statements here may be outdated info, so DYOR I guess.

Dynamic reconfig in 3.5 addresses the "restarting every zookeeper instance" problem. [0] You stand up an initial quorum with seed config, then tie in new servers with "reconfig -add". Not sure how well it would tie into cloudy autoscaling stuff though. I wouldn't start there myself.

A much bigger pain IMO is the handling of DNS in the official Java ZK client earlier than 3.4.13/3.5.5 (and by association, Curator, ZkClient, etc.). [1] The former was released mid 2018 and the latter this year, so tons of stuff out there that just won't find a host if IPs change. If you "own" all the clients it's maybe not a problem, but if you've got a lot of services owned by a ton of teams it's ... challenging.

Even with the fix for ZOOKEEPER-2184 in place I'm pretty sure DNS lookups are only retried if a connect fails, so there's still the issue of IPs "swapping" unexpectedly at the wrong time in cloud environments which can lead to a ZK server in cluster A talking to a ZK server in cluster B (or worse: clients of cluster A talking to cluster B mistakenly thinking that they're talking to cluster A). I'm sure this problem's not unique to ZK though.

Authentication helps prevent the worst-case scenarios, but I'm not sure if it helps from an uptime perspective.

TL;DR: ZK in the cloud can get messy (even if you play it relatively "safe").

[0] https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.ht... [1] https://issues.apache.org/jira/browse/ZOOKEEPER-2184

thomaslee

8 lat temudiscuss

> I think you’re retroactively claiming that Google actively anticipated this in their choice at the beginning of using Perforce as an SCM.

Oh I didn't mean to imply exactly that, but really good point. I just meant that it seems like folks don't typically _anticipate_ these issues so much as they're forced into it by ossifying velocity in the face of sudden wild success. I know at least a few examples of this happening -- but you're right, those folks were using Git.

In Google's case, maybe it's simply that their centralized VCS led them down a certain path, their tooling grew out of that & they came to realize some of the benefits along the way. I'd be interested to know too. :)

thomaslee

8 lat temudiscuss

Exactly this. Or at least it's a way this can be achieved, assuming solid testing & some tooling in the mix.

For folks unfamiliar with it, the issue is something like:

1. You find a bug in a library A.

2. Libraries B, C and D depend on A.

3. B, C and D in turn are used by various applications.

How do you fix a bug in A? Well, "normal" workflow would be something like: fix the bug in A, submit a PR, wait for a CI build, get the PR signed off, merge, wait for another CI build, cut a release of A. Bump versions in B, C and D, submit PRs, get them signed off, CI builds, cut a release of each. Now find all users of B, C and D, submit PRs, get them signed off, CI builds, cut more releases ...

Now imagine the same problem where dependency chains are a lot more than three levels deep. Then throw in a rat's nest of interdependencies so it's not some nice clean tree but some sprawling graph. Hundreds/thousands of repos owned by dozens/hundreds of teams.

See where this is going? A small change can take hours and hours just to make a fix. Remember this pain applies to every change you might need to make in any shared dependency. Bug fixes become a headache. Large-scale refactors are right out. Every project pays for earlier bad decisions. And all this ignores version incompatibilities because folks don't stay on the latest & greatest versions of things. Productivity grinds to a halt.

It's easy to think "oh, well that's just bad engineering", but there's more to it than that I think. It seems like most companies die young/small/simple & existing dependency management tooling doesn't really lend itself well to fast-paced internal change at scale.

So having run into this problem, folks like Google, Twitter, etc. use monorepos to help address some of this. Folks like Netflix stuck it out with the multi-repo thing, but lean on tooling [0] to automate some of the version bumping silliness. I think most companies that hit this problem just give up on sharing any meaningful amount of code & build silos at the organizational/process level. Each approach has its own pros & cons.

Again, it's easy to underestimate the pain when the company is young & able to move quickly. Once upon a time I was on the other side of this argument, arguing against a monorepo -- but now here I am effectively arguing the opposition's point. :)

[0] https://github.com/nebula-plugins/gradle-dependency-lock-plu...

thomaslee

9 lat temudiscuss

> The answer would be no in general, I think, since it is unsafe. > ... > for example, the statement may include an rpc, and we don't want to make that rpc under the lock

I do agree that it's not a "safe" optimization in the extreme general case (so don't go rewriting your code assuming it's equivalent!), but in the case where the loop is a candidate for unrolling it works just fine. Imagine you had a more CPU- or memory-bound workload, and these benchmarks are a whole lot more interesting.

Put another way: if there's an RPC call in the for loop, the time spent in the RPC will dwarf the work involved in executing loop itself so ... odds are good it's not going to be a candidate for unrolling anyways. :)

thomaslee

9 lat temudiscuss

If any Python devs are out there reading: my understanding is that removing the GIL itself isn't the hard part so much as removing the GIL while satisfying certain constraints deemed necessary by GvR and/or the rest of the community. I know some of those constraints relate to compatibility with existing C extensions -- but there must be others too?

The reason I ask is Larry's attempt buffered ref counting surely has implications for single-threaded code that maybe relies on the existing semantics -- e.g. a program like this may no longer reliably print "Deallocated!":

  Python 2.7.13 (default, Mar  5 2017, 00:33:10) 
  [GCC 6.3.0 20170205] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> class Foo(object):
  ...     def __del__(self):
  ...             print 'Deallocated!'
  ... 
  >>> foo = Foo()
  >>> foo = None
  Deallocated!
  >>>

A bad example in some ways since in this particular case we could wait for all ref counting operations to be processed before letting the interpreter exit, but hopefully my point is still clear.

Similarly, what about multi-threaded Python code that isn't written to operate in a GIL-free environment -- absent locks, atomic reads/writes, etc.? At best, you might expect some bad results. At worst, segfaults.

Are these all bridges that need to be crossed once a realistic solution to the core GIL removal issue is proposed? As glad as I am that folks are still thinking hard about this problem, I'm personally sort of pessimistic that the GIL can be killed off without a policy change wrt backward compatibility. Still, I do sort of wonder if some rules of engagement wrt departures from existing semantics might help drive a solution.

thomaslee

9 lat temudiscuss

> The goal should be, and is kind of what Larry Hastings is looking for, is that any program should run 8 times faster on a 8-core CPU compared to a 1-core.

A program that's inherently single-threaded it's unlikely to benefit from more CPUs. When you say "any program" here, you mean "any program with >=8 threads", right?

thomaslee

9 lat temudiscuss

> Servers are where the problem is. The GIL makes python functionally single-threaded, which is a bummer for your server at any kind of scale.

Right, agreed. I can imagine some of the frustration you might experience using CPython for high throughput systems: kind of like NodeJS without the benefits of a standard library written with async/non-blocking I/O in mind.

A bit curious about a few things you mention here, though:

> Or that a python process never really releases memory back to the system, just within itself, so the process slowly grows over the course of a few weeks.

I'm not sure this is true in general, is it? Can you elaborate? It's been a while since I've dug around in Python innards, but if Py_DECREF(x) leads to a refcount of zero IIRC free(x) is ultimately called -- albeit in an indirect manner via a layer or six of tp_dealloc calls and tp_free. :) I suppose calling free(x) may only return the memory associated with x to (g)libc's free list and not necessarily back to the OS [0]. No different to C/C++ in that regard, I guess.

> I came to the conclusion that python is unsuitable for servers, but until Go came out, there wasn't a realistic alternative, since C++ and Java are too heavyweight, and Ruby suffers from similar problems (don't know about a GIL).

"Too heavyweight" in that they're relatively difficult to write in comparison? Maybe true of Java-the-language, but the JVM itself is an absolute workhorse when it comes to high performance. Plenty of languages to choose from there, typically without a GIL. Jython, for example, has no GIL [1].

And yep, Ruby/MRI has a GIL (but JRuby does not).

[0] https://www.gnu.org/software/libc/manual/html_node/Freeing-a... [1] https://stackoverflow.com/questions/1120354/does-jython-have...

thomaslee

9 lat temudiscuss

I can't speak to the rest of Facebook's stuff, but I think the build tool problem is a special case. Per their docs:

> Buck is designed for building multiple deliverables from a single repository (a monorepo) rather than across multiple repositories. It has been Facebook's experience that maintaining dependencies in the same repository makes it easier to ensure that all developers have the correct version of all of the code, and simplifies the process of making atomic commits.

Having been on the "other side" of the monorepo argument where we tried to make do with improving/extending existing build tools etc. in a rapidly growing engineering org, let me say that Facebook (with Buck), Twitter (Pants? I think?) and Google (with Bazel/Blaze) almost certainly built these to deal with the problem of scaling build management with an ever-growing organization.

The popular model of a dozen or so small repositories in GitHub + Jenkins with Maven/NPM/Rake+Bundler/whatever works fine for maybe a few dozen engineers or more, but one day you wake up and realize there hundreds of repositories spread across dozens of _teams_ and hundreds of developers. Obviously you've then got a big ol' dependency graph between repos to deal with, so if you need to fix something near the root suddenly you need to run off bumping version numbers and/or fixing intermediate libraries all the way down the graph. Plus version incompatibilities between the dependencies of different libraries ... it's a total mess, and it doesn't make for an org that can easily "move fast and break things", so to speak.

So then to avoid paralysis your options are basically either to silo up (this team owns their stuff, that team owns their stuff, don't bother with shared dependencies) or you go the monorepo route. If you do, then maybe you go and pull all your hundreds of smaller repos into a monorepo. Having everything in one place makes it easier to police the dependency issues within the org & makes it easier for a single engineer to deal with those sort of "cascading changes" instead of shunting that problem onto the entire organization. But in exchange for this "agility" you've then got the problem that builds take multiple hours & the associated tools are often highly language-centric (Maven+Java, NPM+Node, Ruby+Rake, etc.). They don't typically make any reproducibility guarantees either.

Anyway, to make a short story really long: at the time FB, Google and Twitter were hitting these organizational scaling walls, making these decisions and building these tools internally, there really weren't any great tools out there for the monorepo use case. I think that's why all these tools have appeared as side-by-side alternatives rather than improvements on one another or to tools like Maven et al.

Whether or not consolidation is warranted, for the folks who have the problems that Buck/Bazel/Pants solve, it's likely to save 'em a hell of a lot of time, effort and money IMO. It's a good thing that they have been published, even if the value's maybe not immediately obvious.

thomaslee

10 lat temudiscuss

You could, for example, write a compiler in Python that generated native code (or LLVM bitcode to be passed to llc or whatever) via the LLVM API. Writing a compiler in Python vs. C/C++ would be a lot easier in a number of ways.

I'm not aware of any "production grade" compilers that do this, but no hard reason why not, I guess. Seems like it'd be nice for prototyping etc. if nothing else.

thomaslee

10 lat temudiscuss

Exactly. You can keep the brand while still communicating the fact it's an overhaul.

thomaslee

10 lat temudiscuss

I'm not the parent, but I sort of like the idea at first glance. I mean, it's a fine line: if I have "v1.0.0" and I break one API in one module, I'm compelled to release "v2.0.0" even though v2.0.0 is otherwise entirely compatible with v1 -- it's not as big of an upgrade as the version numbering system would imply.

But if I go and drastically change things in a way likely to impact many users, that gets a brand new version number too. So v1.0.0 -> v2.0.0 only really communicates "something might break".

The scheme proposed by the parent would be able to communicate "expect many things to break because I refactored the heck out of stuff to fix some long-standing design deficiencies" -- though admittedly when to bump that first version number is likely to be a subjective topic. :)

Perhaps this isn't as valuable as it seems at a first glance, but if anybody's tried something like this I for one would be interested to hear about it.

thomaslee

10 lat temudiscuss

> So, what's a decent laptop for running Linux? Good keyboard, good trackpad support, good sound . . .

Sadly enough, I think the lack of really great options is what may have pushed some engineers to macs in the first place. :)

Admittedly, last I looked was several years ago -- I'm interested to know too! Keep hearing good things about the XPS series, but last I had one it ran hot enough that I couldn't comfortably keep it on my lap.

thomaslee

10 lat temudiscuss

+1 on the date, but generally relevant to most modern x86/x86-64 hardware IMO.

(The rest of this comment is largely my own inflated opinion.)

From a developer's perspective, whether it's _useful_ when operating at a certain level of abstraction is another question. Really do think it's good to know this stuff, but if you're writing, say, Ruby or Python web apps you probably don't _need_ to know it.

Low latency/high throughput/systems-level stuff it starts to become more relevant. That said, there's certainly been operational scenarios where knowing some of this stuff can help narrow down misconfigured systems and/or squeeze more life out of existing hardware (e.g. configuring a JVM to use huge pages).

thomaslee

10 lat temudiscuss

> It seems like so many "best practices" are really thinly veiled attempts at exploding complexity

I'll bite. :D

It's a bit more nuanced than that IMO, the "deploy often" mantra is only as good as the process around release + deploy. If you half-ass testing and push to production without without a process for verification -- or if, say, your deploy process is half-baked, or your staging environment is worthless -- you can probably expect "interesting" production deploys on a pretty regular basis.

As much as we'd like to pretend we're all good engineers, this happens more often than you'd expect -- even with good engineers: at some point a company transitions from scrappy startup to a shambling beast, and the things that used to work for a scrappy startup (like skimping on testing and dealing with failures in production) are insufficient when you've got more eyes on the product. Further, the engineering culture remains stuck in "scrappy startup" mode long after the shambling has begun.

And all that's ignoring the fact that less frequent deploys with more changes have their own set of problems. We actually got to a point with deploys of a certain distributed system such that we were terrified if we had more than a few days worth of changes queued up. So many things that could go wrong! :)

> We create ourselves so many of the problems we are paid to solve.

This, on the other hand, I completely 100% agree with: if not us, then who? :)

EDIT: minor formatting change

thomaslee

10 lat temudiscuss

I was obsessed with this game when I was a teenager. The concepts for Cloud and Sephiroth being very deliberately drawn from Musashi and Kojiro was news to me: http://shmuplations.com/wp-content/uploads/2015/01/ff707.jpg

thomaslee

10 lat temudiscuss

In theory, somebody could publish a binary that could be downloaded and run standalone -- no need for "go get". Looks like no such binary has been published, though.

thomaslee

10 lat temudiscuss

So, uh, color me selfish. :) Perhaps this isn't a constructive comment, but I just don't understand these articles that take a long, hard, meandering look at human nature only to arrive at some absolute "truth" like the author's some arbiter of what's right and wrong in the world. Whatever gets eyeballs, I guess.

thomaslee

10 lat temudiscuss

Piling on the nitpick train :) If an error occurs along the way popd won't run, so a subshell might be a better way to accomplish the same thing:

    (
      for ...; do
        cd ...
      done
    )
    # script exits, you're back where you started

If you'd prefer to see it in isolation: pwd; ( cd /tmp; pwd ); pwd

thomaslee

10 lat temudiscuss

> When you Produce a Message Set onto the bus, you don't directly get back a response telling you that the messages have successfully been persisted to one or more partitions. Instead, you must also Consume the bus, and you should eventually receive multiple messages acknowledging the persistence of each message in the set.

Maybe this has changed recently, but IIRC this isn't true if your ProducerRequest has the ack bit set to 1 or 2 (i.e. leader or replica acking):

https://github.com/confluentinc/kafka/blob/79aaf19f24bb48f90...

The response/ack is sent directly over the socket sending the request.

> If a Node dies then a "leadership election" happens, ZooKeeper is updated with the new metadata, and your application must react to this and handle the changes. There's a six second delay while this happens

Not that I doubt it, but not sure where six seconds comes from here ... perhaps waiting for partition leader elections? It's been long enough that I can't quite remember exactly what happens during a failover.

> and who knows what happens if you try and send messages to a dead node during that time.

Depends how it died, which client API you're using and how the client is configured. Some combination of:

* data loss if acking is disabled (hint: enable acking) * backpressure and errors in the client until new partition leaders kick in * client socket writes hanging "forever"

If the latter is surprising: no SO_SNDTIMEO in pure Java blocking socket I/O. Think the new clients may address that, but not entirely sure.

As an aside: can't emphasize enough how important it is to get your configuration right early. By the time you run into problems, it's often too late. Pay heed to any tuning guides you can find. Talk to Confluent if you're still unsure.

> AND HAVE THEY OPEN SOURCED THIS MAGICAL SERVER? NO, THEY BLOODY HAVEN'T.

https://github.com/confluentinc/kafka-rest this thing? FWIW, it's kind of a joke for high throughput anyway. Last time we spoke to Confluent they sort of discouraged its use for exactly that reason.

Still, it's an easy bridge for folks who aren't too fussed about throughput. Not sure why you'd be using Kafka if throughput's not your thing, but y'know.

> If you are using Java/Scala/Clojure/Kotlin/whatever and can use the Official Java Client then I'm sure Kafka is a perfectly reasonable choice for a message bus, although there are plenty of others that seem to me to be far less bloody-minded.

Despite all the gotchas, Kafka's capable of pretty incredible throughput in a fault-tolerant HA configuration. I can empathize with some of the frustrations, but past a certain scale the proposed alternatives just aren't IMHO.

Submissions

Spin model checker open sourced under 3-clause BSD

comments