Simplify Service Dependencies with Nodes(blog.twitter.com)
blog.twitter.com
Simplify Service Dependencies with Nodes
https://blog.twitter.com/2016/simplify-service-dependencies-with-nodes
59 comments
This is like the fourth instance of computational graphs that I've seen recently. Can someone recommend some resources on the underlying theoretical ideas? It's not just graph theory, it's something like computational graph theory or maybe data flow but neither seems to be at the level of formalism I'd like.
You can execute that kind of task with a RETE engine: https://en.wikipedia.org/wiki/Rete_algorithm
> Rete algorithm (/ˈriːtiː/ REE-tee or /ˈreɪtiː/ RAY-tee, rarely /ˈriːt/ REET or /rɛˈteɪ/ re-TAY)
I guess we all can at least agree that there is a way to pronounce it :)
I guess we all can at least agree that there is a way to pronounce it :)
The first two forms are more correct, but the last two sound better. I say 'REET networks', so that's almost certainly wrong.
The name seems to come from Latin and courtesy to my native Romanian aka "Slavicized Latin", it should be pronounced: reh-teh, short sounds for both syllables.
Of course, since now it's a loanword in English, you could also just use English rules for pronouncing it (whichever those are :) ).
Of course, since now it's a loanword in English, you could also just use English rules for pronouncing it (whichever those are :) ).
Thanks - I'll try it out. You're correct that the term comes directly from Latin.
Is there something more general?
What do you mean by more general? Data in nodes in the RETE algorithm can be anything.
I highly recommend reading "Production Matching for Large Learning Systems"[0] as it was accessible enough for me to try to make a basic RETE implementation while learning C++ and playing around with several other technologies I was unfamiliar with[1].
[0]http://reports-archive.adm.cs.cmu.edu/anon/1995/CMU-CS-95-11...
[1]https://github.com/cjslep/cpp-rete-prototype (awful, abandoned prototype)
I highly recommend reading "Production Matching for Large Learning Systems"[0] as it was accessible enough for me to try to make a basic RETE implementation while learning C++ and playing around with several other technologies I was unfamiliar with[1].
[0]http://reports-archive.adm.cs.cmu.edu/anon/1995/CMU-CS-95-11...
[1]https://github.com/cjslep/cpp-rete-prototype (awful, abandoned prototype)
I've wondered about this as well. I'd want to cover ZDDs in addition to Rete (forward and back chaining).
https://en.wikipedia.org/wiki/Zero-suppressed_decision_diagr...
https://en.wikipedia.org/wiki/Zero-suppressed_decision_diagr...
Petri nets? Those can be cyclical though - i don't know if there is a special name for acyclical nets.
Yet another instance of this is the Disruptor patter that use it in an event queue.
Yet another instance of this is the Disruptor patter that use it in an event queue.
Maybe I'm missing something, how is this better than simple promises in JS? This seem easy to reason about:
serviceA { Promise.All(b,c).then(...) }
serviceB { (d).then(...) }
serviceC { Promise.All(d,e).then(...) }
serviceA { Promise.All(b,c).then(...) }
serviceB { (d).then(...) }
serviceC { Promise.All(d,e).then(...) }
Yes you are missing something. The purpose of this library is to decouple the "description" from the "computation" itself. I usually call "separating the _what_ from the _how_".
Essentially they construct an AST for whatever services that need to be called. Most of the time they then simply "interpret" the AST with the lower-level Finagle constructs. But, and that's actually the big point, they can use _the same code_ with a different interpreter to generate a GraphViz representation for visual inspection. This super helpful once your call graph becomes more complex .. Another possibility is to write an optimizing interpreter etc.
Anyhow, what I find a bit sad is that they didn't leverage the Scala part if Finagle more .. this would be a prime example for a Free Monad () based DSL. That way they could have avoided writing 10s of classes with less than fortunate names.
() might need a Free Applicative part as well since they inherently want to have parallelism expressed in the language
Essentially they construct an AST for whatever services that need to be called. Most of the time they then simply "interpret" the AST with the lower-level Finagle constructs. But, and that's actually the big point, they can use _the same code_ with a different interpreter to generate a GraphViz representation for visual inspection. This super helpful once your call graph becomes more complex .. Another possibility is to write an optimizing interpreter etc.
Anyhow, what I find a bit sad is that they didn't leverage the Scala part if Finagle more .. this would be a prime example for a Free Monad () based DSL. That way they could have avoided writing 10s of classes with less than fortunate names.
() might need a Free Applicative part as well since they inherently want to have parallelism expressed in the language
It feels like over engineering for most projects, and yet another thing for new hires to learn.
I can't judge Twitter's design without seeing it, but I don't like the sound of this:
"the simplest web page on twitter.com requires collaboration of dozens of network services speaking many different protocols"
Simplest page? Dozens of collaborating services? Many different protocols?
I'm skeptical this is the most elegant architecture possible that meets the requirements of their simplest web page.
I can't judge Twitter's design without seeing it, but I don't like the sound of this:
"the simplest web page on twitter.com requires collaboration of dozens of network services speaking many different protocols"
Simplest page? Dozens of collaborating services? Many different protocols?
I'm skeptical this is the most elegant architecture possible that meets the requirements of their simplest web page.
Monad implies Applicative, so the "might need a free applicative part as well" is already a given.
Yes and no. With Free Applicatives you can express side-effects such as parallelism .. not so with Free Monads afaik
Looks just like that. As dstaten said, "just Promise.all but for microservices". I've been contributing to an open-source project called snappi.io - a microservice blueprint that includes very comparable dynamic RPC functionality. Combining RPC with standardized service contracts allows us to create and inject stubs for distributed systems as they get shipped out to solve the problems described in the article. Happy to see companies like Twitter following a similar path, and we're hoping to bring those patterns to more applications powered by a wide variety of tech-stacks.
that's very interesting. Have you looked at linkerd.io (used in production at large kubernetes deployments)
I have and personally I'm quite fond of it, but it doesn't quite go far enough imo. Linkerd is very lean and requires virtually zero knowledge of how the underlying service it manages works, but the knowledge of interface contracts for services is an important one.
Snappi introduces some mild requirements for how a service exposes it's functions, but once that is done the rest can be automated at deploy time. Not only can service-based load balancers be created dynamically, but RPC stubs can be created for each service and injected into peer services that are dependent on them. For example, if ServiceA needs to access ServiceB, it can do so by referencing this RPC stub knowing that the location of ServiceB will be injected at deploy time to ensure requests are fulfilled for the environment.
There are a number of benefits to this approach for a single application, but one of the larger benefits of consistent specs for contract creation is that individual services will be far easier to share. We're trying to create a structure that will not only empower us to re-use our own services, but also to consume services published by others by simply specifying them as a dependency.
Snappi introduces some mild requirements for how a service exposes it's functions, but once that is done the rest can be automated at deploy time. Not only can service-based load balancers be created dynamically, but RPC stubs can be created for each service and injected into peer services that are dependent on them. For example, if ServiceA needs to access ServiceB, it can do so by referencing this RPC stub knowing that the location of ServiceB will be injected at deploy time to ensure requests are fulfilled for the environment.
There are a number of benefits to this approach for a single application, but one of the larger benefits of consistent specs for contract creation is that individual services will be far easier to share. We're trying to create a structure that will not only empower us to re-use our own services, but also to consume services published by others by simply specifying them as a dependency.
you should get in touch with @beeps of Kubernetes. There has been quite a few discussions on replacing the iptables based Service architecture of k8s with something like Ingress (ingress-all-the-way).
Your work could be perfectly integrated into k8s
Your work could be perfectly integrated into k8s
Thanks for the suggestion! We're actually using a lot of k8s inspired processes to power the tool, and may end up building on top of k8s as we learn more about how other developers use snappi. Definitely a lot of synergies, and I'll be sure to reach out to @beeps to talk about them.
Thank you! This is just Promise.all but for microservices. Every time a large company discovers that its messy infrastructure can be cleaned up a bit by reimplementing a simple programming concept, it's suddenly not just newsworthy but framework worthy
The library they referenced (Finagle) does pretty much just that. A few things that leapt out at me:
- It's hard to figure out what the dependency graph looks like just from promises
- That pattern is verbose and results in gross types when you do it in Java
- With this library, you can easily profile parts of the algorithm to see what's going slow, what's taking a bunch of memory, etc.
- It's hard to figure out what the dependency graph looks like just from promises
- That pattern is verbose and results in gross types when you do it in Java
- With this library, you can easily profile parts of the algorithm to see what's going slow, what's taking a bunch of memory, etc.
I think Fnangle is an abstraction over futures. The selling point I found when looking at Tokio (rust equivalent, references Fnangle) was that you write a driver for each backend and then have generic services to implement things such as timeouts. So you can reuse the timeout code for your network queries and your db queries.
The "simple promises" from JS are (somewhat crippled) copies of Futures from other languages. In Scala the results of those futures can be produced at the same time, even if they are purely computational as Scala and most other languages are both asynchronous AND concurrent.
JS is async and concurrent. I resent it's limitations as much as any self respecting developer, but workers are finally in.
[deleted]
So, between the lines - Twitter looks ahead to move to Java 8 and finally ditch Scala?
Hum .. don't think so. Not quite yet at least. Underneath this, there is Finagle on top of which most services at Twitter is built on. It offers a java API, but remains a Scala library. And there's still a lot of existing code inside to twitter to ditch Scala right away. Java 8 is awesome, but Java is still not an FP language.
IMHO the problem with Java 8 is not the language but framework support. Except SparkJava, most other frameworks use Java 8 as a "upgraded java compiler". Tge canonical way to write code is still Java 6.
I think the best way to upgrade your java codebase is Kotlin. On the server side, Kotlin +vertx is extremely compelling.. on the Android side, Kotlin pretty much kills it.
I think the best way to upgrade your java codebase is Kotlin. On the server side, Kotlin +vertx is extremely compelling.. on the Android side, Kotlin pretty much kills it.
I agree. This is exactly the move I've made when I was CTO of a company needing a move to the JVM from .NET. Given the options of Java, Scala - I moved to picking Kotlin. In many ways, Kotlin for me is Scala without the different ways to hang yourself. I wish it was given more credit that it gets right now outside of the Android circles.
that is very interesting!
could you talk about the framework that you used? im particularly concerned about ORM and database connectivity.
could you talk about the framework that you used? im particularly concerned about ORM and database connectivity.
If by framework you mean web framework, I've used Dropwizard. It's a lean and mean modern-java option, and it's not a Kotlin native thing (you do have these). You can use any Kotlin or traditional Java framework out there transparently (that's a good value of Kotlin).
For ORM, Dropwizard and its ecosystem got you covered.
Also dropping out to Java when ever you want can be done in the same project (which I encouraged) seamlessly both in terms of how Java and Kotlin interoperate, and in terms of Intellij, which is first-class in both languages because Kotlin is Jetbrains' baby, and, because Java has been best supported by Intellij for ages (can't really find that kind of thing anywhere else).
For ORM, Dropwizard and its ecosystem got you covered.
Also dropping out to Java when ever you want can be done in the same project (which I encouraged) seamlessly both in terms of how Java and Kotlin interoperate, and in terms of Intellij, which is first-class in both languages because Kotlin is Jetbrains' baby, and, because Java has been best supported by Intellij for ages (can't really find that kind of thing anywhere else).
Why ditch Scala completely?
I'm also curious about this one — I always found Scala to be a pretty messy language to write (the code starts elegant, but gets convoluted pretty fast), but I'd love to hear how insiders are thinking about it.
Last I've heard, Twitter was "stuck" with their own version of Scala (was it off 2.10?). Typesafe becoming Lightbend (i.e. not Scala focused anymore) for doing business, Odersky moving all of his focus to Dotty, and core Scala contributors leaving. That's my intuition of asking.
That is not true.
Kotlin is a joke, but Scala's mismanagement and treatment of its contributors might be its undoing. Source: #1/#2 top active community contributor (depends on whether you count commits or locs in scala/scala) leaving Scala, because I'm tired of their shit.
Yeah, Scala's a bit of a mess right now, definitely not thrilled that the flagship web framework (Play) won't be supporting Scala 2.12 until the 2.6 release (i.e. 4-6 months from now), which is absurd given that pretty much the entire ecosystem has a 2.12 release at this point :\
As for typed alternatives, there's no substitute for Scala. Maybe Haskell or OCaml if you leave the JVM, but otherwise it's pretty much a language wasteland for anyone that's taken a dip in the ocean of Scala.
> because I'm tired of their shit
heh, think it goes both ways ;-) As an observer none of the powers that be took a liking to you calling them out, justified or not (agree that community contrib takes a back seat to Lightbend/EPFL).
Good luck wherever you've wound up (and thanks for the biased Either contrib in 2.12, finally!)
As for typed alternatives, there's no substitute for Scala. Maybe Haskell or OCaml if you leave the JVM, but otherwise it's pretty much a language wasteland for anyone that's taken a dip in the ocean of Scala.
> because I'm tired of their shit
heh, think it goes both ways ;-) As an observer none of the powers that be took a liking to you calling them out, justified or not (agree that community contrib takes a back seat to Lightbend/EPFL).
Good luck wherever you've wound up (and thanks for the biased Either contrib in 2.12, finally!)
> As an observer none of the powers that be took a liking to you calling them out
The best way to avoid being called out for saying completely different things in public and private is not saying different things in public and private.
If me politely asking for clarification about this sudden change of opinion is "underhanded behavior", or "renouncing from future interactions" means "let's find some flimsy pretext to contact your place of work to tell them what an evil person you are" then Scala might have some issues attracting and retaining contributors in the future.
The best way to avoid being called out for saying completely different things in public and private is not saying different things in public and private.
If me politely asking for clarification about this sudden change of opinion is "underhanded behavior", or "renouncing from future interactions" means "let's find some flimsy pretext to contact your place of work to tell them what an evil person you are" then Scala might have some issues attracting and retaining contributors in the future.
There's always Typelevel, sure you'd be welcome there as that's kind of the anti-establishment wing of the Scala community ;-)
It's also thriving/taking off, tons of contributions from myriad brilliant minds; for this reason I'm not too worried about Scala, it will continue on with or without Lightbend steering the ship.
It's also thriving/taking off, tons of contributions from myriad brilliant minds; for this reason I'm not too worried about Scala, it will continue on with or without Lightbend steering the ship.
Don't think so.
They won't be able to prevent the language from being run into the ground by SIP/SLIP/SPP committees.
Anyway, getting told that I'm not qualified to tell people that their new language extension ideas tick all the boxes of "bad ideas that we are regretting and deprecating since years" after I have more or less managed deprecations and removals for four major versions of Scala ... I guess they need to find someone else to do my job now.
But given past experience, the people who are eager to add more and more features are seldomly the people who clean up after themselves.
Most upcoming language proposals have extremely poor quality, ignore years of lessons learned, repeat many mistakes of the past, reinvent things that have been tried without success elsewhere and have no respect for design principles that made Scala great.
So glad that my name won't be associated with this.
They won't be able to prevent the language from being run into the ground by SIP/SLIP/SPP committees.
Anyway, getting told that I'm not qualified to tell people that their new language extension ideas tick all the boxes of "bad ideas that we are regretting and deprecating since years" after I have more or less managed deprecations and removals for four major versions of Scala ... I guess they need to find someone else to do my job now.
But given past experience, the people who are eager to add more and more features are seldomly the people who clean up after themselves.
Most upcoming language proposals have extremely poor quality, ignore years of lessons learned, repeat many mistakes of the past, reinvent things that have been tried without success elsewhere and have no respect for design principles that made Scala great.
So glad that my name won't be associated with this.
Well, will be interesting to see how Dotty plays out.
DOT may have been proven sound, but compilers are complex beasts; implementing DOT while preserving an upgrade path for Scala will likely be both difficult and limiting (since Dotty will inevitably be shackled by Scala's past).
For now we've got Scala though, in a couple of years maybe Dotty.
DOT may have been proven sound, but compilers are complex beasts; implementing DOT while preserving an upgrade path for Scala will likely be both difficult and limiting (since Dotty will inevitably be shackled by Scala's past).
For now we've got Scala though, in a couple of years maybe Dotty.
Dotty will inevitably implement the same features with the same fundamental issues. This is not about some old stuff we all regret having in Scala, this is about new low-quality proposals. When they get rubberstamped, they will end up in both Scala and Dotty.
why do you call kotlin a joke? because it is extremely popular in the Android world for large production apps.
plus with the upcoming kotlin 1.1 release and vertx's upcoming first class support for kotlin.. it is beginning to build traction on the server side.
what is really exciting is Kotlin scripting. Gradle is the first project to start using it.
plus with the upcoming kotlin 1.1 release and vertx's upcoming first class support for kotlin.. it is beginning to build traction on the server side.
what is really exciting is Kotlin scripting. Gradle is the first project to start using it.
So I don't fully follow it all -- but I find it really fascinating. Perhaps the real value at twitter is not the service twitter offers of letting you communicate 140 characters at a time but the engineering that goes on behind the scenes.
From the article..
> Nonetheless, Finagle still solved a huge problem for us as we now have real data dependencies and a more efficient execution engine. We gradually converted the Blender workflow code from batch style to Future style in 2013. It ran faster, the code became less error-prone but the readability was still not perfect. It was hard to follow, a pain to debug, tricky to test, and there were lots of duplicate function names.
from the git repo for Node...
> However, this library was written in Scala and isn't exactly Java friendly. It naturally involves a lot of callbacks and repeated function signatures. When it comes to waiting on multiple Futures, the code gets ugly very fast. Nodes is a Java library that aims to solve these problems, making the asynchronous code easier to read, to maintain and to test in Java.
what are they suggesting ? use Finagle for Scala and Nodes for java based projects?
> Nonetheless, Finagle still solved a huge problem for us as we now have real data dependencies and a more efficient execution engine. We gradually converted the Blender workflow code from batch style to Future style in 2013. It ran faster, the code became less error-prone but the readability was still not perfect. It was hard to follow, a pain to debug, tricky to test, and there were lots of duplicate function names.
from the git repo for Node...
> However, this library was written in Scala and isn't exactly Java friendly. It naturally involves a lot of callbacks and repeated function signatures. When it comes to waiting on multiple Futures, the code gets ugly very fast. Nodes is a Java library that aims to solve these problems, making the asynchronous code easier to read, to maintain and to test in Java.
what are they suggesting ? use Finagle for Scala and Nodes for java based projects?
Seems like all the languages and frameworks are creeping slowly and cautiously towards the Actor model. Come on in folks, the water's fine!
[deleted]
Stream processing, ladies and gentlemen! Kafka Streams, Apache Beam, and everything else. Great to see so many people publishing their results for the problems addressed here.
[deleted]
Please don't ask me why, but I'm going to go out on a limb and guess that one of the people who "invented" this system at Twitter is a Xoogler.
Actually I wrote the first version of this graph library 3yrs ago. It was to help the search relevance team create reusable components, turn implicit dependency management into a first class citizen and finally add a way to dump the graph state at runtime or statically.
Looks like just one, from the list of major contributors.
But, Google really has no right to whine about this. They make an explicit tradeoff: they manage to stay 3-5 years ahead of the curve of everyone else by not open sourcing their impressive libraries and infrastructure. But the corollary of that is that there's no place to whine "we got there first!" when someone else produces an implementation of X.
But, Google really has no right to whine about this. They make an explicit tradeoff: they manage to stay 3-5 years ahead of the curve of everyone else by not open sourcing their impressive libraries and infrastructure. But the corollary of that is that there's no place to whine "we got there first!" when someone else produces an implementation of X.
Well producers are available in Dagger now https://google.github.io/dagger/producers.html
Yup, sure does look awfully familiar....
[deleted]