Temporal is designed to handle lower latency use cases than data pipeline systems like Airflow. It also has added a feature recently called Update designed for request-response style interactions that allows for communication with Workflows on the order of tens of ms.
There are certainly use cases for which it's more than is required. Like the most simple would be adding a cron string to a GitHub Action or Vercel function, but in most cases, and certainly Slack's case, you want more reliability, scalability, flexibility, and/or observability. And Temporal has extreme levels of those things, including pausing & editing schedules and seeing the current status of all triggered scripts/functions and all the steps a function has taken so far, and guaranteeing the triggered function completes, including ensuring that if the process or container dies, the function continues running on another one. Even if you don't care about all those things, you might care about some of them in the future, and it doesn't hurt to run a system that has capabilities you don't use.
Using Temporal is in the category of building it yourselves, not a billing service. But it makes it take much less time to build, because it makes the changes, scaling, and grandfathering quick to do. Especially with the new Scheduled workflows feature: https://github.com/temporalio/samples-typescript/tree/main/s...
tldr is Temporal is more general-purpose: for reliable programming in general, vs data pipelines. It supports many languages, and combining languages, has features like querying & signaling, and can do very high scale.
You do need to run it on the same code version. There are different ways deploy code changes. If you use one of our in-built versioning systems, the version is recorded in the workflow history (and you can either keep track of version-sha mapping or use a sha as the version). Otherwise, you can add code that adds the current code version as workflow metadata.
Author here, curious if there are any major TTD debuggers I missed? Also let me know if anything in the post didn't make sense, and I'll edit to clarify!
2PC tends to have limited throughput due to the participants needing to hold a lock between the voting and commit phase, and all the participants need to support the protocol. Sagas work across different services and data stores and can have high throughput.
However, if all of the data you need to update is in a single database that supports atomic commits, I'd go with that over sagas.
At some point, if you can't automatically fix something, you have to stop and report to a human for manual intervention/repair. While a saga doesn't guarantee that you avoid manual repair, it significantly reduces the need for it. If each of these has a 1% chance of non-retryable failure:
Step1
Step2
Step1Undo
then this has a 1% chance of needing manual repair (it's okay if step1 fails, but if step1 succeeds and step2 fails, we need to repair):
do Step1
do Step2
and this has a .01% chance (we only repair if Step2 and Step1Undo fails, 1% * 1%):
If I'm getting your point right, I agree! If you have the workflow / durable execution primitive to depend on (a durable function is guaranteed to complete executing), then there are a lot of pieces of distributed systems stacks that you no longer need to use. Your durable code is automatically backed by the event sourcing, timers, task queues, transfer queues, etc that Temporal internally uses to provide the guarantee, so that you don't need to build them yourself.
Sagas are for when you can't do an update in an ACID transaction, for example when updating state across different types of data stores.
If you're asking whether the catch clause in a Temporal Workflow saga is guaranteed to execute, the answer is yes. The way it's able to guarantee this is by persisting each step the code takes and recovering program state if a process crashes, server loses power, etc. For an explanation of how this works, see: https://temporal.io/blog/building-reliable-distributed-syste...
There are definitely ease of use benefits to more tailored solutions. If workflow definitions are really simple and don't change much, JSON might be easy. Most things I prefer the DX of writing the logic in code. And it wouldn't be 50 different codebases—it would usually be a single codebase with 50 different functions.
Temporal has different database options: Cassandra, Postgres, MySQL, SQLite.
> source data in CSV files; - Capillaries script (JSON file) that defines the workflow and the transforms; - Python code that performs complex calculations (only if needed).
Temporal is more general purpose: source data anywhere, and you write code to define workflows and transforms instead of JSON, and the code can be in Go/Java/Python/JS/TS/.NET
Your code is run in a worker process. If the worker fails, another worker will pick up where the first left off. The state of each workflow function is stored by the Temporal Cluster. The Cluster is a SPOF, but each component of it is fault tolerant and horizontally scalable, so it's a low-risk SPOF to have. In choreography, your message bus would be the SPOF, and Temporal provides much better DX than choreography.
The major differences between an event bus and a workflow engine is the workflow engine:
- Creates and updates events/messages for you
- Maintains consistency between the events and timers and the state of the processing flow. More info on why this is important for correctness/resilience: https://youtu.be/t524U9CixZ0
The difference between workflow code and using an event bus is that with the former, the above is done automatically for you, and in the latter, it's done manually, which can be a lot of code to write and get right, and is harder to track/visualize what happened in production and debug. It would also take a lot of events to get an equivalent degree of reliability—the message processor would need to do a single step and then write the next event to the bus. So a 10-line workflow-as-code function would translate to 10 different events in the bus route.
Also, the event bus route doesn't have the new possibilities I listed in the parent comment.
Traditional workflow systems are often used for specific types of business processes, particularly those that are async like human-in-the-loop.
Temporal, while it uses the "workflow" terminology, is a new type of thing. At a basic level, it's "do you want your backend code to run reliably?" If yes, and you're okay with the latency hit of each step getting persisted for you, then the answer is "use Temporal to write your backend code." It's a new programming model that lets you develop at a higher level of abstraction, where you don't have to be concerned about faults in the hardware or network or downstream services/3rd party APIs being temporarily down. Where you no longer have to code retries, timeouts, use task queues, or use a message bus to communicate between services. And oftentimes don't even need to use a database.
Temporal (and similar systems like Cadence, AWS SWF, Azure Durable Functions) allows you more expressiveness and better DX than defining DAGs in a UI or markup file. You can write (almost) arbitrary code, and the library translates the code's actions into workflow steps.
> The Visual nature of the flow, ability to visually diagnose suspended / failed executions.
Temporal has a web UI in which you can see which executions are failing, and see on which step they're failing:
Correct, the workflow's guarantee to always complete executing independent of process/hardware failures is dependent on the database not losing data. You host your workflow code with Temporal's Worker library, which talks to an instance of the Temporal Server [1], which is an open-source set of services (hosted by you or by Temporal Cloud), backed by Cassandra, MySQL, or Postgres. [2] So for instance increasing Cassandra's replication factor increases your resilience to disk failure.