Why isn't everyone using code workflow platforms when creating new applications?
16 comments
These were hot stuff years back in CRM/ERP land - there’s a genre name which I can’t recall. Tibco Businessworks was one. Back in the day it was “drag and drop” coding. And it was marketed heavily to non-tech managers.
The problem I saw was the same problem that (almost) every framework everywhere has. It makes expected use cases easy, small variations difficult, and unexpected use cases impossible. The escape hatch was generally via custom code that used some horrible DSL evolved from a config file.
The problem I saw was the same problem that (almost) every framework everywhere has. It makes expected use cases easy, small variations difficult, and unexpected use cases impossible. The escape hatch was generally via custom code that used some horrible DSL evolved from a config file.
(This is from when I last evaluated Cadence - which is now temporal.io. The state must have changed since then.)
Workflows are not zero-cost, they have their own tradeoffs compared to microservices. State management / bootstrapping logic becomes non-trivial, execution order though easier to visualize is also slightly not deterministic, workflows are not as well suited for request-response style replies due to the latency involved in total execution etc (but I think they are great alternatives to async / background workfllows) - and shared underlying infrastructure means increased chances of SPOFs.
The state must have improved much since then. Also, adoption of anything new to require remodelling your application into a different paradigm must be worth the value delivered. For example, modular monoliths became popular because they reduced operational complexity by reducing # of pieces involved. At the time, that value prop vs effort involved was unclear to our teams IMO
Workflows are not zero-cost, they have their own tradeoffs compared to microservices. State management / bootstrapping logic becomes non-trivial, execution order though easier to visualize is also slightly not deterministic, workflows are not as well suited for request-response style replies due to the latency involved in total execution etc (but I think they are great alternatives to async / background workfllows) - and shared underlying infrastructure means increased chances of SPOFs.
The state must have improved much since then. Also, adoption of anything new to require remodelling your application into a different paradigm must be worth the value delivered. For example, modular monoliths became popular because they reduced operational complexity by reducing # of pieces involved. At the time, that value prop vs effort involved was unclear to our teams IMO
Temporal is designed to handle lower latency use cases than data pipeline systems like Airflow. It also has added a feature recently called Update designed for request-response style interactions that allows for communication with Workflows on the order of tens of ms.
There is a lot of knowledge / community support around microservices already so people find it hard to gravitate automatically towards workflows (same didn’t exist for those. Same thing as GraphQL vs REST)
I think the biggest issue with workflow tools vs code is that a visual representation of a software system doesn’t made it any less complicated, and often the workflow system makes it even worse. I say this while building AWS Step Functions (another workflow system) into multiple applications, so it’s not really “off the cuff”.
I use Step Functions over Lambda for one big reason: no dependencies or runtimes to update. Secondary reasons include the lack of “cold start” and the unique capabilities. However, they have huge downsides and one of them is the complexity of building anything that isn’t choreographing calls to AWS services. It can get crazy ugly, and would quickly if it was the only tool I had.
Used in combination with imperative code, declarative workflow is a gift. On its own, a nightmare.
I use Step Functions over Lambda for one big reason: no dependencies or runtimes to update. Secondary reasons include the lack of “cold start” and the unique capabilities. However, they have huge downsides and one of them is the complexity of building anything that isn’t choreographing calls to AWS services. It can get crazy ugly, and would quickly if it was the only tool I had.
Used in combination with imperative code, declarative workflow is a gift. On its own, a nightmare.
How does Step Functions avoid the cold start problem? Isn’t it just executing on Lambda?
No, StepFunctions is a different beast. I didn't build it, so I'm just guessing but I suspect it's running a shared backend for all state machines so there is no static initialization, etc. to speak of.
Don't be confused that you _can_ call Lambada from StepFunctions, but that's a different beast (and a bit of an anti-patter IMHO).
Don't be confused that you _can_ call Lambada from StepFunctions, but that's a different beast (and a bit of an anti-patter IMHO).
In my experience you have two ends of a spectrum; the “build everything yourself and design everything exactly as you need it”, versus hooking various third party SaaS, PaaS, IaaS and/or FaaS together like Lego.
Both have advantages and disadvantages, but in the case you’re talking about the compromise is that you end up being constrained by someone else’s design goals, concepts and principles.
When you use multiple SaaS systems together the constraints are cumulative.
Hence, you are forced to constrain the constraints and the analogy “jack of all trades, master of none”, springs to mind.
One of the many SaaS offerings I regularly have to work with is for customers leveraging Microsoft D365 CRM. It classically demonstrates this point. There are two types of people who “develop” using it. The first are integrators that have a more technical background writing code, and locoders who sit in enterprises as a quasi business analyst / click-and-drag developer. The solutions they develop work, but often badly through horrific spaghetti designs, because they are forced to workaround the constraints of the locode system.
As long as those trade-offs are understood by CTO’s making these decisions then it’s fine, however many don’t grok it at all.
One thing many of these systems fail at is observability. Connecting your own code up to DataDog or Prometheus is easy, not often the case for a SaaS or third party system. Although in the case of Temporal, that is somewhat available in your own activities and workflows.
Both have advantages and disadvantages, but in the case you’re talking about the compromise is that you end up being constrained by someone else’s design goals, concepts and principles.
When you use multiple SaaS systems together the constraints are cumulative.
Hence, you are forced to constrain the constraints and the analogy “jack of all trades, master of none”, springs to mind.
One of the many SaaS offerings I regularly have to work with is for customers leveraging Microsoft D365 CRM. It classically demonstrates this point. There are two types of people who “develop” using it. The first are integrators that have a more technical background writing code, and locoders who sit in enterprises as a quasi business analyst / click-and-drag developer. The solutions they develop work, but often badly through horrific spaghetti designs, because they are forced to workaround the constraints of the locode system.
As long as those trade-offs are understood by CTO’s making these decisions then it’s fine, however many don’t grok it at all.
One thing many of these systems fail at is observability. Connecting your own code up to DataDog or Prometheus is easy, not often the case for a SaaS or third party system. Although in the case of Temporal, that is somewhat available in your own activities and workflows.
Happy to see this question. Working with this kind of WMS, BPM, integration plattforms (Kind of EAI, API, BPM) for 15 years now.
There is the term "citizen integrator/developer". The business guys who like computers. 1: most of the time real-world problems are to complex for them to solve as it needs more software engineering skills. 2: Low/No Code is a nice argument to sell such products. In the end techies do develop with it and have a hard time working around the supplied "framework" 3: In some cases it does save time, on the other hand you lose the ability to solve problems as you are going proprietary with this platforms and that makes you dependent.
What could work is if you configure the "business view" workflow in no code and do the implementation in a casual programming language, but this isn't a requirement very often, and there are not so many platforms as you can't lock-in customers that way.
Do you guys know any open WMS/BPM, integration platforms from your own experience that you would recommend? Thanks.
There is the term "citizen integrator/developer". The business guys who like computers. 1: most of the time real-world problems are to complex for them to solve as it needs more software engineering skills. 2: Low/No Code is a nice argument to sell such products. In the end techies do develop with it and have a hard time working around the supplied "framework" 3: In some cases it does save time, on the other hand you lose the ability to solve problems as you are going proprietary with this platforms and that makes you dependent.
What could work is if you configure the "business view" workflow in no code and do the implementation in a casual programming language, but this isn't a requirement very often, and there are not so many platforms as you can't lock-in customers that way.
Do you guys know any open WMS/BPM, integration platforms from your own experience that you would recommend? Thanks.
What is a code workflow platform? Is that something I need to think about before it solves a problem I have observed?
The whole point of micro-service architectures is that everything talks over very standardized interfaces with very clear boundaries so you can insert a wide variety of instrumentation. That instrumentation can even be added after the fact and progressively enhanced when chasing problems instead of prematurely optimizing.
The whole point of micro-service architectures is that everything talks over very standardized interfaces with very clear boundaries so you can insert a wide variety of instrumentation. That instrumentation can even be added after the fact and progressively enhanced when chasing problems instead of prematurely optimizing.
That is a benefit of micro-services, but I would argue the point of microservices is to emulate the organizational structure of the company for better flexibility and interoperability. It solves problems big organizations find working collaboratively on large monoliths.
I point it out because I feel this is a common enough misunderstanding, microservices isn't a technical solution it's just an exercise in organization.
To this point, so are workflows. Workflows better emulate business logic and business data flows, depending on how much of that you have to deal with, worflows might be a good way to organize that core requirement.
I point it out because I feel this is a common enough misunderstanding, microservices isn't a technical solution it's just an exercise in organization.
To this point, so are workflows. Workflows better emulate business logic and business data flows, depending on how much of that you have to deal with, worflows might be a good way to organize that core requirement.
In airflow your code has to be a DAG. Not all code can be DAGs. But I agree it can be nice. It is much less computationally efficient though with all the task overhead, scheduler, etc.
Technical debt accumulation and long term maintenance story.
I guess because how can you prove that this workflow chart is 1:1 with the software
I feel like you are conflating things here a bit. In my experience a WMS is typically used for tasks outside the codebase. Normally you would leverage a CI/CD pipeline (Jenkins or similar) to handle the building, testing, and deployment of a project or repository.
You would then leverage a WMS to create sequential tasks to maybe: trigger an app deployment, send notifications to developers and stakeholders, update a database, send out SMS messages to customers, etc... although many ci/cd services offer these capabilities as well, WMSs are more comprehensive in what they can do.
But if you don't need all of those extra bells and whistles, you might actually add more complexity going to a WMS.
For help with a complex codebase, i like to adhere to a lot of principles in the 12 factor app https://12factor.net/
...Although thats easier said than done to "clean up" a codebase
You would then leverage a WMS to create sequential tasks to maybe: trigger an app deployment, send notifications to developers and stakeholders, update a database, send out SMS messages to customers, etc... although many ci/cd services offer these capabilities as well, WMSs are more comprehensive in what they can do.
But if you don't need all of those extra bells and whistles, you might actually add more complexity going to a WMS.
For help with a complex codebase, i like to adhere to a lot of principles in the 12 factor app https://12factor.net/
...Although thats easier said than done to "clean up" a codebase
You mean like kanban boards? If you do, almost every place I consult plans sprints with them.
In my view, a great alternative would be the use of workflow tools, which have a graphical pipeline where teams can follow each step of the process along with its respective input and output, in a visual interface.
I see this type of design as an evolution of how software is developed, but I don't see a wide adoption by companies/startups when they are starting to develop their applications. What I see is the same old recipe being repeated infinitely, new project = microservices + cloud.
Note: to give examples of such tools, I will mention two that I found on the awesome-workflow-engines list on GitHub. - Apache Airflow - temporal(dot)io