I'm curious what validations/checks you find most useful in your workflows? alint basically started with me taking a bunch of make/shell scripts I already had and combining them into one tool, with some added niceties, and much better performance.
I find manually reviewing every change simply does not scale past a certain point. You need to rely on AI reviews and deterministic static analysis tooling, tests, modeling/verification, etc. to do the much of the heavy lifting.
At the end of the day LLM output is not deterministic, and the burden shifts from writing code to reviewing it. It makes sense to cover the review surface with deterministic checks/validations as much as possible - unit test, linters, static analysis tools, e2e integration test etc. one gap I noticed when it comes to such tools is that they are generally language/stack specific and don't cover things like file/directory structure/naming, file relationships/references, use of potentially harmful Unicode [0], general file/config structure, permissions, broken symlinks, files which should not be committed but are, etc. , etc. I've been building a tool - a general repo shape/structure linter, with extensible config/rules - to cover these cases in a deterministic way [1] [2].
The all time overall stats says volume is ~52m, but the table of featured services adds up to < 11m, with one 'x402 Facilitator - Dexter ' being like 95% of that...
I have a 'node/container' abstraction at the infra/engine layer which is essentially either a cloud VM or a local podman container. The engine/infra layer can spin up more of these as needed. I have a relatively beefy dedicated machine for working with AI, which is where I do most of the testing.
I aggressively try to keep costs down so the workflow DSL I have supports configurable limits which can be set at the $, token, or time dimension , at task, workflow and agent/node levels, with some same defaults. I have a pipeline which keeps LLM API pricing data up-to-date, and I use AI to estimate total costs before runs and manually approve those.
I think we need to build open/cooperative products and services to replace anything that is run/owned by less-than-moral entities, while boycotting those entities as much as possible. We need better open, fair and secure methods/protocols/tools/platforms for cooperating/organizing as well.
I'm my case a workflow is basically an active/living graph of nodes/sub-tasks. One node can process a task (with all relevant context) and create multiple fan-out tasks, or it can add additional context/requirements and pass it along to another node. The message/task passing is all implemented as queue - nodes subscribe to messages/tasks addressed to them and execute them, producing more tasks (or zero new tasks). For each task there is a context and a parent task/context, as well as a key/value store of all tasks and their context. Each agent/node gets instructions injected into their prompts that tell them how to look up parents tasks/context as well as how to output new tasks.
There is also a feedback loop - a node can fail to process a task, and pass the reasoning/context for that back to the parent or another node - this might result in a new adjusted task replacing the failed task, or it might require human intervention.
Nice, yeah also I have planner nodes, review nodes and organizer nodes (organizers can mutate the graph/workflow, create new node types, etc.) Trying to automate the node type/role definitions and overall workflow definitions as much as possible.
I split my project into 3 layers - the engine/infra layer (handles task dispatch/queuing, spinning up node/agent containers, etc.), the 'brain' - basically a collection of different workflow models and related stuff (multiple different models for testing/exploration purposes), and the harness.
Yeah mainly deepseek, it performs near top pretty consistently in terms of price/output (with a basic quality measure). I would love to test with more models but that's not cost-realistic for me at the moment.
I'm working an an agentic graph-based workflow execution engine/framework. The concept of the harness is completely abstracted away/generified - a 'node/agent's is a harness (cc, codex, open code, pi, etc) + model (I test different model and harness combinations). I have a set of tasks from trivial to complex - a set workflows (a workflow is a set of initial nodes and their behaviour) is defined and each one is asked to perform each task (multiplied by each harness/model combination roughly). The workflow can include agents/nodes which are able to modify the workflow graph and create nodes. Other nodes can break down tasks and send subtasks to other nodes. Mostly experimental stage at this point. I'm exploring/tracking metrics such as total wall clock time to complete a task, total cost in tokens and $, among others. This gives me a decent amount of data/insight into the abilities/performance of different harness/agents/models for different tasks, and gives me a great testing/dogfooding of my own harness (which is one of the harnesses being tested, and as of now the most efficient one).
The main bottleneck at this point is the cost of all of the tokens in the fairly large test matrix of tasks, harnesses, models.
I hope to release/open source all of this stuff eventually.
Engineers, (and many other types of specialists/professionals) can use it to speed up their work and increase output (even increase quality in some cases if done right).
But it's not gonna make inherently inefficient, political, and corrupted internal/company processes any more efficient, it might actually multiply those existing inefficiencies.
A big part of the US VC operating model these days seems to be just rebuilding existing products with slight changes, then pushing all of their other startups to use that version of it. This is only going to accelerate with AI. Why pay some company you don't own to do thing for you, when you can just copy the company (maybe even improve it in some ways), seed it with with your large existing user base, then have it do the thing for you (while also generating profit from other customers and rapidly scaling in users and valuation itself).
The reality is most of what most tech startups are doing is not actually hard and has no moat. The moat is in getting users/customers - connections/marketing/sales - product quality also matters of course, but there are plenty hyperscaler unicorns who's product is dogshit and vice versa.
How it works is they build a huge virtual strawman which decrypts and reads all of the data for them then posts online about how NSA spying on people is literally impossible.
> Why would I want the data to be decrypted at each point and why would datacenters do that?
When we talk about data that is sent for processing to a 3p server (like anthropic in this case) the data obviously needs to be decrypted to be processed.
As to why data is decrypted at each point in a typical large backend system - because other than network routing there are presumably multiple services that need to receive and act on this data somehow - you're not just sending encrypted data around to random servers.
> there's absolutely no way for them to strip off my encryption.
You don't seem to understand that you have no control over the encryption or decryption done on the backends of cloud services you use. I don't know how to make it more simple and obvious at this point.
Again, the context here is Anthropic and sending your data to their (or any other big tech API). But even if we move away from this model and suppose you are running your own services on rented cloud VM - then it should be obvious that you don't have full control or even access to this VM... any actor with access can install or modify any software, install/modify EBPF, modified crypto libraries, etc. - you have absolutely no control or say over this.
> Things didn't magically get better.
Things didn't get better at all, they got much worse.
You have no control about where TLS is terminated when you're talking to a 3p cloud service (with services you don't control/run like cloud LLM APIs). You also have no control about what spyware is installed on/around VMs you rent (and there's a lot). Also when talking about encryption between servers within datacenters you seem to be missing that in order for such multi -stage/path encryption (separate certs/keys) to be possible the data first has to be decrypted at each point, not to mention every major US tech company generally cooperates with the NSA and gives them access to anything they request (including allowing the installation of dedicated hardware to intercept decrypted traffic as has been publicly exposed documented many times already).
Yours and others' claims that it's impossible and nonsensical is based on lack of understanding.
Yours and others' claims that things somehow got better after Snowden is just a completely baseless statement - if you actually looked into what happened post-Snowden - absolutely nothing was done to prevent NSA spying on any communications they want, in fact it got significantly worse.
Explain to me how you are going to encrypt your LLM API calls with your let's encrypt cert.
There are also multiple ways/places traffic you send to typical cloud/tech company is decrypted and can be intercepted. (Surprised I have to point this out to someone who 'actually works in security ' lol)
Not to mention US tech companies fully cooperate with the NSA in many cases and are aware of this going on.
I find manually reviewing every change simply does not scale past a certain point. You need to rely on AI reviews and deterministic static analysis tooling, tests, modeling/verification, etc. to do the much of the heavy lifting.