How do you keep AI-generated applications consistent as they evolve over time?
4 comments
Consistency in AI generated apps usually comes down to treating prompts + outputs like real software artifacts. What’s worked for us: versioned system prompts, strict schemas (JSON + validators), golden test cases, and regression evals on every change. We snapshot representative inputs/outputs and diff them in CI the same way you’d test APIs. Also important: keep model upgrades behind feature flags and roll out gradually.
Real example: in one LLM-powered support tool, a minor prompt tweak changed tone and broke downstream parsers. We fixed it by adding contract tests (expected fields + phrasing constraints) and running batch replays before deploy. Think of LLMs as nondeterministic services you need observability, evals, and guardrails, not just “better prompts.”
Real example: in one LLM-powered support tool, a minor prompt tweak changed tone and broke downstream parsers. We fixed it by adding contract tests (expected fields + phrasing constraints) and running batch replays before deploy. Think of LLMs as nondeterministic services you need observability, evals, and guardrails, not just “better prompts.”
This makes a lot of sense — especially the “LLMs as nondeterministic services” framing. I agree that versioned prompts, schema validators, regression evals, and contract tests are essential if you're shipping LLM-powered systems.
What I’m wrestling with is a slightly different failure mode though. Even if: prompts are versioned outputs conform to JSON schema contract tests pass …you can still end up with semantic drift inside the application model itself.
Example: Entity field renamed Metric still references old semantic meaning Relationship cardinality changes Permission scope shifts All technically valid JSON. All passing structural validation.
But the meaning of the system changes in ways that silently break invariants.
So the question becomes: Should lifecycle control live only at the “LLM output quality” layer (prompts, evals, CI), or does the runtime itself need semantic version awareness and invariant enforcement?
In other words: Even if the AI is perfectly guarded, do we still need an application-level planner that understands entities, metrics, relationships, and permissions as first-class concepts — and refuses inconsistent evolution? I am definitively into this.
Curious how you think about that boundary.
What I’m wrestling with is a slightly different failure mode though. Even if: prompts are versioned outputs conform to JSON schema contract tests pass …you can still end up with semantic drift inside the application model itself.
Example: Entity field renamed Metric still references old semantic meaning Relationship cardinality changes Permission scope shifts All technically valid JSON. All passing structural validation.
But the meaning of the system changes in ways that silently break invariants.
So the question becomes: Should lifecycle control live only at the “LLM output quality” layer (prompts, evals, CI), or does the runtime itself need semantic version awareness and invariant enforcement?
In other words: Even if the AI is perfectly guarded, do we still need an application-level planner that understands entities, metrics, relationships, and permissions as first-class concepts — and refuses inconsistent evolution? I am definitively into this.
Curious how you think about that boundary.
The structural validation layer is worth separating from the semantic layer, even if the semantic problem is harder. If every AI-proposed change is validated against a JSON Schema before execution, and that validation produces a deterministic, signed result, you get an auditable record of what passed structural checks and when. That doesn't solve semantic drift (a field can be renamed and still pass validation) but it gives you a foundation to build semantic checks on top of. The key property is determinism: identical schema and payload should produce identical hashes and outcomes, regardless of which service runs the check. Without that, you end up debugging whether a failure is in your schema logic or in validator implementation differences across languages. Contract tests and regression evals work better when the structural layer underneath is provably consistent. One reference for this pattern: https://docs.rapidtools.dev/openapi.yaml
Totally agree. We treat structural validation as a separate, deterministic “compiler phase” before anything executes.
Draft (AI output) → normalize/canonicalize → link/type-check (cross-refs, relationship/FK semantics, datasets/metrics) → append-only migration ops → canonical DSL + stable hash.
This is also why we’re building it as a compiler pipeline rather than “prompting harder”: the output is an auditable artifact (canonical DSL + hash + migration checksum) that makes contract tests/regression diffs trustworthy.
Next step is what you call out: a signed attestation per compile (schemaHash + pinned validator/version/options + canonicalHash), so identical input yields identical outcomes across services/languages.
You’re right it doesn’t solve semantic drift alone (renames can still validate), but once the deterministic floor exists we can layer semantic checks + rename-safe migrations on top without debugging the toolchain itself.
Draft (AI output) → normalize/canonicalize → link/type-check (cross-refs, relationship/FK semantics, datasets/metrics) → append-only migration ops → canonical DSL + stable hash.
This is also why we’re building it as a compiler pipeline rather than “prompting harder”: the output is an auditable artifact (canonical DSL + hash + migration checksum) that makes contract tests/regression diffs trustworthy.
Next step is what you call out: a signed attestation per compile (schemaHash + pinned validator/version/options + canonicalHash), so identical input yields identical outcomes across services/languages.
You’re right it doesn’t solve semantic drift alone (renames can still validate), but once the deterministic floor exists we can layer semantic checks + rename-safe migrations on top without debugging the toolchain itself.
I’ve been experimenting with letting LLMs generate and then continuously modify small business applications (CRUD, dashboards, workflows). The first generation usually works — the problems start on the second or third iteration.
Some recurring failure modes I keep seeing: • schema drift that silently breaks dashboards • metrics changing meaning across iterations • UI components querying data in incompatible ways • AI fixing something locally while violating global invariants
What’s striking is that most AI app builders treat generation as a one-shot problem, while real applications are long-lived systems that need to evolve safely.
The direction I’m exploring is treating the application as a runtime model rather than generated code: • the app is defined by a structured, versioned JSON/DSL (entities, relationships, metrics, workflows) • every AI-proposed change is validated by the backend before execution • UI components bind to semantic concepts (metrics, datasets), not raw queries • AI proposes structure; the runtime enforces consistency
Conceptually this feels closer to how Kubernetes treats infrastructure, or how semantic layers work in analytics — but applied to full applications rather than reporting.
I’m curious: • Has anyone here explored similar patterns? • Are there established approaches to controlling AI-driven schema evolution? • Do you think semantic layers belong inside the application runtime, or should they remain analytics-only?
Not pitching anything — genuinely trying to understand how others are approaching AI + long-lived application state.
Thanks.