Agentic Data Engineering: Pipelines That Run Themselves
Data pipelines have been static artifacts — designed once, scheduled on cron, and patched when they break. That era is ending. Agentic data engineering replaces rigid DAGs with intent-driven orchestration: an engineer describes what the data should achieve, and the system determines how to get there, rerouting around failures, optimizing resource allocation, and recovering from upstream changes without a human opening a ticket. Qlik, Snowflake, and startups like Definity are shipping these capabilities now. The architecture is real. The accountability gap it opens is not yet solved.
The Inflexible Pipeline Inheritance
Every data team inherits the same problem: pipelines designed for a specific source, a specific shape, and a specific schedule. When the source changes — a column renamed, a refresh interval shifted, a provider swapped — someone rewrites the job, redeploys, and monitors the result. The cycle repeats hundreds of times per month across an average enterprise.
Three generations of tooling compressed parts of this cycle without eliminating the core pattern.
The first generation — manual ETL with Informatica and DataStage — required a specialist for every pipeline change. The second — cloud-native orchestration with Airflow, dbt, and Spark — abstracted infrastructure but kept pipeline logic rigid. A DAG in Airflow does exactly what it was programmed to do, no more. When upstream conditions change, the DAG does not adapt; it fails or produces incorrect output until a human intervenes.
The third generation, now emerging, changes the control plane. Instead of specifying how to move and transform data, teams specify what the data should look like and when it matters. The system translates that intent into execution plans, adjusts them at runtime, and reports results.
The shift is from imperative coding to declarative intent — and from there, it is a short step to agent-directed orchestration where the system makes decisions no human explicitly programmed.
| Era | Control Model | What Breaks | Who Fixes It |
|---|---|---|---|
| Manual ETL | Imperative: step-by-step jobs | Schema changes, server failures | Specialist Engineer |
| Cloud-native orchestration | Declarative DAG: directed acyclic graph | Upstream drift, SLA breaches | Data Engineer on call |
| Agentic execution | Intent-driven: goal specification + agent routing | Goal misalignment, authority scope | Agent (with human escalation) |
What Agentic Execution Actually Ships
Qlik announced the most comprehensive agentic data engineering platform in April 2026 at its Connect conference, moving beyond coding assistants to what it calls "agentic execution" — runtime systems that take action, not just suggest code. The release includes several concrete pieces worth examining on their technical merits rather than marketing language.
The Data Product Agent accepts natural-language descriptions of what a data product should contain — dimensions, measures, relationships, freshness — and generates the pipeline scaffolding to deliver it. This is not a chatbot that writes SQL snippets. It produces full pipeline definitions that plug into Qlik Talend Studio, with lineage tracking, data contracts, and service-level expectations attached from creation.
The Trust Score is the operational signal that makes agentic execution governable. Each data product receives a composite score derived from accuracy, timeliness, diversity, and completeness signals.
Below the configured Trust Score, downstream agents halt or escalate rather than proceed with degraded data. This is the missing piece that prevented earlier automation attempts: a machine-consumable quality signal that agents can use without human interpretation.
Open Lakehouse Streaming, now generally available, provides the low-latency transport layer that agentic orchestration requires. When an agent reroutes a pipeline because an upstream source degraded, the routing happens in seconds, not the minutes-to-hours batch windows that Airflow schedules assume. Real-time routing also allows agents to prioritize freshness-sensitive workloads over batch-optimized ones without reconfiguring the entire DAG.
Snowflake takes a different architectural approach with the same end goal. Cortex Code, launched in early 2026 and expanded in April, operates as a governed agent across the entire data stack — not just within Snowflake. It now supports dbt, Airflow, Databricks, AWS Glue, and PostgreSQL, with a Model Context Protocol server that lets external agents query Snowflake schemas, lineage, and access policies.
Over 50% of Snowflake customers adopted Cortex Code within two months of launch. The key insight: agents that understand your entire stack can coordinate across tools that were never designed to talk to each other.
Definity, a startup that raised a $12 million Series A in April 2026, approaches from the operations side. Its in-motion architecture runs directly inside production Spark pipelines on Databricks, EMR, Dataproc, and Kubernetes — capturing full-stack signals across infrastructure, pipeline execution, and data characteristics without requiring code changes.
Definity's technical differentiator is operating within production pipelines rather than observing them from outside. Production customers have reduced platform costs by over 30% through job-level optimization and resolved complex Spark issues ten times faster.
The Accountability Gap
When a pipeline fails at 3 a.m. and an agent reroutes traffic through a backup provider, three questions follow: Was the reroute correct? Who approved it? And what happens if the backup provider introduces a subtle data quality issue that passes the agent's checks?
Traditional pipelines answer these questions through audit logs and change management tickets. Every modification to a production DAG goes through code review, deployment, and monitoring. The chain of custody is clear even when the response time is slow.
Agentic pipelines subvert this chain. An agent that reroutes, retries, or replaces a data source is making a production change — often a significant one — without a human approving it first. The benefits are real: faster recovery, fewer on-call pages, and reduced blast radius from upstream failures. But the accountability framework has not caught up.
| Decision | Traditional Pipeline | Agentic Pipeline | Accountability Question |
|---|---|---|---|
| Source change | Engineer submits PR, peer review, deploy | Agent detects upstream drift, selects alternative source | Was the alternative source equivalent? Who validated it? |
| SLA violation | Alert fires, engineer on call investigates | Agent escalates or degrades gracefully based on Trust Score | Is the degraded output acceptable? For how long? |
| Cost spike | Finance flags after invoice, engineer optimizes | Agent re-optimizes job configuration in real time | Did the optimization change output correctness? Where is the audit trail? |
| Schema drift | DAG fails, engineer fixes, redeploys | Agent adapts pipeline logic to new schema | Is the adaptation semantically correct, or just syntactically valid? |
The rows in this table share a pattern: the agent's action is technically competent but semantically uncertain. A reroute to an alternative source may preserve row counts and schema conformance while introducing distribution drift that corrupts downstream model inference. An optimization that reduces compute cost may reorder operations in ways that break idempotency guarantees the team relied on implicitly.
Qlik's Trust Score and data contracts represent one approach to closing this gap. Contracts define what "good enough" looks like in machine-readable terms — acceptable freshness windows, minimum completeness thresholds, permitted deviation ranges. When an agent operates within contract boundaries, its decisions are auditable against an explicit specification rather than an implicit understanding held by the team that built the original pipeline.
Snowflake's governance angle is architectural: Cortex Code operates within Snowflake's access control and audit framework, so every agent action inherits existing role-based permissions and row-level security policies. The agent cannot access data the engineer could not access. But governance of the agent's decisions — not just its access — requires additional controls that most organizations have not yet defined.
The agent cannot access data the engineer could not access. But can it make decisions the engineer would not have made? Access control governs who acts. It does not govern what they decide.
The Three-Phase Adoption Pattern
Agentic data engineering is not a binary switch from manual to autonomous. Organizations that try to skip phases either over-engineer for low-risk workloads or deploy autonomous agents without the observability foundation to make their decisions trustworthy. The pattern emerging across early adopters follows three phases.
Phase 1: Observability and Context (Already Shipping)
Before an agent can make decisions, it needs to understand the current state of the pipeline, the data, and the business context. This is the layer our earlier coverage of data observability addressed — the telemetry, lineage, and quality signals that make automated decisions possible.
Organizations that have not invested in universal collection and semantic analysis (OTel instrumentation, lineage metadata, freshness timestamps) lack the foundation for agentic execution. The agent will make decisions based on incomplete signals, and those decisions will be wrong in ways the team cannot diagnose because the observability layer was never built.
Snowflake's Cortex Code demonstrates this dependency: its effectiveness as a cross-stack agent depends on having schemas, lineage, and catalog metadata available across every connected system. Organizations with bare-bones metadata get shallow agent suggestions. Organizations with rich lineage get production-grade orchestration.
Phase 2: Assisted Execution (Current Leading Edge)
In assisted execution, the agent proposes actions and a human approves. This is where most of the current product announcements sit. Qlik's Data Product Agent generates pipeline definitions that an engineer reviews and deploys. Cortex Code writes dbt models and Airflow DAGs that a developer inspects before committing. Definity's optimization agents surface recommendations that an engineer accepts or rejects.
The value of assisted execution is not in the automation itself but in the speed of iteration. An engineer who would spend two days scaffolding a new data product can review and refine an agent-generated version in two hours. The bottleneck shifts from "how do I express this?" to "is this correct?"
Phase 3: Autonomous Orchestration (Emerging, With Guardrails)
Autonomous orchestration is where the agent takes action without human approval for routine decisions — rerouting around failures, optimizing resource allocation, adapting to schema changes within defined contract boundaries. This is the capability that makes the accountability gap urgent. Qlik's Automate Agent and Snowflake's full-stack Cortex Code execution point in this direction, but both require explicit contract and permission boundaries before enabling full autonomy.
The guardrail pattern that works draws from software circuit breakers: agents operate freely within a defined envelope, and the envelope is set conservatively. An agent can reroute to a pre-approved backup source. It cannot route to a source the team has never reviewed. It can retry a failed job up to three times. It cannot rewrite transformation logic on the fourth retry without escalation. The boundaries are explicit, versioned, and auditable.
Exceptions and Limits
Agentic data engineering does not apply everywhere, and premature adoption creates more problems than it solves.
- Small teams with few pipelines. Organizations with fewer than 15-20 active pipelines and two or three data engineers gain more from solid dbt tests and Airflow alerting than from agentic orchestration. The complexity of defining contracts, trust scores, and permission boundaries exceeds the maintenance burden of manual monitoring at this scale. Definity's own case studies come from organizations running hundreds of Spark jobs on lakehouse deployments — the cost and complexity profile that justifies agent-level automation.
- Highly regulated environments with immutable audit trails. Finance, healthcare, and government pipelines often have compliance requirements that mandate human approval for production changes. Agents can observe, recommend, and prepare changes, but the final approval step must remain human. In these environments, Phase 2 (assisted execution) is the ceiling, and the guardrails are not just technical — they are regulatory.
- Proprietary or undocumented data sources. Agents make decisions based on available metadata — schemas, lineage, quality signals. When a pipeline depends on a vendor feed with no schema documentation, no freshness guarantees, and no programmatic access to metadata, the agent has no reliable signal to act on. The first investment in these cases is observability and documentation, not automation.
- Vendor lock-in risk. Qlik's agentic capabilities operate within the Qlik ecosystem. Snowflake's Cortex Code is most effective within Snowflake. Definity operates inside Spark-based platforms. Each agent's decisions are only as portable as the contract and metadata formats it emits. Before committing, teams should verify that trust scores, data contracts, and agent decision logs export in open formats.
Honest Assessment
| Approach | Best For | Limitation |
|---|---|---|
| Manual pipeline management | Teams with <20 pipelines and strong on-call coverage | Does not scale; every change requires human intervention |
| Declarative orchestration (Airflow/dbt DAGs) | Teams with mature DAG libraries and clear separation of concerns | Rigid: breaks on upstream changes rather than adapting |
| Assisted agentic execution (Phase 2) | Teams scaling from 20 to 100+ pipelines that need faster iteration | Still requires human review; agent suggestions can be confidently wrong |
| Autonomous agentic orchestration (Phase 3) | High-throughput environments with well-documented contracts and observability | Accountability gap; requires explicit permission boundaries and audit trails |
Actionable Takeaways
- Audit your observability before adding agents. If you cannot answer "is this data fresh?" and "what depends on this table?" in machine-readable form, you are not ready for agentic execution. Invest in OpenTelemetry instrumentation, lineage metadata, and quality scores first. The agents that ship on top of observability are only as reliable as the signals they consume.
- Define data contracts before enabling autonomous action. Every agent decision needs a boundary. Contracts specify freshness windows, completeness thresholds, and permitted source alternatives. Without contracts, "the agent decided to reroute" is an explanation, not an audit. Start with your three most business-critical pipelines and define contracts for each. Then let the agent operate within those boundaries.
- Start with assisted execution and measure iteration speed. The clearest early ROI from agentic data engineering is not cost reduction — it is the time from intent to production. Measure how long it takes to scaffold, review, and deploy a new data product today. Then measure it with an agent generating the first version. If the ratio is not improving, the agent is not adding value.
- Prioritize cross-stack context over single-platform depth. Snowflake's Cortex Code and Qlik's agent suite are both expanding beyond their home platforms. The agent that understands your entire stack — Snowflake plus dbt plus Airflow — is more useful than the one that only optimizes within a single tool. When evaluating agentic platforms, weight cross-stack integration above native depth.
- Build escalation paths, not just automation. An agent that fails silently is worse than a pipeline that fails loudly. Every autonomous action needs a defined escalation path: what the agent does when it cannot make a confident decision, how it notifies the responsible team, and how quickly a human can intervene. Design the failure modes before you design the happy path.