A practitioner’s honest take on where AI is genuinely changing how data pipelines are built, monitored, and fixed—and where it’s still just vendor marketing.
THE SITUATION
The noise arrived before the capability did
Something unusual is happening in data engineering right now. The hype curve for AI in data pipelines is running months ahead of the production reality. Every platform vendor has rebranded their product as agentic. Every conference talk promises pipelines that write themselves and heal themselves. The skepticism data engineers bring to this is earned — but so is the interest.
AI is genuinely changing how pipelines are built, how quality is enforced, and how failures are diagnosed. The capability is not uniformly mature — some of it is production-ready today, some is promising but fragile, and some is a demo that collapses the moment it touches real enterprise data. The skill — for a data engineering team in 2026 — is knowing which is which.
The hype curve for AI in data pipelines is running months ahead of the production reality. The skill is knowing which is which.
THE DISTINCTION
AI Tool vs. AI Agent — Where Is the Line?
An AI tool responds when you ask it to. An AI agent runs continuously, observes its environment, makes decisions, and acts—or routes an action for human approval—without being prompted.
dbt Copilot is an AI tool. Anomalo running on your tables at 3 a.m. and flagging an anomaly before you wake up is closer to agent behavior. Most of what is currently marketed as “AI in data engineering” is tooling. Some of it is genuinely agentic. Understanding the difference between the two determines how you evaluate these solutions and where you choose to invest.
WHAT’S REAL
Where AI is genuinely changing how data engineers work
There are three areas where AI assistance in data engineering has moved from interesting prototypes to solutions that teams are running in production. None are fully autonomous, but all are meaningfully useful.
1. LLM-Assisted Pipeline and Transformation Code Generation
The most immediate practical impact is code generation—the tedious, repetitive parts of pipeline work that follow patterns LLMs have seen thousands of times.
dbt Copilot generates SQL models, YAML tests, and column documentation from natural language. Databricks AI Assistant generates PySpark code and debugs failing notebooks. Snowflake Copilot scaffolds Tasks and stored procedures in Snowsight. Microsoft Fabric Copilot covers notebooks, pipelines, and the SQL analytics workload. These are AI tools, not agents, but the productivity gain is real and measurable.
| What Presidio Observes in Client Engagements Code generation is most valuable when scaffolding is predictable and engineers can focus their time on architecture and logic. It is least reliable when business logic is complex or implicit. Generation is fast, but review by a senior engineer remains essential—and that gate is non-negotiable. |
2. Anomaly Detection Agents Running on Pipeline Outputs
This is the most mature and genuinely agentic capability in production today—AI that runs continuously without being prompted, flags deviations from learned behavior, and acts before a human has noticed.
Anomalo and Monte Carlo operate continuously on your data without relying on static thresholds, automatically flagging deviations from learned normal behavior. Databricks Lakewatch, within Unity Catalog, brings this same monitoring natively to Delta Lake tables. Snowflake Trust Center and native data quality metrics provide continuous visibility within the warehouse. Microsoft Fabric Data Activator detects conditions in your data and triggers actions automatically—one of the most genuinely agent-like native features in any cloud data platform today. Soda Cloud complements this layer: you describe what should be true in plain English, and it compiles a production-grade quality check without requiring any YAML.
3. Self-Healing Pipeline Patterns: Partial, Not Full
Automatic retry logic for transient failures is real and already in production across Airflow and Prefect. This is the behavior most teams think of first when they hear “self-healing,” but it is not AI. The AI-enabled version goes further: a LangChain-powered agent connected to Airflow task logs, Snowflake Cortex AI, or Fabric’s monitoring events reads a failure, queries lineage and documentation, identifies the probable cause, and proposes a specific remediation—such as fixing the downstream model or alerting the upstream table owner. This pattern is in early production for narrow, well-defined use cases, but it is not yet a general-purpose capability you can deploy across a complex data estate.
WHAT’S HYPE?
Where the promise still exceeds the production reality
Fully autonomous self-healing—rewriting logic errors without human review
Auto-retry on transient failures is real. However, an agent that detects a logic error, rewrites the SQL, and deploys to production without human review does not exist today. The failure modes in real pipelines are too context-dependent and too consequential. A wrong number fixed by an agent is still a wrong number—and even harder to trace.
For straightforward, pattern-based transformations, LLM-assisted migration from SSIS or Informatica to ADF or dbt works well. But for the bespoke logic and undocumented lookups that make legacy pipelines truly legacy, migration still requires a data engineer who understands both the source and the target.
LLMs’ Reliability in Understanding Complex Business Logic
LLMs are reliable at recognizing structural patterns, but they struggle to understand complex business logic. They cannot grasp the implicit, undocumented business rules that govern most enterprise data—such as what “active customer” means in your organization or whether a contract closes at signature or payment. These definitions often reside in the minds of long-tenured employees or, occasionally, in a comment inside a stored procedure that hasn’t been touched since 2017. This is why agent-generated fixes always require human approval before reaching production.
AI performs reliably on the structural and predictable parts of a pipeline but quickly degrades when faced with business semantic complexity—where most of the real challenges in data engineering are in the semantics.
THE SHIFT THAT MADE THIS POSSIBLE
Why the move from ETL to ELT is the foundation of AI-native data engineering
The shift from ETL to ELT did not start with AI. It started because the economics of data infrastructure changed fundamentally — and AI is now the latest force amplifying a shift that was already well underway. Understanding why matters, because it changes the argument for the modern stack from ‘it is newer’ to ‘it was right for multiple reasons, and now it is right for one more.’
Why the Shift Happened—Before AI Entered the Picture
The original justification for ETL was practical: warehouses were expensive and slow, so data was transformed before loading to reduce the compute burden. When Snowflake, BigQuery, and Redshift arrived with elastic, massively parallel compute billed by the second, that justification collapsed—it became faster and cheaper to load raw data first and transform it inside the warehouse. Cloud object storage made it dramatically less expensive to store all raw data compared to on-prem alternatives, removing cost as a meaningful constraint.
SQL-first frameworks like dbt made transformation a software engineering discipline—version-controlled, tested, and documented—bringing it within reach of any analyst who knew SQL. Teams locked into Informatica or Talend, where logic was trapped in proprietary formats, actively sought open alternatives. The shift was economic, architectural, and cultural before it was anything else.
Why AI Makes the ELT Stack the Only Defensible Choice
Code generation is where the argument crystallizes. A dbt model is a SQL SELECT statement in a plain text file, version-controlled in Git. A Snowflake Task is a SQL statement scheduled inside the warehouse. These are open, text-based, and inspectable—formats an LLM was trained on and can generate, review, and reason about.
Informatica and Talend support SQL within their transformations, but that SQL is locked inside proprietary mapping files that only work within the vendor’s own software. You cannot extract the logic as a plain file, paste it into an LLM, or ask an AI to generate one from scratch. The SQL exists, but it’s not what you’re working with. You work with the vendor’s proprietary artifact, which is opaque to everything outside the vendor’s own runtime.
The teams that moved to ELT for economic and architectural reasons are now also the teams whose pipelines AI can generate, monitor, and help fix. That convergence is no coincidence—it’s the compounding advantage of having chosen open, code-first tooling.
The Zero-Ops direction
The logical endpoint is Zero-Ops data pipelines—where the operational overhead of managing, monitoring, and fixing pipelines is handled almost entirely by AI agents, with humans governing decisions rather than executing them. Snowflake’s native Streams and Tasks, Databricks Lakeflow Pipelines, and Microsoft Fabric’s Data Activator are all moving in this direction: platforms where pipeline logic is AI-readable, monitoring is continuous, and alerting is automated.
The proprietary ETL layer sits outside this ecosystem by design. Teams still running Informatica or Talend are not just carrying legacy technical debt—they are managing a stack that is structurally incompatible with the future of data engineering.
WHAT’S COMING
The emerging agent loop—observe, diagnose, propose, approve, act
The next meaningful shift is toward agent architectures that coordinate multiple AI capabilities in a continuous loop: observe, diagnose, propose, approve, and act. This isn’t just one AI tool helping an engineer work faster—it’s a system running continuously, with humans governing decisions rather than initiating every step.
Early examples are running today. On Snowflake, Cortex AI provides the LLM layer natively within the warehouse. On Databricks, Lakewatch detects anomalies, Unity Catalog provides lineage context, and a LangChain agent coordinates the response. On Microsoft Fabric, Data Activator detects triggers and initiates workflows—with human approval before any changes execute.
What the agent removes is the mechanical coordination overhead: reading logs, querying lineage, deciding who to notify, and writing up context. That work can take hours today; the agent does it in seconds. The engineer still approves before anything reaches production—and the loop only works if lineage is tracked, documentation is current, and ownership metadata is in place. Teams investing in OpenLineage, dbt docs, and data contracts now are building the infrastructure these agents will run on.
HOW TO THINK ABOUT ADOPTION
A practical framework for data engineering teams today
At Presidio, we advise clients to approach AI adoption in data engineering across three layers—not as a sequence, but as simultaneous bets at different maturity levels:
Layer 1 — Augment (ready now)
Embed AI assistance into the tools engineers already use, such as dbt Copilot or GitHub Copilot for model scaffolding, Databricks AI Assistant for notebook development, Snowflake Copilot for SQL and task generation in Snowsight, Microsoft Fabric Copilot for pipeline and notebook scaffolding, and Soda Cloud or Anomalo for continuous anomaly detection and quality check authoring. No architectural change is required, and output is always reviewed before production.
Layer 2 — Instrument (the prerequisite layer)
Track lineage via OpenLineage or dbt docs. Enforce data contracts using dbt or Soda. Attach ownership metadata to every table and model. On Snowflake, leverage the Trust Center and native quality metrics; on Databricks, use Unity Catalog and Lakewatch; on Fabric, utilize Monitoring Hub and Data Activator. The signal to move to Layer 3 is not a date on a calendar, but when lineage is tracked, contracts are enforced, and ownership is assigned.
Layer 3 — Automate (when the foundation is ready)
Once Layer 2 is in place, evaluate agent tooling vendors, run pilots in non-critical pipelines, and build the organizational muscle for reviewing and approving AI-generated changes before production. Start narrow—with one well-understood pipeline, one clear failure mode, and one defined approval workflow. Expand as trust is established through demonstrated reliability.
| The Presidio Principle on AI in Data Engineering AI streamlines the workload, but engineers make the decisions. The architecture, business logic, data contract definitions, and final sign-off before any production change should remain with accountable people. Autonomy grows as trust is built through proven reliability—not as a given from the start. |
THE POINT
The question is not whether AI is changing data engineering, it already has. The real question is whether your team is on the right side of that change.
The work that AI cannot replace is precisely what makes data engineering valuable: understanding what a business truly means by its data, making architectural decisions with long-term consequences, defining the contracts that govern how data moves through a system, and deciding whether an AI-generated proposal is appropriate for this pipeline, in this context, with these upstream dependencies.
The data engineering teams that will lead in the next few years are not those waiting for the tooling to stabilize. They are the ones already using code generation to reduce time spent on boilerplate, already running anomaly detection agents on their pipelines, already building on open, code-first ELT stacks that AI can read, generate, and reason about, and already instrumenting lineage and contracts so the next generation of agent tooling has something reliable to work with.
The real question for data engineering teams in 2026 isn’t whether to use AI tooling, but whether their approach is structured enough to maintain control over AI’s actions and honest enough to recognize where AI is not yet sufficient.
The infrastructure you implement today will be the foundation for the intelligence that your agents run on tomorrow.
About This Point of View
This point of view comes from Presidio’s Data & Analytics practice. We help organizations evolve from being dashboard-ready to agent-ready across Snowflake, Databricks, and Microsoft Fabric. Before committing to any AI outcome, we thoroughly assess the maturity of each domain. If this perspective resonates with you or if you see things differently, we welcome the conversation.
