The Next Decade Of Data Engineering: Agentic Systems And GenAI Pipelines

This article discusses the future of data engineering, highlighting the transition to agentic systems and GenAI pipelines. Sohag Maitra shares insights on overcoming data fragmentation and embracing modern architectures for enhanced analytics.

Interviewer: Joining us today is Sohag Maitra, Senior Data Analytics Engineer at Rabobank with over 15 years spanning product management, software engineering, and data innovation. She is a published researcher, co-author, a patent holder, and a respected reviewer advancing trustworthy data engineering practices. Welcome, Sohag. Could you take us through your journey and what drew you to data engineering in particular?

Sohag: Thanks so much for having me. You know, my journey into data engineering wasn't exactly a straight line. I've been doing this for over 15 years now, and what I love most is how it sits right at the intersection of strategy and getting your hands dirty with code. I've worked across product management, software development, and innovation—basically anywhere data touches business decisions. What's really kept me engaged all these years is watching how we've gone from these massive, monolithic systems that took forever to change, to the agile, cloud-based architectures we have today. It's been quite a ride, honestly.

Interviewer: That's fascinating. You've also co-authored a book, "Advanced Data Engineering Architectures for Unified Intelligence." What gap were you and your co-author trying to fill that wasn’t already covered by existing books or vendor documentation?

Sohag: Sure! So, my co-author and I wrote this book because we kept seeing the same struggles across different organizations. Everyone's trying to figure out how to get from their old SAP systems or legacy databases to modern platforms like Snowflake, but there's not a lot of practical guidance out there for senior engineers and architects. We wanted to create something that tells the real story not just the glossy vendor pitches, but the actual journey of how enterprise data infrastructure has evolved. We cover everything from those rigid on-premises databases through Amazon Redshift and into what we call the modern data stack. But we also look ahead to where things are going with AI and these emerging agentic systems. The goal was to write the book we wished we'd had when we were navigating these transitions ourselves.

Interviewer: In your experience working with large enterprises, what do you see as the most common roadblock preventing organizations from fully leveraging AI and advanced analytics today?

Sohag: Oh, there are plenty! But if I had to pick the biggest one, it's data fragmentation. I see this all the time companies have data scattered everywhere, and it's killing their ability to do anything meaningful with AI or analytics. There's this stat that ML teams spend about 80% of their time just wrangling data and building features instead of innovating. That's crazy when you think about it. The other big thing is that moving to the cloud sounds great on paper, but it's complicated in practice. You've got to think about governance, data quality, security across these distributed systems. The flexibility and speed you get from modern architectures is amazing, but it also means you need to fundamentally rethink how you approach data architecture. It's not just a lift-and-shift situation.

Interviewer: Sohag, you've been involved in cutting-edge research throughout your career. Could you elaborate on one of your significant contributions and how it has impacted the broader field?

Sohag: Yeah, one project I'm proud of is the work I presented at ML Con in New York about enterprise feature stores. The talk was called "From Data Silos to AI Excellence: Building Enterprise Feature Stores on Unified Foundations." Here's the thing I kept seeing ML teams essentially reinventing the wheel every single time they started a new project. They'd spend months building features that someone else had already built six months ago in a different department. It was incredibly inefficient. So, I demonstrated how you can use unified data architectures with tools like Delta Lake, Apache Iceberg, Databricks, and Feast to basically industrialize feature engineering. We showed real numbers of 70% reduction in time to deploy models, 3x improvement in feature reuse. These aren't just theoretical gains; they translate directly to faster time-to-market and better business outcomes.

Interviewer: You’ve started talking about “intelligent, self-managing data systems” and agentic architectures. How close are we to that becoming mainstream in enterprise settings, and what needs to happen first?

Sohag: I think we're moving toward something exciting, what I call intelligent, self-managing data systems. Right now, data engineering requires a lot of manual orchestration and maintenance. But increasingly, we're seeing systems that can manage themselves, optimize their own performance, and even make decisions about how to handle data. It's this idea of agentic systems. The modern data stack isn't just going to be a place where you store and process data; it's becoming this intelligent platform that understands context, enforces policies automatically, and adapts to changing needs without someone having to reconfigure everything manually. And with large language models, we're seeing people interact with data in completely new ways asking questions in plain English and getting meaningful answers without needing to write SQL. That democratization is huge.

Interviewer: Looking ahead three to five years, which technologies or architectural patterns do you believe will have the biggest impact on how enterprises build and manage data platforms?

Sohag: There are a few that I'm really watching closely. Lakehouse architectures are becoming the standard; they give you the flexibility of a data lake with the governance of a data warehouse, which is exactly what most organizations need. Delta Lake, Iceberg, Apache Hudi these are the technologies making that happen. Real-time streaming is another big one. We're getting to a point where real-time AI is actually feasible at scale. And then there's generative AI, which I think will change data engineering itself. Imagine having AI that can help you design pipelines, monitor data quality, even suggest architectural improvements. We're not quite there yet, but we're getting close. Oh, and privacy-preserving AI techniques like federated learning that's going to be crucial for handling sensitive data responsibly.

Interviewer: You've managed to publish multiple scholarly articles, co-author a book, and speak at conferences while working full-time at Rabobank. How do you make time for it all, and what advice would you give others trying to do the same?

Sohag: Honestly? Coffee helps! But seriously, it's about passion and finding the synergies. I'm genuinely fascinated by this stuff, so working on research or writing doesn't feel like a burden. It's energizing. I'm disciplined about time management, blocking out specific times for writing or research. But the real key is that my professional work and my research interests feed into each other. Things I learn at Rabobank inform my writing, and the research I do makes me better at my day job. Also, having a great co-author made a huge difference. We brought different strengths to the table, which made the book better and the process more manageable. You can't do everything alone, and you shouldn't try to.

Interviewer: You've published research on practical applications of machine learning and generative AI. What motivated you to focus on these areas?

Sohag: I've always been more interested in applied work than pure theory. Don't get me wrong, theoretical research is important, but I get excited when I can see how something helps a business or solves a real problem. Like my work on using large language models for smart retail shopping assistants that's about making the shopping experience better for actual people. The research I did on machine learning frameworks for stock trading shows how you can combine technical indicators with economic data to make better investment decisions. And the generative AI work I've done around product design in cloud-native platforms, especially for EVs and automation that's about accelerating real innovation in industries that are transforming right now. I always ask myself: how can we take these powerful AI capabilities and make them accessible and useful in the real world?

Interviewer: Why do you believe modern data engineering has become such a critical competitive advantage for organizations and even entire economies in today’s environment?

Sohag: Data is basically the foundation of everything now. If you can't collect, process, and learn from your data faster than your competitors, you're going to fall behind. It's that simple. Modern data engineering gives American companies the ability to innovate quickly. Think about it: better customer experiences, optimized supply chains, breakthrough AI applications all that run on good data infrastructure. And it's not just about business. In healthcare, good data architectures mean better patient outcomes. In finance, it means more stability and better risk management. In national security, it's about making smarter, faster decisions. To stay competitive globally, we need to keep investing in data infrastructure, support open-source technologies, and make sure we're training the next generation of data engineers. The countries that figure out unified, intelligent data architectures are going to lead in AI, and that's where the future is.

Interviewer: What policy decisions do you see as most critical to sustaining global leadership in data engineering and artificial intelligence?

Sohag: We need a multi-pronged approach. First, continued investment in R&D particularly around distributed systems, real-time processing, and AI-ready architectures. Second, stronger partnerships between companies, universities, and government. We need to get modern data practices adopted across all industries, not just tech companies. Education is huge too. We need to teach not just coding, but systems thinking, architectural design, and the ethical implications of AI and data. The open-source ecosystem is another critical piece—so much innovation happens in open source, and America should keep leading there. Apache projects have been game-changers, and we need to keep that momentum going. Finally, we need smart regulation that protects people's privacy and security without strangling innovation. It's a balancing act, but it's doable.

Interviewer: What advice would you give to aspiring data engineers and entrepreneurs in this space? What developments are you most excited about?

Sohag: For people just starting out in data engineering, I'd say build depth in the fundamentals Python, SQL, cloud platforms but also get a broad understanding of business and emerging technologies. Don't just chase the latest shiny tool; focus on solving real problems. Learn to think about systems, not just individual components. For entrepreneurs, look for places where better data architecture can unlock completely new business models. There's a lot of opportunity in embedded analytics, industry-specific data platforms, and AI-powered data operations. Personally, I'm excited about where generative AI and data engineering are converging. Imagine data pipelines that can basically design themselves, or governance systems that explain compliance requirements in plain language. That's coming. Edge computing for real-time processing is another area that's going to explode, especially for IoT and autonomous systems. The next few years are going to be wild, in a good way. If you stay curious and adaptable, there's never been a better time to be in this field.

Interviewer: Sohag, thank you this has been an incredibly insightful conversation.

Sohag: Thanks for having me! This was fun. I love talking about this stuff, and I'm optimistic about where data engineering is headed. The next generation of engineers is going to do amazing things.