profile

Start Data Engineering

The 2 Principles Behind AI-Assisted Pipeline Development


Hello Reader,

As data engineers, we are constantly bombarded with new AI terms such as skills, MCPs, agents, evals, etc. But it all comes down to 2 principles: Define & Refine.

Define your workflow (i.e., how you’d approach a problem) as a Markdown file
Refine it with additional information and make it more specific over time

Learn: How to Use AI to Speed up Data Pipeline Development

In this post, you will learn

  • How to create skills specific to your workflow
  • How to provide additional information about your data for better code design (with an MCP server)

Question for you: Do you feel unable to keep up with DE advances? If so, what is the main area (modeling, code design, AI dev, etc.) of your concern?

Hit reply and let me know. I read every email.

Best,

Joseph

Start Data Engineering

Over the last decade, I've built highly scalable distributed data platforms and helped companies scale to processing multiple exabytes of data. My mission is to bring software practices followed by top tech companies to data engineering and help data engineers level up. I help data engineers land high paying tech jobs and significantly up skill themselves.

Share this page