Start Data Engineering

Hello Reader, AI can write code now. So is learning Python still worth it? If you've wondered that, or are finding learning Python increasingly demotivating, this one's for you. In this post, I break down where AI-written code falls short in data engineering, and how much Python you actually need to know. Do I Need to Learn Python for Data Engineering? -> P.S. If you found this post helpful, I’d really appreciate a share on your socials. Best, Joseph

Hello Reader, As data engineers, we are constantly bombarded with new AI terms such as skills, MCPs, agents, evals, etc. But it all comes down to 2 principles: Define & Refine. Define your workflow (i.e., how you’d approach a problem) as a Markdown file Refine it with additional information and make it more specific over time Learn: How to Use AI to Speed up Data Pipeline Development In this post, you will learn How to create skills specific to your workflow How to provide additional...

Hello Reader, There is a lot of uncertainty about our future as data engineers. Will AI take our jobs? How do we continue growing in our careers? The answer (as it always has been) is to focus on delivering easy-to-use and reliable data to end users. In this post, I explain what skills to focus on, the learning resources for each, and how to use AI to supplement your work (not replace you). The 6 Skills that AI can’t fully replace (click here) → Beyond that, I am planning on doing a free live...

Hello Reader, Too many small files are one of the most common performance killers in large-scale Spark pipelines. When a dataset is split into hundreds or thousands of tiny files, Spark spends more time opening files than processing data. N small files = 2*N Spark API calls In this week's deep dive, we see how to detect & fix the issue with working code. Identify & fix small files with Spark and Iceberg → You’ll come away knowing how to: Detect the problem using the Spark UI. Compact small...