Start Data Engineering
Over the last decade, I've built highly scalable distributed data platforms and helped companies scale to processing multiple exabytes of data. My mission is to bring software practices followed by top tech companies to data engineering and help data engineers level up. I help data engineers land high paying tech jobs and significantly up skill themselves.
Free Workshop: Data Pipeline Design Patterns
Hello Reader, We work in a speed-obsessed industry. Pipelines are generated with AI super fast. But when they break, no one knows what happened or how to fix them. We spend days fixing pipelines instead of taking a few hours to design them right from the start. What if we could build reliable, easy-to-fix pipelines quickly? Most pipelines follow similar patterns; it's about choosing the right pattern for your use case. Join me in my Free Live Workshop on July 4th, 2026, 1 PM EST. We will go...
6 Data Engineering Skills To Progress in the Age of AI
Hello Reader, There is a lot of uncertainty about our future as data engineers. Will AI take our jobs? How do we continue growing in our careers? The answer (as it always has been) is to focus on delivering easy-to-use and reliable data to end users. In this post, I explain what skills to focus on, the learning resources for each, and how to use AI to supplement your work (not replace you). The 6 Skills that AI can’t fully replace (click here) → Beyond that, I am planning on doing a free live...
How to Transition From A Data Analyst to Data Engineering
Hello Reader, Are you a data analyst (or similar) struggling to break into DE? Are you tired of doing multiple courses that don’t translate to real-world experience? Then this post is for you. Build data projects at work starting today. Here’s how to build a data project at work → Read this and join me for a live YouTube workshop on Saturday, June 20th at 1 PM EST (10 AM PST). I’ll cover the article in-depth (with open Q&A): Building DE projects at your current job Strategies for internal...
3 Design Decisions for Maintainable Incremental Pipelines
Hello Reader, Incremental pipelines are notoriously difficult to get right. When they fail, they often require painful manual intervention. But with the right design decisions, they’ll just chug along. Read the 3 design decisions → You’ll learn: How your source’s timestamp columns dictate extract design Load patterns depending on the type of output (fact, dimension, etc.) Scaling strategies for backfilling Was this helpful? If yes, share it with a colleague who is designing incremental...
How to manage data infrastructure with Terraform (not notebook scripts)
Hello Reader, AI can write Terraform code, but without the fundamentals, you are building a house of cards. Terraform (IaC) is basically git for infrastructure. In this post, we will go over what IaC is, Terraform fundamentals & setting up AWS infrastructure with Terraform Setting up AWS infrastructure with Terraform → In order to truly understand a concept, you need to be able to explain it plainly. In your own words, share your main takeaway from this post. Regards, Joseph Machado...
How to detect and fix the small files problem with Spark and Iceberg
Hello Reader, Too many small files are one of the most common performance killers in large-scale Spark pipelines. When a dataset is split into hundreds or thousands of tiny files, Spark spends more time opening files than processing data. N small files = 2*N Spark API calls In this week's deep dive, we see how to detect & fix the issue with working code. Identify & fix small files with Spark and Iceberg → You’ll come away knowing how to: Detect the problem using the Spark UI. Compact small...
How Companies Ingest Data: 2 Key Patterns
Hello Reader, Most companies ingest data in one of two ways. Stream data into a cloud store via an event log like Kafka. Extract data from source systems in batch. In this article 👇, we go over a high-level design of how they are built and their tradeoffs. > Data Ingestion Patterns Please reply to this email if you have any questions/thoughts/ideas and I will get back to you. Regards, Joseph Machado startdataengineering.com
Free Sample Chapter From The Data Engineering Course
Hello Reader, Over the past few weeks, we covered: 4 concepts to land a DE job 3 Data storage techniques for your warehouse Scheduling and orchestration in Airflow Being in demand as a data engineer by learning fundamentals and best practices The Data Engineering Course is where it all comes together. Learn how to build production-grade pipelines using industry-standard tools and best practices. Enrollment opens tomorrow, April 26th. Only 64 spots left. Newsletter subscribers get an early...
4 Steps To Becoming A Valuable Engineer
Hello Reader, Data engineering job requirements keep getting longer. And with LLMs generating code, it's getting harder to know which skills are actually worth your time. The engineers who stay in demand are the ones who know how to build pipelines that just chug along, no matter what's thrown at them. I made a cheatsheet that breaks down the fundamentals and best practices that set them apart. 👉 [Download it free here] In my Data Engineering Course launching April 26th, I cover the...
Free Live Airflow Webinar (April 11th, 1PM EST)
Hello Reader, Most high-paying data engineering jobs require a good understanding of pipeline orchestration. Understanding & practicing the problems that Airflow was built to solve will help you understand orchestrators in depth. I will guide you through these concepts live on Saturday, April 11th, at 1PM EST. → Join the live session here Code for the workshop: https://github.com/josephmachado/airflow-tutorial We will cover: Why Airflow exists and why not just cron? Creating pipelines with...
Over the last decade, I've built highly scalable distributed data platforms and helped companies scale to processing multiple exabytes of data. My mission is to bring software practices followed by top tech companies to data engineering and help data engineers level up. I help data engineers land high paying tech jobs and significantly up skill themselves.