Start Data Engineering

Hello Reader, Most companies ingest data in one of two ways. Stream data into a cloud store via an event log like Kafka. Extract data from source systems in batch. In this article 👇, we go over a high-level design of how they are built and their tradeoffs. > Data Ingestion Patterns Please reply to this email if you have any questions/thoughts/ideas and I will get back to you. Regards, Joseph Machado startdataengineering.com

Hello Reader, Data engineering job requirements keep getting longer. And with LLMs generating code, it's getting harder to know which skills are actually worth your time. The engineers who stay in demand are the ones who know how to build pipelines that just chug along, no matter what's thrown at them. I made a cheatsheet that breaks down the fundamentals and best practices that set them apart. 👉 [Download it free here] In my Data Engineering Course launching April 26th, I cover the...

Hello Reader, Most high-paying data engineering jobs require a good understanding of pipeline orchestration. Understanding & practicing the problems that Airflow was built to solve will help you understand orchestrators in depth. I will guide you through these concepts live on Saturday, April 11th, at 1PM EST. → Join the live session here Code for the workshop: https://github.com/josephmachado/airflow-tutorial We will cover: Why Airflow exists and why not just cron? Creating pipelines with...

Hello Reader, Are your data costs skyrocketing? Are your queries timing out? You know techniques like partitioning and bucketing exist — but knowing when to use which is where most engineers get stuck. What if you had an easy way to determine what technique to use to make your queries fast and cheap? This post gives you exactly that, a rule-of-thumb pattern for analytical tables that is useful in real life and in interviews. > 3 Data Storage Techniques Every Data Engineer Should Know If you...

Hello Reader, Data Engineering job requirements are insane. Every role now requires multiple tools and years of experience. It can feel overwhelming just thinking about it. But what if you can make potential employers excited to hire you? Companies are hiring for one thing: problem solvers. Here are the 4 concepts that map directly to the problems data teams face every day — and that interviewers actually test for. > 4 Concepts To Land A High-Paying Data Engineering Job Want to get proficient...

Hello Reader, It's a tough market for data engineers right now. Companies expect a lot from their data engineers, and the hiring bar is exceptionally high. Take a look at some job postings, and you will find a variation of "advanced distributed data systems knowledge" as a requirement. But, the catch is that there is no single definition of what this means. Additionally, there is a perceived impact of LLMs on the job market. However, it's not all lost; let's take a step back and consider what...