profile

Start Data Engineering

How Companies Ingest Data: 2 Key Patterns


Hello Reader,

Most companies ingest data in one of two ways.

  1. Stream data into a cloud store via an event log like Kafka.
  2. Extract data from source systems in batch.

In this article 👇, we go over a high-level design of how they are built and their tradeoffs.

​> Data Ingestion Patterns​

Please reply to this email if you have any questions/thoughts/ideas and I will get back to you.

Regards,

Joseph Machado

​startdataengineering.com​

Start Data Engineering

Over the last decade, I've built highly scalable distributed data platforms and helped companies scale to processing multiple exabytes of data. My mission is to bring software practices followed by top tech companies to data engineering and help data engineers level up. I help data engineers land high paying tech jobs and significantly up skill themselves.

Share this page