Treat data engineering as product/service, rather than process

  • toc: true
  • badges: true
  • comments: true
  • sticky_rank: 1
  • author: Yen
  • categories: [fastpages, jupyter]

Video

Key Takes Away

  • Understand the data consumers
  • Validation
  • Keep the raw data and intermediate steps data
  • Validate input and output
  • Monitoring is part of the product

Ref

  • https://easydata.engineering/data-engineering-principles-according-to-gatis-seja
  • https://medium.com/@rchang/a-beginners-guide-to-data-engineering-the-series-finale-2cc92ff14b0