Securing and automating Machine Learning with Airflow
Emmi approached me with an amazing team of datascientists and limited
infrastructure and backend heavy lifting experience. I spent time with the data scientists and head of
product to find a pragmatic solution. Their larger customers demanded a security baseline - processing
production data in the cloud.
After careful consultation with experts in the field, requirements gathering, and analysing various data pipeline platforms, it wasn't hard to choose Apache Airflow - and in particular MWAA (AWS's offering for Airflow as a managed service). This allows production workloads to build off the work of data scientists, and allows them to focus on their core work. It is an example of not reinventing the wheel, and choosing the right tool for the job. All of this was built with terraform and is running in production at Emmi today.
After careful consultation with experts in the field, requirements gathering, and analysing various data pipeline platforms, it wasn't hard to choose Apache Airflow - and in particular MWAA (AWS's offering for Airflow as a managed service). This allows production workloads to build off the work of data scientists, and allows them to focus on their core work. It is an example of not reinventing the wheel, and choosing the right tool for the job. All of this was built with terraform and is running in production at Emmi today.