Setting up your git repo

We recommend that you manage your Airflow files in one central git repository. We provide a starter project to help with the base files that you need to get started. Copy these files into a new git repo that you own and manage.

Cloud build

An advantage of using Cloud Composer to manage your Airflow instance is that Google keeps the /home/airflow/gcs folder in your Airflow workers in sync with the Cloud Storage bucket that was created when you created your Cloud Composer instance. You’ll notice a file, cloudbuild.yaml, in the root of the project directory. Update this file to reference your Cloud Storage bucket name.

Create trigger

Head to Cloud Build and click Connect Repository to connect to your Bitbucket or GitHub repository. Once you’ve mirrored your repository to Cloud Source Repositories, you will be asked to create a trigger. the example settings below will sync your dags and plugins folders whenever a commit is made to the master branch.

• Trigger type: branch
• Branch (regex): ^master$
• Build configuration: Cloud Build configuration file

Airflow variables

Complete the variables.json file under dags/airflow and import to Airflow via Admin –> Variables.