Workflow

Project structure image

Project can have many workflow.

Roles

Roles of workflow are:

  • organizing tasks
  • managing execution.

Contents

Workflow has information like:

  • workflow name
  • task pipeline
  • execution start date
  • schedule interval

And while execution, workflow also has information like:

  • workflow parameters includes execution date
  • execution id

Features

Workflow has features like:

  • generating tasks
  • building data loader
  • building pipeline

Executing

DSF has web-based task execution manager (apache airflow), and workflow can be managed by it.

Airflow ss

And also you can execute whole workflow by using dsfrun workflow command.

Please visit here for more information.

Task Pipeline

You can write data analysis logic pipeline in a workflow.

Workflow image

And In a project, workflow can be connected to pipeline by referring other workflow.

Workflow pipeline

Workflow Parameters

In DSF, there is a concept Workflow parameter, which store information determined at execution.

Simplest one is execution date. You must want to use execution date to control behavior of calculation. In that case please use workflow parameter.