By this command, you can generate new project from templates.
In {root-dir} that you specify, you can see some files that include a directory named as {name} that you specify.
dsfg project
--name
--owner
[--start-date]
[--root-dir]
[--dwh-root]
[--dw-class]
[--dw-param-json]
[--sub-component-dw-class]
[--sub-component-dw-param-json]
[--partition-dw-class]
[--partition-dw-param-json]
[-- --help]
| parameter | required | description |
|---|---|---|
| name | YES | project name |
| owner | YES | project owner name |
| start-date | no | project start date by iso format |
| root-dir | no | project root directory |
| dwh-root | no | root directory of project data store |
| dw-class | no | class name of dw interface that is used to store data to dwh |
| dw-param-json | no | parameter that is passed to initialize dw-class |
| sub-component-dw-class | no | class name of sub-component-dw interface |
| sub-component-dw-param-json | no | parameter for sub-component-dw-class |
| partition-dw-class | no | class name of partition-dw interface |
| partition-dw-param-json | no | parameter for partition-dw-class |
| help | no | show help (dsfrun project -- --help is proper command to see help) |
After you generated new project with dsfg project, You can generate workflow by using this command.
Under the {root-dir}/{project-name}/workflow directory, there is a directory that is named according to {namespace} that you specify. And In that directory, you can see workflow class file.
And as a workflow class file is, a script file is created under the {root-dir}/{project-name}/script directory.
dsfg workflow
--project-name
--workflow-name
--namespace
[--root-dir]
[-- --help]
| parameter | required | description |
|---|---|---|
| project-name | YES | project name |
| workflow-name | YES | workflow name |
| namespace | YES | dot separated namespace for workflow |
| root-dir | no | project root directory |
| help | no | show help (dsfrun workflow -- --help is proper command to see help) |
Execflow is used to specify an execution order of workflows in a project.
When you run this command, you can get a file execflow-{name} in your {root-dir}.
In this file, all of workflow names are listed in the order of alphabet.
You can edit this execflow file and you can use dsfrun project command by specifying it.
dsfg execflow
--project-name
[--name]
[--root-dir]
[-- --help]
| parameter | required | description |
|---|---|---|
| project-name | YES | project name |
| name | no | execflow name (default name is all) |
| root-dir | no | project root directory |
| help | no | show help (dsfrun execflow -- --help is proper command to see help) |
You can update workflow catalogue by using this command.
This command is automatically executed after you generate project or workflow. So please use this command when you want to fix broken catalogue file.
dsfg catalogue
--project-name
[--root-dir]
[-- --help]
| parameter | required | description |
|---|---|---|
| project-name | YES | project name |
| root-dir | no | project root directory |
| help | no | show help (dsfrun catalogue -- --help is proper command to see help) |
You can use jupyter-based debugger to debug calculations that are programmed in your workflow.
By using this command, you can get a jupyter notebook that contains programs imitating workflow and you can run this notebook with small test data.
Please see here for how to run jupyter notebook.
| parameter | required | description |
|---|---|---|
| project-name | YES | project name |
| workflow-name | YES | workflow name |
| execution_date | no | execution date that is passed when build workflow |
| root-dir | no | project root directory |
| debug-targets | no | list of tasks to be debugged |
| help | no | show help (dsfrun debugger -- --help is proper command to see help) |
You can use web-based task manager to control task execution or to check the progress of it.
By using this command, you can initialize the web service.
| parameter | required | description |
|---|---|---|
| project-name | YES | project name |
| root-dir | no | project root directory |
| help | no | show help (dsfrun workflow -- --help is proper command to see help) |
Please see here for how to run web service.