Workflow examples

Below are examples of how data is generated.

Data generation

For the data generation of a bot, it must first be compatible with the AiDataGenByLeo structure. As an example we will use this project: https://forge.mql5.io/nique_372/EasySbAi

Prerequisites

You will need to have setfiles already defined and structured to be compatible with the FolderToTaskCsv2 EA. For more info: https://forge.mql5.io/nique_372/AiDataTaskRuner/wiki/FolderToTaskCsv

Workflow

In this workflow the idea is to first generate a .csv file with all the tasks that we will then pass to the panel to execute.

1. Create the YAML file
2. Define the name and environment variables that will be used (useful to avoid repetition):

name: "Simple data generation"
env:
  main_folder: "EasySbAi" 
  task_csv: "task_from_folder.csv"
  common_folder: true

3. Let's define the steps of this workflow:

• First we will delete the move_files.json file that we will create later, to avoid errors on its next creation. Also, when we run the EA that collects the setfile info and transfers it to a CSV it won't accidentally read that JSON. We also add a "valid_results" field — this is an array and we will add the single value "any", which tells the orchestrator to consider any result value as success for this step.

  - name: "FileDelete"
    module: "File"
    valid_results: ["any"]
    with:
      file_name: "${{env.main_folder}}\\Config\\move_files.json" # Delete the json file that will be alongside the sets (to avoid EA failure)
      common_flag: ${{env.common_folder}} 

• Next we run the EA that will create the task file, so we execute FolderToTaskCsv2:

  - name: "RunEA"
    module: "Generic"
    with:
      fixed:
        name: "Experts\\Shared Projects\\AiDataTaskRuner\\Utils\\FolderTaskToCSV2.ex5"   # adjust to real path
        symbol: _Symbol
        timeframe: _Period
        run_flags: "DLL"
        ms_sincronizacion: 500
        timeout_segundos: 120
      ea_params:
        - InpFolder:                      ["TYPE_STRING", "${{env.main_folder}}\\"]
        - InpFolderIsInCommon:            ["TYPE_BOOL", "${{env.common_folder}}"]
        - InpParamActiveGenerateName:     ["TYPE_STRING", "InpStrategyAiGenerateDataMode"]
        - InpDataStartDate:               ["TYPE_DATETIME", "2021.01.01 00:00"]
        - InpDataEndDate:                 ["TYPE_DATETIME", "2025.06.01 00:00"]
        - InpCsvFileNameOutput:           ["TYPE_STRING", "${{env.task_csv}}"]
        - InpCsvFileNameOutputIsInCommon: ["TYPE_BOOL", "${{env.common_folder}}"] # using the common folder we are working with  
        - InpStrMatchExcluyed:            ["TYPE_STRING", ""] # not excluyed 
        - InpStrMatchIncluyed:            ["TYPE_STRING", "*.set|*\\"] # Incluyed all .set and foldes\

• Then we create the JSON config file. In this file we define which files (generated by the bot) the panel will move to the final folders — in this case the 6 files resulting from data generation:

  - name: "WriteInFile"
    module: "File"
    with:
      open_flags: ["${{aidata.common_flag_val}}", "FILE_TXT"] # Using the panel's flag since the panel expects the file to be there
      file_name: "${{env.main_folder}}\\Config\\move_files.json"
      # Write to the file with \\ since internally when reading the JSON it is escaped, so be careful with that
      content: |
       {
        "files_to_move": [
          "${{env.main_folder}}\\data_tp.csv",
          "${{env.main_folder}}\\data_sl.csv",
          "${{env.main_folder}}\\data_pred.csv",
          "${{env.main_folder}}\\saclertp_median_iqr.csv",
          "${{env.main_folder}}\\saclersl_median_iqr.csv",
          "${{env.main_folder}}\\saclerpred_median_iqr.csv"
        ]
       }

• Before telling the panel to run all tasks from the task CSV we generated in step 2, we need to first load the initial config, so we call the AiDataTaskRunner API function LoadConfigInPlace with the necessary parameters:

  - name: "LoadConfigInPlace"
    module: "AiDataTaskRunner"
    with:
      file_name: "${{aidata.task_folder}}Wf\\config.txt"  
      progress_csv: "temp.csv"
      main_folder: "${{env.main_folder}}\\"
      expert_path: "Experts\\Shared Projects\\EasySbAi\\Ea.ex5"    
      file_name_json_config: "${{env.main_folder}}\\Config\\move_files.json"

• Now we tell the panel to load all tasks from the CSV we generated:

  - name: "LoadTaskInFile"
    module: "AiDataTaskRunner"
    with:
      file_name: "${{env.task_csv}}"

• Then we tell the panel to run all pending tasks:

  - name: "RunAllTask"
    module: "AiDataTaskRunner"
    with:
      strict: true     
      ms_pool: 2500

• Finally we want the panel to notify us when it has finished generating data, so we can use any function from https://forge.mql5.io/nique_372/AiDataTaskRuner/wiki/Wf_Message. In this case we'll use basic alerts:

  - name: "Alert"
    module: "Message"
    with: 
      message: "All tasks are done, check the terminal"   

Done! Now we can save this file in YAML format and load it into the panel.