Forum Discussion

KurtMayer's avatar
KurtMayer
Contributor
12 days ago

Is there guidance on the setting for the number of parallel executions for Harvest Batch Loads?

When using the BRAPi.Utilities.ExecuteFileHarvestBatchParallel function in an Extender BR, does anyone have any guidance and/or opinions on the proper setting of the parallelBatchCount parameter? 

My understanding has always been to set it to a maximum of the number of CPUs - 1 , available for the server performing data management or batch load processing.

Data Integration
Data Management
Import
Workflow

2 Replies

  • KH1's avatar
    KH1
    Contributor II
    10 days ago

    How did you get "parallelBatchCount" = "maximum of the number of CPUs - 1"?
    - An earlier post suggested using a Count < "max CPUs - 1".
    - We raised a Case: Support suggested using a "trial and error".

    Please review our findings below:
    1. Parallelism:
    Raise a Case: to set Parallelism = the number of CPU Cores of Servers if not already.

    Parallelism = the number of CPU Cores (32 Cores for our Servers)

    2. Trial and Error:
    Run jobs: parallelBatchCount = 16, 18, 20, 22, 24 (and higher as you wish).
    - Our tests showed that jobs ran longer when parallelBatchCount >= 20.

    Please share your findings.

    Thank you and Good luck, Kurt.