Forum Discussion

Kelley_M's avatar
Kelley_M
New Contributor
3 years ago

Direct Connect Loads into OneStream

When loading data into OneStream, our longest workflow takes ~40 minutes to load. We would like to try to kick off the 5 direct connect loads within this workflow to start simultaneously instead of s...
Data Sources
  • MarcR's avatar
    MarcR
    3 years ago

    Hi Kelley, we just did actually.
    It is important that the workflow steps are in different workflows (different workflow steps in 1 workflow don’t run in parallel). You also need to make the load ID (the first item of your load file) dynamic.
    OneStream advises to have a max of 3 in parallel per server so you could create two dm jobs to two different servers with both 3 threads (that’s what we do).
    We also use the direct load (if you don’t expect tranformation issues) that is also 20 to 30% faster with large data sets.
    Next to that the transformation itself is important so you could check the log after the load what takes most time. If it is the parsing or the writing to cube then parallel will help you, if it is the complex transformation then you can try to fix that in the SQL/ source.

MarcR's avatar
MarcR
VIP

Hi Kelley, we just did actually.
It is important that the workflow steps are in different workflows (different workflow steps in 1 workflow don’t run in parallel). You also need to make the load ID (the first item of your load file) dynamic.
OneStream advises to have a max of 3 in parallel per server so you could create two dm jobs to two different servers with both 3 threads (that’s what we do).
We also use the direct load (if you don’t expect tranformation issues) that is also 20 to 30% faster with large data sets.
Next to that the transformation itself is important so you could check the log after the load what takes most time. If it is the parsing or the writing to cube then parallel will help you, if it is the complex transformation then you can try to fix that in the SQL/ source.

Kelley_M's avatar
Kelley_M
New Contributor
3 years ago

Thank you Mark! I’ve copied this and sent to my OneStream consultant.