Forum Discussion

fc's avatar
fc
New Contributor III
2 months ago

Issue with multiple data load steps at year end

Hi everyone,

My client loads data through a classing import step throughout all the months of the year. In addition to that, they asked to set up 3 additional import steps (P13, P14, P15) to be used in December that will serve as adjustments to the previous data loaded on the cube.

At each import step the data will be loaded in full and should overwrite the data loaded through previous import steps (and this is why we don't use forms but we use multiple import steps).

For example, when P13 is loaded, it should overwrite the data loaded through the Import step, and when P14 is loaded it should overwrite the data loaded through P13, and so on.

I already set the "Load overlapped siblings" setting of the parent profile to false:

The loading seemed to work fine, as each load process overwrote the previous one.

But there is an exception that we don't understand how to fix.

We set up a mapping rule for a specific source account is such a way that it can be mapped into one or another target account depending on the presence or absence of specific parameters provided in the source load file. If the parameters are present, the amount of that source account is mapped into a specific Asset account in Onestream, and if the parameters are not present, the amount is mapped into a Liability account. During each load, the account can be mapped either into one or the other target account (therefore if we map into the Asset account, the liability account should be = 0, and the other way around).

Now, what we noticed that if in P13 the source parameters are present, the system maps the source account into the Asset account in Onestream (which is correct). But then, if in P14 we remove the parameters, the system correctly maps the source account into the Liability target account in onestream, but the amount loaded during P13 on the Asset account is still present in the cube and visible in the reports.
We would expect that amount not to be present, since each load step should completely override the previous step.

Does anyone has any idea why this might happen?

I know it's quite a specific question, if you need further info please ask.

Thanks in advance for any hint! 

 

 

 

Workflow
  • ckattookaran's avatar
    ckattookaran
    VIP
    2 months ago

    Wait till you hit level 3 :). I would do that as a specific clear, not a wide blanket clear. If you know upfront what the mappings do, then do that clear from accounts and the corresponding entities. I'm so hoping that you guys are not doing calculations on Import member.

  • fc's avatar
    fc
    New Contributor III
    2 months ago

    Thanks for the answer

    Is this the only way to do it? We would like to ask the client to perform the least number of actions possible, so ideally we would like to automate this action. But if there is no other way we'll certainly do that

    • Henning's avatar
      Henning
      Valued Contributor II
      2 months ago

      Well, as always, there are many ways to Rome. You could also set up a Clear Data Data Management step and assign this to a Process step in the workflow or trigger it in a different way.

      Or you can clear the stage data, retransform and load 'nothing'.

      Depending on the requirements, the latter might be better as it is more pointed and does not clear out all data on O#Import in a given data unit.

       

       

      Make sure you document everything for your client. (Fully) Automating this bares the risk of users not knowing what is happening to their data when clear, reload, ... processes are running in the background without the users specifically and knowingly triggering any of that.

  • fc's avatar
    fc
    New Contributor III
    2 months ago

    Hi Henning, how do you suggest to clear all data from the previous load step? 

    Is there a way to do it that does not involve the use of a custom BR?

    • Henning's avatar
      Henning
      Valued Contributor II
      2 months ago

      You can e.g. clear all imported data by right-clicking in the workflow:

      And then you load the data you need in the cube.

  • Henning's avatar
    Henning
    Valued Contributor II
    2 months ago

    Hi, as you are loading data via the workflow, the data uses the concept of the Level 2 Workflow Data Unit.

    Data Units (onestream.com)

    "The workflow data unit is the default level used by the workflow engine to control, load, clear, and lock data. Workflow level data loads from the staging data mart to the cube, and is cleared and locked at a granularity level that includes the account dimension by default."

    Because you are mapping data to different accounts, the previous loads to accounts that hold (previous) data but are not mapped to anymore as part of the following load (e.g. P13), the data is not cleared and remains.

    Make sure to clear and complete the previous load step to clear all data from the corresponding load.