Forum Discussion

Keyser_Soze's avatar
Keyser_Soze
Contributor
2 years ago

Import Step: Where is Data stored before being saved to 'SourceStageData' Table

I am using an Event Handler to manipulate loaded data before it gets saved to the staging area table. More specifically, I am duplicating some rows while adding new logic to the dimensions (I know it...
Import
Workflow
MarcusH's avatar
MarcusH
Valued Contributor
2 years ago

I think it uses INSERT BULK direct from the source somehow so there's no intermediate table (at least that's what the profiler is saying). If you are using a text file as a source you could edit that file before it is imported. I have done it on the InitializeTransformer Before event.

  • Keyser_Soze's avatar
    Keyser_Soze
    Contributor
    2 years ago

    Thanks for replying,
    For AuditTrail purposes, I think it would be wise to keep the original file intact. Also, it won't work for every workflow Profile since the data sources aren't the same ... It would be hard parsing each file by dimension etc

    What method did you use to achieve this though ? 
    - Retrieving the .csv file from the harvest folder and modify it
     Or

    - A property within these objects;

          Dim objTransformer As Transformer = DirectCast(args.Inputs(0), Transformer)

          Dim objParser As Parser = DirectCast(args.Inputs(0), Parser)

     

    • MarcusH's avatar
      MarcusH
      Valued Contributor
      2 years ago

      It was much simpler than that. It was not a batch process so:

      - take a backup of the original file, read the file, make the changes and then save.