Forum Discussion

ST1's avatar
ST1
New Contributor III
3 months ago

Source Data Condition check before Import

Is there a way I can check my source records before importing data into a stage? If I am in a workflow for Import, Validate and Load, Even before the data is imported, I would like to check my source record for a condition like if it is less than a particular value. Only if the condition satisfies, I need to proceed to import else stop the workflow.

If someone has done it, can you share the setup and code for the same.

Appreciate your help!

Data Integration
Import
  • JoakimK's avatar
    JoakimK
    Contributor
    3 months ago

    Is this a file load or a direct connect import?

    It sounds to me that you are asking for a check before import to evaluate if it should even bother with the import at all

    So, if using a file based import a parser rule is good for such a thing, a parser rule is a rule that will run on a record by record basis and is already part of the import, so your import would have already started when it runs, and it will evaluate each record on its own merit. This works fine if the evaluation you want to perform is on the first line, but for anything else it will likely import some items and some not using this method.

    If you are using a direct connect method, you can evaluate the dataset you are wanting to import before even initiating the import at all, and you would need to do this in the connector business rule. Alternatively, if its connector based you could even make it a requirement for the data to even be returned at all if the criteria is not satisfied (saving the time needed to even transfer any data in the first place) by using a where clause (for SQL) or parameter if using an API.

    Joakim

  • ludodepaz's avatar
    ludodepaz
    Contributor
    3 months ago

    Hi, you can write a Parser Rule, it will run when you Import data. You create your rule and you call it in the Data Source like this: 

    • ST1's avatar
      ST1
      New Contributor III
      3 months ago

      Thanks! I will try it. Is there a way that we can also write error email function in parser rule or call an extender rule from parser rule? I need to send an error email if source condition does not satisfy. Would really appreciate your help

      • ludodepaz's avatar
        ludodepaz
        Contributor
        3 months ago

        Yes, you can send an email. I see 2 options

        1. if you execute the Import from a DM job and you're on V8.2 or later, you can send an email from the Data Management Sequence.
        2. There is another option, which is very slick I believe, you can log the errors in a file and then send the file by email and with a little bit of code (like add a boolean with 'has Errors'), you can send the email only if there is an error and send the details. 

        I posted a blog on how to log with a file and send the file via email, I hope it helps!
        Log like a champ with OneStream - OneStream Community (onestreamsoftware.com)