Forum Discussion

photon's avatar
photon
Contributor
3 days ago

Process Cube automation

We have a business rule that, after loading data, runs a Process Cube.

When using this with the batch harvest API, it behaves pretty much as expected in that, after the process is complete, it returns a status and execution time and all that.

However, I have grown to dislike the file clutter that results from years and years of harvest file processing and I'm updating the code to use the BRApi.DataQuality.Process functions. Both ExecuteProcessCube and StartProcessCube can be used for this with the main difference being the objects needed to kick them off and the object types they return: Start returns a TaskActivityItem and Execute returns ProcessCubeProcessInfo.

Superficially, I could walk away from this effort right now and it would work but I've noticed one really annoying issue: neither returns a real status of the process itself. This appears to be because our Process also runs a Calc. Both Start and Process "succeed" nigh-instantly while the Calc continues to run for five more minutes. I'd rather prefer to wait, monitor the calc, and only send the notification of the full load/calc being complete when it's... actually complete, you know, the way it already works with the harvest automation method.

So, am I using these tools incorrectly? Have I missed an obvious feature? The fact that they both seem to have effectively the same behavior actually makes me think one of them should behave differently than the other by stopping, waiting, and observing before returning a status and its failure to do so is a bug. Could I perhaps use one of these returned objects to sniff out the calc and monitor it manually?

Extenders

8 Replies

  • ckattookaran's avatar
    ckattookaran
    VIP
    3 days ago

    All those process return a task activity item which you can check for status. I would do it as a for loop to wait till the status becomes successful after the calc execution. 

    • photon's avatar
      photon
      Contributor
      3 days ago

      Both return success instantly even though the calc is still running.

    • photon's avatar
      photon
      Contributor
      2 days ago

      The network security is preventing me from uploading a screenshot but even Task Activity shows the "Process" completes in a fraction of a second but there's a calc running in a DM job for a full five minutes afterwards.

    • photon's avatar
      photon
      Contributor
      2 days ago

      There we go. Process: Complete. Calc: still running.

      The BR only seems to be able to report on the on Process itself and I can't find a way to access the details/status of the child calc.

      • ckattookaran's avatar
        ckattookaran
        VIP
        2 days ago

        Aha, so you are using a no calculate and running a dm job from it. I'm don't know whether there is a process which can get the status of an already running task that was not initiated within the rule. Not in front of a system to check for this. But your best bet is going to be that, look for a method that can give you the status of a running DM job and then wait for that one, or you might have to tweak the event handler to do it as an execute dm instead of start dm. 

  • photon's avatar
    photon
    Contributor
    2 days ago

    Maybe I can attack this from a different angle.

    Is there a way to get the calc info from the WF properties somehow? If I can get the name of the DM step (which is in the Filter field), I'm pretty sure I can figure out how to query the status. There's a snippet for GetUserWorkflowInitInfo that seems expose a lot of properties but it will take me a while to dig through everything that gives me.

    Provided I can get what I need from there, the next step is figuring out how to check for running Task Activity by that name. This is where the visibility seems to break down even further. The only ways I can see to interface with the Task Activity seems to require knowing the task activity ID in advance/being inside the DM job already and accessing it through the args. Maybe I'm missing something here as well though.

    Worst case, because it's not at all dynamic, is just querying the task activity table directly for a task with that name and running the query every 30 seconds or so until the status shows complete. It's gross but it will work if there are no better methods.