Forum Discussion

photon's avatar
photon
Contributor
3 days ago

Process Cube automation

We have a business rule that, after loading data, runs a Process Cube.

When using this with the batch harvest API, it behaves pretty much as expected in that, after the process is complete, it returns a status and execution time and all that.

However, I have grown to dislike the file clutter that results from years and years of harvest file processing and I'm updating the code to use the BRApi.DataQuality.Process functions. Both ExecuteProcessCube and StartProcessCube can be used for this with the main difference being the objects needed to kick them off and the object types they return: Start returns a TaskActivityItem and Execute returns ProcessCubeProcessInfo.

Superficially, I could walk away from this effort right now and it would work but I've noticed one really annoying issue: neither returns a real status of the process itself. This appears to be because our Process also runs a Calc. Both Start and Process "succeed" nigh-instantly while the Calc continues to run for five more minutes. I'd rather prefer to wait, monitor the calc, and only send the notification of the full load/calc being complete when it's... actually complete, you know, the way it already works with the harvest automation method.

So, am I using these tools incorrectly? Have I missed an obvious feature? The fact that they both seem to have effectively the same behavior actually makes me think one of them should behave differently than the other by stopping, waiting, and observing before returning a status and its failure to do so is a bug. Could I perhaps use one of these returned objects to sniff out the calc and monitor it manually?

Extenders

8 Replies

  • photon's avatar
    photon
    Contributor
    2 days ago

    Maybe I can attack this from a different angle.

    Is there a way to get the calc info from the WF properties somehow? If I can get the name of the DM step (which is in the Filter field), I'm pretty sure I can figure out how to query the status. There's a snippet for GetUserWorkflowInitInfo that seems expose a lot of properties but it will take me a while to dig through everything that gives me.

    Provided I can get what I need from there, the next step is figuring out how to check for running Task Activity by that name. This is where the visibility seems to break down even further. The only ways I can see to interface with the Task Activity seems to require knowing the task activity ID in advance/being inside the DM job already and accessing it through the args. Maybe I'm missing something here as well though.

    Worst case, because it's not at all dynamic, is just querying the task activity table directly for a task with that name and running the query every 30 seconds or so until the status shows complete. It's gross but it will work if there are no better methods.

  • ckattookaran's avatar
    ckattookaran
    VIP
    3 days ago

    All those process return a task activity item which you can check for status. I would do it as a for loop to wait till the status becomes successful after the calc execution. 

    • photon's avatar
      photon
      Contributor
      2 days ago

      There we go. Process: Complete. Calc: still running.

      The BR only seems to be able to report on the on Process itself and I can't find a way to access the details/status of the child calc.

      • RobbSalzmann's avatar
        RobbSalzmann
        Valued Contributor II
        2 days ago

        Specifically, what object.method are you calling that returns a status of "Success"?

        Here is code I use to get the ProcessCubeProcessInfo result of WorkFlowDataQuality.ExecuteProcessCube.  Unfortunately AFAIK there is no handle to the actual process, and therefore no way to monitor it's progress.  As you can see here, the method to launch the process simply fires it off and waits for a ProcessCubeProcessInfo result, then based on the value of ProcessCubeProcessInfo.Status, returns a bool.  

        What it sounds like you're saying is WorkflowDataQuality.ExecuteProcessCube returns immediately after being called?

        result in the code below is a ProcessCubeProcessInfo object.  Among other things, it should contain a List(of DataUnitInfo), process start, end, and duration times, status, and error messages.

        I'm not handling any of that info, other than to validate status completed.  Are you saying the status is returned before the calc finishes?

        public bool Execute(WorkflowExecutionContext context)
{
	var result = WorkflowDataQuality.ExecuteProcessCube(si, context.ClusterPk, context.StepType, false);
	
	if (result.Status != WorkflowStatusTypes.Completed)
	{
	    throw new WorkflowStepException(string.Format(
	        context.BaseMessage,
	        context.WorkflowInfo.CurrentStep().Name,
	        context.ProfileName,
	        result.DataUnitInfos,
	        result.ErrorMessage));
	}
	
	return true;
}

         

    • photon's avatar
      photon
      Contributor
      3 days ago

      The network security is preventing me from uploading a screenshot but even Task Activity shows the "Process" completes in a fraction of a second but there's a calc running in a DM job for a full five minutes afterwards.

    • photon's avatar
      photon
      Contributor
      3 days ago

      Both return success instantly even though the calc is still running.