Dramatic change in execution time for Validate Intersections step

Mike1613
New Contributor II

Hello, we are 'suddenly' seeing an increase in the time for a matrix data source import to complete the validate intersections step.  It has gone from 5 minutes to over 2.5 hours to complete one step during the import step of the workflow.  The subsequent cube load takes another 2+ hours of time once the validation step completes (for a total execution time now over 5 hours).

We have backed out all changes since the previous execution that completed in 5 minutes with no change to the time.  We currently are not suppressing zeros on the data source, however even with that enabled in our QA instance, the time to complete the validate intersections step is still much longer than 5 minutes (40+ minutes).

Any suggestions on what else we can check?  We are currently on platform version 7.0.1. 

5 REPLIES 5

SamRichards
Contributor

Hey Mike,

Have the mappings changed at all, some types of mappings take a considerable amount longer than others?  Is the overall file size the same between the two loads? Also is the 5 minutes in your production environment and the 40 min in the QA? Are the two environments specd out the same from a hardware standpoint? It isn't uncommon to have a QA/DEV scaled down from a hardware perspective. One more item I would look at is the logs of the load itself to see what step is taking the longest. You can click on the three bars in the far left to see more detail of the task in task activity to help to point you in the right direction.

Mike1613
New Contributor II

Thank you for your reply, no mapping changes since this was first live over a year ago.  The file used for the PROD and QA loads was the same set of data (in the app too).  You are correct, the Production environment is a larger configuration, however, that is where the most recent run completed in over 5 hours (see below).

The time increase is in the validate source and validate intersections steps that are 'suddenly' chewing up the time.  We thought maybe the audit / activity / error logs or such need to be cleaned up to help speed up that process but TBD.

We did notice that suppress zeros was false for the data source, which in our testing has helped cut the time down to about an hour, but that still does not explain the overall jump in time.

 

Mike1613_1-1666970482667.png

Hey Mike, that is interesting... Is there anything else running on the system at the same time? I honestly am not sure if nothing has changed and would submit a ticket to the support team to see if they can help you run this down...It might be on the hardware/memory side like you put in your previous comment.

Yes it is, we are working with OS support via a ticket opened last week.  Posted here to explore all possible avenues to see if anyone else has also seen a similar behavior and was able to identify the reason why it started taking so much longer.  More to come if we find it...   Happy Halloween!!!!  😉

Mike1613
New Contributor II

Quick update on this topic, we identified some 'corrupt' data that is causing an error when the workflow throws this message in the error log (shown below).  When this occurs, the execution of the workflow impacted dramatically slows down with no other messages that have been identified.  We are evaluating options to see if we can identify the 'corrupted' data outside of clearing everything and rebuilding (ideally worse case since we are in a production application).  More to come...

Mike1613_0-1675272240928.png