How to speed up force calculate step?

Ankit_Shah
New Contributor II

Hi all,

We are running "force calculate" step to perform force calculation of 80% base entities  . It runs for 4-6 hours. How to improve performance , any way run this in multi thread/parallel by entity (servers are hosted by OneStream) ? Thanks!

8 REPLIES 8

JackLacava
Community Manager
Community Manager

The system already parallelizes everything it can. There isn't a magic "go faster" button.

Force Calculate will run on all previous periods, ignoring calculation status of every dataunit. So, in December, you're recalculating the whole year.

That, plus (likely) inefficient rules, can result in bad performance like you're seeing. 

You'll need to look at the performance of your rules, scouring the log for timings (you can use the With Logging option for more details, although it will slow down things further) and checking the various Application Reports for dense dataunits, bad rules using #All, excessive reliance on Business Rules rather than Member Formula, etc etc.

Ankit_Shah
New Contributor II

Thanks Jack, I am already ware of what you said , may I should rephrase my question. I want to know a way to trigger same calc.step in parallel for 5-6 different group of entities that way I can achieve faster execution. Currently we pass value1 and all base entities of value1 is force calc. it has appx 10000 entities underneath. Now value1 is a parent in entity dimension so I am looking for a way to launch from dashboard button in single click, if there is way to run force calc for value1.1, value1.2, value1.3 at same time?

Ah, I see what you mean. I was going to suggest using a Data Management job, but it looks like it works in the same way you mention - i.e. when you use a filter on  a Force Calculate step, the resulting members are calculated sequentially.

The most efficient approach would probably be to just Consolidate parents instead, letting OneStream do what it does best (i.e. parallelizing dataunit calculation).

If that's not possible, depending on what you're doing in those calculations and how long they take individually, it *might* make sense to play around with .Net parallelization yourself, as shown in the example extender below. However, in my tests this adds significant overhead - although they are nominally started at the same time, having to set up internal machinery for each additional thread actually slows down the total execution time by several seconds per thread. If each calculation can take minutes, that's a price worth paying; otherwise, we're making things worse. It may or may not be faster to launch parallel Custom Calcs with ExecuteCustomCalculateBusinessRule instead of going through DataManagement, but at that point you cannot use regular Calculate. Also, it's completely possible that some stuff will work (or not work) in unexpected ways, as it's always the case when one plays with threads; in that case, maybe just launch the sequences one after the other in a simple loop, and hope that OneStream will parallelize some of them (and likely remove some of the threading overhead).

 

Case Is = ExtenderFunctionType.ExecuteDataMgmtBusinessRuleStep
	' define which dimension and parent to get base members for
	Dim myDimPk As DimPk = brapi.Finance.Dim.GetDim(si, _
							args.NameValuePairs("Dimension")).DimPk
	Dim parentId As Integer = brapi.Finance.Members.GetMemberId(si, 
									DimType.Entity.Id, _
									args.NameValuePairs("ParentEntity"))
	' retrieve base members
	Dim baseMembers As List(Of Member) = brapi.Finance.Members.GetBaseMembers(si, _
																myDimPk, _
																parentId)
	' set up parameters for our parallel jobs, 
	' in this case containing the name of the member we want to calc
	Dim paramDicts As New List(Of Dictionary(Of String, String))
	For Each M As Member In baseMembers
		paramDicts.Add(New Dictionary(Of String, String) From {{"Child", M.name}})
	Next
	' launch execution in parallel. The sequence will have one step, calculating the member
	Parallel.ForEach(paramDicts, _
		Sub(paramObj)
			BRApi.Utilities.StartDataMgmtSequence(si, "FCSeq", paramObj)
		End Sub)

 

 

dipawar
New Contributor II

Will this above code work with finance business rule??

ChristianW
Valued Contributor

The standard business rule and member formula calculation are already calculated in parallel. Therefore, I wouldn't run the code in a financial business rule or a member formula.

But if you have custom calculations (FinanceFunctionType.CustomCalculate) you need to call for a set of entities or scenarios, you can use the code to launch them in parallel.

You can run calculations on different servers using data management jobs. There is a selection box for server in data management jobs. 

Other tips:

  • Member formulas run in parallel.
  • You can aggregate instead of consolidate.
  • You can run custom calcs in a parallel.ForEach loop from a extender business rule.
  • You can avoid aggregating/consolidating multiple periods. Set the base entities (or a subset of it) to dirty with a business rule and then run a standard aggregate/consolidate instead of force. This can be done with a business rule and a data management job.

And of cause you can combine some of these ideas.

Christian, out of curiosity, would triggering custom calculate using a parallel.ForEach run faster compared to triggering a DM Step for 100 entities? I thought the DM Step would also include some sort of parallelization

ChristianW
Valued Contributor

In general probably not, but in special cases when your knowledge about the calculations allows you to optimise it, it can help.

OneStream tries to calculate entities in parallel and even within a single entity, it tries to calcs all member formulas with the same calc level at the same time, but if your calculations are single entity (I.e top member of a consolidation) and if it is business rule based, it definitely helps, because OneStream can’t parallelise business rules without your help.