Forum Discussion
dbeavon
4 years agoContributor
FDX methods (like FdxExecuteCubeView) ... Why are they faster?
Recently I heard in these forums that FDX methods (like FdxExecuteCubeView) should be used from a BR whenever the performance of the data is critical.
Can someone help me understand why they are fa...
- 4 years ago
To add to Peter's points, the FDX methods were specifically created to programmatically extract data from the appropriate OneStream tables or other source systems in the most efficient way. The FDX connectors were primarily created in conjunction of the BI Blend process but can be used for any process that requires data to be extracted from OneStream
FDX, Fast Data Extract, BRAPIs allow a variety of options for connecting to DataSources for BI Blend. A key differentiator between the FDX BRAPI’s and other collection methods is support of parallel processing, in memory processing, and management of the Time dimension.
FDX BRAPI’s provide functionality to build Connectors to extract data from:
- Cube Views: Extract data through a Cube View definition. Ideal for defining data definitions through a Cube View, including Dynamic Calc results.
- Across Cube Data Units: Extract Cube data to a BI Blend target table through defined Data Unit filters.
- Stage Workflow Imports: Ability to leverage existing Stage Data. Uses may be reporting on existing “attribute” records contained in Stage, or simply enhanced dashboard reporting on Stage data.
- Source Systems / Data Warehouses: Performance oriented solution to connect to source system.
Performance is gained through the BRAPI’s ability to parallel process. For example, extracting data by Cube Data Unit will parallel process all the Data Units defined in the filter. Second, the FDX BRAPI’s do not generate a “.CSV” file as do Data Management File “Export Data” or “Export File” processes. The results of the export are managed during the BI Blend “in-memory” processing.
In cases of very large data sets, which where multiple periods are loaded, the processing time can be slow because each period is reflected as a data record. FDX BRAPI’s offer solutions to pivot the Time records to columns in order to create a matrix data layout. The Datasource can associate each of the periods with an “Attribute Value” field within the Integration settings. The design will treat each record as a collection of 12 periods when processing.
- FdxExecuteCubeView: Extract data defined through a Cube View. Any data presented in the Cube View will be extracted, such as Dynamic Calculated results.
- FdxExectuteCubeViewTimePivot: Cube View Data will generate all time as Columns which can be assigned as Attribute Value members in the Data Source.
- FdxExecuteDataUnit: Cube Data extract solution to extract data from Data Unit members.
- FdxExecuteDataUnitTimePivot: Cube Data extract solution to extract data from Data Unit members. Generate all time as Columns, which can be assigned as Attribute Value members in the Data Source.
- FdxExecuteStageTargetTimePivot: Extract existing Workflow’s Stage Data. Generate all time as Columns, which can be assigned as Attribute Value members in the Data Source.
- FdxExecuteWarehouseTimePivot: Extract data from an external source system.
- FdxGetCubeViewOrDataUnitColumnList: Connector BRAPI used to return field names.
- FdxGetStageTargetColumnList: Connector BRAPI used to return field names.
- FdxGetWarehouseColumnList: Connector BRAPI used to return field names.
TonyToniTone
OneStream Employee
4 years agoTo address your questions, I would recommend submitting a ticket with Technical Support to go through your situation. In general, you should not be responsible for clearing memory. The garbage disposal of IIS and Memory Manager should be clearing out memory as the execution of the function is complete. The assumption of a Cube View rendering not exceeding 1 GB is not entirely a correct assumption. It all depends on the amount of data pulled into cache and then inserted into a data table. Each data record in a database takes 3,200 bytes. If you have 1 million data records from the result of the Cube View definitions, then you will use 3.2 GB of memory just for the data records not including the additional memory for the new object of a data table. Every Cube View can be wildly different so I can't give you a definitive answer on any of this. It would be a good idea to meet with Technical Support to run through all of this. The memory not releasing could be a concern but we won't know unless Technical Support gets involved to troubleshoot. OneStream has a lot of memory intensive processes. Cube Views being one of them. 16 GB's on a server is ok for general development but not optimal for a Production environment. 16 GB's of memory can be consumed very quickly depending on the process, data volumes, number of large data tables in memory, etc.
dbeavon
4 years agoContributor
Thanks TonyToniTone
In the past the tech support team was not terribly interested in having our memory dumps. Moreover, a memory dump of 16 GB that demonstrates a problem is pretty unwieldy, even after you are able to download it and open it with a tool like in WinDbg. I was just going to watch counters, and take some smaller memory dumps after running a few cubeviews and triggering the garbage collection.
Ideally it would be possible to confirm a memory problem before 3 or 5 GB. If necessary I could work on creating an artificial repro with tiny cubeviews. I just wanted to know what to look for, to determine whether things were "leaking" or just "caching". (I know that this distinction can sometimes be based on lots of factors.)
Based on your response, it sounds like you have no knowledge of FDX-specific memory issues. I will hold off on assuming that there is a problem for now.... The only reason for my suspicion is based on the behavior I noticed, and the requirement for IIS to be reset every night. Yes, our production servers do have quite a lot more RAM (32 or 64 I think).
Related Content
- 2 years ago
- 3 years ago
- 4 months ago
- 3 years ago