Forum Discussion
Solved
FDX methods (like FdxExecuteCubeView) ... Why are they faster?
Recently I heard in these forums that FDX methods (like FdxExecuteCubeView) should be used from a BR whenever the performance of the data is critical.
Can someone help me understand why they are fa...
To add to Peter's points, the FDX methods were specifically created to programmatically extract data from the appropriate OneStream tables or other source systems in the most efficient way. The FDX connectors were primarily created in conjunction of the BI Blend process but can be used for any process that requires data to be extracted from OneStream
FDX, Fast Data Extract, BRAPIs allow a variety of options for connecting to DataSources for BI Blend. A key differentiator between the FDX BRAPI’s and other collection methods is support of parallel processing, in memory processing, and management of the Time dimension.
FDX BRAPI’s provide functionality to build Connectors to extract data from:
- Cube Views: Extract data through a Cube View definition. Ideal for defining data definitions through a Cube View, including Dynamic Calc results.
- Across Cube Data Units: Extract Cube data to a BI Blend target table through defined Data Unit filters.
- Stage Workflow Imports: Ability to leverage existing Stage Data. Uses may be reporting on existing “attribute” records contained in Stage, or simply enhanced dashboard reporting on Stage data.
- Source Systems / Data Warehouses: Performance oriented solution to connect to source system.
Performance is gained through the BRAPI’s ability to parallel process. For example, extracting data by Cube Data Unit will parallel process all the Data Units defined in the filter. Second, the FDX BRAPI’s do not generate a “.CSV” file as do Data Management File “Export Data” or “Export File” processes. The results of the export are managed during the BI Blend “in-memory” processing.
In cases of very large data sets, which where multiple periods are loaded, the processing time can be slow because each period is reflected as a data record. FDX BRAPI’s offer solutions to pivot the Time records to columns in order to create a matrix data layout. The Datasource can associate each of the periods with an “Attribute Value” field within the Integration settings. The design will treat each record as a collection of 12 periods when processing.
- FdxExecuteCubeView: Extract data defined through a Cube View. Any data presented in the Cube View will be extracted, such as Dynamic Calculated results.
- FdxExectuteCubeViewTimePivot: Cube View Data will generate all time as Columns which can be assigned as Attribute Value members in the Data Source.
- FdxExecuteDataUnit: Cube Data extract solution to extract data from Data Unit members.
- FdxExecuteDataUnitTimePivot: Cube Data extract solution to extract data from Data Unit members. Generate all time as Columns, which can be assigned as Attribute Value members in the Data Source.
- FdxExecuteStageTargetTimePivot: Extract existing Workflow’s Stage Data. Generate all time as Columns, which can be assigned as Attribute Value members in the Data Source.
- FdxExecuteWarehouseTimePivot: Extract data from an external source system.
- FdxGetCubeViewOrDataUnitColumnList: Connector BRAPI used to return field names.
- FdxGetStageTargetColumnList: Connector BRAPI used to return field names.
- FdxGetWarehouseColumnList: Connector BRAPI used to return field names.
TonyToniTone
OneStream Employee
OneStream EmployeeTo add to Peter's points, the FDX methods were specifically created to programmatically extract data from the appropriate OneStream tables or other source systems in the most efficient way. The FDX connectors were primarily created in conjunction of the BI Blend process but can be used for any process that requires data to be extracted from OneStream
FDX, Fast Data Extract, BRAPIs allow a variety of options for connecting to DataSources for BI Blend. A key differentiator between the FDX BRAPI’s and other collection methods is support of parallel processing, in memory processing, and management of the Time dimension.
FDX BRAPI’s provide functionality to build Connectors to extract data from:
- Cube Views: Extract data through a Cube View definition. Ideal for defining data definitions through a Cube View, including Dynamic Calc results.
- Across Cube Data Units: Extract Cube data to a BI Blend target table through defined Data Unit filters.
- Stage Workflow Imports: Ability to leverage existing Stage Data. Uses may be reporting on existing “attribute” records contained in Stage, or simply enhanced dashboard reporting on Stage data.
- Source Systems / Data Warehouses: Performance oriented solution to connect to source system.
Performance is gained through the BRAPI’s ability to parallel process. For example, extracting data by Cube Data Unit will parallel process all the Data Units defined in the filter. Second, the FDX BRAPI’s do not generate a “.CSV” file as do Data Management File “Export Data” or “Export File” processes. The results of the export are managed during the BI Blend “in-memory” processing.
In cases of very large data sets, which where multiple periods are loaded, the processing time can be slow because each period is reflected as a data record. FDX BRAPI’s offer solutions to pivot the Time records to columns in order to create a matrix data layout. The Datasource can associate each of the periods with an “Attribute Value” field within the Integration settings. The design will treat each record as a collection of 12 periods when processing.
- FdxExecuteCubeView: Extract data defined through a Cube View. Any data presented in the Cube View will be extracted, such as Dynamic Calculated results.
- FdxExectuteCubeViewTimePivot: Cube View Data will generate all time as Columns which can be assigned as Attribute Value members in the Data Source.
- FdxExecuteDataUnit: Cube Data extract solution to extract data from Data Unit members.
- FdxExecuteDataUnitTimePivot: Cube Data extract solution to extract data from Data Unit members. Generate all time as Columns, which can be assigned as Attribute Value members in the Data Source.
- FdxExecuteStageTargetTimePivot: Extract existing Workflow’s Stage Data. Generate all time as Columns, which can be assigned as Attribute Value members in the Data Source.
- FdxExecuteWarehouseTimePivot: Extract data from an external source system.
- FdxGetCubeViewOrDataUnitColumnList: Connector BRAPI used to return field names.
- FdxGetStageTargetColumnList: Connector BRAPI used to return field names.
- FdxGetWarehouseColumnList: Connector BRAPI used to return field names.
Thanks Tony.
Sorry to tag on another question. One issue that came up when swapping over to the FDX for cubeviews is a concern about memory usage . Is there any additional responsibility that our BR may have insofar as memory is concerned? Will the owner of the BR be responsible for release memory used by the FDX, or possibly flushing a cache? Sometimes .Net API's provide deliberate Dispose() methods for this purpose, if the release of memory needs to happen proactively.
I was working with our pre-prod environment the other day (I was the only one in there), and was running a massive number of FDX operations. Memory usage continually increased until it reached the capacity of the server and then we eventually needed to restart IIS since the dashboard adapters started failing and returning the word "null". If this memory in IIS was related to legitimate "caching" then I would expect that memory usage would level out after repeatedly executing the same FDX-cubeviews. We are talking about 16 GB of RAM on the server, and each cubeview would only conceivably use 1 GB at the very most.
I am preparing to deploy my work to production and this will become someone else's problem. 😉 But I just want to make sure that I've done my own due diligence, to limit the need for technical support tickets with onestream in the future. I'm pretty good at working with .Net memory dumps so if you can give me an idea about what particular FDX class instances should be the focus of my attention, then I should be able to do more to test for possible memory leaks.
We are using 6.5 in pre-prod and 6.3 in prod.