What's the best way to pull 16M+ rows of data
Hello! I have a user group who needs a specific set of data pulled just after close every month in order to populate other reporting tools they use. Here's the formatting they require: I'm ...
Forum Discussion
You can use this function "BRApi.Import.Data.FdxExecuteCubeView(si, cubeViewToExtract, entityDimName, entityMemFilter, scenarioDimName, scenarioMemFilter, timeMemFilter, nameValuePairs, includeCellText, useStandardFields, filter, parallelQueryCount, logStats)" to retrieve the data into a DataTable. Then, using a StreamWriter to a CSV.
Hope it helps!
Thanks - is there more instruction somewhere that you know of which I could review to see if this will allow the flexibility I need?
It's part of the so-called FDX (Fast Data eXtract) APIs. If you search around the forum you should find quite a few examples. We also cover them in the BiBlend class, so if you know anyone who attended that you can ask him for the material. They are in the latest docs at https://documentation.onestream.com - they show them applied to Data Source Connectors, but will work as well in Dashboard Data Set rules or any other type of rule.
Basically, they allow fast extraction of Cube Views, Data Unit records, or Stage records, and also allow to pivot them on the Time dimension if necessary (i.e. getting a column for January, one for February, etc, which can compress the number of records pretty dramatically). I don't think flexibility will be a problem, but they do require a bit of coding to set up all the parameters they require.
Thanks so much for the direction! I've set up the BR and the data adapter and when I test, I get the same error I received when I built and attempted to export the initial CV:
What am I doing wrong? Here's the code (which I shamelessly stole from this post Solved: FDX Cube View Rule - OneStream Community (onestreamsoftware.com)):
And my data adapter set up:
Any direction would be very much appreciated!!
Some thoughts:
1. FdxExecuteCubeViewTimePivot might get you closer with periods in your columns but 16mil/12 is still more than 1mil so probably not helpful
2. Have you tried the other FDX options that don't rely on a cube view? E.g., BRapi.Import.Data.FDXExecuteDataUnit or BRapi.Import.Data.FDXExecuteDataUnitTimePivot? Not sure what their limits are.
