What's the best way to pull 16M+ rows of data
Hello! I have a user group who needs a specific set of data pulled just after close every month in order to populate other reporting tools they use. Here's the formatting they require: I'm ...
Forum Discussion
Some thoughts:
1. FdxExecuteCubeViewTimePivot might get you closer with periods in your columns but 16mil/12 is still more than 1mil so probably not helpful
2. Have you tried the other FDX options that don't rely on a cube view? E.g., BRapi.Import.Data.FDXExecuteDataUnit or BRapi.Import.Data.FDXExecuteDataUnitTimePivot? Not sure what their limits are.
Hi!
Thanks - I don't want periods in the columns though so that wouldn't be an option anyway.
I haven't but that was my next step! Thanks 🙂
Well I wouldn't forget that you're just trying to get the data out here. You can always sort out the format/structure of it at a later stage.
I guess another option is splitting your extract by one of your dimensions and then putting it back together after (or letting the user do it).
I'm trying to solve without manual user manipulation, not just trying to get the data out. I can do a flat file DM job if I wasn't looking for a one button solution.
Yes but my point is your flat file doesn't need to be the end of your process. So if you hit a brick wall with other extraction options you have the option to use one of the imperfect options and manipulate your extracted data either in OS, the target system(s), or somewhere in between.