Forum Discussion

DanielWillis's avatar
DanielWillis
Valued Contributor
2 years ago

Re: What's the best way to pull 16M+ rows of data

Well I wouldn't forget that you're just trying to get the data out here. You can always sort out the format/structure of it at a later stage.

I guess another option is splitting your extract by one of your dimensions and then putting it back together after (or letting the user do it).

3 Replies

  • NicoleBruno's avatar
    NicoleBruno
    Valued Contributor

    I'm trying to solve without manual user manipulation, not just trying to get the data out. I can do a flat file DM job if I wasn't looking for a one button solution. 

    • DanielWillis's avatar
      DanielWillis
      Valued Contributor

      Yes but my point is your flat file doesn't need to be the end of your process. So if you hit a brick wall with other extraction options you have the option to use one of the imperfect options and manipulate your extracted data either in OS, the target system(s), or somewhere in between.