Extract Row count for Import only stage load
Hello all - Looking to capture the actual # of rows inserted into the StageSourceData, NOT just the # of rows that the api.Parser.ProcessDataTable(si, dt, True, api.ProcessInfo) command tries to inse...
Forum Discussion
I started down that path previously and it wasn't giving me what I was looking for. In the example I am working on here, it breaks down like this:
# of Source Rows - 42738 (when i query the external source)
# of Lines Read - 42738
#of Lines Valid - 42738
However, in the processing log there are 42000+ rows that are showing with the 5 ZP (Zero Suppression). I can confirm these are in fact being suppressed in the staging area because there are only 100 or so rows showing in the import screen.
So, I believe the import itself is working correctly by applying zero suppression, but the processing log and the API.Parser methods are not correctly capturing the # records that are being skipped. If I can get the # of rows being skipped, then I can easily subtract that from the # of lines ready to accurately report how many rows actually got loaded.
api.Parser.LinesSkippedZeroSuppress is returning zero, which I think it should be returning the 42k+ rows that are zero suppressed in the log.
Can you attach screenshot of your log?
I had to edit all the other suppressed rows out of the picture to make it fit, but assume 40k+ were suppressed
can i see the bottom of the log?
is that file loading to multiple periods?
Here is the rest of the log, only loading to 1 period:
------------ Skipped --------------
Lines Blank ....................... 0
Lines Bypassed .................... 0
Lines Bypassed DynaCode ........... 0
Lines Non-Numeric ................. 0
Lines Zero Suppressed ............. 0
Lines Unmarked Value .............. 0
Lines Invalid DataKey.............. 0----------- Duration --------------
1. Load Rule Cache ............... 44 ms
2. Parse + One-To-One Rules....... 31096 ms
3. Load Cache (Recalc/Append)..... 0 ms
4. Transformation Rules .......... 2 ms
5. Delete Existing Stage Data .... 15 ms
6. Delete Rule History ........... 1 ms
7. Write Data Pages To Stage ..... 166 ms
8. Post Summary Target Data ...... 18 ms
9. Post Rule History ............. 6 ms
10. Full Process .................. 31348 ms----------- Data Keys ------------
(Global) - Model | (Global) - 2024M1