File upload issue

radhakrishna
New Contributor

Hi,

We are seeing the below issue on data load process.

we always see the last file uploaded, it seems if you upload another before one is deleted.

If you upload another file you will see only that one and not the rest of the files.

We want to see the all upload files .Please suggest.

Regards,

Rk

1 REPLY 1

JackLacava
Community Manager
Community Manager

The way DataSources work, they will associate a Source ID with each record loaded. This value will be defined in the DataSource configuration, as you can see highlighted here:

JackLacava_0-1675440870749.png

Whenever you load a new file, if the system identifies that the SourceID is the same as one already loaded, it will look at what you've told it to do in the new load.

JackLacava_1-1675441016978.png

By default, with Replace, it will remove all records coming from a previous load with the same Time and Source ID.

So the solution is either to specify different Source ID values for each file (which is what most people will do), or use the Append option (which tends to be bad from an audit perspective).

In order to specify a new SourceID, typically you'd set the Logical Operator for that field to be a Complex Expression, which generates values according to some criteria; the classic example is the Snippet "Make File Name the Source ID", as shown here:

JackLacava_2-1675441280911.png

This way, every time you load a new file, the Source ID will be different, and the new dataset will be added to the old one rather than replacing it.