Forum Discussion

bjornliow's avatar
bjornliow
New Contributor III
4 days ago

Cannot access the file because it is being used by another process

I'm trying to export data via a Data Management step.

I am running into frequent errors stating: 

Error processing Data Management Step '[DM STEP NAME]'. Unable to create or open file '[FILE PATH]'. The process cannot access the file '[FILE PATH]' because it is being used by another process.

My servers were unoccupied as no tasks were running in the background in the task activity log.

I've also tried via DM sequence and selecting my own server which did not help.

 

  • MarcusH's avatar
    MarcusH
    Valued Contributor

    The DM jobs write files to the FileShare which is a folder on your OS server. I have had this error when another user had the file open (Excel file) when the process was being executed. I don't think this would be a problem for text files. Might be worth checking.

    • bjornliow's avatar
      bjornliow
      New Contributor III

      I've renamed the file and have tried with overwriting existing files set to True/False. I'm the only one accessing this file and I've made sure to close them before running the DM step. Still no luck.

  • You don't offer any clues as to why there could be more than one process or user trying to simultaneously access a common file.  Regardless, once you figure out what is causing the simultaneous access, you are going to need to make use of concurrent locking or otherwise ensure no two tasks or users can access the same file.

    A common example of how you can run into the problem you are seeing is if let's say you have a Custom Calculate Data Managment step that is supposed to write some data to a common [FilePath].  That will work perfectly if you only ever run the package for a single Data Unit at a time, or each DU writes to a unique file.  But if run for multiple DU's, you now have the risk of multiple parallel threads all trying to hit the common [FilePath] at the same time.  The solution in this instance is to make use of a OS Global lock when updating the common [FilePath], so that only one thread can update the file at a time.  OS Global locks are automatically dropped should a task fail, and remove any Global data when the overall controlling process completes or fails, making them pretty easy/low risk to work with.

    If you have multiple different users or different processes that could be trying to hit the same common [FilePath], then OS Global locks will not work, as they only work for tasks sharing the same overall task.  At that point, I would suggest writing a "lock" row to a custom SQL table, where if the row exists, you are to treat it as locked.  This option gets trickier, as you need to consider what happens if a task fails or hangs before it releases the lock (deletes the row).

    • bjornliow's avatar
      bjornliow
      New Contributor III

      I'm the only one accessing this file and running an export DM to the file path, I've renamed the file in the DM step and have tried the enabling/disabling overwrite existing files and still no luck.

      File Path:
      [ServerName]\OneStreamShare\FileShare\Applications\[ApplicationName]\DataManagement\Export\[UserName]\[CurrentDate]\[FileName]

      My export files have all dimensions set to Top level except for the following which are at base level:
      Entity, Time, Account, UD1 (Business Line)