How to improve the file server's time-consuming problem of deleting large folders

When deleting large folders in the file server, it is often time-consuming. For example, on less fast storage, deleting a folder that contains 100,000 files may take 1 to 3 minutes. If waiting for the deletion result in the original thread synchronously, it will block the running of the original thread. If deleting the folder asynchronously in another thread, while it does not block the operation of the original thread, for those users who pay attention to the deletion result, they still need to wait for the result of asynchronous operation, so from the user's perspective, the deletion operation is still slow.

How to improve this status

Given that the folder in the file server, from a user's perspective, is invisible to the user, it has achieved the effect of being "deleted". Therefore, if we can hide the display of the folder on the premise that the server successfully deletes the file, and then respond to the user's delete request immediately, finally complete the actual deletion operation, then in the eyes of users, the deletion speed of the server has been greatly improved.

As mentioned earlier, we can rename the folder before actually deleting it, adding a specific pending deletion tag (this could be a predetermined string), and then hide the display of the folder with the pending deletion tag on the server side. Contrary to actual deletion operation, folder renaming is very fast, hence in the user's eye, we rapidly deleted the target object. Afterwards, to ensure the folder to be correctly deleted, we can set up a database, record the actual path of the renamed folder, and then actually delete the object in a specific thread.

Measures to Improve Time-Constant Deletion of Large Folders from File Servers by Raysync

Raysync is a file system based on a self-developed Raysync transfer protocol. It provides a secure and efficient file transfer and synchronization solution for users. For the issue of time consumption for users to delete large folders on the server side, Raysync has well practiced the above idea. Its practice is as follows:

 

Step 1: After receiving the deletion request from the user side, the server side first renames the target folder, adding a specific string starting with .delete as a pending deletion tag.

Step 2: Block the user side display of folders with a pending deletion tag.

Step 3: Respond to the user side deletion request.

Step 4: Record the actual path of the folder to be deleted into a specific database.

Step 5: In dedicated threads, read the database record regularly, completing the actual deletion operation for the folder to be deleted.

Step 6: For each successful folder deletion, delete the corresponding record in the database. For failed deletions, keep the record for the next deletion.

For a folder that contains hundreds of thousands or even millions of files, the Raysync can respond to the user's repair request quickly (usually within 1s) while ensuring the correct deletion of the folder. From a user's perspective, the speed improvement of this completely asynchronous deletion method compared to directly deleting the target folder is significant (from minutes to within 1s).

Conclusion

Deleting large folders on a file server often takes a considerably long time, which impacts the server's performance and efficiency. To address this problem, one can adopt the solution provided by the file transfer software - Raysync. Raysync is renowned for its ultra-high speed and stability, capable of achieving rapid transfer and deletion of large files or folders of several hundred GB or even TB scale.