Introduction

In the previous post I was telling about the synchronization of big size items from Azure BLOB to Sitecore. It worked fine for not very large size unicorn projects and for some big, but at the same time, it crashed on other large size projects.

Investigation

During the investigation I found that the problem happens when:

• There are some issues with items. Parent items or template items are not found. Unicorn adds items to the queue to process them later expecting the template will be loaded after.
• The files are located in one very big folder. Looks like all the files in the folder are loaded to the memory before they start to be processed, and only after it the Unicorn synchronizes them to Sitecore and release from memory.

Implementation

The easiest solution was to create lazy Rainbow.Model.IItemData objects for big files that have metadata data is loaded to memory right away (this is some GUIDs and paths) and the other part loaded only when it’s required (or not to load at all if there is a data issue and the item cannot be synced).

I created it for AzureBlob data storage. To enable this option set useBigFilesLazyLoad to true.

The same improvement can be done to Rainbow.Storage.SerializationFileSystemDataStore - more preciously, ReadItem method of Rainbow.Storage.SerializationFileSystemTree.

Links

The source code of AzureBlob data storage: https://github.com/fpodshivadlov/Rainbow.Storage.AzureBlob