Migrating 9 million documents to a new SharePoint platform is a big migration. In this blog you read how you can execute a large migration. This blog is a case study of a migration project which was completed in 2018 for a large international firm.
Scenario outline
A large international firm with over 5.000 employees in the Netherlands used SharePoint 2010 to support document collaboration for their customer engagements. Based on the administration in their ERP system, the solution would create and configure customer- and case sites (for each project) using standard site templates and metadata definitions.
Site creation was performed automatically when a new customer or case was created in the ERP system. Within the same approach permissions were automatically granted for specific sites whenever employees were assigned to certain customers and cases.
As this solution provided the primary location for storing and sharing documents, the utilized storage had grown to 14.5 Terabytes. As the existing software versions were end-of-life, the platform required an update to SharePoint 2013 to be (Cloud) ready for future versions of SharePoint.
Main challenges
Business continuity and scale were the primary challenges to perform the migration to the new platform. As this platform was used daily by all employees, any significant downtime would have an impact on customer facing activities. Furthermore, the initial analysis of the data showed a large amount of URL length exceptions which would cause a problem during the migration.
These were caused by end-users creating in-depth folder structures and/or storing files with large names. Any file with a Url over 260 characters would be blocked from the migration due to its length.
Approximately 40 sites were labelled as large sites, as each of these sites would contain more than 15 GB (and some over 100 GB) in terms of documents and versions.
Finally, due to the type of services that were provided to customers, a full proof of successful migrations was a key requirement.
Solution approach
As Sharegate had already been used successfully in previous migration projects for this organization, there was a preference of using it again.
ETTU was involved in this project to design and execute the migration approach together with the internal customer team. Due to the large number of sites and documents, defining the migration strategy required extensive testing to result in a successful outcome and optimal performance.
Ultimately, the migration strategy consisted of 3 stages:
| Stage | Initial full migration | Recurring incrementals | Final cut-over |
|---|---|---|---|
| Activities | The creation of all sites and the first full migration of all existing content during business hours. | Copy new and changed content since the start of the full migration. The recurring incrementals would looking back at the start of each preceding one making them shorter with each run. | Performed during weekends where each of the four divisions would go live after a joined Go-decision with the business representatives. |
| Timelines | Several months | Several months | 3 weeks |
To support this approach, four instances of Sharegate were installed on 2 separate servers with 2 different accounts on each server. This provided a stable setup with 4 parallel sessions, targeting separate web applications for each division. The Sharegate servers would target separate SharePoint webservers in both the source and destination environments, which were not hit by end-users.
As Sharegate comes with a rich PowerShell module, a script library was created to support all copy, validation and reporting tasks required to perform a migration. The script logic would connect to each site in the source and destination using an input (“to-do”) list with site URL’s and iterate through each library to copy the content to the destination. Completion of each migration job would result in the addition of a row in a migration result report.
This would contain all relevant information for the business to approve the migration, including the source/destination site pairs and migration result in terms of success, warning and errors.
This report would also include an item count measured in both source and destination showing potential difference between the two. The migration team would use this information to focus on resolving issues with items missing in destination.
Working towards the cut-over migration stage, the team collaborated intensively with the business representatives to shorten the URL’s of the file/folder combination exceeding the threshold blocking the migration.
Result & lessons learned
With the final cut-over, the full migration of over 52.000 sites and 9 million documents was completed successfully with minimal disruption of the platform availability.
The scale and characteristics of this migration resulted in some valuable lessons learned:
- Large scale migrations require extensive analysis and testing (especially with large data sets) to define the optimal migration strategy. Choosing the right tooling and extending this to automate as much as possible, is essential to ensure continuity of copying content;
- Be aware of how a collaboration platform can reflect organizational changes, which may result in additional effort to ensure that “hidden” data is also copied over;
- Large scale migrations are not only about copying large amounts of data, but also about the impact on the underlying infrastructure. Something that needs to be considered to avoid capacity issues or data loss during the migration;
- Invest in stakeholders as these know the business best and can support in resolving issues or inform the user groups;
- Due to the amount of data, large scale migrations require a process approach. Many activities are recurring in nature and require continuity, which should be provided in terms of availability within the team and keeping up-to-date on progress and issues;
- Sharegate provided us with a cost-effective migration solution with the flexibility to create a fully automated approach for this scale and to fulfill the reporting requirements.
This article was first published in Dutch on the Wortell website.
