Next-Level Drupal Hosting Migration Tools at Work

From time to time, a web development agency—any agency for that matter—finds itself faced with a huge ask for which the resources, the tools, and the know-how appears to exceed the realm of what’s realistic.

It’s tempting to decline and return to the safe zone of the known, but stepping up to help a client who is in a jam, and committing to meeting a deadline based on determination and a drive to innovate can serve as an opportunity to discover what the team is made of.

Once a successful outcome is achieved, agency capabilities and offerings are catapulted to the next level. 

That was the case for Promet Source, when in 2022, our partner Pantheon approached us with an urgent request to migrate 525 University of California Riverside sites from an existing hosting relationship. 

Our solution: build a net new application that used terminus commands to perform web hosting migration tasks.

Expediting migration via automation

Steps that we automated for purposes of expediting migration processes included: 

1. Backup of the sites prior to migration; 
2. Receipt of an alert when the site archives were backed up and ready for the next step; 
3. Extraction of needed data from the archives, while minimizing required space on the hard drive. Some of the archived sites were eight to nine gigabytes in size;
4. Spin up of a new site on Pantheon hosting;
5. Push files and the database to the new site;
6. Push all environments (dev, test, and life) to the new site;
7. Run cleanup on the new Pantheon site;
8. Uninstall modules;
9. Update configuration;
10. Deploy code updates; and 
11. Run automated visual regression tests so that we knew whether we were successful or if anything failed along the way.

How our team of experts did it

We created a PHP application that runs on an Apache web server that enabled us to initiate backups site by site. When backups were ready, the system informed the PMT endpoint with the associated URL, and saved the files on a server from which the files were readied for migration.

The new tool allowed for the creation of multiple functions within the UI that functioned as a bulk deployment tool that utilized terminus commands as if they were API endpoints for managing the many tasks associated with the migration of a site.

Leveraging the functionality provided by the terminus command line tool, in conjunction with Pantheon custom up streams, enabled for full automation of the site migration onto the Pantheon hosting platform. 

Migrating 15-20 sites simultaneously

We added group IDs to the site’s database so that we could target specific groups, which enabled us to run migrations for 15-20 sites at the same time.

This process also enabled efficient sequencing. As we received a backup on one group, we immediately initiated deployment on the next. 

We set it up so that the process could run itself, and was executed by Cron jobs.

Monitoring and troubleshooting

Results were monitored from a front-end UI, which refreshed every 60 seconds and reported on multiple states.

Visual regression testing during deployment allowed for pixel by pixel screenshot comparisons as sites were migrated. This provided immediate insight and flagged failures that required attention—ensuring that issues were not lost or overlooked within the sheer volume of sites being migrated simultaneously.

High functioning interface

From a single interface and dashboard, we were able to initiate and monitor the status of every migration. 

The interface provided a comprehensive list of sites with color coding to indicate current states in all environments, with red indicating the detection of an error. 

Color bars on the dashboard functioned as clickable links from which we were able to access the sites in order to individually address error indicators. 

Visual Regression Testing (VRT), proved key to success of the project. It enabled us to move forward at a rapid clip, with the confidence that errors at any point in the process would be detected quickly and addressed immediately without ever becoming buried in the flow and requiring complex and time-consuming troubleshooting.

The system allowed for approximately 15 to 20 site migrations in an hour, depending on size. When sites were between 400 and 500 megabytes, the process ran quickly. Sites that exceeded four gigabytes slowed things down.

The ability to effect site migrations in bulk—up to 25 at a time—allowed for the unprecedented levels of efficiency and accuracy. The process drove home the power of our Pantheon partnership, and resulted in a tool that is uniquely positioned to fuel successful migrations in the future.

Interested in putting rapid deployment web hosting migration tools that work for your organization and ignite new digital possibilities?

That’s what we do! Let’s talk.