The insides of AM:A's recommendations engine

December 19, 2023
9 minute read
Like 2

Dislike 0

Acquia Migrate: Accelerate has been open source for 2 months (see the announcement, drupal.org project and tutorial).

Time for a peek behind the curtain now that the entire source code is visible — I already showed how the module actually works, so now it's time for the other crucial part: the "recommendations engine"? 🕵️

Our internal goal was: provide vetted migrations for 80% of the installed non-custom modules of Drupal 7 sites running on Acquia Cloud. We reached this! (For context: the site with least benefit still had 64% coverage, the site with most got to a satisfying 100%.)

What is a "vetted migration"? It's one that somebody on the AM:A team manually tested and either found to be working well out of the box, or contributed an upstream patch for to get to that point. We typically tested with real-world data, but always explored what the multitude of ways was that a module could be used, to ensure a satisfactory Drupal 9 migration for all imaginable Drupal 7 source data.

It was very labor-intensive (I estimated at least 5,000 hours went into this 🤯) to get the recommendations to this point!

`recommendations.json`

Surprise! The engine isn't really an engine, it's essentially just a JSON file (with a JSON schema to validate correctness 🤓). Later, you'll find out how this JSON is actually used.

The recommendations.json file is the source of truth for Drupal 7 → 9 migration recommendations.

This file defines which Drupal 7 modules correspond to which Drupal 9-compatible packages plus which modules within those packages ought to be installed. Only if a migration has "vetted": true set will it get automatically installed on the destination Drupal 9 site.

There also is the ability to specify patches, notes, and more. The goal was to get as many Drupal 7 modules as possible to have a "vetted" migration, which may require patches. Over time, those patches should be committed to the relevant packages, to keep the AM:A maintenance burden manageable.

For example, this defines that the Drupal 7 menu module needs to have the drupal/core package installed, any Drupal 9 version of it, and specifies two modules to install:

        {
            "package": "drupal/core",
            "constraint": "*",
            "install": [
                "menu_link_content",
                "menu_ui"
            ],
            "replaces": {
                "name": "menu"
            },
            "vetted": true
        },

And this specifies that the Drupal 7 field_collection module needs to have the drupal/entity_reference_revisions module installed:

        {
            "package": "drupal/entity_reference_revisions",
            "constraint": "1.8",
            "replaces": {
                "name": "field_collection"
            },
            "install": [
                "entity_reference_revisions"
            ],
            "vetted": true
        },

There are three key files in the recommendations branch of the AM:A repository:

generated.json — generated by a script that checks every single Drupal 7 module on drupal.org to check if a Drupal 9-compatible release is available. Will never include patches or note. And will never have "vetted": true.
curated.json — curated recommendations generated by hand. Entries here supersede the entries in generated.json, and this is where "vetted" migrations will be found.
recommendations.json — the merged end result — with recommendations in curated.json that cover the same Drupal 7 module as a recommendation in generated.json will get priority over the automatically generated recommendation.

"Will the recommendations help my site?"

Per the auto-generated statistics.txt, there are currently:

743 recommendations (curated)
- (of which 693 contrib recommendations)
2091 recommendations (generated)
2529 recommendations (total)
- (of which 280 vetted recommendations)
  - (destination: targeting 189 composer packages)
  - (destination: targeting 270 Drupal modules)
  - (source: migrating from 404 Drupal modules)
  - (source: obsoleting 160 Drupal modules)
- (of which 160 obsolete recommendations)
- (of which 94 patched recommendations)
- (of which ~ 438 generated recommendations overridden by curated recommendations
128 unique patches, or 2.0 (avg) per patched recommendation for a total of 190 (39 max)
- (of which 6 patches are not migration-specific, but necessary bugfixes)

We didn't want to just provide 1:1 migrations, because some best practices have changed. For example, in Drupal 7 there were a wide range of competing file & media management modules in existence. In Drupal 8/9/10, the recommended solution are the Media + Media Library modules in Drupal core. So, we worked to provide migration paths from virtually all of these to Media, putting great care into ensuring each file/media item ends up with the appropriate media type. Our team made significant contributions to the Media Migration module to make that a reality.
In that sense, AM:A and its recommendations is opinionated. But … it's not some quirky opinion: it's the opinion that it will be easier in the long term to maintain and update a Drupal site if it follows Drupal core's recommendations. 😊

How much these hundreds of manually curated recommendations help you, depends on which modules are used in a given Drupal 7 site. Sites that used best practices in Drupal 7 (used widely adopted stable contrib modules, installed updates, did not build bewildering data models) will have great success and will get to >80% migrated within hours. The further a site deviates from this path, the more Migrate API expertise and knowledge about both Drupal 7 and 9 internals will be needed.

The glue between `recommendations.json` and the module: `acli app:new:from:drupal7`

The Acquia CLI (acli) command-line tool gained a new app:new:from:drupal7 command which fulfills the task of a formerly closed source tool. It

inspects a Drupal 7 site to determine the set of modules it has installed and some fundamentals such as where its public files are stored, whether it uses private files (and where those are stored)
uses that set of Drupal 7 modules to look up corresponding recommendations in recommendations.json
uses the recommendations it found to generate a composer.json file for the closest possible Drupal 9 equivalent to the Drupal 7 site it is asked to operate on
… including which modules ought to be installed upon installing Drupal. Only modules with vetted migration paths will end up in there, because unvetted migration paths can completely break migrations.

Keeping recommendations up-to-date

Like I said, it was very labor-intensive to get the recommendations to the point they are currently at. Which is also why the recommendations have an extensive CI pipeline:

As you can see, it verifies:

JSON schema conformance
that every vetted recommendation is installable on PHP 7.4, 8.0 and 8.1
same for unvetted recommendations — which installs 2752 Composer packages 😄 (before Composer 2 was out, AM:A had a crazy work-around for this)
does and end-to-end test where it utilizes acli app:new:from:drupal7 using the recommendations.json being tested, to generate a Drupal 9 site from a Drupal 7 site and then verifies the AM:A module is working (the AM:A module goes even further: it literally installs Drupal 7 using an old Drush version)
… and it automatically informs us of stale recommendations: if >=1 patch of our contributed upstream patches has been committed and a newer release of that module is available, it'll cause this test to show an exclamation point: a warning, because it probably should be updated (although it's harmless not to do so).

We have documentation for how to update curated recommendations, as well as for how to release and use recommendations. Plus, sample merge requests for each.

What about Drupal 10?

The question everybody's asking: why not update AM:A to target Drupal 10 directly?

Well, it'd involve redoing much (but not all) of the >5,000 hours it took to get to this point. Because most migration paths do not have explicit test coverage, and they were written during the PHP 7 era. Drupal 10 requires PHP 8.1, and PHP 8 contains breaking changes.

This is why for now, we recommend to first migrate to Drupal 9 (with or without AM:A) and then update from Drupal 9 to 10.

But … if there is enough interest, I will create a recommendations-10 branch where we can gradually grow a new set of recommendations for Drupal 10 — chime in on the proposed plan if you're interested, I'm looking forward to supporting you! 🤝 😄