File import improvements for Migrate 2.4

The Migrate module is the leading tool for migrating data from an external application into Drupal. Migrate has been used to bring many world class sites onto Drupal, including The Economist, Martha Stewart and thousands more. The main theme of the upcoming Migrate 2.4 release is improved file handling on Drupal 7.


Media files and attachments are often the trickiest part of a site migration, and the previous approach taken by Migrate did not help as much as it could.

  1. File migration mappings supported a large and obscure array of arguments, without a clear distinction between options and data, or much clarity on which ones were relevant in any given context.
  2. The available options and behavior of the file field handler and the file destination differed significantly, with no code shared between the two (being in two distinct class hierarchies).
  3. The hard-coded “file functions” provided no mechanism for overriding or extending their behavior.
  4. The file field handler implementation in particular was very convoluted, making it difficult to add enhancements or figure out exactly how it would behave in various circumstances.

For Migrate 2.4, now in the release candidate stage, we were determined to significantly improve the developer experience around file migration.

File classes

The key to addressing these issues was to introduce file classes, more-or-less equivalent to the former file functions (which, truth be told, were never real functions). By implementing the primary tasks of file migration - moving (or creating, or linking to) the actual file, and creating the corresponding file entity - in separate classes, we have achieved better isolation of the different implementations, as well as the ability to share them between the field and destination handlers. Each file class is responsible for interpreting the incoming file representation (such as a URI) and, based on its parameters, returning a corresponding file entity which can serve as the result of a file destination import, or a file field within another entity.

File classes are implementations of MigrateFileInterface, and are required to implement two methods - fields() (documenting any options or subfields they support) and processFile() (which takes some value representing a file or file data, and returns the corresponding file entity). The file field handler and the file destination instantiate their configured file class and have it do its work, so each can focus on its specific work - constructing a field array or managing the file entity. There are three file classes implemented directly in Migrate:

MigrateFileUri is the default file class, and does not need to be specified. It accepts a local file path or a remote URI, and copies (if necessary) the file at that location into the Drupal file system.

MigrateFileBlob accepts a variable containing the actual file data (presumably coming from a database blob field) and saves it to a file in the Drupal file system.

MigrateFileFid is something of a degenerate case, and only applicable to file fields - when you create a separate file migration, and need to link a file field in a later migration to one of the previously-migrated files, you simply pass its fid in the mapping to the file field with this class specified to make the link.

Field documentation

Different options and subfields are applicable to the different file classes. In the original implementation, all of these were mixed together in the static arguments() method, with little help on which ones would work with which “file function”. In the new world, each file class implements its own fields method() to document what options and subfields it supports, and the file field handler and destination class incorporate them into the Migrate UI’s detail pages, alongside the regular fields available for mapping. Thus, once you have selected a particular file class, you only see the options and subfields that are relevant to that file class. In addition, we have documented each option and subfield on, and linked directly to the documentation from the field descriptions:

Migrate web UI


File destinations

To migrate to a file destination (i.e., to create file entities directly from source data), the key points are to map the representation of the file (usually a URI/file path) to the ‘value’ field on the file destination, and to pass the file class as the second argument to the MigrateDestinationFile constructor (optional for MigrateFileUri). The options and subfields supported by the chosen file class can be mapped directly.

fields('f', array('attachmentid', 'attachmentblob', 'filename', 'file_ownerid'));
   $this->source = new MigrateSourceSQL($query);
   $this->destination = new MigrateDestinationFile('file', 'MigrateFileBlob');
   $this->addFieldMapping('value', 'attachmentblob');
   $this->addFieldMapping('destination_file', 'filename');
   $this->addFieldMapping('uid', 'file_ownerid')

File fields

Now, if the blobs we migrated above are referenced in a node’s file field, we can easily reference them by using the MigrateFileFid file class:

addFieldMapping('field_blob_attachment', 'attachmentid')

As you can see, the relevant options and subfields are expressed in the mapping using the new subfield syntax in Migrate 2.4, following the parent field and a colon.

Let’s consider migrating image files directly through the field mappings, with the file entities being automatically created. In this example, the source files are mounted at /mnt/files and the image_filename source field is a file path relative to that directory. The source database also contains alt and title information for each file. Because the default file_class is MigrateFileUri we don’t need to specify it:

$this->addFieldMapping('field_image:alt', 'image_alt');
$this->addFieldMapping('field_image:title', 'image_title');

Try it!

We believe these changes will substantially improve the developer experience in implementing file migrations, as well as making this support more maintainable in the long run. They are, however, a complete rewrite, and given the myriad of real-world scenarios and complexity of the options and subfields supported, we can’t test it all ourselves - please install the Migrate 2.4 release candidate and try it in your migration project. Any bug reports, or improvement suggestions, are welcome in the Migrate issue queue.

It is important to note that these changes are incompatible with the previous file support - if you have existing file migrations (particularly if you use MigrateFileFieldHandler::arguments, which has been removed), you must change your mappings to use the new techniques.

For more information, please see the documentation at