wiki:quintagroup.transmogrifier

Context Navigation

Version 6 (modified by olha, 13 years ago) (diff)
--

quintagroup.transmogrifier

quintagroup.transmogrifier
1. Links
Content migration from Plone 2.1 to Plone 3.2

Content migration from Plone 2.1 to Plone 3.2

Before initiating migration you have to create buildouts for old Plone 2.1 site and new Plone 3.2 site. These buildouts should include some extra packages, since we will use a special tool - transmogrifier for content migration. See the following explanations.

Create buildout for Plone 2.1 site

Plone 2.1-based websites were not managed by buildout. Zope instances were created manually and Plone 2.1 archive with all required products was simply extracted into the Products subdirectory. That's why old Zope instance should be transformed to buildout. To accomplish this - do the following steps:

Our buildouts require Python 2.4 with several additional packages. If you don't have this version of Python, you will have to install it. Find more about this at http://plone.org/documentation/tutorial/buildout/prerequisites page.

Download libxml2-python package from ftp://xmlsoft.org/libxml2/python/libxml2-python-2.6.21.tar.gz, unpack and install it:
```
tar -xzf libxml2-python-2.6.21.tar.gz
cd libxml2-python-2.6.21
python setup.py install
```

Create a blank buildout:
```
paster create -t plone2.5_buildout
```

where paster - is a special python script (it goes with ZopeSkel bundle), used for creating different Python projects and in our case it creates Plone buildout.

Note the project template given in -t plone2.5_buildout option. We use Plone 2.5 template because there is no template for Plone 2.1 buildout. We can use the existing Plone 2.5 template with minor modifications. After typing the command and pushing Enter button you will need to answer some questions:

project name - directory, where buildout will be created (for example plone21)
path to Zope 2 installation - if you use Zope 2.8 for your Zope instance set this to Zope 2.8 directory, for example ~/zope-2.8.5. If Zope 2.7 is used - leave it blank.
path to the directory, containing Plone products - set this to Products subdirectory of your Zope instance, for example ~/instance/Products

Go to src subdirectory of the created buildout:
```
cd plone21/src
```

and check out required transmogrifier packages from svn repository:

svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/branches/plone-2.1/collective.transmogrifier collective.transmogrifier
svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/branches/plone-2.1/plone.app.transmogrifier plone.app.transmogrifier
svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/branches/plone-2.1/quintagroup.transmogrifier quintagroup.transmogrifier

Then go to Products sudirectory of the buildout and check out Marshall product:

cd ../products
svn co http://svn.plone.org/svn/archetypes/Marshall/tags/1.0.0 Marshall

Go to buildout directory:
```
cd ..
```

and edit buildout.cfg file, removing or commenting plone value in parts option of buildout section and adding transmogrifier packages in required places. Final buildout.cfg might look like this:

[buildout]
parts = 
#    plone
    productdistros
    instance
    zopepy
...
eggs =
    ...
    quintagroup.transmogrifier
...
develop =
    src/collective.transmogrifier
    src/plone.app.transmogrifier
    src/quintagroup.transmogrifier
...
[instance]
...
zcml =
    quintagroup.transmogrifier
...

Additional modification if you dont' haven Zope 2.8:

[zope2]
...
url = http://www.zope.org/Products/Zope/2.8.5/Zope-2.8.5-final.tgz

Run buildout:
```
python bootstrap.py
bin/buildout
```

Copy Data.fs file from Zope instance to buildout:

cp /path/to/instance/var/Data.fs var/filestorage/

Check whether everything works fine by starting Zope:
```
bin/instance fg
```

Content export

Content migration is carried out with transmogrifier - a tool for creating configurable pipelines with sections for transforming some input data. In our case pipeline is used for exporting/importing content; pipeline configuration contains setting for sections, that do all work. quintagroup.transmogrifier package contains default pipeline configuration for content migration. Configuration has ini-syntax, that's why it looks in the same way as buildout configuration.

Now let's get to exporting process. In the browser open the following address: http://localhost:8080; in ZMI go to the Plone site, which content you want to migrate.

Add CMF Setup tool in this site, which will be added with a portal_setup name. Open this tool and on Properties tab in Active site configuration field select Transmogrifier, then click on Update button. Then go to Export tab and from the list of Available Export Steps select Content and click on Export selected steps button. Content exporting process will begin, its progress will be displayed in the terminal, in case Zope was started in fg mode. As a result content will be exported and you'll get content archive in the browser. Save this archive somewhere in a file system add unpack it to some directory. Then add empty quintagroup.transmogrifier-import.txt file in that directory and pack it. This can be done with next commands:

mkdir content
cd content
tar -xzf path/to/setup_tool-xxxxxxxxxxxxxx.tar.gz
touch quintagroup.transmogrifier-import.txt
tar -czf content.tgz quintagroup.transmogrifier-import.txt structure

After that content.tgz is the archive which will be used for importing content.

Creating buildout for Plone 3.2 site

For this buildout use the same Python interpreter as for Plone 2.1 buildout.

Create a blank buildout:
```
paster create -t plone3_buildout
```

Set project name - directory where buildout will be created (for example: plone32). Also, set HTTP port to different value than 8080, if you plan to run two buildouts at the same time (for example 8081).

Add newer versions of products to the buildout that were used in old Plone 2.1 site and were related to content.

Go to src subdirectory of buildout:
```
cd plone32/src
```

and check out required transmogrifier packages from repositories:

cd plone32/src
svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/branches/plone-3.1/collective.transmogrifier collective.transmogrifier
svn co http://svn.plone.org/svn/collective/plone.app.transmogrifier/trunk plone.app.transmogrifier
svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/trunk quintagroup.transmogrifier

Go to the buildout directory
```
    cd ..
```

and edit buildout.cfg file, adding configuration for transmogrifier packages. After this buildout.cfg may look like this:

[buildout]
...
eggs =
    ...
    quintagroup.transmogrifier
...
develop =
    src/collective.transmogrifier
    src/plone.app.transmogrifier
    src/quintagroup.transmogrifier
...
[instance]
...
zcml =
    quintagroup.transmogrifier
...

Run buildout:
```
python bootstrap.py
bin/buildout
```

Start zope:
```
bin/instance fg
```

Content Import

Open the following address http://localhost:8081/ in the browser, add new Plone site and install all required products. Go to portal_setup tool and on Import tab select archive with content and then click on Import uploaded tarball button. Content importing will start, dislpaying its progress in the terminal, in case Zope was started in fg mode.

Migration Configuration

Export and import processes are controlled by configuration files. Default configuration was used in previous sections, which is located in file system - in quintagroup.transmogrifier package. It's registered with ZCML in configure.zcml file of this package:

<configure
    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">
...
    <transmogrifier:registerConfig
        name="export"
        title="Export pipeline configuration"
        description="This is a Plone content export pipeline configuration."
        configuration="export.cfg"
        />

    <transmogrifier:registerConfig
        name="import"
        title="Import pipeline configuration"
        description="This is a Plone content import pipeline configuration."
        configuration="import.cfg"
        />
...
</configure>

You may notice in this configure.zcml fragment that transmogrifier pipeline configuration is registered by transmogrifier:registerConfig ZCML directive, that has 4 parameters:

name - not required (if not provided - configuration will be named as dafault, but it is recommended to provide the name, since name must be unique, name duplication is not accepted);
title - not required;
description - not required;
configuration - required path to file with pipeline configuration.

In this fragment two configurations with names export (export.cfg file) and import (import.cfg file) were registered. They can be overridden by registrating configurations with the same names in overrides.zcml file. quintagroup.transmogrifier.simpleblog2quills and quintagroup.transmgorifier.pfm2pfg packages referenced in "Additional packages" section of this document perform such overriding.

You can also modify configuration in ZMI. For that open the following address: http://localhost:8081/site/@@pipeline-config in browser, replacing site with your site id. In Plone 3.2 you can go to this page from Site setup page http://localhost:8081/site/plone_control_panel (if quintagroup.transmogrifier product was installed by QuickInstaller), where it is named as Content migration. When you click Save button modified configuration is saved in database and will be used instead of default (that registered with ZCML).

You can do a little change in export configuration for storing content not in tarball archive, but in the some file system directory (this is required for big sites migration, because when exporting to archive, it's stored in memory and this will requires its big amount). Configuration of such export is the following:

[transmogrifier]
pipeline =
    sitewalker
    manifestexporter
    fileexporter
    marshaller
    propertiesexporter
    commentsexporter
    datacorrector
    writer
    EXPORTING

[sitewalker]
blueprint = quintagroup.transmogrifier.sitewalker

[manifestexporter]
blueprint = quintagroup.transmogrifier.manifestexporter

[fileexporter]
blueprint = quintagroup.transmogrifier.fileexporter

[marshaller]
blueprint = quintagroup.transmogrifier.marshaller

[propertiesexporter]
blueprint = quintagroup.transmogrifier.propertiesexporter
exclude = title

[commentsexporter]
blueprint = quintagroup.transmogrifier.commentsexporter

[datacorrector]
blueprint = quintagroup.transmogrifier.datacorrector
sources =
    marshall

[writer]
blueprint = quintagroup.transmogrifier.writer
context = directory
path = /path/to/export/directory/
prefix = structure

[EXPORTING]
blueprint = quintagroup.transmogrifier.logger
keys = 
    _type
    _path

The only change is in [writer] section in which two extra options were added - context = directory (determines, that files will be created in file system instead of the default archive storage) and path = /path/to/export/directory/ (directory, where they will be created).

I recommend you to modify configuration only if you have a good idea of what you are doing and how transmogrifier works. Read docs in collective.transmogrifier, plone.app.transmogrifier and quintagroup.transmogrifier packages for more information.

Additional packages

Explained migration procedure works fine if your site contains default Plone content types. For cases when additional Plone products add new content types - their migration should be accomplished in different way. The following are packages used for such cases:

Blog migration from SimpleBlog to Quills:

quintagroup.transmogrifier.simpleblog2quills

Form migration from PloneFromMailer to PloneFormGen:

quintagroup.transmogrifier.pfm2pfg

These two packages are added into buildout in the same way. For example, buildout.cfg with added quintagroup.transmogrifier.pfm2pfg package looks like this:

[buildout]
...
eggs =
    ...
    quintagroup.transmogrifier.pfm2pfg
...
develop =
    src/collective.transmogrifier
    src/plone.app.transmogrifier
    src/quintagroup.transmogrifier
    src/quintagroup.transmogrifier.pfm2pfg
...
[instance]
...
zcml =
    quintagroup.transmogrifier.pfm2pfg
...