Changes between Initial Version and Version 1 of quintagroup.transmogrifier/plone3-4

Timestamp:

Nov 23, 2010 10:11:44 AM (13 years ago)

Author:

olha

Comment:

--

quintagroup.transmogrifier/plone3-4

@@ +1 @@
+= Content migration from Plone 3 to Plone 4 =
+
+[[PageOutline]]
+
+The following example describes content migration from Plone 3.2.3 instance to Plone 4.0 buildout.
+
+-------------------------------------------------------------------------------
+
+Before initiating migration you have to create buildouts for old Plone 2.1 site and new Plone 3.2 site. These buildouts should include some extra packages, since we will use a special tool - transmogrifier for content migration. See the following explanations.
+
+== 1. Create buildout for Plone 2.1 site ==
+
+Plone 2.1-based websites were not managed by buildout. Zope instances were created manually and Plone 2.1 archive with all required products was simply extracted into the Products subdirectory. That's why old Zope instance should be transformed to buildout. To accomplish this - do the following steps:
+
+1. Our buildouts require '''Python 2.4''' with several additional packages. If you don't have this version of Python, you will have to install it. Find more about this at http://plone.org/documentation/tutorial/buildout/prerequisites page.
+
+2. Download '''libxml2-python''' package from ftp://xmlsoft.org/libxml2/python/libxml2-python-2.6.21.tar.gz, unpack and install it:
+{{{
+tar -xzf libxml2-python-2.6.21.tar.gz
+cd libxml2-python-2.6.21
+python setup.py install
+}}}
+
+3. Create a blank buildout:
+{{{
+paster create -t plone2.5_buildout
+}}}
+where '''paster''' - is a special python script (it goes with !ZopeSkel bundle), used for creating different Python projects and in our case it creates Plone buildout. 
+
+Note the project template given in '''-t plone2.5_buildout''' option. We use Plone 2.5 template because there is no template for Plone 2.1 buildout. We can use the existing Plone 2.5 template with minor modifications. After typing the command and pushing ''Enter'' button you will need to answer some questions:
+
+ * project name - directory, where buildout will be created (for example plone21)
+ * path to Zope 2 installation - if you use Zope 2.8 for your Zope instance set this to Zope 2.8 directory, for example ~/zope-2.8.5. If Zope 2.7 is used - leave it blank.
+ * path to the directory, containing Plone products - set this to Products subdirectory of your Zope instance, for example ~/instance/Products
+
+4. Go to '''src''' subdirectory of the created buildout:
+{{{
+cd plone21/src
+}}}
+and check out required '''transmogrifier packages''' from svn repository:
+{{{
+svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/branches/plone-2.1/collective.transmogrifier collective.transmogrifier
+svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/branches/plone-2.1/plone.app.transmogrifier plone.app.transmogrifier
+svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/branches/plone-2.1/quintagroup.transmogrifier quintagroup.transmogrifier
+}}}
+Then go to '''Products''' sudirectory of the buildout and check out '''Marshall''' product:
+{{{
+cd ../products
+svn co http://svn.plone.org/svn/archetypes/Marshall/tags/1.0.0 Marshall
+}}}
+
+5. Go to buildout directory:
+{{{
+cd ..
+}}}
+and edit '''buildout.cfg''' file, removing or commenting '''plone''' value in '''parts''' option of buildout section and adding transmogrifier packages in required places. Final buildout.cfg might look like this:
+{{{
+[buildout]
+parts = 
+#    plone
+    productdistros
+    instance
+    zopepy
+...
+eggs =
+    ...
+    quintagroup.transmogrifier
+...
+develop =
+    src/collective.transmogrifier
+    src/plone.app.transmogrifier
+    src/quintagroup.transmogrifier
+...
+[instance]
+...
+zcml =
+    quintagroup.transmogrifier
+...
+}}}
+Additional modification if you dont' have Zope 2.8:
+{{{
+[zope2]
+...
+url = http://www.zope.org/Products/Zope/2.8.5/Zope-2.8.5-final.tgz
+}}}
+
+6. Run buildout:
+{{{
+python bootstrap.py
+bin/buildout
+}}}
+
+7. Copy Data.fs file from Zope instance to buildout:
+{{{
+cp /path/to/instance/var/Data.fs var/filestorage/
+}}}
+
+8. Check whether everything works fine by starting Zope:
+{{{
+bin/instance fg
+}}}
+
+== 2. Content export ==
+
+Content migration is carried out with '''transmogrifier''' - a tool for creating configurable pipelines with sections for transforming some input data. In our case pipeline is used for exporting/importing content; pipeline configuration contains setting for sections, that do all work. quintagroup.transmogrifier package contains default pipeline configuration for content migration. Configuration has ini-syntax, that's why it looks in the same way as buildout configuration.
+
+Now let's get to exporting process. In the browser open the following address: http://localhost:8080; in ZMI go to the Plone site, which content you want to migrate.
+
+Add '''CMF Setup tool''' in this site, which will be added with a '''portal_setup''' name. Open this tool and on '''Properties''' tab in '''Active site configuration''' field select '''Transmogrifier''', then click on '''Update''' button. Then go to '''Export''' tab and from the list of '''Available Export Steps''' select '''Content''' and click on '''Export selected steps''' button. Content exporting process will begin, its progress will be displayed in the terminal, in case Zope was started in fg mode. As a result content will be exported and you'll get content archive in the browser. Save this archive somewhere in a file system add unpack it to some directory. Then add empty '''quintagroup.transmogrifier-import.txt''' file in that directory and pack it. This can be done with next commands:
+{{{
+mkdir content
+cd content
+tar -xzf path/to/setup_tool-xxxxxxxxxxxxxx.tar.gz
+touch quintagroup.transmogrifier-import.txt
+tar -czf content.tgz quintagroup.transmogrifier-import.txt structure
+}}}
+After that '''content.tgz''' is the archive which will be used for importing content.
+
+== 3. Create buildout for Plone 3.2 site ==
+For this buildout use the same Python interpreter as for Plone 2.1 buildout.
+
+1. Create a blank buildout:
+{{{
+paster create -t plone3_buildout
+}}}
+Set project name - directory where buildout will be created (for example: plone32). Also, set HTTP port to different value than 8080, if you plan to run two buildouts at the same time (for example 8081).
+
+2. Add newer versions of products to the buildout that were used in old Plone 2.1 site and were related to content.
+
+3. Go to '''src''' subdirectory of buildout:
+{{{
+cd plone32/src
+}}}
+and check out required '''transmogrifier''' packages from repositories: 
+{{{
+cd plone32/src
+svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/branches/plone-3.1/collective.transmogrifier collective.transmogrifier
+svn co http://svn.plone.org/svn/collective/plone.app.transmogrifier/trunk plone.app.transmogrifier
+svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/trunk quintagroup.transmogrifier
+}}}
+
+4. Go to the buildout directory 
+{{{
+    cd ..
+}}}
+and edit '''buildout.cfg''' file, adding configuration for transmogrifier packages. After this buildout.cfg may look like this:
+{{{
+[buildout]
+...
+eggs =
+    ...
+    quintagroup.transmogrifier
+...
+develop =
+    src/collective.transmogrifier
+    src/plone.app.transmogrifier
+    src/quintagroup.transmogrifier
+...
+[instance]
+...
+zcml =
+    quintagroup.transmogrifier
+...
+}}}
+
+5. Run buildout:
+{{{
+python bootstrap.py
+bin/buildout
+}}}
+
+6. Start zope:
+{{{
+bin/instance fg
+}}}
+
+== 4. Content Import ==
+Open the following address http://localhost:8081/ in the browser, add new Plone site and install all required products. Go to '''portal_setup''' tool and on '''Import''' tab select archive with content and then click on '''Import uploaded tarball''' button. Content importing will start, dislpaying its progress in the terminal, in case Zope was started in fg mode.
+
+== Migration Configuration ==
+Export and import processes are controlled by configuration files. Default configuration was used in previous sections, which is located in file system  - in '''quintagroup.transmogrifier''' package. It's registered with ZCML in configure.zcml file of this package:
+
+{{{
+<configure
+    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">
+...
+    <transmogrifier:registerConfig
+        name="export"
+        title="Export pipeline configuration"
+        description="This is a Plone content export pipeline configuration."
+        configuration="export.cfg"
+        />
+
+    <transmogrifier:registerConfig
+        name="import"
+        title="Import pipeline configuration"
+        description="This is a Plone content import pipeline configuration."
+        configuration="import.cfg"
+        />
+...
+</configure>
+}}}
+You may notice in this configure.zcml fragment that transmogrifier pipeline configuration is registered by '''transmogrifier:registerConfig''' ZCML directive, that has 4 parameters:
+ * name - not required (if not provided - configuration will be named as '''default''', but it is recommended to provide the name, since name must be unique, name duplication is not acceptable);
+ * title - not required;
+ * description - not required;
+ * configuration - required path to file with pipeline configuration.
+
+In this fragment two configurations with names '''export''' (export.cfg file) and '''import''' (import.cfg file) were registered. They can be overridden by registrating configurations with the same names in '''overrides.zcml''' file. '''quintagroup.transmogrifier.simpleblog2quills''' and '''quintagroup.transmgorifier.pfm2pfg''' packages referenced in "Additional packages" section of this document perform such overriding.
+
+You can also modify configuration in ZMI. For that open the following address: http://localhost:8081/site/@@pipeline-config in browser, replacing ''site'' with your site id. In Plone 3.2 you can go to this page from '''Site setup''' page http://localhost:8081/site/plone_control_panel (if quintagroup.transmogrifier product was installed by !QuickInstaller), -> Add-on products configuration -> '''Content migration'''. When you click Save button modified configuration is saved in database and will be used instead of default (that registered with ZCML).
+
+You can do a little change in export configuration for storing content not in tarball archive, but in the same file system directory (this is required for big sites migration, because when exporting to archive, it's stored in memory and this will require its big amount). Configuration of such export is the following:
+{{{
+[transmogrifier]
+pipeline =
+    sitewalker
+    manifestexporter
+    fileexporter
+    marshaller
+    propertiesexporter
+    commentsexporter
+    datacorrector
+    writer
+    EXPORTING
+
+[sitewalker]
+blueprint = quintagroup.transmogrifier.sitewalker
+
+[manifestexporter]
+blueprint = quintagroup.transmogrifier.manifestexporter
+
+[fileexporter]
+blueprint = quintagroup.transmogrifier.fileexporter
+
+[marshaller]
+blueprint = quintagroup.transmogrifier.marshaller
+
+[propertiesexporter]
+blueprint = quintagroup.transmogrifier.propertiesexporter
+exclude = title
+
+[commentsexporter]
+blueprint = quintagroup.transmogrifier.commentsexporter
+
+[datacorrector]
+blueprint = quintagroup.transmogrifier.datacorrector
+sources =
+    marshall
+
+[writer]
+blueprint = quintagroup.transmogrifier.writer
+context = directory
+path = /path/to/export/directory/
+prefix = structure
+
+[EXPORTING]
+blueprint = quintagroup.transmogrifier.logger
+keys = 
+    _type
+    _path
+}}}
+The only change is in '''[writer]''' section in which has two extra options added - '''context = directory''' (determines that files will be created in file system instead of the default archive storage) and '''path = /path/to/export/directory/''' (directory, where they will be created).
+
+I recommend you to modify configuration only if you have a good idea of what you are doing and how transmogrifier works. Read docs in collective.transmogrifier, plone.app.transmogrifier and quintagroup.transmogrifier packages for more information.
+
+== Additional packages ==
+
+Explained migration procedure works fine if your site contains default Plone content types. For cases when additional Plone products add new content types - their migration should be accomplished in different way. The following are packages used for such cases:
+
+1. Blog migration from !SimpleBlog to Quills:
+[http://svn.quintagroup.com/products/quintagroup.transmogrifier.simpleblog2quills/trunk/ quintagroup.transmogrifier.simpleblog2quills]
+
+2. Form migration from !PloneFromMailer to !PloneFormGen:
+[http://svn.quintagroup.com/products/quintagroup.transmogrifier.pfm2pfg/trunk/ quintagroup.transmogrifier.pfm2pfg]
+
+These two packages are added into buildout in the same way. For example, buildout.cfg with added quintagroup.transmogrifier.pfm2pfg package looks like this:
+{{{
+[buildout]
+...
+eggs =
+    ...
+    quintagroup.transmogrifier.pfm2pfg
+...
+develop =
+    src/collective.transmogrifier
+    src/plone.app.transmogrifier
+    src/quintagroup.transmogrifier
+    src/quintagroup.transmogrifier.pfm2pfg
+...
+[instance]
+...
+zcml =
+    quintagroup.transmogrifier.pfm2pfg
+...
+}}}