Changes between Version 8 and Version 9 of quintagroup.transmogrifier


Ignore:
Timestamp:
Nov 23, 2010 10:10:13 AM (9 years ago)
Author:
olha
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • quintagroup.transmogrifier

    v8 v9  
    88 * PYPI - http://pypi.python.org/pypi/quintagroup.transmogrifier/ 
    99 
    10 = Content migration from Plone 2.1 to Plone 3.2 = 
     10== How-to guides == 
    1111 
    12 Before initiating migration you have to create buildouts for old Plone 2.1 site and new Plone 3.2 site. These buildouts should include some extra packages, since we will use a special tool - transmogrifier for content migration. See the following explanations. 
     12 * [wiki:quintagroup.transmogrifier/plone2-3 Content migration from Plone 2.1 to Plone 3.2]  
     13 * [wiki:quintagroup.transmogrifier/plone3-4 Content migration from Plone 3 to Plone 4] 
    1314 
    14 == 1. Create buildout for Plone 2.1 site == 
    15  
    16 Plone 2.1-based websites were not managed by buildout. Zope instances were created manually and Plone 2.1 archive with all required products was simply extracted into the Products subdirectory. That's why old Zope instance should be transformed to buildout. To accomplish this - do the following steps: 
    17  
    18 1. Our buildouts require '''Python 2.4''' with several additional packages. If you don't have this version of Python, you will have to install it. Find more about this at http://plone.org/documentation/tutorial/buildout/prerequisites page. 
    19  
    20 2. Download '''libxml2-python''' package from ftp://xmlsoft.org/libxml2/python/libxml2-python-2.6.21.tar.gz, unpack and install it: 
    21 {{{ 
    22 tar -xzf libxml2-python-2.6.21.tar.gz 
    23 cd libxml2-python-2.6.21 
    24 python setup.py install 
    25 }}} 
    26  
    27 3. Create a blank buildout: 
    28 {{{ 
    29 paster create -t plone2.5_buildout 
    30 }}} 
    31 where '''paster''' - is a special python script (it goes with !ZopeSkel bundle), used for creating different Python projects and in our case it creates Plone buildout.  
    32  
    33 Note the project template given in '''-t plone2.5_buildout''' option. We use Plone 2.5 template because there is no template for Plone 2.1 buildout. We can use the existing Plone 2.5 template with minor modifications. After typing the command and pushing ''Enter'' button you will need to answer some questions: 
    34  
    35  * project name - directory, where buildout will be created (for example plone21) 
    36  * path to Zope 2 installation - if you use Zope 2.8 for your Zope instance set this to Zope 2.8 directory, for example ~/zope-2.8.5. If Zope 2.7 is used - leave it blank. 
    37  * path to the directory, containing Plone products - set this to Products subdirectory of your Zope instance, for example ~/instance/Products 
    38  
    39 4. Go to '''src''' subdirectory of the created buildout: 
    40 {{{ 
    41 cd plone21/src 
    42 }}} 
    43 and check out required '''transmogrifier packages''' from svn repository: 
    44 {{{ 
    45 svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/branches/plone-2.1/collective.transmogrifier collective.transmogrifier 
    46 svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/branches/plone-2.1/plone.app.transmogrifier plone.app.transmogrifier 
    47 svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/branches/plone-2.1/quintagroup.transmogrifier quintagroup.transmogrifier 
    48 }}} 
    49 Then go to '''Products''' sudirectory of the buildout and check out '''Marshall''' product: 
    50 {{{ 
    51 cd ../products 
    52 svn co http://svn.plone.org/svn/archetypes/Marshall/tags/1.0.0 Marshall 
    53 }}} 
    54  
    55 5. Go to buildout directory: 
    56 {{{ 
    57 cd .. 
    58 }}} 
    59 and edit '''buildout.cfg''' file, removing or commenting '''plone''' value in '''parts''' option of buildout section and adding transmogrifier packages in required places. Final buildout.cfg might look like this: 
    60 {{{ 
    61 [buildout] 
    62 parts =  
    63 #    plone 
    64     productdistros 
    65     instance 
    66     zopepy 
    67 ... 
    68 eggs = 
    69     ... 
    70     quintagroup.transmogrifier 
    71 ... 
    72 develop = 
    73     src/collective.transmogrifier 
    74     src/plone.app.transmogrifier 
    75     src/quintagroup.transmogrifier 
    76 ... 
    77 [instance] 
    78 ... 
    79 zcml = 
    80     quintagroup.transmogrifier 
    81 ... 
    82 }}} 
    83 Additional modification if you dont' have Zope 2.8: 
    84 {{{ 
    85 [zope2] 
    86 ... 
    87 url = http://www.zope.org/Products/Zope/2.8.5/Zope-2.8.5-final.tgz 
    88 }}} 
    89  
    90 6. Run buildout: 
    91 {{{ 
    92 python bootstrap.py 
    93 bin/buildout 
    94 }}} 
    95  
    96 7. Copy Data.fs file from Zope instance to buildout: 
    97 {{{ 
    98 cp /path/to/instance/var/Data.fs var/filestorage/ 
    99 }}} 
    100  
    101 8. Check whether everything works fine by starting Zope: 
    102 {{{ 
    103 bin/instance fg 
    104 }}} 
    105  
    106 == 2. Content export == 
    107  
    108 Content migration is carried out with '''transmogrifier''' - a tool for creating configurable pipelines with sections for transforming some input data. In our case pipeline is used for exporting/importing content; pipeline configuration contains setting for sections, that do all work. quintagroup.transmogrifier package contains default pipeline configuration for content migration. Configuration has ini-syntax, that's why it looks in the same way as buildout configuration. 
    109  
    110 Now let's get to exporting process. In the browser open the following address: http://localhost:8080; in ZMI go to the Plone site, which content you want to migrate. 
    111  
    112 Add '''CMF Setup tool''' in this site, which will be added with a '''portal_setup''' name. Open this tool and on '''Properties''' tab in '''Active site configuration''' field select '''Transmogrifier''', then click on '''Update''' button. Then go to '''Export''' tab and from the list of '''Available Export Steps''' select '''Content''' and click on '''Export selected steps''' button. Content exporting process will begin, its progress will be displayed in the terminal, in case Zope was started in fg mode. As a result content will be exported and you'll get content archive in the browser. Save this archive somewhere in a file system add unpack it to some directory. Then add empty '''quintagroup.transmogrifier-import.txt''' file in that directory and pack it. This can be done with next commands: 
    113 {{{ 
    114 mkdir content 
    115 cd content 
    116 tar -xzf path/to/setup_tool-xxxxxxxxxxxxxx.tar.gz 
    117 touch quintagroup.transmogrifier-import.txt 
    118 tar -czf content.tgz quintagroup.transmogrifier-import.txt structure 
    119 }}} 
    120 After that '''content.tgz''' is the archive which will be used for importing content. 
    121  
    122 == 3. Create buildout for Plone 3.2 site == 
    123 For this buildout use the same Python interpreter as for Plone 2.1 buildout. 
    124  
    125 1. Create a blank buildout: 
    126 {{{ 
    127 paster create -t plone3_buildout 
    128 }}} 
    129 Set project name - directory where buildout will be created (for example: plone32). Also, set HTTP port to different value than 8080, if you plan to run two buildouts at the same time (for example 8081). 
    130  
    131 2. Add newer versions of products to the buildout that were used in old Plone 2.1 site and were related to content. 
    132  
    133 3. Go to '''src''' subdirectory of buildout: 
    134 {{{ 
    135 cd plone32/src 
    136 }}} 
    137 and check out required '''transmogrifier''' packages from repositories:  
    138 {{{ 
    139 cd plone32/src 
    140 svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/branches/plone-3.1/collective.transmogrifier collective.transmogrifier 
    141 svn co http://svn.plone.org/svn/collective/plone.app.transmogrifier/trunk plone.app.transmogrifier 
    142 svn co http://svn.quintagroup.com/products/quintagroup.transmogrifier/trunk quintagroup.transmogrifier 
    143 }}} 
    144  
    145 4. Go to the buildout directory  
    146 {{{ 
    147     cd .. 
    148 }}} 
    149 and edit '''buildout.cfg''' file, adding configuration for transmogrifier packages. After this buildout.cfg may look like this: 
    150 {{{ 
    151 [buildout] 
    152 ... 
    153 eggs = 
    154     ... 
    155     quintagroup.transmogrifier 
    156 ... 
    157 develop = 
    158     src/collective.transmogrifier 
    159     src/plone.app.transmogrifier 
    160     src/quintagroup.transmogrifier 
    161 ... 
    162 [instance] 
    163 ... 
    164 zcml = 
    165     quintagroup.transmogrifier 
    166 ... 
    167 }}} 
    168  
    169 5. Run buildout: 
    170 {{{ 
    171 python bootstrap.py 
    172 bin/buildout 
    173 }}} 
    174  
    175 6. Start zope: 
    176 {{{ 
    177 bin/instance fg 
    178 }}} 
    179  
    180 == 4. Content Import == 
    181 Open the following address http://localhost:8081/ in the browser, add new Plone site and install all required products. Go to '''portal_setup''' tool and on '''Import''' tab select archive with content and then click on '''Import uploaded tarball''' button. Content importing will start, dislpaying its progress in the terminal, in case Zope was started in fg mode. 
    182  
    183 == Migration Configuration == 
    184 Export and import processes are controlled by configuration files. Default configuration was used in previous sections, which is located in file system  - in '''quintagroup.transmogrifier''' package. It's registered with ZCML in configure.zcml file of this package: 
    185  
    186 {{{ 
    187 <configure 
    188     xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier"> 
    189 ... 
    190     <transmogrifier:registerConfig 
    191         name="export" 
    192         title="Export pipeline configuration" 
    193         description="This is a Plone content export pipeline configuration." 
    194         configuration="export.cfg" 
    195         /> 
    196  
    197     <transmogrifier:registerConfig 
    198         name="import" 
    199         title="Import pipeline configuration" 
    200         description="This is a Plone content import pipeline configuration." 
    201         configuration="import.cfg" 
    202         /> 
    203 ... 
    204 </configure> 
    205 }}} 
    206 You may notice in this configure.zcml fragment that transmogrifier pipeline configuration is registered by '''transmogrifier:registerConfig''' ZCML directive, that has 4 parameters: 
    207  * name - not required (if not provided - configuration will be named as '''default''', but it is recommended to provide the name, since name must be unique, name duplication is not acceptable); 
    208  * title - not required; 
    209  * description - not required; 
    210  * configuration - required path to file with pipeline configuration. 
    211  
    212 In this fragment two configurations with names '''export''' (export.cfg file) and '''import''' (import.cfg file) were registered. They can be overridden by registrating configurations with the same names in '''overrides.zcml''' file. '''quintagroup.transmogrifier.simpleblog2quills''' and '''quintagroup.transmgorifier.pfm2pfg''' packages referenced in "Additional packages" section of this document perform such overriding. 
    213  
    214 You can also modify configuration in ZMI. For that open the following address: http://localhost:8081/site/@@pipeline-config in browser, replacing ''site'' with your site id. In Plone 3.2 you can go to this page from '''Site setup''' page http://localhost:8081/site/plone_control_panel (if quintagroup.transmogrifier product was installed by !QuickInstaller), -> Add-on products configuration -> '''Content migration'''. When you click Save button modified configuration is saved in database and will be used instead of default (that registered with ZCML). 
    215  
    216 You can do a little change in export configuration for storing content not in tarball archive, but in the same file system directory (this is required for big sites migration, because when exporting to archive, it's stored in memory and this will require its big amount). Configuration of such export is the following: 
    217 {{{ 
    218 [transmogrifier] 
    219 pipeline = 
    220     sitewalker 
    221     manifestexporter 
    222     fileexporter 
    223     marshaller 
    224     propertiesexporter 
    225     commentsexporter 
    226     datacorrector 
    227     writer 
    228     EXPORTING 
    229  
    230 [sitewalker] 
    231 blueprint = quintagroup.transmogrifier.sitewalker 
    232  
    233 [manifestexporter] 
    234 blueprint = quintagroup.transmogrifier.manifestexporter 
    235  
    236 [fileexporter] 
    237 blueprint = quintagroup.transmogrifier.fileexporter 
    238  
    239 [marshaller] 
    240 blueprint = quintagroup.transmogrifier.marshaller 
    241  
    242 [propertiesexporter] 
    243 blueprint = quintagroup.transmogrifier.propertiesexporter 
    244 exclude = title 
    245  
    246 [commentsexporter] 
    247 blueprint = quintagroup.transmogrifier.commentsexporter 
    248  
    249 [datacorrector] 
    250 blueprint = quintagroup.transmogrifier.datacorrector 
    251 sources = 
    252     marshall 
    253  
    254 [writer] 
    255 blueprint = quintagroup.transmogrifier.writer 
    256 context = directory 
    257 path = /path/to/export/directory/ 
    258 prefix = structure 
    259  
    260 [EXPORTING] 
    261 blueprint = quintagroup.transmogrifier.logger 
    262 keys =  
    263     _type 
    264     _path 
    265 }}} 
    266 The only change is in '''[writer]''' section in which has two extra options added - '''context = directory''' (determines that files will be created in file system instead of the default archive storage) and '''path = /path/to/export/directory/''' (directory, where they will be created). 
    267  
    268 I recommend you to modify configuration only if you have a good idea of what you are doing and how transmogrifier works. Read docs in collective.transmogrifier, plone.app.transmogrifier and quintagroup.transmogrifier packages for more information. 
    269  
    270 == Additional packages == 
    271  
    272 Explained migration procedure works fine if your site contains default Plone content types. For cases when additional Plone products add new content types - their migration should be accomplished in different way. The following are packages used for such cases: 
    273  
    274 1. Blog migration from !SimpleBlog to Quills: 
    275 [http://svn.quintagroup.com/products/quintagroup.transmogrifier.simpleblog2quills/trunk/ quintagroup.transmogrifier.simpleblog2quills] 
    276  
    277 2. Form migration from !PloneFromMailer to !PloneFormGen: 
    278 [http://svn.quintagroup.com/products/quintagroup.transmogrifier.pfm2pfg/trunk/ quintagroup.transmogrifier.pfm2pfg] 
    279  
    280 These two packages are added into buildout in the same way. For example, buildout.cfg with added quintagroup.transmogrifier.pfm2pfg package looks like this: 
    281 {{{ 
    282 [buildout] 
    283 ... 
    284 eggs = 
    285     ... 
    286     quintagroup.transmogrifier.pfm2pfg 
    287 ... 
    288 develop = 
    289     src/collective.transmogrifier 
    290     src/plone.app.transmogrifier 
    291     src/quintagroup.transmogrifier 
    292     src/quintagroup.transmogrifier.pfm2pfg 
    293 ... 
    294 [instance] 
    295 ... 
    296 zcml = 
    297     quintagroup.transmogrifier.pfm2pfg 
    298 ... 
    299 }}}