~
📡
RSS feed
~
✉
torstein.k.johansen @ gmail
~
🐘
@skybert@hachyderm.io
~
🎥
youtube.com/@skybert
Import and especially big data migration is a domain full of dark art and vodoo magic. Here, I've written done some things which will make your life easier when working with Escenic Content Engine imports.
I've created the following command to analyse the import related errors: import-error-log-analyzer
Among other things, it will output things like:
Empty multimedia files.
Images with the wrong colour space (images imported into Escenic Content Engine must use the RGB colour space).
SAX parsing errors
Resulting Escenic XML (after parsing it from the origin XML format) having unknown/unhandled fields++
The script will also suggest shell commands on how to fix some of these errors.
$ cd /var/spoo/escenic/import/my-import-job/error
$ i=0; for el in *.xml; do i=$(( $i + 1 )); mv $el ../new/${i}.xml; done
The ECE importer will fail to import images that uses e.g. the CMYK colour space. If you've got images that come from a print CMS, there are good chances for these to have pictures using CMYK as this is the de facto standard in the print and photography worlds.
It's easy fixing this using ImageMagick, though. I've written a script which converts all pictures to RGB.
The ECE importer will move imported (and failed) images to the archive and error diretories. If you wish to keep them in their original folder, remove the writable bit on the files:
$ find /var/exports/import-data/images/ -iname "*.jpg" | while read f; do chmod 444 "$f"; done
$ xsltproc transform-from-3rd-party-content-to-escenic.xsl 3rd-party-content.xml
$ xmllint --format my-file.xml
Use my import-status script and pass the base directory of your imports:
$ import-status --dir /var/spool/escenic/import/dn
Status of imports in /var/spool/escenic/import/dn
Seconds since last import status: 317
New : 124813 since last: -2529
Error : 1244 since last: 6
Archive: 466398 since last: 2528
Speed : 7 files/s
Estimated finish: 4h 32m
The Escenic Content Engine will fail if you try importing an image with a CMYK colour profile. To remedy this, you can first run my wee script convert-images-from-cmyk-to-rgb-colour-space :
$ convert-images-from-cmyk-to-rgb-colour-space
Press ENTER to convert all images in the current directory and all sub
directories to standard RGB colour space if they're currently CMYK.
To abort, press Ctrl + c now.
Started: Thu Sep 27 16:36:29 IST 2012
Number of pictures in /var/spool/escenic/import/images: 1892
Picture #23 uses CMYK color profile, converting it to RGB ...
Picture #43 uses CMYK color profile, converting it to RGB ...
Picture #45 uses CMYK color profile, converting it to RGB ...
Picture #72 uses CMYK color profile, converting it to RGB ...
Picture #526 uses CMYK color profile, converting it to RGB ...
Picture #651 uses CMYK color profile, converting it to RGB ...
Picture #690 uses CMYK color profile, converting it to RGB ...
Picture #1371 uses CMYK color profile, converting it to RGB ...
Picture #1667 uses CMYK color profile, converting it to RGB ...
Picture #1675 uses CMYK color profile, converting it to RGB ...
Picture #1678 uses CMYK color profile, converting it to RGB ...
Picture #1679 uses CMYK color profile, converting it to RGB ...
Picture #1727 uses CMYK color profile, converting it to RGB ...
Picture #1736 uses CMYK color profile, converting it to RGB ...
Picture #1740 uses CMYK color profile, converting it to RGB ...
Picture #1770 uses CMYK color profile, converting it to RGB ...
Picture #1830 uses CMYK color profile, converting it to RGB ...
Picture #1838 uses CMYK color profile, converting it to RGB ...
Picture #1875 uses CMYK color profile, converting it to RGB ...
Finished: Thu Sep 27 16:38:02 IST 2012