D6 to D7 Upgrade Checklist

  1. Complete standard upgrade steps of taking offline and disabling all modules and themes.
  2. Run the ‘Repair’ SQL script agains the database.
  3. Do the D6 to D7 core upgrade process documented at https://www.drupal.org/node/570162. Steps 15 and 16 are critical! So take backups of the sites folder and the database after Step 14.
  4. Do Step 15 and be sure to take a backup. CCK does use:
    • Node Reference and User Reference so needs the References module installing.
    • Content Permissions so needs the Field Permissions module installing.
    • Field Groups so needs the Field Group module installing. Note the lack of the ‘s’ on the new module.
    • (NOT SURE ABOUT THIS? Start with the References module update and take a backup. Then do the References to Entity References update/migration and take another backup. This implies that the Entity References module installing. See the specific migration instruction page at https://www.drupal.org/project/entityreference_migration.)
  5. Do Step 16 and be sure to take a new backup after each module update.
  6. Next to do is the C-N-U theme. See https://c-n-u.eu/wp/2014/01/07/zen-upgrade/ and delete the C-N-U theme, upgrade the Zen theme and create a new C-N-U theme for D7 by copying the STARTERKIT folder up to the Themes level at sites/all/themes. It’s very simple to install following the README.txt file, especially as I’ve saved the normalize.css file from the sites/all/themes/cnu/css folder and the responsive.css file from the sites/all/themes/cnu/css/layouts folder.

Drupal 6 to 7 Take 2

The objective here is to validate the D6 db and get its fields back to pristine condition.

The plan is to:

  1. Build a clean D6 and clone it.
  2. Restore Dec 2010 db to the clone and check the differences between it and the clean D6.
  3. Make another clone of the clean D6 and restore the latest backup to it – taken especially
  4. Run the ‘repair SQL script’ on that and validate against line 2.
  5. If really happy, take a back up of that and restore to Production on UK2.

Then use a locally restored copy of UK2 Production to build a migration script in SQL for D6 to D7 so that the images are not totally lost.

Yet Another D8 Beta Attempt

The point of this is to do a GUI driven install on a Fusion VM

  1. Update the VM Linked Clone
  2. Install Apache 2 but in Debian 7 Wheezy the Rewrite Module is not enabled so do that using a2enmod rewrite, which edits the Apache 2 config file in the process so the Rewrite Module is loaded through subsequent reboots. Then add
    <Directory />
    AllowOverride All
    </Directory>

    to the apache2.conf file so that the rewrite rules in the .htaccess are applied.

Not sure where this one got to, but its safe to say it was not a completed success!

ReWrite Module Tribulations

One of the ‘challenges’ of Drupal 8 is that Clean URLs are set on by default. Which means if the LAMP server is not configured the ReWrite module the the site breaks below the home page with 404 errors.

I haven’t yet got my head around reliably configuring this Apache module in Debian so I am suffering many more install failures than successes.

Drupal Needs Mod_Rewrite in Apache2

This one was a real PITA!

When I got Drupal installed anything beyond the home page threw a ‘404’ error. The reason being is that Drupal needs clean URLs enabling and Debian / Apache2 default hasn’t got it enabled. Much reading and experimenting found this process to work:

  1. Add Options All after the

    <IfModule mod_rewrite.c>
    RewriteEngine on

    section

  2. Uncomment the line # RewriteBase /
  3. Insert the following before the last </IfModule tag>

    # Rewrite old-style URLS of the form 'node.php?id=x':
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{QUERY_STRING} ^id=([^&]+)$
    RewriteRule node.php index.php?q=node/view/%1 [L]

    # Rewrite old-style URLs of the form 'module.php?mod=x':
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{QUERY_STRING} ^mod=([^&]+)$
    RewriteRule module.php index.php?q=%1 [L]

    # Rewrite URLs of the form 'index.php?q=x':
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]

Reference: https://www.drupal.org/node/43783

Installing VMware Tools on Debian 7

  1. Install make and GCC using the Add/Remove Software in the GUI
  2. Do what this tip says about the kernel headers by running the following command with elevated privileges.
    “I encountered this problem when upgrading from 12.10 to 13.04. No matter what I did, the VMware Tools installer couldn’t seem to find the headers. For the record, here’s how I installed the headers:
    sudo apt-get update && sudo apt-get install build-essential linux-headers-$(uname -r)
  3. Then install the VMware tools as per the text file in the distribution.

Digital Ocean Success

After a week of frustration at Drupal 8 Beta failing to install on the Raspberry Pi and Raspian after a couple of initial successes I decided, sat in the Fox with a very good pint of Phoenix Red Ale (that is actually a very pale straw colour and totally delicious), to have a go at installing it on a Digital Ocean Debian 7 32 bit Droplet. Despite the delays due to beer-driven typos, the Digital Ocean web page guide that I have been following worked perfectly! First time in about 45 mins.
Caveat: I haven’t checked below the home page and may need to refer to Drupal Needs Mod_Rewrite in Apache2

So the inescapable conclusion is that something must have changed in Raspbian that makes it Drupal 8 Beta unfriendly? If it passes the CIBA test I’ll try a stable release of Drupal 7 on Raspbian to see what happens. Or I won’t and will continue to invest 1 cent per hour to get to know this version and see if I can skip a generation from D6 to D8. They are making a lot of noise about ease of upgrade, but we’ll see.

‘Resetting’ Raspbian

Loading a new build onto the SD card is really quite straightforward.

  1. Using these steps:
    • Follow this instruction page. (Just remember the em command doesn’t show any progress indication and it takes about and hour and twenty minutes to complete.)
      Update: I have found that using rdisk in the of=/dev/diskX path speeds it up fantastically.
    • Immediately run sudo raspi-config and at the very least expand the root volume to fill the SD card.Followed by sudo bash (needed for this to work properly) and atp-get update && apt-get upgrade to bring it bang up to date.
    • Option: run sudo dpkg-reconfigure tzdata to set the correct time zone.
    • To be able to connect to the GUI later add TightVNC with the command sudo apt-get install tightvncserver.
  2. The next step is to reinstall the LAMP stack.
    (The php installer recommends the installation of mailx, so need to see what that is and add it to process here if necessary?)
    Optional: Additionally I add phpMyAdmin with the command sudo apt-get install phpmyadmin.
  3. The last step is to reinstall the latest version of Drupal 8 (still currently in Beta – v3 at time of writing) following the INSTALL.txt file.
    Issue: Unfortunately this does not provide a working Drupal installation. I think that the many install fails started around the time I switched to this script rather than using VNC and the GUI to install the LAMP stack. Need to go back and try the GUI again. Also look more closely at the suggested and recommended packages for PHP shown by the installer and try installing all of them. Or is this a Raspbian issue? Too many variables!
  4. Update:Now starting to think the failure is more to do with lack of mod_rewrite support, need to nail down how that is reliably implemented across servers.

Building A Droplet LAMP Server

First lesson learned about Digital Ocean; for temporary Droplets using public / private key authentication in SSH for ‘root’ login is not worth it! In fact its a real PITA! But as turning off root access by SSH is strongly recommended there’s really no point anyway. Better to do steps 2 to 7 as a matter of routine.

Once I know that I will keep a Droplet longer term then I will set up an RSA key to authenticate to it using my ‘normal’ account, at least from Braeburn, using the method of configuring public / private key authentication here by running the command ssh-copy-id {remote_host} to install the local machine public key on the remote server.

My build method basically follows this page and this page.

  1. Select the lowest configuration option, nearest available datacentre (London or Amsterdam) and Debian 7.0 x32 as the O/S giving it an appropriate hostname. The root password is emailed, which for a temporary Droplet is fine.
  2. Log in using SSH and use passwd to immediately change the root password to the usual long Rob Roy
  3. Add garrathe as a normal user by running adduser
  4. Configure garrathe as a sudo capable user following the above build page and running visudo and adding the line:

    garrathe ALL=(ALL:ALL) ALL

    in the section

    # User privilege specification
    root ALL=(ALL:ALL) ALL
  5. Log out the root account and log in as me
  6. To make the server more secure open the configuration file

    sudo nano /etc/ssh/sshd_config

    Find the following sections and change the information where applicable:

    Port yyyymmdd
    (use a memorable date e.g. birthday or wedding anniversary)
    Protocol 2
    PermitRootLogin no
    (changing this from yes to no prevents future root login)

    Add these lines to the bottom of the document. AllowUsers will limit login to only the users on that line.

    UseDNS no
    (at this time I’m not sure what this line does?)
    AllowUsers garrathe

  7. Patch using sudo apt-get update to freshen the database of ‘stuff’ followed by an sudo apt-get upgrade to get things patched up to date.
  8. Confirm the name of the packages using apt-cache search {package name}.
  9. Install the AMP using sudo apt-get install apache2 mysql-server php-pear php5-mysql php5 libapache2-mod-php5.
    Info: The last one might be installed by one of the preceding ones and so may be redundante, update when sure.
  10. Confirm that Apache is working by http://{IP address}.
  11. Enter mysql -p in the shell to confirm MySQL is working.
  12. To check whether php is installed and running properly, use sudo nano /var/www/test.php to create a test.php in the /var/www folder with the phpinfo() function exactly as shown:

    # test.php


    Point a browser to http://ip.address/test.php and this should show all your php configuration and default settings.
  13. Optional: To install phpMyAdmin just execute sudo apt-get install phpmyadmin.

Building A Local LAMP Server

This is based on a local VMWare Guest starting with Debian 7.4 (i386) because the server has less than 2GB RAM. Its actually 512MB RAM and 20GB disk, so that it matches the lowest Digital Ocean Droplet configuration. The reason I’m doing this is because I want to move away from CentOS onto Debian longer term.

  1. Download the internet installer as its only about 240MB and boot a new VM from that. Use the text installer UI to install.
  2. Add install options of Web and SQL to the offered list.
  3. Install VMware Tools, but this requires the installation of GCC using apt-get install build-essential.
  4. {Next step is to test Apache and MySQL then update the snapshot}

Image Assist Repair Script

This post will contain a copy of the T-SQL script that I use(d) to ‘upgrade’ the D7 database to show the images that were placed in the D6 database nodes using the retired Contributed Module Image Assist.

Copying Tables Between Databases

To copy tables between database (on the same server) use this syntax:

CREATE TABLE New Table_name
SELECT * FROM Original Table_name;

So my code snippet is:

CREATE TABLE `xfer_db.image`
SELECT * FROM `cnueu_drpldb.image`;
CREATE TABLE `xfer_db.files`
SELECT * FROM `cnueu_drpldb.files`;

Note to self: try always to create snippets that can be run in phpMyAdmin or in the mysql command shell. Hence one reason to use back ticks to enclose field names in MySQL as shown.

The Image Assist Solution

The high level process ought to be this.

  1. Take a backup of the production database cnueu_drpldb to a .sql file named cnueu_drpldb6.sql to explicitly identify the Drupal version.
  2. Restore this to a non-production MySQL server so that the restored database name matches the restored dump file.
  3. Create an ‘export’ database called xfer_db. Extract what the needed from the D6 database by copying the images, files and node_revisions tables from the restored backup of the production database.
  4. Identify the page nodes that have images embedded in them from the Node Revisions table by the presence of the Image Assiste modules HTML string starting [Image_assist and extract the image node IDs from which the Image Assist module embeds the image in the posted node using the image NID value in the string in the node_revisions table. Write the page NID and image NID to the transfer database.
  5. Use the image NID to identify its FID (file ID) in the Files table.
  6. Enumerate the two tables and write the node ID, the file ID, the filepath and other metadata into a new custom table.
  7. Upgrade the D6 non-production website to D7 using my ‘discovered’ upgrade process.
  8. Use that new table to write the file path into the correct node record of a copy of the migrated databse within CSS to place the picture at top right of the poem area in an upgraded D7 version of the website on the non-production webesite.

Sounds simple, but I suspect the development of the necessary T-SQL script will be far from such. I have started a T-SQL techno-blog to capture this and other ‘snippets’, and store tips and reminders too. I will build this ‘Image Repair’ script in a posting in this techno-blog.

Moving Databases

phpMyAdmin includes import and export facilities but the size of the file is limited by a number of differing locations across the system. Its a real PITA to remember, find and change them all each time I run up a new system on a VM.

So I have opted not to use them, but instead to use command line options where possible. (The one exception is our current hosting service UK2, which doesn’t give SSH access to non-business users.) So I have book marked this location that shows how to do it in the mysql command line (inside a shell).

http://stackoverflow.com/questions/859313/import-sql-dump-into-mysql

Then this one shows how to use mysqldump from the shell command line

http://stackoverflow.com/questions/8444108/how-to-use-mysql-dump

The nice thing about both of these is ‘>’ dumps out to a file and ‘<' imports from an existing dump file. Where there is no access to a terminal shell, e.g. with SSH then this is possible in the SQL window of phpMyAdmin as well as within a mysql command shell:
use db_name;
source backup-file.sql;

See http://stackoverflow.com/questions/17666249/how-to-import-a-sql-file-using-the-command-line-in-mysql

It also occurs to me that this last is also the way to run any script from a command line or within phpMyAdmin.

How To Change Hostname

The command to view the hostname in a terminal is simple; just typing hostname and hitting return will show you what the hostname for the system is.

The way to change the hostname is not obvious. There is a configuration file in a folder called /etc/sysconfig called network. By default it contains:

NETWORKING=yes
HOSTNAME=hostname

where hostname is the name you chose while running the original installer, just change the name using nano and save. The change appears to be effected immediately.

Remember: you need to be root to do this.

The Image Assist Dilemma

So the next thing to do is to develop an automated process for replacing the images in those nodes where Image Assist had been used to insert them in D6. I think the process is:

  1. Find the nodes that have images inserted by Image Assist in D6.
    To find the nodes with images inserted using ImageAssiste use:
    SELECT * FROM `node_revisions` WHERE `body` LIKE '[img_assist%'
    where the LIKE operator looks for a pattern in the field. In this instance the string starting with [img_assist where the % is a wildcard so that all are found. A test of the query returns 715 rows, which is rising as we add more posts to the online live site.
  2. Determine the Node ID for the Image Node the image is located in. The Node ID is the next part of the Image Assist string in the Body of the Poem (or other) node.
  3. Determine the file path of the image file. The Image table has three columns, the first two are the Node ID (i.e. the Node that contains the image not the node that the image is embedded in) and the File ID. So using SELECT * FROM `image` WHERE `nid`=1389 returns:

    nid fid image_size
    1389 2608 preview
    1389 2595 _original
    1389 2607 thumbnail

    And looking in the Files table using SELECT * FROM `files` WHERE `fid`=2595 returns the line with the filepath filed containing the path to the original image file on disk.
  4. Insert the image file into the Poem node so that it floats to the right with padding like Image Assist used to do

There is then the issue that the text layout of the poems themselves has been lost too!

Another Gotcha! Permissions and a SQL Solution

I’ve just realised that there is a consequence to the loss of Author (where blank = Anonymous) in the online D6 site when it is upgraded. Anonymous users can at the least change the outline configuration. I need to ‘craft’ some SQL code to ‘inject a user’ into the blank fields in the database.

So looked in W3Schools and will try this on the live database.

UPDATE node_revisions
SET uid=175
WHERE uid=0;

This has set the uid field to 175 when it is 0, which is the anonymous uid and 175 is the uid of a new user account I have set up for this exercise. But that didn’t work, so tried it again using:

UPDATE node
SET uid=175
WHERE uid=0;

And now if the node / revision isn’t set to me or Lynne it is now set to C-N-U, which means that it isn’t Anonymous and therefore editable by anonymous users after the upgrade.

Node Gallery Investigation

This seems to integrate galleries and inline insert but is not intuitive or straightforward and is going to need much investigation and experimentation to understand and document.

And a weekend later I am not much the wiser! And there is a severe lack of documentation on this as an option.

And then somehow I’ve cracked it. Unfortunately no clear idea how. Something to do with node gallery content types?

What is clear is that placing images inline and creating galleries are two separate workflows. Both seem simple enough. I just need to crack floating images to the right of text for Lynne to place them in poems and recipes.

Gallery Options

Now I need a gallery solution and the options at first cast are Media Gallery and Node Gallery.

As I have Media already installed investigate that first but download Node Gallery anyway. When I go to activate Media it reminds me it needs Ctools (Chaos Tools) installing to be able to activate it. Ctools has lots of options so start simple and only activate Ctools and Media itself, which tells me I must activate its File Entity option too. Again there is lots more to investigate.

Media Gallery appears to be mutually exclusive with the Insert Module. If I choose the File upload widget to be Media browser I lose the Insert button, if I choose the File widget I cannot browse the Media Gallery. I will devote a bit more time seeing if they will integrate before disabling and trying Node Gallery. And decide to disable for now.

Activating Node Gallery shows me that I need two more modules Entity and Entityreference. So download those as well.

Colorbox Configuration

Colorbox is a one-shot applies everywhere configuration type module. So far the default Colorbox style is the one I like but I will need to consult. One change to,default configuration is to enable the inline option so the tapping inline images displays the image in color box.

This one will need more experimentation to find any nuances.

Finally! Cracked Insert

Finally, last night, the Insert penny dropped! To get images inline two conditions need to be met:

  1. the Content Type must have a File Upload field included at Home > Structure > Content types > Travel > Manage fields. The example here is the Travel Content type where the label is Add Image.
  2. At the bottom of the Travel settings is (contracted) the Insert control. Expand it and tick the Enable insert button box.

Then a couple of hours of fiddling determined how to hide the Add Image file attachment when viewing. How to pick an image style for the insertion. Now I think I need to learn how to configure styles so that a thumbnail is positioned as desired and is enlarged in Colorbox when tapped.

WYSIWYG Editor Solution Solved

So the new theory is:

  • Install the WYSIWYG module first
  • and the Insert module
  • and then, as WYSIWYG says, the CKEditor libraries package, not “the CKEditor for Drupal module”

First two are straightforward, download, unzip and activate. Then make a new folder sites/all/libraries/ckeditor. And get an error message which I google, to find that the latest stable version of CKEditor is not yet supported by the latest of WYSIWYG!

So ‘downgrade’ to v3.x and now the editor can be configured at admin/config/content/wysiwyg. So we have a content authoring solution but still no simple way of adding images inline as yet, nor of viewing them within the web site.

CKEditor for Drupal Module Not The Solution

Found this article at Cocomore.Com which claims to solve the problem so try that.

After reading this through the article is very useful using IMCE and Lightbox2 as the examples. But in a reply to a comment the author also says that Colorbox is an alternative and that he may well end up preferring Insert. But perhaps more importantly I find in the Wysiwyg configuration page this warning.

CKEditor (Download) Not installed.
Extract the archive and copy its contents into a new folder in the following location:
sites/all/libraries/ckeditor

So the actual library can be found at:
sites/all/libraries/ckeditor/ckeditor.js

Do NOT download the “CKEditor for Drupal” edition.

So now I’m thinking I can uninstall IMCE and CKEditor modules and install the CKEditor library? Need to test tomorrow.

Set Up Zen & C-N-U Theme Procedure

  • Step 16 continued: now would be a good time to turn my attention to the theme, which should be straightforward. Delete the C-N-U theme, upgrade the Zen theme and create a new C-N-U theme for D7 by copying the STARTERKIT folder up to the Themes level at sites/all/themes. It’s very simple to install following the README.txt file, especially as I’ve saved the normalize.css file from the sites/all/themes/cnu/css folder and the responsive.css file from the sites/all/themes/cnu/css/layouts folder.

    So having sorted the C-N-U theme its back to Modules and the image, galleries and inline insertion thereof conundrum. Install Media and Insert modules. The latter at the recommendation of the Drupal Core Image documentation as the solution to inserting images inline. WYSIWYG wants an URL that I currently don’t have as the images don’t seem to be associated with any nodes (and of course there are nodes that now do not have their images!) I’m really starting to miss the D6 Image module and all its bits like ImageAssist!

    Start to explore the Media and Insert modules and find that Media module has the command Rebuild type information for media, so run that but not sure what the success message “All files in the system have been assigned types. Media installation complete.” really means? Can I see the uploaded images now? So next look at Media browser settings to see what clues lie there? Answer, not a lot – still bamboozled!

Complete CCK Fields Migration & Start on Modules Installation

  • Step 15 continued: so install the References module and see if that permits more fields to be migrated from CCL to Core. But that does not add any migrate-able fields to the Structure | Migrate fields page.
  • Step 16 is to upgrade the modules that were not uninstalled and upgrade the theme.

    Start with ACL, Book Access and Forum Access. Forum Access requires Chain Menu Access. Then move on to Views which requires Chaos Tools (aka Ctools).

    ThickBox has been retired and the recommended ‘upgrade’ is Colorbox which is composed of a module and an additional plugin. Also learn it needs the Libraries module from reading the installation instructions in the README.txt file. This doesn’t appear to be working just now and could be a lot to learn, so move on for the sake of getting this whole shebang finished. Then I find I need to install the Colorbox Plugin, so give that a whirl, which involves creating a new folder sites/all/libraries.

    So this brings this upgrade test to the point of identifying a replacement for the D6 Image module. First candidate is CKeditor for WYSIWYG content editing to make bullet and numbered lists, bold and italic much more user friendly. Upload it and run update.php to install it. It took a while to suss show to associate a CKeditor profile with a text format like Filtered HTML but once understood it looks good.

Complete Core Repeat Upgrade

  • Step 7 is to remove default.settings.php in the folder /var/www/sites/default
  • Step 8 is to remove all the core files and folders except the sites folder from /var/www
  • Step 9 is to remove all files from the sites/all/modules folder, which I did earlier
  • Step 10 is to download to a folder outside the web server and install the latest version of Drupal core, which at the time of writing is 7.26 and extract.
  • Step 11 is to re-apply modifications to core files, which does not apply here as I consider it good practice to avoid it where possible, and have done so in so far
  • Step 12 is to make the settings.php file writeable
  • Step 13 is to run update.php on the upgraded installation and get an error that the settings.php file is not writeable. I keep on having to set permissions for everyone not just the pi user (or group), which seems to indicate that apache is not running under the pi user on Raspberry Pi? This is quite slow on the RPi but completes…Successfully with a page of messages – see attached PDF file. But Garland appears broken so switch to Bartik which does work.
  • Step 14 is to take a backup immediately of the upgraded database before attempting anything further. Do this using MYSQLDUMP utility in the shell rather than phpMyAdmin to keep the file on the RPi server.
  • Step 15 is to upgrade the fields, i.e. move stuff out of CCL into the replacements in Core. The first thing to do is to replace the CCL in the sites/all/modulesfolder and run update.php. Looked at the new install facility but it needs FTP access to the server to work, and that is just plain difficult. So do it the manual way in the terminal shell. then I can go to Structure | Migrate field and there is one field available for migration. As D6 included User References and Node References (dunno why) I guess I also need the References module?

Repair Database Then Start To Repeat Upgrade Test

As I know the script will ‘repair’ the database after replacing D6 core with D7 core its worth a wee bit of time to try it before.

So in the MYSQL shell run source DB_repair.sql and see what happens? And nothing seems to have broken! Despite one small error message at the end ERROR 1017 (HY000): Can’t find file: ‘./cnueu_drpldb/d6_date_formats.frm’ (errno: 2).

So next is to repeat the Upgrade process from the web site page https://drupal.org/node/570162.

  • Starting with steps 1 through 5 to get into a ‘clean’ state so to speak
  • Step 6 is disable all non-core modules, and if they are not going to be used in D7 to uninstall them from that tab on the page, like if they are deprecated/retired/discontinued, i.e. Image and associated modules. The list for confirmation of uninstalling is:
    • Advanced Help
    • Image
    • ImageAPI
    • ImageCache
    • ImageField
    • Image Attach
    • Image Gallery
    • Image Import
    • Image assist
    • Thickbox
    • Variables API

    This last one because I have no idea what Variables API is for or why its installed! So click the Uninstall button and…
    I’m confused! The page stated the selected modules have been uninstalled, there are one or two permissions errors for the Image module only but all the modules still appear on the Modules List page but not in the Uninstall page! Not sure how to proceed, so will google this issue before proceeding. It occurred to me, do the modules need to be deleted from the disk to disappear from the Modules List page? And the answer is a resounding “Yes!” Also deactivate, uninstall and delete FileField as that was only installed because ImageFiled required it. This appears to leave the system in the perfect state to be upgraded.

Back To Square One!

So it’s back to,square one. Lessons learnt and needing validation by starting again.

And the lesson? Do what it says in the upgrade steps. My mitigation is that I didn’t really know for sure which Modules needed to go but now I do.

  • Anything to do with the Images Module, which means everything in the Image section and the ImageCache section
  • Also everything on the Other Section as I never use the Advanced Help, Thickbox is deprecated (to be replaced by Colorbox) and I haven’t a clue what the Variables API is for.

So do a complete clean down by deleting everything in /var/www and dropping the database with the idea of starting with a fresh ‘snapshot’ of Production from UK2. But Drupal have released a new security update, so that needs to be applied online first.

So with Drupal 6.30 applied online I took a database back up ready to start over again. Moved it to Raspberry Pi via Pippin and in mysql shell dropped the old cnueu_drupldb database, created it again with the same name and used source localhost.sql to add the content of the latest backup to it. It is now ready for the first rehearsal of the upgrade process I’ve come up with. The only question I haven’t answered yet is “should I run the repair script again the database before I upgrade D6 core to D7 or not?

Module Upgrade

So now that the Zen Upgrade has achieved about 95% of the old site appearance the 80-20 rule says move on to the Modules upgrade.

  1. First task is to uninstall Image and all the associated modules. Then delete their folders in sites/all/modules. Except ImageCache and ImageField won’t uninstall, preventing Image API and FileField from being uninstalled too. (Leave ImageCache and ImageField uninstalling in session overnight. But it still ended up as ‘White screen of Death’!) Thinking back I believe I should have uninstalled and deleted all the D6 unwanted after I’d deactivated them before upgrading to D7. Looking now like a big mistake. May need to go for a re-run! 🙁
    And by rolling back the VM to Snapshot D6 Step 1-5 I proved this to be true. Using uninstall on all the Image and related modules worked fine before the upgrade.
  2. Replace old (D6) ACL, Book Access and Forum Access folders in sites/all/modules folder with new (D7) copies. Run https://c-n-u.eu/wp/update.php to update them in the database. These completed successfully. Then activated ACL, Book Access and Book Access UI, but Forum Access requires the Chain_menu_access Module installing and activating to be activated.
  3. Replace old (D6) Views folders in sites/all/modules folder with new (D7) copies. Run https://c-n-u.eu/wp/update.php to update them in the database. These completed successfully. But before Views and Views UI can be activated Ctools needs to be downloaded, installed and activated, so do so.

Zen Upgrade

After many hours over many evenings I have arrived at the knowledge that most of my branding edits are added to the Normalize.css file in the CSS folder of the CNU theme folder in sites/all/themes folder of the Drupal installation. For now (but I might learn how to ‘normalise’ it later) the banner edit is in the Responsive.css file in the sub-folder Layouts. It’d be nice if I could get all the edits into the Normalize.css file eventually. In the meanwhile…

The mark up for the C-N-U branding as captured is:

/* Insert into Body selector */
background-color:#eefae5; /* C-N-U edit: make all pages pale green */
color:#435444; /* C-N-U edit: sets font color to dark green on all pages */
font-family:Arial,sans-serif; /* C-N-U edit: sets the typeface for the whole site */


a:link {color:#435444;} /* C-N-U edit: Link Dark Green Font Colour */
a:visited {color:#432444;} /* C-N-U edit: Visited Link Very Dark Green Font Colour */
a:hover {color:#438444;} /* C-N-U edit: Hover Over Link Mid Green Font Colour */


color:#438444; /* C-N-U edit: Headings Mid Green Font Colour in H1 and H2 etc class declarations */


/* Insert into menu region classes */
background-color: #daefc9; /* C-N-U Edit: set menu background to slightly darker pale green for menus in Responsive.css file */

First Customisation Experience

The first thing that I discovered was that I could not VNC onto the RPi. A bit of googling revealed that a VNC server is not a standard part of the build. SUDO APT-GET INSTALL TIGHTVNCSERVER quickly downloaded and installed it, followed by VNCSERVER to start the first session running on port 5901. Access from VNC clients on Mac and iOS works beautifully.

Next was to transfer the Drupal file system and database from the VM to it. Wheezy proved to be not at all straightforward. USB devices do not auto amount in the shell and I want to use the GUI as little as possible. SUDO APT-GET INSTALL USBMOUNT adds the needed auto-mount facility. But then I hit the next ‘snag’.

All my transfer devices are formatted in EXFAT for maximum sized files and cross compatibility. But Raspbian does not support it as supplied. However a quick SUDO APT-GET INSTALL EXFAT-FUSE overcame this last obstacle.

Looking back: all the above was not such a good idea, I’ve learned to use FTP via Pippin and just wish I could mount its SAMBA share to make it even easier, but hey-ho that time-bandit has had enough of my ‘leisure’ time for now. Rebuilds have blown away the expat-fuse and usbmount additions.

First Experiences

Assembly is a doddle. It took less than 10 minutes to unpack and assemble the kit ready to connect and power up. The PSU is neat and the SD card was 8GB not the advertised 4GB but it is micro-USB with a snazzily Raspberry logo’ed adapter provided

As I was unsure whether SSH access was possible when it booted into the NOOBS installer I dug out a USB keyboard and mouse and connected it to the HDMI port on the telly. The install of Raspbian (a shortening of Raspberry Debian) Wheezy took less than twenty minutes. I took it upstairs and hooked it up to the network in its working location next to the two NAS boxes.

Access using the SSH command in a Mac Terminal session is easy and the iOS app WebSSH is a delight to use.

(*** Tried it and it don’t work despite much Googling and troubleshooting – tried MOUNT command and editing /ETC/FSTAB ***)

C-N-U Zen Theme Notes

This page is a bunch of snippets from the D6 Zen-based C-N-U Theme CSS file for reference in building a D7 replacement.

body
{
margin: 0;
padding: 10px;
background-color: #eefae5 /* A very pale green */
}

————————————————————————————————-

#header-inner
{
background-image:
url(‘/sites/all/themes/cnu/images/CandyRide.png’);
background-repeat: repeat-x
}

————————————————————————————————-

.breadcrumb /* The path to the current page in the form of a list of links */
{
padding-bottom: 0; /* Undo system.css */
font-size: 75%; /* Make breadcrumb smaller */
}

————————————————————————————————-

h1.title, /* The title of the page */
h2.title, /* Block title or the title of a piece of content when it is given in a list of content */
h3.title /* Comment title */
{
margin: 0;
color: #438444 /* Font dark green */;
}

————————————————————————————————-

/** sidebar-left **/
#sidebar-left
{
}

#sidebar-left-inner
{
background-color: #DAEFC9 /** A mid-green **/
}

————————————————————————————————-

/** sidebar-right **/
#sidebar-right
{
}

#sidebar-right-inner
{
background-color: #DAEFC9 /** A mid-green **/
}

————————————————————————————————-

/** footer **/
#footer
{
}

#footer-inner
{
text-align: center;
font-size: 70%
}

#footer-message /* Wrapper for the footer message from Drupal’s “Site information”
and for any blocks placed in the footer region */
{
}

————————————————————————————————-

#footer-inner
{
text-align: center;
font-size: 70%
}

Module Upgrade Woes

There are two parts to the modules upgrade. The Themes Module and the Contributed Modules. All need to be replaced with D7 versions, which is where the woes began. Thinking that the database corruption was going to be the worst of my problems and once resolved all would be plain sailing I was not expecting to find a bigger problem! Namely that not all our Contributed Modules had made it into the D7 world.

The worst being Images and associated modules that had become the well worn path trodden to inserting an image into the content via Gallery. Post upgrade we are left with the dozens of images still in the web site and no way to view them or exploit them.

Formatting content was a pain needing manual insertion of HTML tags so we have agreed that a WYSIWYG module is needed to make formatting text easier and capable of inserting images inline too. As yet a gallery option has not offered itself either but when it does Thickbox will also need to be replaced by something like Colorbox or similar.

I decided to tackle the Theme Module upgrade first. Fortunately Zen has made it to the D7 world and deploying it was the usual process of download, extract, delete old and move new into place. But that doesn’t apply to the C-N-U sub-theme as that is my customisation. The online documentation for ‘upgrading’ a sub-theme from D6 to D7 is highly confusing and even contradictory so I decided that the cleaner approach was to do it again from scratch. So all my free time has gone on reverse engineering what I did the first time round in the D5 world that did upgrade easily into the D6 world. D7 is such a major upgrade in preparation for D8, HTML5 and responsive web sites for device independence that it didn’t just roll forward this time, hence the conclusion the to start again. I have now got to the point where I have the core ‘branding’ analysed and captured in a web page.

NB: In studying how to do the new C-N-U Zen sub-theme it is becoming clear that much of the good practice that Zen follows has now been written into Drupal itself so a future option may be to write a ‘native’ sub-theme and drop Zen too?

Core Upgrade Success

I now have completed the core upgrade successfully by following the upgrade steps 1 – 13 in https://drupal.org/node/570162 using D7.24 and the latest version of the ‘repair’ script as I now call it.

Step 14 was to take a database backup, substituted by a snapshot. Then step 15 was to Upgrade Fields, meaning migrate data from D6 CCK into D7 Fields within core. (There are some notes in Pages on my iMac I need to insert here later.) And that got me a snapshot of a completed Core Upgrade that I can roll back to in a pinch

Roll-back Script Test

So I need to test this before letting it loose on the live site. So the plan is this:

  1. Take a new backup from live site database
  2. Restore the database from live and do a new snapshot of the C-N-U on D6.29 VM
  3. Run the roll-back SQL script
  4. Validate the site afterwards

If all goes well then I should do the D6 to D7 upgrade again at least once to ‘validate’ at a repeatable process, and that VM can become my ongoing pre-prod system by cleaning out all previous snapshots. Or be transferred to the Raspberry Pi now?

Post-upgrade Notes

Need to allow our users to set their own time zones – upgrade page contains a reminder to set this.

Also default time zone is UTC so need to set it to GMT (Europe/London) to ensure that summer time is applied

D7.24 is out so need to apply that to be compliant! (Mebbes tomorrow?)

This is worth trying on premise at some point:
Upload progress Not enabled
Your server is capable of displaying file upload progress through APC, but it is not enabled. Add apc.rfc1867 = 1 to your php.ini configuration. Alternatively, it is recommended to use PECL uploadprogress, which supports more than one simultaneous upload.

Upgrade Blues Rollback SQL Script

— This script is probably a one-off!
— Created to undo the damage of the original D6 to D7 upgrade to the live site before rehearsing on premise

USE cnueu_drpldb;
ALTER TABLE system DROP INDEX system_list;
ALTER TABLE `url_alias` DROP `alias`;
ALTER TABLE `url_alias` DROP `source`;
ALTER TABLE `menu_router` DROP `delivery_callback`;
ALTER TABLE `menu_router` DROP `context`;
ALTER TABLE `menu_router` DROP `theme_callback`;
ALTER TABLE `menu_router` DROP `theme_arguments`;
DROP TABLE `role_permission`;
DROP TABLE `date_formats`;
DROP TABLE `date_format_locale`;
DROP TABLE `date_format_type`;
RENAME TABLE d6_date_formats TO date_formats, d6_date_format_locale TO date_format_locale;
CREATE TABLE IF NOT EXISTS `actions_aid` (`aid` varchar(255) NOT NULL DEFAULT ‘0’);

And that’s it! Database repaired enough for the update utility to run. What next?

Upgrade Blues!

I am in the process of a long and complicated repair of the web site MySQL database on a pre-prod Drupal VM server because when I attempted this Drupal 6 to 7 upgrade on the live site last year it all went wrong. I managed to get it back but unfortunately it has left a ‘legacy’ of a partly upgraded database that now, following common sense by using a pre-prod environment, won’t upgrade! I am having to invoke each error and write a back-out SQL command and add it to a rollback script and execute it so I can invoke the next error. I grindingly repetitive roundabout! I will post the resulting script here.

Basically this is the process documented here: https://drupal.org/node/570162 but… It all went horribly wrong at step 13! The update.php threw a SQL error and would not complete. After a few hours of investigation I find I am having to invoke each error and write a back-out SQL command and add it to a rollback script and execute it so I can invoke the next error. I grindingly repetitive roundabout!

I will post the resulting script here. After which I will test it on a VM of D6.29 and the latest C-N-U backup to make sure I get a repaired and upgradeable database.

Beware of Ownership

When a Drupal distribution is downloaded and put into place it seems to come with the distributers ownership and permissions.

Ideally in a VM the owner of everything needs to be Apache, so running a chown -Rvf apache:apache * from the root folder of the web site (usually /var/www/html in CentOS) looks like a good idea and habit to get into.

But now we have the Raspberry Pi for on premise that changes to chown -Rvf pi:pi * and the root is /var/www In Raspbian.

Restoring Drupal 6

After last year’s disastrous attempt to upgrade our hosted web site I decided to build a ‘pre-prod’ copy of the site on a VM hosted on my iMac to play with before touching the actual ‘production’ site hosted at UK2.NET again.

It has been a long and arduous process just to build the basic LAMP server because the CentOS 6 (and earlier) distribution doesn’t include the phpMyAdmin tool for MySQL which makes life really hard setting MySQL up. (The sting in the tail is that phpMyAdmin won’t backup or restore databases larger than about 10MB due to a bug that hasn’t yet been addressed by the developers. Crazy!)

But eventually I have managed to get the production web site and database migrated to pre-prod, only to find that while I get the home page okay clicking any link through to another page always throws a 404! ARGH!

It was only with the help of this page https://drupal.org/node/753760 that I found out why, the ‘clean URLs’ feature. And finally an appearance of a document in the Drupal site on how to migrate or restore Drupal here https://drupal.org/node/776864 confirms the need to turn off ‘clean URLs’ before backing up the source web site.

Which makes me wonder “If you have to turn them off every time you want to take a back up, what is the point of them in the first place?” But actually the answer is that my LAMP setup doesn’t support clean URLs for some un fathomable reason that is too low priority to sort out now.

Xerox Phaser 6100 Driver

Apparently there is a new OS X 10.6 driver on the Xerox website at http://www.support.xerox.com/support/phaser-6110mfp/downloads/enus.html?operatingSystem=macosx and this works in Lion too. Also the Samsung CLP 300 driver works as the Phaser 6100 is just a Samsung with Xerox cartridges apparently?