Download all files from a Google Cloud Storage bucket (in OSX)

To download all files from a Google Cloud Storage bucket (if you are using Google Cloud services like App Engine etc) is actually quite easy but you’ll need to use gsutil to do it. See the docs at https://cloud.google.com/storage/docs/gsutil. You will need the Google Cloud SDK installed so you can run “gcloud”.

I’m using OSX and did the following:

Download gsutil from https://storage.googleapis.com/pub/gsutil.tar.gz

Extract to your home directory:

tar xfz gsutil.tar.gz -C $HOME

Go to $HOME/gsutil and check the version, it may ask if you want to update Google Cloud components:

gsutil version -l

If you need to, update Google Cloud components:

gcloud components update

Authenticate with your Google Cloud account:

gcloud auth login

Set the correct project (in this case “project101″):

gcloud config set project project101

Copy everything recursively from the bucket “bucketname” to the directory “bucketname”:

gsutil -m cp -R gs://bucketname

Done!

Installing a Network Shared HP LaserJet 4050 in Windows 7 x64

This is fairly specific but was difficult to find a fix for. The HP universal print driver cannot be used when installing a LaserJet 4050 that is not attached directly to your PC. The problem is down to the plug-n-play ID for the printer not being found and the driver not installing correctly. Thanks to the guys on the HP Support Forums I found a solution.

Initially I tried just installing the universal printer driver and adding a network printer (with the correct machine and printer name) but it couldn’t find the driver. I even manually selected the download folder for the driver “C:\HP Universal Print Driver\pcl5-x64-5.5.0.12834″ but it wouldn’t have it.

Next I tried adding a local printer on LPT1 (with no physical printer present) and using the same driver. This didn’t work.

The solution is to add a local printer using LPT1 then click the “Windows Update” button, wait a few minutes then select Manufacturer “HP” and Printers “HP LaserJet 4050 Series PCL 5″. This will install a working driver, which is then used automatically when you add a network printer.

Quickly add a Virtual Host to Apache Tomcat to Map URLs to Java Web Applications

Adding a virtual host to Apache Tomcat is really easy and just involves pointing the address at the right directory. This means you can map web addresses pointing at your server IP to Java web applications running on Apache Tomcat. This assumes you already have a hostname such as “websiteaddress.com” pointing to your server with an IP (for example) of 1.2.3.4.

Just modify you server.xml file (in our Ubuntu Server tomcat7 setup in “/usr/share/tomcat7/conf/server.xml”) to include the following at the end of the file in the “Engine” element after the “Host” element for localhost:


      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html
             Note: The pattern used is equivalent to using pattern="common" -->
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

      </Host>

      <Host name="websiteaddress.com" appBase="/usr/share/tomcat7/webapps/myapplication">
        <Context path="" docBase="."/>
      </Host>

Note here that the web address is http://websiteaddress.com and is being resolved to the application at “/usr/share/tomcat7/webapps/myapplication”. More information about this “Host” element is available at the Apache Tomcat website.

Extracting and Copying Mail and Calendar Appointments from a Corrupted Microsoft Windows Live Mail Installation (Calendar EDB/ESE Database Files)

In order to install another one of the Live series of packages (Movie Maker) on Windows 7 a colleague updated their Window Live Mail package as part of the install process. The install process then hung and crippled the Mail program, stopping it from booting with a useless “Windows Live Mail has stopped working” message.

As they were using POP3 to download mail and not synchronising with the calendar server very often they were worried about losing everything, including all their very important calendar appointments. First step was to try System Restore, which didn’t work (all the system restore points we tried came back with the same error in Windows Live Mail. Next we had to look at manually moving and editing files.

We decided that the next best thing would be to set up Windows Live Mail on another machine before manually copying the data from the corrupted machine over to the new one. You can alternatively just make backups of your data and try uninstalling/re-installing Windows Live Mail again.

Mail retrieval from the existing install is easy as all you need to do is copy some physical files across from the USERNAME C:\Users directory. Just move the subfolders (containing mail) from “C:\Users\USENAME\AppData\Local\Microsoft\Windows Live Mail” to the other machine with a good copy of Windows Live Mail. You will need to set up your email retrieval settings again but your old mail should just appear in all the correct folders.

Retrieval of a corrupted calendar is a lot more tricky and needs some free third party tools in order to work. Despite the availability of several (very poor) tools I couldn’t find a way of extracting the calendar data in a format that could be easily imported into the working Windows Live Mail program. Most tools simply refused to open the data file. As a result I had to extract the data and my colleague had to manually enter the appointments again (which is still better than losing everything). I went a step further and wrote a little PHP script to display the data more easily so they didn’t have as hard time of it.

By far the most success I had was with NirSoft ESEDatabaseView which could open the corrupted Live Mail Calendar database file where all other programs failed. The “WLCalendarStore.edb” file containing the database of calendar appointments was found at “C:\Users\USERNAME\AppData\Local\Microsoft\Windows Live Mail\Calendars\DBStore\WLCalendarStore.edb”. I downloaded ESEDatabaseView and ran the executable from the zip. Then I opened the corrupted “WLCalendarStore.edb” and selected the “calendarItem” table from the dropdown. Now this is great, but everything is in HEX format and needs to be converted to normal text!

I first extracted the HEX encoded CalendarItem data to CSV (click on an item in the list of CalendarItem, ctrl-A, ctrl-S then Save as Type “Comma Delimited Text File (*.csv)”). It’s up to you how best to convert this HEX information from the output CSV but I used the following PHP script to convert and display the easy to understand “ServerIcal” column which was output as column 17 in the CSV. Note that I have two parts to this PHP, one part that exports nicely human readable data and the other that just outputs the raw iCal style data:


<?php

// helper function to convert hex value to string
function hex2str($hex) {
    for($i=0;$i<strlen($hex);$i+=2) $str .= chr(hexdec(substr($hex,$i,2)));
    return $str;
}

// show appointments as easy to read HTML
echo "<h1>Appointments</h1>";

if (($handle = fopen("calendaritem.csv", "r")) !== FALSE) {
    while (($data = fgetcsv($handle, 1000000, ",")) !== FALSE) {

			// get the ServerIcal column
			$ical = $data[17];

			// strip out all '00' and ' ' strings from the output
			$ical = str_replace(" 00 ","",$ical);
			$ical = str_replace(" ","",$ical);

			// convert to string
			$line = hex2str($ical);

			// tidy up HTML to make it easily human readable
			$line = substr($line,strpos($line,"DTSTART"));
			$line = substr($line,0,strpos($line,"UID"));
			$line = str_replace("SUMMARY:","SUMMARY: <strong>",$line);
			$line = str_replace(PHP_EOL,"</strong>" .PHP_EOL,$line);
			$line = str_replace("DTSTART;VALUE=DATE:","DTSTART;VALUE=DATE: <strong>",$line);
			$line = str_replace(" DTEND;","</strong>  DTEND;",$line);
			$line = str_replace("DTEND;VALUE=DATE:","DTEND;VALUE=DATE: <strong>",$line);
			$line = str_replace("  SUMMARY: ","</strong>   SUMMARY: ",$line);
			$line = str_replace("DTSTART;VALUE=DATE:","Start Date:",$line);
			$line = str_replace("DTEND;VALUE=DATE:","End Date:",$line);

			// output as HTML
			echo "<br/>$line";
        }

    fclose($handle);
}

// show raw appointment data
echo "<h1>Raw Data</h1>";

if (($handle = fopen("calendaritem.csv", "r")) !== FALSE) {
    while (($data = fgetcsv($handle, 1000000, ",")) !== FALSE) {

			// get the ServerIcal column
            $ical = $data[17];

			// strip out all '00' and ' ' strings from the output
			$ical = str_replace(" 00 ","",$ical);
			$ical = str_replace(" ","",$ical);

			// convert to string
			$line = hex2str($ical);

			// output as HTML
			echo "<br/><br/>$line";
        }

    fclose($handle);
}

?>

The output as HTML was easily readable enough that my colleague could manually enter all the appointments again.

Obviously I would much prefer to output from the database to something that could be directly imported but this functionality isn’t available in Windows Live Mail and the database was so corrupted we couldn’t even just copy it over to the new machine. My colleague is now looking at some of the many alternatives to POP3 and manual sync of calendar items using Windows Live Mail.

Quickly and Safely Move a Microsoft SQL Server Database (MDF and LDF Files) to a New Physical Location (Including Setting Read Write Access)

To move a Microsoft SQL Server database to a new physical location you need to detach, copy, reattach and set permissions for the database MDF and log LDF files associated with the database. I needed to do it as the drive on which each type of database required file (MDF/LDF) was located was to be separately backed up, as per the site backup regulations. I was using SQL Server 2008 R2.

Microsoft’s recommended way of doing this is to use SQL Management Studio. Create a query and detach your database “mydb” by entering and running the following:

use master
go
sp_detach_db ‘mydb’
go

Now copy the MDF and LDF files to their new location and reattach (locations and file names are for this example only):

use master
go
sp_attach_db ‘mydb’,’E:\DATA\mydb.mdf’,’F:\DATA\mydb_log.ldf’
go

You can check the basic properties of the database (and that the file locations have been correctly set) using:

use mydb
go
sp_helpfile
go

The final thing I needed to do was to set the file/folder permissions so that the database could go from read-only to read/write. I first set the folder permissions for my two new folders (“E:\DATA\” and “F:\DATA\” as above). To do this I needed to add the following user to the security settings with full control:

sqlservermssqluser$COMPUTERNAME$mssqlserver

This didn’t actually set the files (server permissions error) so I set the permissions for the “E:\DATA\mydb.mdf” and “F:\DATA\mydb_log.ldf” files individually the same way.

Now open up another query window and type the following to enable read/write access to the database:

use master
go
alter database mydb set read_write with no_wait
go

Now your database is set up exactly as it was previously, only the associated files have moved physical location.

Run .bat batch and .cmd files as scheduled tasks in Windows with a local user (avoid the “Could not start” error)

Running scheduled tasks as a local user means you can lock down user permissions and avoid giving broad admin rights to your local users. I have a scheduled task that needed to be run by a local user by running a .cmd (.bat works as well) batch file every day.

I created the local user with a password and added the scheduled task to run my .cmd file every day at 4am. When adding the scheduled task I put in the correct user details and password and then tried to run it, which failed with a “could not start” error.

The reason for this is that by default, new local users do not have read and execute permissions on “cmd.exe” which is used by Windows task scheduler to start .cmd and .bat files in scheduled tasks. The fix is to navigate to your “system32″ directory (probably “c:\windows\system32″) and right click on the “cmd.exe” application, go to the security tab and add your new local user with “Read & Execute” permissions.

Once the security settings for “cmd.exe” are set to allow your local user to run it, the task scheduler will now allow your .cmd/.bat scheduled task to run with that local user and everything will work fine.

Use Google PageSpeed Insights and WebPagetest to benchmark your website speed and get performance suggestions for free

As part of optimising the speed of this site I found a link to Google PageSpeed Insights from their Webmaster Tools page. This is a free service and gives you some great feedback on high, medium and low priority changes you can make to your site to improve response and overall user experience. You can use their web interface and just type in a URL to check the PageSpeed score.

For reference, as of today this site got a PageSpeed score of 94 out of 100, which is excellent.

I also found another good free one, WebPagetest, which breaks down the speed of your site from various locations and allows you to really see which part of your site load is slowing things down. You can do this kind of thing in your browser but the location awareness of this one makes it a lot more useful.

Test your Polycom and Tandberg video conference hardware is working from your desktop using Polycom PVX

I needed to test that our (fairly ancient) Tandberg 6000 video conferencing system was working but our portable Polycom unit typically used for this purpose had failed. Trying to find a software alternative to do this is like pulling teeth as Polycom have discontinued their PVX software and replaced it with RealPresence Desktop and a suite of other software with no trial or free edition. The Polycom PVX software is the easiest way of testing our video conferencing hardware using software on the desktop but has not been updated since 2007.

Polycom PVX does still exist out there and a trial version can be downloaded from Polycom themselves (from their support site). You can also find the software to download from a few other sites such as Software Informer. With this trial edition you only get 5 minutes of video but it is enough to connect to your hardware and check everything is working. The interface is awful but it does work and the software connected to our Tandberg system fine using my computer’s webcam and mic etc.

Moving from wordpress.com to wordpress.org

I finally made the decision to move my blog to my own hosting after a few years of hosting on wordpress.com. There are a lot of good reasons to use wordpress.com such as the fantastic “googleability” of wordpress.com – it seems like search engines are excellent at searching through blogs hosted there. The disadvantage is that you are heavily restricted as to what you can do and you have to pay extra for a lot of features that I think should come standard.

First I downloaded wordpress, set up my database and ran the install to set up a wordpress.org installation on my own server. Then I went in to the admin panel in my wordpress.com site and went to “Store” then “Site Redirect”. It cost $13 for a year of redirection and I pointed the existing jamesrossiter.wordpress.com site at my new site blog.jamesrossiter.co.uk. This is a permanent redirect (302) so Google should pick up on this and update the listings for my site.

I then exported my site from wordpress.com by going to “Tools” -> “Export” and imported into my new site at blog.jamesrossiter.co.uk using “Tools” -> “Import” which included all the comments on the blog..

Finally, in order to make sure all my posts were correctly forwarded to the right URL I set the permalinks at blog.jamesrossiter.co.uk to be the same format as those used by blogs at wordpress.com which is “Day and name” and is set in “Settings” -> “Permalinks”.