“unblock” an entire directory of files (rather than individually) when copying files between NTFS locations in Windows Server

We copied a few hundred files between a Windows Server 2003 machine and a Windows Server 2008 machine in order to migrate a ASP.NET web application. There are a few hoops to jump through as once it was set up I instantly hit:

Request for the permission of type ‘System.Web.AspNetHostingPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b27b5d561934e089′ failed. (C:\inetpub\wwwroot\WebApp\web.config line 90)

This was related to a “<httpModules>” element in the “Web.config” file.

On searching for similar problems I found a blog post on MSDN which stated that this could be due to a DLL file that needed to be unblocked (right click, “Unblock” button) after copying from another location. Unfortunately, we couldn’t go through and unblock each individual file to try and get round this as there were so many.

The solution (thanks to superuser/StackExchange) is to zip up all the files into one compressed archive before transferring them. Then the NTFS flagging of “unsafe” files that need to be unblocked only includes the one file, simple!

This solved our System.Web.AspNetHostingPermission error straight away, implying that the blocking of files by NTFS for security reasons can affect ASP.NET migration.

Simple and secure MySQL database backup to gzip using mysqldump in Linux

As part of a larger daily backup cron job script I needed to quickly backup my MySQL databases to individual compressed “gzip” .GZ files. The command to do this is very easy, just run the command and pipe it to “gzip”:

mysqldump -u USERNAME -pPASSWORD DATABASENAME | gzip > OUTPUTFILE.gz

This requires you to actually put in the USERNAME and PASSWORD on the command line, which is obviously a bad idea due to logging of commands and other security reasons.

The MySQL recommended way of doing this is to instead use a separate file containing the login details. You use “mysqldump” with the argument “–defaults-extra-file” and specify the location of a configuration file such as “/root/mysqldetails.cnf”. It is a good idea to create this file and “chown” as root and “chmod” it to be “0400″ which will make it read-only by the “root” user.

chown root:root /root/mysqldetails.cnf
chmod 0400 /root/mysqldetails.cnf

The file itself is a very simple text file and just looks something like:

[client]
host = localhost
user = USERNAME
password = PASSWORD

So now this file has been created and the permissions set correctly, the mysqldump command looks like:

mysqldump –defaults-extra-file=/root/mysqldetails.cnf DATABASENAME | gzip > OUTPUTFILE.gz

The result is OUTPUTFILE.gz which is a compressed copy of your DATABASENAME database, without showing anyone the username and password required to access the database. The “mysqldump” command is very useful and more information can be found in the MySQL documentation.

Encrypt a USB drive in linux and automatically mount it on startup using a keyfile and dm_crypt

The easiest way of doing this is to use dm_crypt‘s “cryptsetup” on your USB drive, create a keyfile then set the options in “/etc/fstab” and “/etc/crypttab”. By using a keyfile you can get the drive to automatically mount without having to type in your encryption password. I was doing this on a bare install of CentOS 6.3 but the steps should be similar on other distros with “cryptsetup” installed.

I needed to back up some important (and confidential) files to a USB portable drive that I wanted to encrypt with full disk encryption. You can do this in a variety of ways but the method here was the easiest I found. More information can be found at Brad’s Blog and HowtoForge.

Encrypting and mounting your USB drive

First you need to physically plug in your USB drive to the machine and then unmount it if it automatically mounts. I performed all the commands here using the root user. In my case, when I plugged in the USB drive it was found as “/dev/sdb” and automatically mounted by CentOS. To unmount:

umount /dev/sdb

Now the USB drive needs to be formatted using “cryptsetup” and the “luksFormat” command:

cryptsetup luksFormat /dev/sdb

The tool will give you a warning about overwriting data, which you need to confirm by typing an uppercase “YES”. You then type in and confirm your LUKS passphrase, which will be used to unlock the drive in future. This passphrase is also used later when creating the keyfile.

Now you can create a device mapper for the drive using “cryptsetup” and the “luksOpen” command. I called my mapper “secretvol” in this example so the drive will be mapped to “/dev/mapper/secretvol”. You will be prompted for the passphrase:

cryptsetup luksOpen /dev/sdb secretvol

Now before you can mount your newly mapped device you need to format the file system (I used ext3):

mkfs.ext3 /dev/mapper/secretvol

Now you can mount the USB drive. Make sure you have created the mount point (in my case “/mnt/encrypteddrive”) first then mount it with:

mkdir /mnt/encrypteddrive
mount /dev/mapper/secretvol /mnt/encrypteddrive

To test this all works properly reboot your machine before unlocking and mounting your USB drive manually (requiring entry of the passphrase):

cryptsetup luksOpen /dev/sdb secretvol
mount /dev/mapper/secretvol /mnt/encrypteddrive

To unmount and lock the drive by closing the device mapper with the “luksClose” command:

umount /dev/mapper/secretvol
cryptsetup luksClose secretvol

Creating a keyfile to avoid entering your passphrase manually

A keyfile is good as it means you can unlock your USB drive without having to manually type the passphrase. To create a keyfile “/root/keyfile” for your device using “cryptsetup” and the “luksAddKey” command enter the following (you will need to enter your passphrase). The first command creates a random 4096 byte file, the second makes it read only to root and the third stores your passphrase in the keyfile using “luksAddKey”:

dd if=/dev/urandom of=/root/keyfile bs=1024 count=4
chmod 0400 /root/keyfile
cryptsetup luksAddKey /dev/sdb /root/keyfile

Now you can unlock your previously created drive without manually entering the passphrase using:

cryptsetup luksOpen –key-file /root/keyfile /dev/sdb secretvol

And mount with:

mount /dev/mapper/secretvol /mnt/encrypteddrive

Automatically unlock and mount your encrypted USB drive at system startup

Now that you have a keyfile you can set up your linux install to automatically unlock and mount the USB drive by editing a couple of files.

Edit your “/etc/crypttab” file:

nano /etc/crypttab

Add the line below to add the “/dev/mapper/secretvol” device:

secretvol /dev/sdb /root/keyfile luks

NOTE: You can also use the UUID of your drive in “/etc/crypttab” to make sure that the right disk as detected by the kernel is used. In cases where you may be adding or removing disks this is really important as you may have “sdb” or “sdc” or “sdX” depending on what order the disks are detected by your linux install. To find the right UUID type:

ls -l /dev/disk/by-uuid

Which in my case told me that my UUID for “sdb” (my USB drive) was “6858274d-2370-4377-9426-d786c3e7a410″. The line in “/etc/crypttab” that you should use in this case to add “/dev/mapper/secretvol” is:

secretvol /dev/disk/by-uuid/6858274d-2370-4377-9426-d786c3e7a410 /root/keyfile luks

Now edit your “/etc/fstab” file:

nano /etc/fstab

Add the line below to automatically mount the device to “/mnt/encrypteddrive”:

/dev/mapper/secretvol /mnt/encrypteddrive ext3 defaults 0 2

Now to test this, reboot your machine and navigate to “/mnt/encrypteddrive” where your USB drive will be mounted automatically for you. Easy!

Run .bat batch and .cmd files as scheduled tasks in Windows with a local user (avoid the “Could not start” error)

Running scheduled tasks as a local user means you can lock down user permissions and avoid giving broad admin rights to your local users. I have a scheduled task that needed to be run by a local user by running a .cmd (.bat works as well) batch file every day.

I created the local user with a password and added the scheduled task to run my .cmd file every day at 4am. When adding the scheduled task I put in the correct user details and password and then tried to run it, which failed with a “could not start” error.

The reason for this is that by default, new local users do not have read and execute permissions on “cmd.exe” which is used by Windows task scheduler to start .cmd and .bat files in scheduled tasks. The fix is to navigate to your “system32″ directory (probably “c:\windows\system32″) and right click on the “cmd.exe” application, go to the security tab and add your new local user with “Read & Execute” permissions.

Once the security settings for “cmd.exe” are set to allow your local user to run it, the task scheduler will now allow your .cmd/.bat scheduled task to run with that local user and everything will work fine.

Simple setup of Oracle 11g Release 2 on CentOS 6.3, including pdksh and all dependencies, in VirtualBox

I’ve installed Oracle Database 11g Release 2 a few times on various Linux installs and apart from a few quirks it is a pretty similar process on most. The absolute bare bones default install, as described here, is easy to set up and doesn’t take that long. You can see more detail, including all the recommended steps if you follow the instructions in the Oracle install guide. I will describe installing 32bit Oracle Database 11g Release 2 on CentOS 6.3 32bit with the UI installed so we can use the Oracle installer directly. My computer’s name was “localhost.localdomain” as I was testing this in a development VirtualBox install.

First download Oracle 11g Release 2 from their website. For a linux install it comes as 2 zip files which you must first accept the license for before downloading. The exact version I downloaded was “Oracle Database 11g Release 2 (11.2.0.1.0) for Linux x86″.

Now you need to prepare your CentOS install by adding the required users and user groups for the install process. In my setup I am following oracle and running the following commands to add the “oinstall” and “dba” user groups:

groupadd oinstall
groupadd dba

Now add the “oracle” user, who we will be using to run the Oracle 11g install and give the user the correct group membership:

useradd -g oinstall -G dba oracle

Now create a directory and set the appropriate permissions where you are going to install Oracle. In my case I have installed it in the “oracle” user’s home directory under “/home/oracle/app”:

mkdir -p /home/oracle/app
chown -R oracle:oinstall /home/oracle/app/
chmod -R 775 /home/oracle/app/

Now extract the Oracle ZIP files downloaded earlier into somewhere sensible. I chose “/home/oracle/database”. Navigate to the directory and run the install script as your new oracle user:

su oracle

cd /home/oracle/database
./runInstaller

NOTE: In my case, because this CentOS install was a VirtualBox virtual machine I needed to explicitly set the $DISPLAY variable to the local machine before the UI for the installer would run. This is done by running the following command and restarting my shell:

export DISPLAY=:0.0

Now the installer will start up. You can ignore entering your email in the first step “Configure Security Updates” and leave the default setting of “Create and configure a database” in the second step “Installation Option”.

For the “System Class” step of the install I just left it as the default “Desktop Class” and in the “Typical Installation” step I left everything as default apart from setting the Administrative password. The default settings puts the oracle base in “/home/oracle/app/oracle” with a global database name of “orcl.localdomain”. For the “Create Inventory” step I left the default folder of “/home/oracle/app/oraInventory” and the group name “oinstall”.

Now we get on to the interesting part of the install, which is the “Prerequisite Checks” stage. If you are running the install on a brand new copy of CentOS you will need to set a few system variables and install a set of prerequisites.

NOTE: You may not need to, but I needed to add more swap space to my CentOS install this time around in order to meet the prerequisites. Run the following commands as root to create a 2048mb swap file called “/swapfile” on your harddrive and set CentOS to use it for swap space:

dd if=/dev/zero of=/swapfile bs=1024 count=2097152
mkswap /swapfile
swapon /swapfile

Now set CentOS to always use this swap space at boot by editing your “/etc/fstab” file using the command:

nano /etc/fstab

And add the following line:

/swapfile  swap  swap  defaults  0  0

So if you have passed the swap space test in the “Prerequisite Checks” in the Oracle install you can start to fix all those “Failed” messages. Click on the button “Fix & Check Again” and a window will pop up to tell you about the handy “runfixup.sh” script that will be placed in “/tmp/CVU_11.2.0.1.0_oracle/runfixup.sh”. So in your shell, navigate to the directory as root and run the script:

cd /tmp/CVU_11.2.0.1.0_oracle/
./runfixup.sh

The “runfixup.sh” script will fix all the system variables for you so you don’t need to set them manually. Now all that remains is to fix the dependencies, most of which can be installed using “yum” with the following command:

yum install gcc gcc-c++ compat-libstdc++-33 elfutils-libelf-devel libaio-devel libstdc++-devel unixODBC unixODBC-devel

Now the only remaining prerequisite that causes a “Failed” message is “pdksh-5.2.14″ which has been removed from the CentOS repositories after CentOS 5 (see here). The replacement is “ksh” but if you install this package using “yum install ksh” you will get the same dependency check “Failed” in the Oracle install for “pdksh-5.2.14″ and “ksh” will conflict with “pdksh” if you then go to install it.

The solution is to install “pdksh” manually from RPM, which can be found at a variety of mirrors. I used the following command to install the “pdksh” package:

rpm -q ftp://ftp.pbone.net/mirror/archive.download.redhat.com/pub/redhat/linux/6.1/en/os/i386/RedHat/RPMS/pdksh-5.2.14-1.i386.rpm

Now Oracle should pass all the prerequisite checks and you will see the “Summary” step of the install where you can click the “Finish” button. It may take a while but Oracle Database will install with all the required settings ready for you to use out of the box.

The final step is to execute the configuration scripts as root, which will pop up after you have unlocked any users you might need other than the defaults (you don’t need to though at this stage). The two scripts can be run as follows:

cd /home/oracle/app/oraInventory/
./orainstRoot.sh

cd /home/oracle/app/oracle/product/11.2.0/dbhome_1/
./root.sh

To test your install worked you can log in to the web based management interface for your computer “localhost.localdomain” with the user name “SYS” connecting as “SYSDBA” and using the password you set during the install of Oracle. Remember to open port 1158 on your firewall if you need to:

https://localhost:1158/em/

Now you can start to use Oracle. I highly recommend looking through the documentation from Oracle themselves to help get yourself used to the Oracle way of doing things. There are loads of client applications that can help, like the command line based Oracle Instant Client and the Oracle SQL Developer UI program. Oracle have a lot of good walkthroughs for working with their tools which are available as part of their Learning LIbrary.

Quickly enable NTFS support in CentOS 6.3 using EPEL, yum and ntfs-3g

Enabling NTFS support in CentOS 6.3 is only 2 commands in a shell script and can be done in seconds by installing the EPEL repository and the “ntfs-3g” package.

I needed to transfer some files from my USB drive to CentOS 6.3 (using the USB device option in VMware) and got an error message about an unknown filesystem “ntfs”. The drive I was using was formatted in Windows using the NTFS filesystem and couldn’t be read by my CentOS 6.3 install by default.

First you need to install the Extra Packages for Enterprise Linux (EPEL) repository, which is done by typing the following into the shell as root assuming you are using 32bit CentOS 6.3:

rpm -Uvh http://mirror.overthewire.com.au/pub/epel/6/i386/epel-release-6-7.noarch.rpm

If you are using 64bit CentOS use:

rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-7.noarch.rpm

Accept all the prompts to install the repository to get access to a large number of extra packages for CentOS. You can add the “ntfs-3g” package with the following command:

yum install ntfs-3g

Accept all the prompts and you are done, your NTFS formatted drives can now be read from and written to using CentOS.

Use Google PageSpeed Insights and WebPagetest to benchmark your website speed and get performance suggestions for free

As part of optimising the speed of this site I found a link to Google PageSpeed Insights from their Webmaster Tools page. This is a free service and gives you some great feedback on high, medium and low priority changes you can make to your site to improve response and overall user experience. You can use their web interface and just type in a URL to check the PageSpeed score.

For reference, as of today this site got a PageSpeed score of 94 out of 100, which is excellent.

I also found another good free one, WebPagetest, which breaks down the speed of your site from various locations and allows you to really see which part of your site load is slowing things down. You can do this kind of thing in your browser but the location awareness of this one makes it a lot more useful.

Easily set up and automatically start Apache Tomcat 7 Java web server in Ubuntu Linux

Apache Tomcat is actually easier than the standard Apache webserver to set up, which is great news if you are working with Java based web applications. All you need to do is download it and make sure it starts with whichever linux distribution you are using. Deploying applications in standard WAR format is really easy as well due to the simple web based management interface.

In my case I wanted Tomcat to start with Ubuntu and sit on the default port 8080 so I could have it running alongside my standard Apache webserver for PHP. We were developing a Spring application and used Maven to build and compile to a single deployable WAR file. You must have Java installed and set up for this to work. To check you have Java set up type:

java -version

This should tell you what version of java you have installed (hopefully Java 1.7). You also need to check that the “JAVA_HOME” variable is set by typing:

echo $JAVA_HOME

If you don’t get something like “/usr/lib/jvm/jdk1.7.0_09″ please install Java following my installation instructions in a previous post.

To install Apache Tomcat first of all I downloaded the latest copy of Tomcat 7 from mirrorservice.org using wget run from my home directory:

wget http://www.mirrorservice.org/sites/ftp.apache.org/tomcat/tomcat-7/v7.0.32/bin/apache-tomcat-7.0.32.tar.gz

Please note that the version I downloaded may not be available or there may be a newer version so check http://www.mirrorservice.org/sites/ftp.apache.org/tomcat/tomcat-7/ first before running the wget.

“wget” will download the file, which then needs to be extracted:

tar xvzf apache-tomcat-7.0.32.tar.gz

Now you will have a folder “apache-tomcat-7.0.32″ in your home directory. This needs placing somewhere sensible so copy it to “/usr/share/tomcat7″ using:

sudo mv apache-tomcat-7.0.32/ /usr/share/tomcat7

Now you can test your Tomcat install works with its default settings by starting it up. Note: before you do this you need to set the “JAVA_HOME” variable otherwise you will get errors (see my previous post).

To start up Tomcat navigate to “/usr/share/tomcat7″ and run “startup.sh”:

cd /usr/share/tomcat7

./startup.sh

With the default settings you should now be able to reach your Tomcat server home page by navigating to “http://your.ip.add.ress:8080″ where you should hopefully see the homepage and a nice message saying:

“If you’re seeing this, you’ve successfully installed Tomcat. Congratulations!”

Now we need to set up management users for the manager app so we can easily deploy our WAR files containing our Java web applications. You need to edit “/usr/share/tomcat7/conf/tomcat-users.xml”:

sudo nano /usr/share/tomcat7/conf/tomcat-users.xml

Now add the following lines within the “<tomcat-users>” block to give access to the manager GUI:

<role rolename=”manager-gui”/>
<user username=”MANAGERUSER” password=”YOURPASSWORD” roles=”manager-gui”/>

Now you will be able to log in to the manager GUI at “http://your.ip.add.ress:8080/manager/html” using the login details MANAGERUSER and password YOURPASSWORD. You can deploy applications and generally manage your Tomcat install from here.

The final thing to do is to set up Tomcat so that it starts every time your server starts. This is pretty easy as all you need to do in Ubuntu is edit the “/etc/init.d/tomcat7″ file:

sudo nano /etc/init.d/tomcat7

Now enter the following lines:


# Tomcat auto-start
#
# description: Auto-starts tomcat
# processname: tomcat
# pidfile: /var/run/tomcat.pid

case $1 in
start)
sh /usr/share/tomcat7/bin/startup.sh
;;
stop)
sh /usr/share/tomcat7/bin/shutdown.sh
;;
restart)
sh /usr/share/tomcat7/bin/shutdown.sh
sh /usr/share/tomcat7/bin/startup.sh
;;
esac
exit 0

Set the permissions for the file:

sudo chmod 755 /etc/init.d/tomcat7

Add Tomcat to system startup as a service using the command:

sudo update-rc.d tomcat7 defaults

Now you can test that Tomcat is set up as a service using:

sudo service tomcat7 restart

Now to check everything is working on system startup reboot your machine using:

sudo reboot now

Navigate to “http://your.ip.add.ress:8080″ where the Tomcat home page should appear with no problems. Note: If you are having problems reaching your Tomcat home page make sure you have opened port 8080 on your server’s firewall.

It’s definitely worth reading some of the documentation on Tomcat, plenty of which is linked off your newly installed Tomcat home page. You should now have all you need to deploy your Java web applications as WAR files which is really easy using the manager GUI provided by Tomcat.

Install the latest Java 7 JDK on Ubuntu Linux Server 10.04 without apt-get

I was trying to set up Apache Tomcat on an older server running Ubuntu 10.04 and noticed that Java wasn’t actually installed by default. Also, the licensing agreement with Oracle seems to have changed and it is no longer possible to just use apt-get to install it. You have to manually accept the licensing agreement so even “wget” wont work any more (you just get a “download-fail-XXXXXXX.html” file instead.

First up, you have to go to the Java JDK download page and manually accept the licensing agreement. This must be done from a PC with a UI and a browser so no “wget”. You need to get the correct version, which in my case was the x86 tar.gz version. When this is downloaded you should have a “jdk-7u9-linux-i586.tar.gz” file which then needs to be copied to your Ubuntu server (I copied to my user’s home directory).

Now you have the file on your Ubuntu server you can extract it using:

tar -xvf jdk-7u9-linux-i586.tar.gz

This should give you a directory “jdk1.7.0_09″ which we need to move to somewhere sensible such as “/usr/lib/jvm/jdk1.7.0_09″:

sudo mv jdk1.7.0_09 /usr/lib/jvm/jdk1.7.0_09

Now we need to set up a symbolic link so that we can run Java from everywhere:

 sudo ln -fs /usr/lib/jvm/jdk1.7.0_09/bin/java /usr/bin/java

Now check that Java is all installed correctly by checking the version using:

java -version

Which in this case should give:

java version “1.7.0_09″
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) Client VM (build 23.5-b02, mixed mode)

Now you can set up your JAVA_HOME variable at a system level so other applications can use Java by editing “/etc/environment”:

sudo nano /etc/environment

Now add the following line to point to your newly installed Java:

JAVA_HOME=/usr/lib/jvm/jdk1.7.0_09

Now if you open up a new session (not your currect session) and type “echo $JAVA_HOME” you should see the path “/usr/lib/jvm/jdk1.7.0_09″ which means the variable has been set correctly.

Test your Polycom and Tandberg video conference hardware is working from your desktop using Polycom PVX

I needed to test that our (fairly ancient) Tandberg 6000 video conferencing system was working but our portable Polycom unit typically used for this purpose had failed. Trying to find a software alternative to do this is like pulling teeth as Polycom have discontinued their PVX software and replaced it with RealPresence Desktop and a suite of other software with no trial or free edition. The Polycom PVX software is the easiest way of testing our video conferencing hardware using software on the desktop but has not been updated since 2007.

Polycom PVX does still exist out there and a trial version can be downloaded from Polycom themselves (from their support site). You can also find the software to download from a few other sites such as Software Informer. With this trial edition you only get 5 minutes of video but it is enough to connect to your hardware and check everything is working. The interface is awful but it does work and the software connected to our Tandberg system fine using my computer’s webcam and mic etc.