Forward ports in Ubuntu Server 12.04 using ufw

We needed to forward port 3307 to port 3306 to get around a new company wide firewall restriction blocking access to port 3306 (our MySQL server). It was a pain to find how to get port forwarding working in Ubuntu Server 12.04, which uses “ufw” as a front end to “iptables”. I couldn’t get it working without specifically forwarding to my IP, which I shouldn’t need to do (but at least it works).

This will forward port 3307 to 3306 so you can connect to your.ip.add.ress:3307 and have it automatically connect to a server (such as MySQL) on port 3306.

To do this you need “ufw” to be enabled, which you can check with “sudo ufw status”.

Make sure the ports you need are allowed:

sudo ufw allow 3307

Now open up “/etc/ufw/before.rules”:

sudo nano /etc/ufw/before.rules

Go to the bottom of the file and put:

# nat Table rules
*nat
:PREROUTING ACCEPT [0:0]
-A PREROUTING -i eth0 -p tcp -m tcp -d your.ip.add.ress –dport 3307 -j DNAT –to-destination your.ip.add.ress:3306
COMMIT

Restart “ufw” to make sure everything worked ok:

sudo ufw disable
sudo ufw enable

Now when you connect to port 3307 it will forward to 3306.

Quick Subversion (SVN) Server Setup on Ubuntu Server 12.04

Setting up an Apache Subversion (SVN) server for access using svn:// with client applications like TortoiseSVN is actually pretty simple. The official Ubuntu Documentation covers a lot more than this simple setup but this is enough to get something up and running quickly without worrying about WebDAV or HTML access. The odyniec.net tutorial is also really useful and provides the init.d startup script I used to make the SVN server run at boot.

The steps are; install subversion, create the repository directories, set access control, set subversion to run at boot.

To install subversion in ubuntu just run:

sudo apt-get install subversion

Now create a directory to hold your subversion repositories, in my case I used “/home/svn”:

sudo mkdir /home/svn

Create a repository folder, for example “svnrepo1″, within this directory:

sudo mkdir /home/svn/svnrepo1

Now you can use the “svnadmin” program that comes as part of the subversion package to create a SVN repository within this folder:

sudo svnadmin create /home/svn/svnrepo1

The configuration file for the repository is created as “/home/svn/svnrepo1/conf/svnserve.conf” and contains the option to enable password protection as well as a lot of other useful settings. The important lines to uncomment to force password access are:

anon-access = none
auth-access = write

By setting “anon-access” to “none” you force people to enter passwords on connecting to the SVN. Now set up password protected access by uncommenting the following:

password-db = passwd

Settings “password-db” to “passwd” means the list of users and passwords in the “/home/svn/svnrepo1/conf/passwd” file will be used to check if someone has access. In a lot of cases it makes sense to keep this “passwd” file somewhere else so it can be used for all your repositories. In my case I set it to:

password-db = /home/svn/passwd

Just make sure to set the passwd file to be only readable by root:

sudo chmod 600 /home/svn/passwd

The “passwd” file is actually a very simple text file and looks something like:

[users]
harry = harrypassword
sally = sallypassword

Once the SVN server is configured and a repository set up as above you can run the SVN server using:

svnserve -d –foreground -r /home/svn

To make sure the SVN server starts at boot you need to set up init.d. Do this by creating and editing a file “/etc/init.d/svnserve” (I use “nano” to do my text editing on the command line):

sudo nano /etc/init.d/svnserve

Now paste in the contents of the odyniec.net init.d script. This script covers everything you need to start, stop and restart the “svnserve” program at boot so your SVN server can listen to all svn:// connections. There are alternatives to using this script, but this works and is simple to set up. Make sure you change the line with “DAEMON_ARGS” to point to the right place of “/home/svn”:

DAEMON_ARGS=”-d -r /home/svn”

Now tell Ubuntu to update its startup routine to include this new script:

sudo update-rc.d svnserve defaults

Reboot the server to make sure everything is working as expected.

You can now start, stop or restart the automatically booted SVN server using the following commands:

sudo /etc/init.d/svnserve start
sudo /etc/init.d/svnserve stop
sudo /etc/init.d/svnserve restart

Connecting to your SVN server can be done using something like TortoiseSVN and the URL you use to connect to the “svnrepo1″ repository you just set up is:

svn://your.serv.er.ip/svnrepo1

There is a lot more you can do with the SVN configuration, such as adding group support etc, but this is the quickest way to set up a standard SVN server on Ubuntu to accept svn:// connections using “svnserve”.

Easily set up and automatically start Apache Tomcat 7 Java web server in Ubuntu Linux

Apache Tomcat is actually easier than the standard Apache webserver to set up, which is great news if you are working with Java based web applications. All you need to do is download it and make sure it starts with whichever linux distribution you are using. Deploying applications in standard WAR format is really easy as well due to the simple web based management interface.

In my case I wanted Tomcat to start with Ubuntu and sit on the default port 8080 so I could have it running alongside my standard Apache webserver for PHP. We were developing a Spring application and used Maven to build and compile to a single deployable WAR file. You must have Java installed and set up for this to work. To check you have Java set up type:

java -version

This should tell you what version of java you have installed (hopefully Java 1.7). You also need to check that the “JAVA_HOME” variable is set by typing:

echo $JAVA_HOME

If you don’t get something like “/usr/lib/jvm/jdk1.7.0_09″ please install Java following my installation instructions in a previous post.

To install Apache Tomcat first of all I downloaded the latest copy of Tomcat 7 from mirrorservice.org using wget run from my home directory:

wget http://www.mirrorservice.org/sites/ftp.apache.org/tomcat/tomcat-7/v7.0.32/bin/apache-tomcat-7.0.32.tar.gz

Please note that the version I downloaded may not be available or there may be a newer version so check http://www.mirrorservice.org/sites/ftp.apache.org/tomcat/tomcat-7/ first before running the wget.

“wget” will download the file, which then needs to be extracted:

tar xvzf apache-tomcat-7.0.32.tar.gz

Now you will have a folder “apache-tomcat-7.0.32″ in your home directory. This needs placing somewhere sensible so copy it to “/usr/share/tomcat7″ using:

sudo mv apache-tomcat-7.0.32/ /usr/share/tomcat7

Now you can test your Tomcat install works with its default settings by starting it up. Note: before you do this you need to set the “JAVA_HOME” variable otherwise you will get errors (see my previous post).

To start up Tomcat navigate to “/usr/share/tomcat7″ and run “startup.sh”:

cd /usr/share/tomcat7

./startup.sh

With the default settings you should now be able to reach your Tomcat server home page by navigating to “http://your.ip.add.ress:8080″ where you should hopefully see the homepage and a nice message saying:

“If you’re seeing this, you’ve successfully installed Tomcat. Congratulations!”

Now we need to set up management users for the manager app so we can easily deploy our WAR files containing our Java web applications. You need to edit “/usr/share/tomcat7/conf/tomcat-users.xml”:

sudo nano /usr/share/tomcat7/conf/tomcat-users.xml

Now add the following lines within the “<tomcat-users>” block to give access to the manager GUI:

<role rolename=”manager-gui”/>
<user username=”MANAGERUSER” password=”YOURPASSWORD” roles=”manager-gui”/>

Now you will be able to log in to the manager GUI at “http://your.ip.add.ress:8080/manager/html” using the login details MANAGERUSER and password YOURPASSWORD. You can deploy applications and generally manage your Tomcat install from here.

The final thing to do is to set up Tomcat so that it starts every time your server starts. This is pretty easy as all you need to do in Ubuntu is edit the “/etc/init.d/tomcat7″ file:

sudo nano /etc/init.d/tomcat7

Now enter the following lines:


# Tomcat auto-start
#
# description: Auto-starts tomcat
# processname: tomcat
# pidfile: /var/run/tomcat.pid

case $1 in
start)
sh /usr/share/tomcat7/bin/startup.sh
;;
stop)
sh /usr/share/tomcat7/bin/shutdown.sh
;;
restart)
sh /usr/share/tomcat7/bin/shutdown.sh
sh /usr/share/tomcat7/bin/startup.sh
;;
esac
exit 0

Set the permissions for the file:

sudo chmod 755 /etc/init.d/tomcat7

Add Tomcat to system startup as a service using the command:

sudo update-rc.d tomcat7 defaults

Now you can test that Tomcat is set up as a service using:

sudo service tomcat7 restart

Now to check everything is working on system startup reboot your machine using:

sudo reboot now

Navigate to “http://your.ip.add.ress:8080″ where the Tomcat home page should appear with no problems. Note: If you are having problems reaching your Tomcat home page make sure you have opened port 8080 on your server’s firewall.

It’s definitely worth reading some of the documentation on Tomcat, plenty of which is linked off your newly installed Tomcat home page. You should now have all you need to deploy your Java web applications as WAR files which is really easy using the manager GUI provided by Tomcat.

Install the latest Java 7 JDK on Ubuntu Linux Server 10.04 without apt-get

I was trying to set up Apache Tomcat on an older server running Ubuntu 10.04 and noticed that Java wasn’t actually installed by default. Also, the licensing agreement with Oracle seems to have changed and it is no longer possible to just use apt-get to install it. You have to manually accept the licensing agreement so even “wget” wont work any more (you just get a “download-fail-XXXXXXX.html” file instead.

First up, you have to go to the Java JDK download page and manually accept the licensing agreement. This must be done from a PC with a UI and a browser so no “wget”. You need to get the correct version, which in my case was the x86 tar.gz version. When this is downloaded you should have a “jdk-7u9-linux-i586.tar.gz” file which then needs to be copied to your Ubuntu server (I copied to my user’s home directory).

Now you have the file on your Ubuntu server you can extract it using:

tar -xvf jdk-7u9-linux-i586.tar.gz

This should give you a directory “jdk1.7.0_09″ which we need to move to somewhere sensible such as “/usr/lib/jvm/jdk1.7.0_09″:

sudo mv jdk1.7.0_09 /usr/lib/jvm/jdk1.7.0_09

Now we need to set up a symbolic link so that we can run Java from everywhere:

 sudo ln -fs /usr/lib/jvm/jdk1.7.0_09/bin/java /usr/bin/java

Now check that Java is all installed correctly by checking the version using:

java -version

Which in this case should give:

java version “1.7.0_09″
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) Client VM (build 23.5-b02, mixed mode)

Now you can set up your JAVA_HOME variable at a system level so other applications can use Java by editing “/etc/environment”:

sudo nano /etc/environment

Now add the following line to point to your newly installed Java:

JAVA_HOME=/usr/lib/jvm/jdk1.7.0_09

Now if you open up a new session (not your currect session) and type “echo $JAVA_HOME” you should see the path “/usr/lib/jvm/jdk1.7.0_09″ which means the variable has been set correctly.

Easily connect and use PHP with SharePoint lists using cURL, RSS and NTLM authentication

Connecting to SharePoint from PHP is actually not that difficult if you have the cURL extension installed on your web server. In the case of my XAMMP windows development server I just made sure the following line in php.ini (c:xammpphpphp.ini in my case) was uncommented before restarting Apache:

extension=php_curl.dll

In Ubuntu/Linux you can usually just install the packages for cURL and after restarting Apache it will become available. Just type the following on the command line:

sudo apt-get install curl libcurl3 libcurl3-dev php5-curl

Then restart Apache

sudo /etc/init.d/apache2 restart

Now the following code comes from both Norbert Krupa’s comment on David’s IT Blog and a question about parsing HTML on StackOverflow. The important thing to note is that I needed to use cURL to authenticate my domain user when connecting to my secure SharePoint Services 3.0 test site. Apparently you can get away without using cURL on sites that don’t need authentication but the same cURL code listed below can be used with a blank username and password for the same effect.

The goal of this listing is to connect to SharePoint using a domain user (can also be a local user if SharePoint is set up that way) and retrieve the contents of a SharePoint list. The trick is to supply the RSS feed url, which allows PHP to parse the RSS feed and neatly list the contents of a SharePoint list. An advantage of using RSS feeds of SharePoint lists is that they are secured using the same method as the list itself and require no extra configuration on the SharePoint side of things. You can also set the RSS feed to only show a set number of items or days, which is useful for regularly updated lists.

// generic function to get the contents of an HTML block
function get_inner_html( $node ) {
    $innerHTML= '';
    $children = $node->childNodes;
    foreach ($children as $child) {
        $innerHTML .= $child->ownerDocument->saveXML( $child );
    }
    return $innerHTML;
}

// username and password to use
$usr = 'DOMAINUSERNAME';
$pwd = 'PASSWORD';
// URL to fetch, this is the address of the RSS feed (go into a list and click "Actions" -> "View RSS Feed" to get the url)
$url = "http://www.sharepointsite.com/_layouts/listfeed.aspx?List=%7BCED7CDDC-49C0-4C46-BDE6-CFC2BA993C84%7D";
//Initialize a cURL session
$curl = curl_init();
//Return the transfer as a string of the return value of curl_exec() instead of outputting it out directly
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
//Set URL to fetch
curl_setopt($curl, CURLOPT_URL, $url);
//Force HTTP version 1.1
curl_setopt($curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
//Use NTLM for HTTP authentication
curl_setopt($curl, CURLOPT_HTTPAUTH, CURLAUTH_NTLM);
//Username:password to use for the connection
curl_setopt($curl, CURLOPT_USERPWD, $usr . ':' . $pwd);
//Stop cURL from verifying the peer’s certification
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
//Execute cURL session
$result = curl_exec($curl);
//Close cURL session
curl_close($curl);

$xml = simplexml_load_string($result);

// display results on screen
foreach($xml->channel->item as $Item){
    echo "<br/>($Item->title)";
    $doc = new DOMDocument();
    $doc->loadHTML($Item->description);
    $ellies = $doc->getElementsByTagName('div');
    foreach ($ellies as $one_el) {
        if ($ih = get_inner_html($one_el))
        {
            echo ", $ih";
        }
    }
}

The SharePoint RSS feed is a little interesting as the “$Item->title” object is the main column in the list but the rest of the list is encapsulated in <div> within “$Item->description”, hence the requirement to parse the html.

For a SharePoint list with 3 columns the output will look something like:

(Item 1 Title) , Other Column A: xxxx, Other Column B: yyyy
(Item 2 Title) , Other Column A: zzzz, Other Column B: kkkk

Now the potential for this is great as it allows us to securely synchronise SharePoint lists with external databases, use SharePoint for authenticating PHP applications etc . We are going to be using this for automatically pulling users from a SharePoint list to populate a separate PHP application, whilst keeping user-identifiable data locked away on SharePoint.

Copy VMWare Server virtual servers (Ubuntu) between physical machines, avoid locking errors and fix missing eth0

I needed to copy a virtual Ubuntu Server install from our backup store to a new physical server (server A) as our old server (server B) suffered a major failure and wouldn’t even boot. This meant copying the whole VMWare directory containing the Ubuntu virtual machine from the backup store (a USB harddrive in this case) to the new physical server (server A). I had installed the free VMWare Server on both server B and A so that if anything happened to server B I would be able to get up and running quickly on server A.

Note: I am assuming you can install and configure VMWare Server for Windows, which is what I’m using here.

Once I copied the whole directory from the backup store to server A’s VMWare directory (in this case c:vmware) I then used the VMWare web interface to “Virtual Machine – Add Virtual Machine to Inventory” which allowed me to select the .vmx file in my datastore that corresponded to my copied Ubuntu virtual machine.

On trying to start the virtual machine I get a warning message and a choice:

msg.uuid.altered: This virtual machine may have been moved or copied. In order to configure certain management and networking features VMware Server needs to know which. Did you move this virtual machine, or did you copy it? If you don’t know, answer “I copied it”.

I select “I copied it” and then click “ok”, which then brings up an error:

“Power On Virtual Machine” failed to complete

If these problems persist, please contact your system administrator.
Cannot open the disk ‘C:vmwareVIRTUALSERVERNAMEVIRTUALSERVERNAME.vmdk’ or one of the snapshot disks it depends on. Reason: Failed to lock the file.

Ok, so it seems that because I copied the virtual server from the backup store it also included the .lck directories, which are used while the server is running. The simple fix is to delete these directories in C:vmwareVIRTUALSERVERNAME and allow VMWare server to rebuild them. Once I had deleted the two directories “VIRTUALSERVERNAME.vmdk.lck” and “VIRTUALSERVERNAME.vmem.lck” I could start the virtual machine.

I then ran into another problem, which was that my network instance eth0 was not being set up correctly even with VMWare Server set to use “Bridged” networking which worked on the old server. Running “ifconfig -a” showed that eth0 wasn’t there and it didn’t have the static IP I had given it previously in “/etc/network/interfaces”. The loopback interface, l0, was there but not eth0.

Orzeszek has an easy solution for this, which is to delete the “/etc/udev/rules.d/70-persistent-net.rules” file and “sudo reboot” to allow Ubuntu to rebuild the file with the correct MAC address, set up by VMWare Server when we originally added the virtual machine to the inventory. Now everything should be working perfectly.

Some people have reported other errors, which can be fixed by changing the name of eth0 in “/etc/network/interfaces” to eth1, which I didn’t need to do but you might.

 

Using ADODB to easily connect to MySQL and Microsoft SQL Server from PHP

In previous posts I’ve talked about connecting to Microsoft SQL Server from PHP using ODBC and using mssql_connect to connect directly. The most important thing I’ve realised is not to re-invent the adodb wheel, which has shown to be a really great cross-platform way of connecting to databases.

The best thing about adodb is that just by changing the connection string I can talk to MySQL and Microsoft SQL Server without changing any of my code. I’ve used this for performance and migration testing and it’s been perfect, with none of the strange hangs I noticed with a standard ODBC connection. Currently I am using Ubuntu Server 10.10 to host all the PHP, which has overall been a pretty good experience, apart from these strange hangs.

If you are in Ubuntu Server you may need to install some ODBC packages in order to get adodb to work. Some of these packages are not totally necessary but I would recommend them for testing ODBC connections etc:

sudo apt-get install php5-odbc unixodbc unixodbc-dev freetds-dev sqsh tdsodbc

Now you need to create a SQL user in either your MySQL or SQL Server database. Make sure you map the SQL Server user as a database owner on the DB you want to connect to and give them the same default schema. The reason for this is I couldn’t get the connection in adodb to work with the actual DB specified (unlike in MySQL).

Download adodb5 and place the directory in the root of your webserver and add the following to the top of your PHP file:

include(‘adodb5/adodb.inc.php’);

Now open up your PHP file and insert the following for SQL Server with IP “the.ser.ver.ip”:

$dbcstring = “mssql://username:password@the.ser.ver.ip”;

Or the following for MySQL:

$dbcstring = “mysql://username:password@the.ser.ver.ip/databasename?persist”;

Create a connection to the database:

$DB = NewADOConnection($dbcstring);

Now you can use the built in commands in adodb to easily pull data from the database and display it on screen. It doesn’t matter which database you are connecting to, the commands are now the same. This is the main advantage of adodb in my opinion, along with the ease of use:

$arr = $DB->GetAll(“SELECT column FROM table”);
for($i=0;$i<count($arr);$i++)
{
echo $arr[$i][0]  . “<br/>”;
}

It’s worth reading the guide on adodb for the commands you can use, such as $DB->GetRow(), $DB->GetOne() and $DB->Execute(). The great thing is that adodb makes everything so simple you can treat your data as arrays or you can go further and make use of some of the more advanced functionality. Either way, by using adodb you massively cut down on code rewrites when you migrate to a new database.

Connecting to Microsoft SQL Server from PHP in Ubuntu using mssql_connect()

As part of ongoing testing to find the best way of storing large amounts of data we are considering Microsoft SQL Server, which needs to be accessible from our PHP5 application. The site is hosted on Ubuntu Server 10.10, which makes things a little more interesting. We currently use adodb5 to talk to a MySQL server, which works fine, apart from MySQL being far too slow. It is possible to create an ODBC connection in Ubuntu to talk to SQL Server but for testing the speed of our queries I just used mssql_connect(). In the near future I will move the database connection entirely to ODBC/adodb so we don’t have to rewrite any of our existing code.

Ubuntu doesn’t come with the packages needed for mssql_connect() by default so you need to install them:

sudo apt-get install php5-sybase

Then restart the apache server to apply the changes:

sudo /etc/init.d/apache2 restart

Now you can use mssql_connect and its associated functions in your PHP to connect to your Microsoft SQL Server, as in this example:

ini_set(‘display_errors’, 1);
$server = ‘my.server.ip:1433DATABASEINSTANCE’;
$link = mssql_connect($server, ‘username’, ‘password’);

if (!$link) {
die(‘<br/><br/>Something went wrong while connecting to MSSQL’);
}
else {
$selected = mssql_select_db(“databasename”, $link)
or die(“Couldn’t open database databasename”);
echo “connected to databasename<br/>”;

$result = mssql_query(“select name from table”);

while($row = mssql_fetch_array($result))
echo $row["name"] . “<br/>”;
}

Transfer SSL Certificates from Microsoft IIS to Linux Apache

Thanks to a few different guides online (mainly Pete Frietag’s) I got an SSL certificate transferred from IIS on Server 2003 to Apache on Ubuntu Server 10.04. There was an extra step I needed to do that seems obvious now.

First Export your IIS certificate into a pfx file (this is something you should do anyways for backup)

  • Run mmc.exe
  • Click the ‘Console’ menu and then click ‘Add/Remove Snap-in’.
  • Click the ‘Add’ button and then choose the ‘certificates’ snap-in and click on ‘Add’.
  • Select ‘Computer Account’ then click ‘Next’.
  • Select ‘Local Computer’ and then click ‘OK’.
  • Click ‘Close’ and then click ‘OK’.
  • Expand the menu for ‘Certificates’ and click on the ‘Personal’ folder.
  • Right click on the certificate that you want to export and select ‘All tasks’ -> ‘Export’.
  • A wizard will appear. Make sure you check the box to include the private key and continue through with this wizard until you have a .PFX file.

Next run openssl to extract the private key, and the cert file.

# Export the private key file from the pfx file

openssl pkcs12 -in filename.pfx -nocerts -out key.pem

# Export the certificate file from the pfx file

openssl pkcs12 -in filename.pfx -clcerts -nokeys -out cert.pem

# This removes the passphrase from the private key so Apache won't
# prompt you for your passphase when it starts

openssl rsa -in key.pem -out server.key

Now to make this work (the extra step) you need to go to your apache virtual host configuration file (or default if you dont have any virtual hosts) located at “/etc/apache2/sites-available/virtualhostname.conf” and make sure the following lines are present and correct:

SSLEngine on
SSLOptions +StrictRequire
SSLCertificateFile /path/to/certificate/cert.pem
SSLCertificateKeyFile /patch/to/key/server.key

Then restart apache with the standard command

sudo /etc/init.d/apache2 restart

Install Webmin on Ubuntu Server 10.10

Content from Andrai Daneasa, thanks!

1. Download webmin from http:www.webmin.com . Currently 1.520
wget http://www.webmin.com/download/deb/webmin-current.deb

2. Execute webmin package. It will fail but it’s easier for us, since we’ll just force installing dependencies, rather than installing them one by one.
sudo dpkg -i webmin-current.deb

3. Install missing dependencies and it will automatically recompile webmin
sudo apt-get -f install

Enjoy webmin on https://[serverIP]:10000 .

So easy when you don’t have to worry about dependancies.

UPDATE:

There were issues with Apache configuration under webmin in Ubuntu Server 10.10. Webmin couldn’t see that apache was already started so you have to do the following (thanks hrpr on the Ubuntu forums):

I just installed Webmin on Ubuntu 10.10 desktop and had same problem. Fixed in my case by settng path to Apache PID to “/var/run/apache2.pid” rather than using the default setting of “Work out automatically.” I did not make any changes to the envvars file.

This also worked for me.

On another note, my fresh install of Ubuntu Server 10.10 with a LAMP server and webmin installed, on a vmware virtual server, came to 1.64GB, a tiny amount!