Hello!

MySQL replication has been known to easily break, as a result of a large multitude of potential causes.

Sometimes the replication can even break if an erroneous query is executed on the master server.

With all the potential issues that may break replication, we thought it prudent to write an automated check script that can run on a scheduled basis (i.e. every 10-15 minutes), check the Slave status, report on any errors if applicable and attempt to repair replication.

We have built this script to exit and send mail alerts if any step of the checking and repairing process fails or generates an error in itself.

The script also generates a lock file to ensure that no more than one check process can run at any given time. We feel this script could be best used for scenarios for remote MySQL slaves, for example. Adding this extra layer may ensure a more reliable replication.

The repair process is simply 3 MySQL Commands :

stop slave;
reset slave;
slave start;

The above directives assume that you have a master.info with the mysql master server information statically set. No CHANGE MASTER commands have to be executed as a result. Resetting the slave clears the error and resumes replication, and all the queries missed during the time it failed should be queued and applied after it starts again.

Here is the script :

#!/bin/sh
# Slave replication auto recovery and alert
# Star Dot Hosting 2012

currentmonth=`date "+%Y-%m-%d"`
lock_file=/tmp/slave_alert.lck

echo "MySQL Replication Check Script" > /var/log/replication_check.log 2>&1
echo "------------------------------" >> /var/log/replication_check.log 2>&1
echo "$currentmonth" >> /var/log/replication_check.log 2>&1
echo "" >> /var/log/replication_check.log 2>&1

# Check if lock file exists
if [ -f $lock_file ];
then
        echo "Lock file exists! Possible conflict!" >> /var/log/replication_check.log 2>&1
        mail_alert
        exit 1
else
        touch $lock_file
fi

# Fix slave
function fix_replication () {
        mysql -u root --password="XXXXX" -Bse "stop slave" >> /var/log/replication_check.log 2>&1
        if [ "$?" -eq 0 ];
        then
                echo "Stop slave succeeded..." >> /var/log/replication_check.log 2>&1
        else
                echo "Slave recover function failed" >> /var/log/replication_check.log 2>&1
                mail_alert
                exit 1
        fi
        mysql -u root --password="XXXXX" -Bse "reset slave" >> /var/log/replication_check.log 2>&1
        if [ "$?" -eq 0 ];
        then
                echo "Reset slave succeeded..." >> /var/log/replication_check.log 2>&1
        else
                echo "Slave recover function failed" >> /var/log/replication_check.log 2>&1
                mail_alert

                exit 1
        fi
        mysql -u root --password="XXXXX" -Bse "slave start" >> /var/log/replication_check.log 2>&1
        if [ "$?" -eq 0 ];
        then
                echo "Slave start succeeded." >> /var/log/replication_check.log 2>&1
        else
                echo "Slave recover function failed" >> /var/log/replication_check.log 2>&1
                mail_alert
                exit 1
        fi
}

# Alert function
function mail_alert () {
        cat /var/log/replication_check.log | mail -s "Replication check errors!" kkutzko@n49.com
}

# Check if Slave is running properly
Slave_IO_Running=`mysql -u root --password="XXXXX" -Bse "show slave status\G" | grep Slave_IO_Running | awk '{ print $2 }'`
Slave_SQL_Running=`mysql -u root --password="XXXXX" -Bse "show slave status\G" | grep Slave_SQL_Running | awk '{ print $2 }'`
Last_error=`mysql -u root --password="XXXXX" -Bse "show slave status\G" | grep Last_error | awk -F \: '{ print $2 }'`

# If no values are returned, slave is not running
if [ -z $Slave_IO_Running -o -z $Slave_SQL_Running ];
then
        echo "Replication is not configured or you do not have the required access to MySQL"
        exit 1
fi

# If everythings running, remove lockfile if it exists and exit
if [ $Slave_IO_Running == 'Yes' ] && [ $Slave_SQL_Running == 'Yes' ];
then
        rm $lock_file
        echo "Replication slave is running" >> /var/log/replication_check.log 2>&1
        echo "Removed Alert Lock" >> /var/log/replication_check.log 2>&1
elif [ $Slave_SQL_Running == 'No' ] || [ $Slave_IO_Running == 'No' ];
then
        echo "SQL thread not running on server `hostname -s`!" >> /var/log/replication_check.log 2>&1
        echo "Last Error:" $Last_error >> /var/log/replication_check.log 2>&1
        fix_replication
        mail_alert
        rm $lock_file
fi

echo "Script complete!" >> /var/log/replication_check.log 2>&1
exit 0

Hello!

We thought it would be important to share this information as it might be interesting to someone who wants to replicate the same VPS across many instances in order to create a farm of web servers (for example).

This uses very similar concepts to our LVM XEN backup post a while back.

Step 1: Take a snapshot of the current VPS

This is simple. Use the lvcreate command with the -s option to create a snapshot of the running VPS. We assume your VPS is 5GB in size, so just replace that with however large your VPS is :

lvcreate -s -L 5GB -n snapshot_name /dev/VolGroup00/running_vps_image

Step 2: Create your new VPS

This is important. You want to create a new vps, assign a MAC and IP address first and let the creation process fully finish. Then shut the VPS down.

Step 3: Copy the snapshot to the new VPS

All you have to do is use the dd command to transfer the snapshot image to the newly created VPS image :

dd if=/dev/VolGroup00/snapshot_name of=/dev/VolGroup00/new_vps_image

All done! Dont forget to remove the snapshot after your done with it :

lvremove -f /dev/VolGroup00/snapshot_name

Start up the new vps and you should have a carbon copy of the previous vps!

Hey there,

Before I update regarding my continued experiences with Xen, KVM, Varnish, FreeBSD and all the other (exciting) things that I do, I thought I’d remind anyone here who is looking for linux vps hosting, virtual hosting or specifically xen vps hosting or even cpanel vps hosting to check out our company website : www.stardothosting.com

If you’re looking for managed hosting services, dedicated hosting or anything along those lines, feel free to fill out our managed hosting quotation form here : www.stardothosting.com/managed-hosting

Thanks!

Hello There!

One of the exciting new technologies to come out in the last few years is a tremendously efficient and dynamic caching system called Varnish (see : http://www.varnish-cache.org).

We have been employing the use of Varnish for high traffic websites for the purposes of user experience improvements as well as for redundancy and load balancing purposes.

Varnish can do it all – complex load balancing and polling based on many different weighting methodologies for fail over, as well as holding on to a “stale” cache in the event of a back end web server outage, or perhaps for geographic redundancy (holding a stale cache in a secondary data center).

One of the challenges we have faced in the many different implementations of varnish into web stacks, is dealing with dynamic and user session (i.e. “logged in”) content.

If the Internet was full of only static (see 1995) html files, varnish would work beautifully out of the box. Unfortunately the web is a complicated mess of session based authentication, POSTS, GETS and query strings among a few things.

One of our recent accomplishments was getting the Joomla 1.5 content management system to work with Varnish 2.1.

The biggest challenge for Joomla was that it creates a session cookie for all users. This means the session is created and established for any guest visiting the site, and if they decide to log in , that same session is used to establish a logged in session through authentication. This is an apparent effort to deter or avoid session hijacking.

The problem with this is that Varnish ends up caching all the logged in session content, as well as the anonymous front page content.

I spent a significant amount of time fine tuning my VCL (varnish configuration language) to play nice with Joomla. Unfortunately it became apparent that some minor modifications to the Joomla code was necessary in order for it to communicate properly with Varnish.

Step 1 : Move the login form off the front page

I realize this might be a hard decision. I cant offer an alternative. If you have an integrated login form on the front page of your site, and you wish to cache that page with varnish, you will likely have to chose one or the other. It would probably be ideal to replace that login form with a button to bring the user to a secondary page off the main page.

For the sake of argument, lets call our site “example.com” and the login page url within Joomla should look like the following :

http://www.example.com/index.php?option=com_user&view=login

Take note of login URI in this string.

The reason we need the login form on a secondary page is because we need an almost “sandboxed” section of the site where the anonymous session cookie can be established, and passed through the authentication process to a logged in session. We will tell varnish to essentially ignore this page.

Step 2 : Modify Joomla to send HTTP headers for user/guest sessions

This isn’t that hard. In the Joomla code, there is a section where it defines the HTTP headers it sends to the browser for cache variables such as expire times and whatnot. I’m going to assume you have turned off the built-in Joomla caching system.

What you need to do is tell Joomla to send a special HTTP header that will give either a True or False value if the user is logged in or not. This is useful information. It will allow varnish to not cache any logged in content such as “Welcome back, USERNAME” after the user is passed back to the front page from logging in.

In my joomla installation, I modified the following file :

The parent folder being the public_html / root folder for your Joomla installation. In this file, please find the line that determines if the Joomla caching system is disabled :

After this line, you will see about 5 HTTP header declarations (expires, last-modified, cache-control, cache-control again and pragma). Above those declarations , add the following 6 lines of code :

If you read the above code, its fairly straight forward. I do a check to see if the user is a guest (aka anonymous) or not. If they are logged in I send an HTTP header called “X-Logged-In”, and assign a “True” value to it. If the user is not logged in, it sets it to “False”.

Pretty easy, right?

This will allow varnish to avoid caching a logged in user’s page.

Step 3 : Configure Varnish

This is the part that took the most time during this entire process. Mind you patching the Joomla code and whatnot took some time as well, this process took a lot of experimentation and long hours examining session cookies and host headers.

What I will do is break down the generalized configuration directives into two groups : VCL_RECV and VCL_FETCH.

VCL_RECV

In here, I set a bunch of IF statement directives to tell varnish what it should look up in the cache and what it should pipe to the backend and what it should pass. This could probably be optimized and improved upon, but it works for me :

VCL_FETCH

The fetch section is a little bit easier. I only have about 5 directives. The first one is the most important one you want to look at. It “unsets” the cookie from any page on the site, EXCEPT the login page. This allows varnish to properly establish the logged in session. The subsequent rules determine what to deliver and what to pass based on URI or HTTP header checks :

Thats it! I just saved you many sleepless nights (I hope!). Hopefully your headers will look something like this after you implement varnish in front of Joomla :

UPDATE : 12/08/2011

I realize I made a mistake and have corrected this post. In vcl_fetch, i had the following :

Well I realize I should be unsetting the response cookie, not the set cookie. For some reason, the above (erroneous) directive works only right after you login. If you start clicking around the site, your logged in session disappears. I suspect this is because either joomla or varnish is mistakenly unsetting a logged in session.

This is the correct entry (I have fixed it in my original post as well) :

After making the above change, I can login and browse the site and my session stays intact. Mind you, the Joomla site I am testing with is definitely not a vanilla Joomla installation.

I’d love to hear from anyone who has accomplished the above scenario either way!

We are excited about a new promotion for our web hosting catalog here at *.hosting.

We are offering a 50% discount for ANY of our SHARED or VPS hosting plans for the first 3 (thats THREE) months!!!!

Simply enter the following promotional code during the signup process :

It’s that simple! The 50% discount will then be immediately applied to the first three months of hosting — no matter what the hosting plan you chose!

Happy Holidays from Star Dot Hosting!

I decided to write this little guide to provide the relatively simple steps needed to migrate your linux system to a Xen (HVM) virtual instance.

It is assumed that on your source and destination boxes, that you only have one root “/” partition. If you partitioned out your file system differently, you will have to accommodate that based on these instructions.

The following steps walk you through the process of migrating linux to Xen from start to finish :

1. Install the exact same version of linux on your destination server
This isn’t really 100% necessary, obviously. You could always boot into Finnix, partition your disk and install Grub. If you are uncomfortable doing that, install the distribution from start to finish. The file system will be overwritten anyways.

2. Boot into finnix on the destination system
If you have never used Finnix, it is a “self contained, bootable linux distribution”. I like it alot actually and have used it for similar purposes, rescue operations and the like.

3. Setup networking on both destination and source systems
If both systems are on the same network, you could assign local IP addresses to ensure the process of synchronisation is speedy and unobstructed.

Ensure you configure networking either way and that you set a root password and start ssh :

4. Mount the partition that you want to copy to on the destination server
Remember, so far everything you are doing has been on the destination server. Mount the destination partition within finnix :

5. On the source server, rsync all the files of the source partition to the destination partition
When logged into the source server, simply issue the following rsync command and direct it to the destination server’s partition you just mounted :

The rsync process will complete and the partition on the destination server should be ready to boot into. Remember to change the networking configuration if you dont want any IP conflicts to happen.

I hope this helps!

This post is intended to be a general guide for configuring “stickied” load balanced HTTP servers. Whether it’s F5 load balancers, foundry load balancers or open source based load balancers (keepalived/lvs), the concepts are the same and can be migrated across said platforms.

If you have a paid of foundry’s and are looking to configure stickied load balanced HTTP servers, hopefully this guide will provide some assistance.

Logging into the load balancer

Telnet to the box and ‘enable’ to allow admin access. The first thing you want to do is show the current configuration to view the existing setup for other working boxes :

Real servers : defining the multiple load balanced boxes

Show the existing configuration on the foundary :

Take a look at the configuration of two “real” servers, which are the two servers that are behind the load balancer that will have balanced sticky connections :

The above example is balancing TCP 8001 traffic, which is for TOMCAT. Here are entries for two servers doing simple HTTP traffic :

This example is similar to the tomcat example, except you have several options. “port default disable” disables all other ports. “port http keepalive” and “port http url “HEAD /”" define the http checks that take place to ensure apache is running on that box. If not , it will fail over to the second box and stop sending traffic to it.

SSL Connections

SSL incoming connections are handled by the load balancer initially, then passed off to the actual server as regular http / port 80 traffic. The internal box configuration would be similar to the above configuration examples :

Configuring the external IP to NAT to the internal virtual

Typically, you will have a firewall in front of the load balancer that actaully holds the external ip addresses. The traffic is filtered initially by the firewall, then NAT’d to the virtual ip (VIP) of the load balancer, which then handles balancing the traffic.

You will need to either establish a new external ip , or use an existing one (for instance, if you are moving from 1 web server to 2 web servers , and want to balance the traffic using the load balancer). You need to setup the external IP address, and NAT it to the internal VIP.

Verifying the configuration works

Once everything is setup properly, and the external IP is being NAT’d to the load balancer, it is time to ensure the load balancer is seeing the connections. You could do this before doing the switchover on the firewall as well, just to ensure everything looks right before actually doing the switchover.

To see the active connections being load balanced, issue the following command (replacing the servername for whichever one you want to check) :

That should display information similar to this :

The above is displaying the specific connection details for a single real server. To check the VIP / Virtual server :

Which will display the following :

You can see that “ServerConn” is displaying 46 connections. Thats it!

Hello there,

Just thought I’d share an exclusive coupon / discount for all of our shared / vps hosting plans that allows for 60% off the first month of hosting fee’s :

COUPON CODE : SDHTWT2010

Take a look at our main site for plan details. This coupon expires and there is only a limited number of them available!

Don’t say we never gave you nothin’

Through our research and implementation of many Xen environments, it has become necessary to develop a reliable and secure method for backing up our Xen instances that are mounted on Logical Volumes (LVM).

The underlying problem is that the logical volume is usually a live file system that cannot be directly mounted / backed up or imaged safely.

We have written a script that processes all running Xen logical volumes, creates a snapshot of the volume and through that snapshot , uses dd to image the snapshot to another server over ssh.

You would be surprised at how well these dd images compress. Piping dd to bzip2 then to ssh to receive the image produces a very substantial compression ratio.

The initial trouble was writing the logic in the script to properly go through each Xen LV , create the snapshot, image and then remove the snapshot. Obviously extensive testing had to be completed to ensure reliability and proper error reporting.

This script should work with any 3rd party Xen control panel implementation (HyperVM, FluidVM, SolusVM to name a few). They all use the same underlying technology / framework. Since our script is a simple bash / shell script, it will run on any linux based system with little modification.

If you are using a LV for another purpose on the same box, it is probably a good idea to modify the script to ignore that so it doesn’t inadvertently get backed up.

Before implementing the script, it is probably a good idea to go through the motions manually just to see how it performs :

lvcreate -s -L 5G -n vm101_img_snapshot /dev/vps/vm101_img
dd if=/dev/vps/vm101_img_snapshot | bzip2 | ssh xenbackup@x.x.x.x "dd of=vm101_img.bz2"

One thing that you cant get around is space — you need to leave as much room as the largest Xen image on your logical volume — otherwise the script will fail at the snapshot creation process.

Find the script below. Hopefully it will help make your life easier (as well as being able to sleep at night) :

#!/bin/bash
# XEN Backup script
# Written by Star Dot Hosting

todaysdate=`date "+%Y-%m-%d"`

echo "XEN Backup Log: " $currentmonth > /var/log/backup.log
echo -e "------------------------------------" >> /var/log/backup.log
echo -e "" >> /var/log/backup.log

for obj0 in $(lvs --noheadings --separator ',' -o lv_name,lv_size | grep -v "swap" | awk -F "," '{printf "%s\n", $1}');
do

#grab the snapshot size
snapsize=`lvs --noheadings --separator ',' -o lv_name,lv_size | grep -v "swap" | grep $obj0 | awk -F "," '{printf "%s", $2}'`

#create the snapshot
lvcreate -s -L $snapsize -n $obj0_snapshot /dev/xenlvm/$obj0 >> /var/log/backup.log 2>&1

#dd piped to bzip2 to compress the stream before piping it over the network via ssh to the destination box
dd if=/dev/xenlvm/$obj0_snapshot | bzip2 | ssh xenbackup@0.0.0.0 "dd of=/home/xenbackup/xen-backups/$obj0.$todaysdate.bz" >> /var/log/backup.log 2>&1

if [ "$?" -eq 1 ]
then
        echo -e "***SCRIPT FAILED, THERE WERE ERRORS***" >> /var/log/backup.log 2>&1
        cat /var/log/backup.log | mail -s "XEN Backup Job failed" admin@yourdomain.com
        lvremove -f /dev/xenlvm/$obj0_snapshot
        exit 1
else
        echo -e "Backup of $obj0 Completed Successfully!" >> /var/log/backup.log 2>&1
fi

# remove the snapshot
lvremove -f /dev/xenlvm/$obj0_snapshot

done

cat /var/log/backup.log | mail -s "XEN Backup Job Completed" admin@yourdomain.com

If you plan on automating this script in a cronjob, it may be a good idea to utilize ssh key authentication between your destination server and your Xen server.

There seems to be a lot of tutorials with respect to how you can dump/restore FreeBSD implementations. However, none of them appear to be all encompassing what is actually required from start to finish during the entire process.

The one thing that I think is lacking in proper documentation is utilizing FreeBSD in a LiveCD scenario (LiveFS) within a network capacity (necessary for migration).

We decided to write this tutorial so that people could have one place to establish all the necessary things required for this type of migration from start to finish.

In this scenario we actually migrated a FreeBSD implementation on VMWARE to XEN HVM. In the end, there were no technical problems with FreeBSD actually running after it was migrated — it ran beautifully actually.

I should note that this was tested with FreeBSD 7.2-RELEASE disc images.

Please find the guide below :

Prepare OLD Instance

1. Boot into old operating system

2. Take note of partition slices / slice names / sizes / etc

3. Reboot with FreeBSD LiveFS disc

Prepare NEW Xen

1. Boot Xen instance with FreeBSD Disc 1 ISO

2. Partition / install boot loader exactly the same slices as the old instance. To be extra careful, give your slices a bit more disc space than the old implementation.

3. Write changes & reboot with FreeBSD LiveFS disc

Establish FreeBSD LiveFS environment

You need to establish a few things to get SSH / DUMP / RESTORE to work properly on both the ”’old”’ and ”’new”’ instances

1. Boot into FreeBSD LiveFS (Fixit > livefs)

2. Create the following folders :

/etc/ssh
/usr/sbin
/usr/bin
/root
/root/.ssh

3. Copy the following files :

cp /mnt2/bin/ps /bin
cp /mnt2/sbin/sysctl /sbin
cp /mnt2/etc/ssh/* /etc/ssh
cp /mnt2/bin/csh /bin
cp /mnt2/bin/cat > /bin
cp /mnt2/sbin/restore > /sbin

4. Set an IP address on both old and new instances:

new :

ifconfig eth0 10.0.0.50 netmask 255.255.255.0

old :

ifconfig eth0 10.0.0.60 netmask 255.255.255.0

5. Start sshd :

/mnt2/etc/rc.d/sshd forcestart

Start transferring slices

1. To allow for transferring of partitions properly, the /tmp partition should be mounted on the new Xen instance :

mount -t ufs /dev/ad0s1e /tmp

2. For the first partition you wish to transfer, mount the empty slice on the new xen instance :

mount -t ufs /dev/ad0s1a /mnt/ufs.1

Sometimes you have to fsck mark the filesystem clean to mount it :

fsck /dev/ad0s1a

3. On the old instance :

dump -0aLf - /dev/ad0s1a | ssh 10.0.0.50 "cd /mnt/ufs.1 && cat | restore -rf -"

That should dump/restore the slice from old > new.

Final things on the new Xen instance

Dont forget to boot the new instance in single user mode and modify ”’fstab”’ to reflect the new slice names (if applicable), as well as ”’rc.conf”’ for any hard coded interface names, etc. FreeBSD won’t boot if the right slice names / interface names aren’t present. Or at least cause problems.

You can mount the /etc slice while still in the LiveFS for the new FreeBSD instance.

Hopefully this was helpful! Obviously this has nothing to do with Xen, other than the fact that we were migrating the FreeBSD vmware instance to Xen.

You can do this on “real” machines, or from xen to vmware or anywhere. As long as the hardware is compatible.