In development, having a seamlessly integrated process where you can propagate your code through whatever QA, testing and development policy you have is invaluable and a definite time saver.

We work with SVN as well as GIT code repository systems and have developed a web based system to “Export” or “Push” the code through development, staging and production environments as such.

I have already talked about sanitizing your code during the commit process, to ensure commit messages are standard and there are no PHP fatal errors, so now I will be showcasing you a simple web based system for propagating your code through development, staging and production servers.

This system should be on a secure web accessible page on each server. For the sake of argument , I’ll call each server the following :

dev.server.com — development server

staging.server.com — staging server

www.server.com — production server

We will be using PHP for the web based interface, and we will assume that you will be password protecting access to this page via htpasswd, as well as forcing SSL. I am also assuming that within your SVN repository, you have multiple “sites” that you will be individually pushing or exporting (svn export). Once you have the secure, password protected page (lets call it https://dev.server.com/svn) , the following PHP page will be the main index :

svnupdate.php

If you carefully look at the above code, you will see that this page will be dependent on 3 external scripts, which I will describe below. The page itself generates a list of whatever sites you want to include in the push process, within a PHP based Array. The array details important info per site such as the name, svn location, location of the files on the server as well as whatever other notes and additional info you want to provide.

Each time a site is “exported” by clicking the export button, it calls an external script called svnupdate_process.php. This executes the SVN EXPORT command, as well as logging which user requested the action into a simple text based log file. The user is determined by the authentication user that is accessing the page. The htpassword credentials you will be providing to your users should be set per-user so that it can be easier to determine who pushed the code and whatnot.

The other two external scripts are one that will view the log file in an iframe on the same page, as well as a script to extrapolate the pending commits that are in the queue since the LAST code push / svn export. That is really useful, as you can imagine.

Script to view the export log

This script, log.php is used to dump the contents of the log.txt export log file. Very simple

log.php

Simple, right? The log.php code includes a functions.inc.php file, used for writing and reading the log.txt file. The above code depends on it, as well as the svnupdate_process.php code (described below), in order to log each time someone hits the export code button

functions.inc.php

The code of the svn export process is handled by the following script below. Again its self explanatory. PHP executes a shell command to export the svn code based on the array variables defined in the very first script. Make sure all the variables match up to whats in svn and the location of the files, and even execute a test run of the command manually with the variables. If there are problems, you can modify the command to pipe the output to a log file for further analysis. Additionally you may need to alter the permissions of the apache user so that the command can be properly run. Avoid setting the apache user to have a shell (big no-no) ,but maybe a nologin shell or something along those lines. Its completely up to you , but be very careful about the choices you make to get it to run properly.

svnupdate_process.php

Finally the last script will be the script that parses the SVN log output with a date/time range from the last time the export button was pushed, until the current date and time. This will load the output in the same iframe log window on the svn page so the user can see what pending commits are in the code since the last time it was exported. Invaluable information, right?

Note that this has a function to filter out additional illegal characters to avoid cross site scripting injections. This code should be completely 100% restricted from outside public use, however it might be worth it to put this function in the svnupdate_process.php script as well. Can’t be too careful. I thought I’d include it here for you to use.

viewcommit.php

Lets break down the SVN log command, so you know whats going on. I’m grabbing the SVN site array variables when the “view log” link is clicked on the svn page. I am also parsing the export log text file to get the last entry for the particular site in question, grabbing the date and time.

I am then getting the current date and time to complete the date/time range in the svn log query. The finished query should look something like this :

Occasionally with a very busy site, being behind a hefty web stack does not always have enough capacity to mitigate a significant surge in artificial (DoS) requests. Detecting and mitigating denial of service attacks is an important and time sensitive exercise that will determine the next mitigating steps that you may need to take to reduce or null route the offending traffic.

These steps are very basic and use the everyday system utilities and tools that are found in vanilla linux implementations. The idea is to utilize these tools to identify connection and request patterns.

I’m going to assume that your potential or assumed attack is going straight to port 80 (HTTP), which would be a common assumption. A typical DoS attack would just be a generation of thousands of requests to a particular page, or perhaps just to the homepage.

Check your Process and Connection Counts

It is a good idea to get a picture of how overworked your system is currently, other than the initial reports of slow site performance or perhaps mysql errors such as “The MySQL server has gone away”, or anything of the sort.

Count how many apache/httpd processes you currently have to get an idea :

Check what the CPU load is currently on the server :

So you can see the load is quite high and there are 96 apache processes that have spawned. Looks to be quite a busy server! You should take it a step further and perhaps identify how many open port 80 (HTTP) connections you have :

So thats quite a significant number of HTTP connections on this server. It could be a substantial DoS attack, especially when you consider that this may be one server in a 3 server load balanced pool. That means the numbers here are likely multiplied by three.

Still, it could be legitimate traffic! The spike could be attributed to a link on reddit, or perhaps the site was mentioned on a popular news site. We need to look further at what requests are coming in to be able to determine if perhaps the traffic is organic or artificial. Artificial traffic would typically have thousands and thousands of identical requests, possibly coming from a series of IP addresses.

How distributed a DoS attack probably depends on the skill and resources of the offending party (potentially). If its a DoS, hopefully it will be easily identifiable.

Lets take a closer look at the open connections. Maybe we can see how many connections from singular IP addresses are currently open on the server. That may help identify if the majority of the traffic is from a single or single set of sources. This information can be kept aside after our analysis is complete so that we can use that information to report and block the traffic to ultimately mitigate the attack.

What the above line essentially does is scan the open connections specifically to port 80 and filters only the IP addresses that have 45 or more open connections. This number can obviously be adjusted to whatever you like. Take a look at the different results and see what it produces.

For potentially offending IP addresses, whois them and see where they are originating from. Are they from a Country that typically isn’t interested in your site? If the site is an English language site about local school programs in a small North American city, chances are someone from China or Russia has little legitimate interest in the site.

Analyze the Requests

In this scenario, we would be analyzing the apache access logs in order to get a clearer picture of what exactly is happening that is generating the DoS. Access logs in apache are a great resource to get the source IP, request URI and other useful information that may help identify an automated tool such as LOIC or an automated botnet perhaps that is sending thousands of identical requests to your server.

Lets filter our the actual GET requests from the apache logs, sort them and count each request in order to show the highest number of requests for the same URI. If we can take this information and then cross reference it with the connection stats we gathered earlier, we should have a clear picture of who is conducting the attack and how they are doing it.

This code filters GET requests from the logs, cleans them up, sorts the results, counts all the duplicate requests , sorts it by highest number of requests, and prints the results.

Again the 45 in the last portion of the command can be changed to whatever you feel is necessary. It all depends on whats a normal request, what is legitimate traffic and how busy your server is normally.

All the data you have gathered thus far should be enough to complete a preliminary investigation into your DoS attack. As for mitigating , there are many options. I wont go too much into it because that could be considered a completely separate topic. I’ll give some suggestions for you, either way :

Block offending IPs with IPTABLES :

Use software layer mitigation such as mod_evasive or mod_security to reduce the ability of attackers to generate significant numbers of requests. Most importantly of all, use your best judgement!

Backing up your code repository is important. Backing up your code repository to an off-site location in a secure manner is imperative. Throughout our travels and experience utilizing the SVN code repository system, we have developed a quick bash script to export the entire SVN repository, encrypt it, compress it into an archive, and then ship it (over an encrypted network connection) to Amazon S3 storage.

We will be using the (familiar) s3sync Ruby script to do the actual transport to Amazon S3, which you can find here.

Note also that this script also keeps a local copy of the backups, taken each day, for a maximum of 7 days of retention. This might be redundant since all revisions are kept within SVN itself, but I thought it would provide an additional layer of backup redundancy. The script can be easily modified to only backup a single file every night, overwriting the older copy after every backup.

Here’s the script :

Note how I have provided an example , commented out within the script, on how you can go about decrypting the encrypted SVN dump file. You can also modify this script to back up to any offsite location, obviously. Just remove the s3sync related entries and replace with rsync or your preferred transport method.

I hope this makes your life easier!

We have completely redesigned the look, feel and user interface. I think its much nicer, modern and slick

Our web design division is offering an amazing deal for anyone looking to kick off their design (or re-design) project.

Shift8web is offering 30% off All web design services through the Kijiji coupon service. Click the link below for more information :

Dealing with several different development environments can be tricky. With SVN specifically, it is ideal to have some “pre-flight” checks in order to make sure some basic standards have been followed.

Some of the things you would want to check might be :

- Does the code generate a fatal PHP error?
- Is there any syntax errors?
- Has valid commit messages been attached to the code commit?

I thought I’d share our pre-commit hook in one of our SVN code repositories in order to let you utilize and perhaps expand on it to include many more checks. Additional checks that may be specific to your code environment might benefit you. Feel free to share if improvements are made!

You may also be interested in their intro youtube video : Shift8 Youtube Video

	Fully Managed 24x7x365	Our Managed Dedicated Server hosting service catalog is designed to offload the responsibility of hiring a systems administrator to manage your systems. Our staff are highly trained and experienced in a wide assortment of high availability dependent industries including financial, health, news and more!
	Integrated Reporting and Alerting	Growth planning and trending can be tough. We offer an integrated SNMP based graphing system to track all aspects of your dedicated hardware : traffic, hard disks, cpu load, database reads/writes, web server statistics and many more metrics. Synthetic monitoring from multiple geographic end-points give a full picture when ensuring your critical web property is not only up, but responding in a reasonable amount of time and rendering the page to your own parameters.
	Dedicated Firewall and IPS	With ALL of our managed dedicated hosting packages, we offer dedicated firewall and intrusion prevention devices. We utilize dedicated high availability security and firewall protection devices on the edge of your network. This service is fully managed by us to give you the reassurance that all traffic is stringently monitored and blocked appropriately.
	Offsite Secure Backups	We offer offsite secure backup services to all our managed dedicated hosting clients. This can be arranged for an extra fee.
	Proactive maintenance and Support	Support is offered 24x7x365 through our ticketing system. Proactive maintenance, configuration, installation and administration is offered for all of our managed dedicated hosting packages. We also can assist in migrating all of your web properties to our systems as part of your transition strategy. Our support ticket response time is consistently fast and highly reliable.
	99.99% Uptime SLA	Through our managed hosting service agreement we offer a 99.99% uptime guarantee with an crediting system in the unlikely event of downtime, as well as hardware replacement guarantees.

With Stardot’s Fully Managed Infrastructure services we offer complete end-to-end management of your mission-critical IT environment. We provide data retention, IPS, Load balancing, Database clustering among many more offerings from our service catalog.

Star Dot Hosting is a leading provider of managed hosting services in Canada for American and Canadian clients. We work with you to develop your mission critical strategies and ensure all of your requirements are met.

Our network is designed to provide you with the fastest and most reliable connections possible. Our routing policy is designed to choose the highest quality and fastest connection to any destination, not simply the network that offers the cheapest rates. We connect our network to as many other networks as possible. The closer you are to other networks, the faster and more reliable your service will be.

Between our peering and transit relationships, we are directly connected to every major cable company and telecommunications company across Canada.

Fill out the form below to give our sales associates the initial information required to generate a quotation for our managed services.

Managed Hosting Services Quote Form

	Fully Managed 24x7x365	Our managed VPS hosting packages provide you with the peace of mind that you need, so that you can focus on running your business. Our multiple endpoint monitoring, graphing and alerting systems are just one of the many systems we utilized to ensure your systems are up and attended to.
	Integrated Control Panel	Need to see how much bandwidth you’re using? Want to attach to the server’s console? Need to initiate a power cycle? Our integrated control panel allows you to connect to your managed VPS instances to get critical information, statistics and control.
	Dedicated Firewall and IPS	With ALL of our managed VPS packages, we offer dedicated firewall and intrusion prevention devices. We utilize dedicated high availability security and firewall protection devices on the edge of your network. This service is fully managed by us to give you the reassurance that all traffic is stringently monitored and blocked appropriately.
	Integrated Weekly Backups	For additional redundancy, your managed VPS instance is imaged and backed up on a weekly basis. Secure offsite backups are offered at an additional fee.
	Proactive maintenance and Support	Support is offered 24x7x365 through our ticketing system. Proactive maintenance, configuration, installation and administration is offered for all of our managed VPS hosting packages. Our response time is consistently fast and reliable.
	Fully Managed Amazon Cloud	Want to establish your business with the Amazon Cloud infrastructure? We’ve been managing Amazon Cloud instances for our clients for years. We consider ourselves highly experienced at implementing, monitoring, integrating, maintaining and transitioning businesses to the Amazon Cloud infrastructure. Amazon offers a wide assortment of services that could help bring your business to the next level on the largest cloud platform in the world.

Star Dot Hosting’s Managed Virtual Private Server service catalog is the preferred solution for clients who need to have the full control similar to a dedicated server, and the assurances of a fully staffed 24x7x365 managed services team ensuring their services are stable.

Managed VPS customers can choose any Operating System that meets their requirements (Linux,Windows,Unix). Our monitoring services are included that provides 24x7x365 alerting, trending, monitoring and graphing of their VPS instances health and also helps growth and expansion planning.

We provide the exact same service catalog as our managed dedicated services, except on a virtual platform. Everything from failover, load balancing, data and database replication, monitoring/alerting, trending and graphing are fair game.

We would love to work with you to design your architecture from the ground up, without the added expenses of deploying dedicated hardware. Please fill out the contact form below and one of our technical agents will quickly follow up with you to ensure we have all the information we need to provide an accurate and thorough quotation for managed vps hosting services. Please fill out the form below to have one of our sales associates build a Managed VPS quotation to suit your needs.

Create A Custom Quote

99.99% Uptime Guarantee

Star Dot Hosting will provide a minimum 99.99% uninterrupted access to your web site, email, VPS and other related services.

Should your services become unavailable for a cumulative period beyond the allowed 0.01% in any month of service, the client will receive a credit equivalent to 5% of the client’s pro-rated recurring monthly fees for that month and then an additional 5% for every additional 15 minutes the web site and/or other related services are unavailable.

  ]]> http://www.stardothosting.com/unmanaged-vps/feed/ 0 http://www.stardothosting.com/company/?utm_source=rss&utm_medium=rss&utm_campaign=company http://www.stardothosting.com/company/#comments Sun, 19 Aug 2012 04:52:15 +0000 Kevin http://www.shift8web.com/projects/stardothosting.com/?page_id=8
Tired of navigating an endless sea of “resellers” utilizing larger managed hosting company resources and pretending to be something they are not?

Star Dot Hosting was founded over 6 years ago with 3 goals in mind : simplicity, trust and reliability.

We are a small full-service facility built around ensuring ongoing client satisfaction in the face of constantly changing business requirements.

We were built on the idea that we would like to offer our clients the best return for their IT endeavors without the maximum investment.

Our professional team of experts can help your company find an affordable solution that fits your hosting needs. If web hosting is a serious part of your business then choose a company who is serious about providing quality web hosting solutions. We have worked hard to build a reputation of customer satisfaction through technical excellence, implicit reliability, and friendly staff.

Our Team

Our team has several decades of collective experience in the hosting industry. We have been around to see the evolution of the industry and have the clarity to see where it is headed. We have the confidence and vision to take our customers into new ventures while utilizing bleeding edge technologies with a concise understanding of the underlying structure.

We believe that one of the main distinguishing factors of any service oriented company is customer service and customer satisfaction. Our #1 goal is ensuring that the service you receive is exemplary and that it exceeds your expectations.

We are passionate about this industry. We love solving problems, developing new solutions and watching your business grow.

We take great pride in knowing that our customers are consistently impressed with our high standards and fast response times. Our ultimate goal with our clients is to ease uncertainty, reduce stress and solve problems. Period.

We threw together a quick & simple script to dynamically update your .htaccess files within apache to add your dynamic IP address to the allow / deny fields.

If you’re looking to password protect an admin area (for example) but your office only has a dynamic IP address, then this script might be handy for you.

Its an extremely simple script that polls your dynamic hostname (if you use no-ip.org or dyndns.org for example) every 15 minutes as a cron job and, if it has changed, updates the .htaccess file

Hopefully it will make your life just a little bit easier

Sample Cron entry :

And now the script :

We have recently been implementing several different backup strategies for properties that reside on the Amazon cloud platform.

These strategies include scripts that incorporate s3sync and s3fs for offsite or redundant “limitless” backup storage capabilities. One of the more recent strategies we have implemented for several clients is an automated Amazon EBS volume snapshot script that only keeps 7 day retention on all snapshot backups.

The script itself is fairly straightforward, but took several dry-runs in order to fine tune it so that it would reliably create the snapshots, but more importantly would clear out old snapshots older than 7 days.

You can see the for loop for deleting older snapshots. This is done by parsing snapshot dates, converting the dates to a pure numeric value and comparing said numeric value to a “7 days ago” date variable.

Take a look at the script below, hopefully it will be useful to you! There could be more error checking, but that should be fairly easy to do.

Hello,

I thought I’d update our blog to let everyone know that we offer our extensive managed services catalog on our VPS infrastructure!

If you are looking for high quality managed services from systems administrators with extensive experience, then contact our sales department for a consultation / quotation!

Find below just a SAMPLE of some of the managed services we provide :

Support Services
- 24/7/365 Technical Support (Ticketing system)
- 24/7/365 Alert monitoring on-call rotation pager system
- Online Customer Service Center

Linux Administration
- Installs, reinstalls and updates
- Trouble Shooting and configuration
- Apache configuration, optimization & Setup
- Linux X64 installation, optimization and maintenance
- Security optimizations
- Proactive and reactive maintenance
- Installs, reinstalls and updates
- Trouble Shooting and configuration

Managed MySQL and MSSQL
- MySQL and MS SQL first time install service
- MySQL maintenance and troubleshooting

24/7/365 System Availability
- Availability monitoring (Ping and Synthetic Transactions)
- Performance monitoring of key server parameters
- Graphing and trending

Security Services
- Routine Security Patching
- Routine Security scanning and auditing
- Penetration testing, SQL Injection
- Quarterly security audit reporting
- Dedicated hardware firewalls for all clients

Hardware Maintenance
- Hardware failure alerting and monitoring

Offsite Backups
- Remote off-site backups per 50gb nightly

Occasionally some of our managed services work has us dealing directly with other cloud providers such as Amazon. One of our clients set a requirement to migrate over 5,000 domain’s to Amazon’s Route53 DNS service.

There was little doubt that this could be automated, but since we have never done this massive of a deployment through Amazon’s API directly, we thought it might be interesting to post the process as well as the script through which we managed the import process.

Essentially the script utilizes a master domain name list file as its basis for looping through the import. The master list refers to the bind zone files and imports them into Amazon’s Route53 via the Cli53 tool package.

One final note, the script outputs all completed domain imports into a CSV file with the following format :

This is because when facilitating the actual nameserver change request, all the nameservers assigned to domains when imported to Route53 are randomly generated, so the script has to keep track of these nameserver/domain associations.

The script isn’t perfect and could benefit from some optimizations and more error checking (it does a lot of error checking already, however), but here it is in its entirety. We hope you will have some use for it!

MySQL replication has been known to easily break, as a result of a large multitude of potential causes.

Sometimes the replication can even break if an erroneous query is executed on the master server.

With all the potential issues that may break replication, we thought it prudent to write an automated check script that can run on a scheduled basis (i.e. every 10-15 minutes), check the Slave status, report on any errors if applicable and attempt to repair replication.

We have built this script to exit and send mail alerts if any step of the checking and repairing process fails or generates an error in itself.

The script also generates a lock file to ensure that no more than one check process can run at any given time. We feel this script could be best used for scenarios for remote MySQL slaves, for example. Adding this extra layer may ensure a more reliable replication.

The repair process is simply 3 MySQL Commands :

The above directives assume that you have a master.info with the mysql master server information statically set. No CHANGE MASTER commands have to be executed as a result. Resetting the slave clears the error and resumes replication, and all the queries missed during the time it failed should be queued and applied after it starts again.

Here is the script :

This update is to highlight the managed hosting services and systems administration services offered by Star Dot Hosting.

We are proud to offer some of the highest standards of managed hosting services in the industry. Here are some of the services we offer :

- 24x7x365 Monitoring, Alerting & Trending
- 24x7x365 ~15minute response time support ticketing system
- Quarterly Security Auditing & Reporting
- Patch management & Maintenance
- Administration, troubleshooting, deployment and code pushes
- Hardware upgrades, deployments, expansion
- Dedicated load balancing
- Dedicated firewalls
- Database replication & clustering
- Dedicated edge content caching

Those are just some of the services we would provide to you! Our quotation process involves rigorous consultations which includes technology assessment and especially a complete understanding of how your scenario can be designed and where improvements can be made, if any.

Feel free to click the link below to visit our managed hosting page, and fill out a quotation request form!

I thought I’d share a quick script in the scope of backing up GIT repositories for the purposes of encrypted and compressed off-site backups.

Unfortunately git does not have an equivalent of svnadmin dump or export, which can conveniently be piped to stdout.

What the above scenario would do is shorten the amount of commands a script would require in order to accomplish a similar task.

Find below a quick bash script that clones a repository, tar/gzip’s it, encrypts the archive and keeps 7 days worth of archive files :

FYI if you ever needed to decrypt the openssl encrypted backup archive, the command below should do the job :