Web Log of Greg Mathews

OK its been a fun week, server load was still high on the 3 servers I had and I couldn’t pin point the problem so I threw a 4th server at it.

This server was just a NFS server (file server), before this all files where stored on the web server and the database servers connect to that, now with a dedicated nfs server the load should reduce.

Yes it did but  not by enough so I then ordered another web server and 2 database servers. I now have 4 db servers all using replication to keep them in sync and 2 web servers. I was going to use a load balancer but this proved costly after just a short test period so I opted to try a basic Round robin DNS, which although not a true form of load balancing it should work for my needs.

OK things not good, the webservers pick a random database to connect to each time they are accessed, this just does not work. You end up at times with one db server overloading which in turn overloads the web server so I needed an effective means of load balancing the db servers.

I then came up with a small script which runs on one of the web servers. This reads the load of each of  of the db servers and writes the ip address of the db server with the lowest load to a file, each of the scripts on both web servers then ‘requires’ this file so they access the db server with the lowest load.

It’s early days yet but it seems to be working well, the required file is updated a couple of times a minute so the db servers should never end up with a high load and hence the same for the web servers.

In heinsite if I had done this a week ago then I might not have needed the extra servers but its better to have too  much then too little and I do plan to get more traffic so the servers will be utalised.

Happy new year one and all

OK it’s been heck of a week.

It was the general concesus that using an iSCSi was not suitable for my needs so that has gone out the window and I have reverted to using NFS accross the network.

I moved  the scripts over to the new server after making a few minor changes to it and tested the scripts worked fine before I updated the DNS which all appeared to go great.

I had decided not to move all the html files thinking these would be recreated by the script. I was right, but as there were 700,000 html files  this took so much resources that the site became unaccessable, what a disaster.

I switched the dns back and spent the next few hours transferring the files accross. Read the rest of this entry »

I have been talking to a System Admin company and they are giving the impression that using GSF and iSCSi may not be the best solution for what I want, they seem to be saying that NFS should not be causing a bottleneck if set up correctly.

Will see what they say today

Well I have decided to upgrade some of my servers, I have ordered 3 new servers from a different host. Why changing host, well I am not exactly I will still have some servers here in the UK with my existing host but the host I have just ordered from are in the USA and offering a better spec server for the same money and I don’t have to order a private network as their servers are connected to a private network by default.

For those a bit more tech savy, I have also ordered a iSCSi drive and intend to use that with GFS rather than the NFS system I am using at the moment which I hope will reduce any bottlenecks that are currently occuring.

I will update as I progress.

Well eventually 123reg.co.uk have admitted taking to much money from me and not renewing my two domains. It took a whole week for them to admit it but they have.

Their support desk have told me that they have credited my card for the amount in full but what about the phone calls at 10p a minute? the amount of time I was on the phone to them they have probably got most of that back in profit from the calls.

Oh well lets wait to see how long it is before I actually get the money back into my account.

123reg.co.uk is apparently ‘the UK’s largest domain name registrar, web hosting and email provider’ and was voted ‘Best Web Host 2006′ by the readers of Webuser.

I’ve used them for years to register my .UK domains and have not had a problem until a few days ago. On Friday I tried renewing 2 domains via the 123reg.co.uk control panel, renewing two domains for two years cost £13.58 (about $27 USD). Everything went through as it should but I noticed that after the renewel process had completed and my card had been charged both domains still showed as ‘due for renewel’

Read the rest of this entry »

When discussing web sites, we all talk about getting the content right, this in turn leads to better chance of getting traffic and obviously planning the content is important.

One thing that doesn’t appear to be mentioned that often and can be extremely important if the content does its job and attracts a lot of content and that is planning how you are going to host your site, and being prepared for whats ahead.

If your site is busy, this could be because it gets a lot of traffic and/or is dynamic so uses more resources than a basic static site then you need to monitor the server hosting your site to ensure that you are not going to run out of ‘allowed’ resources.

Read the rest of this entry »

This is a strange one. Over the past month the traffic to my site has been growing nicely thank you, reaching 140,000 visitors last tuesday. This was causing my server to overload so I thought I would get a 2nd server and use round robin DNS to rotate the traffic between the two servers.

So I did just that on Tuesday, over the next day or so I spent some time playing with the new server, stopping cron jobs running that used a lot of resources and I didn’t really need, mainly ‘Direct Admin’ tasks but by Thursday I had everything set up.

Great, more servers, more resources to cope with more traffic. But No, since I set up the round robin DNS my taffic has dropped drastically. On Tuesday 5th June I was seeing an average of 5800 visits an hour, by Saturday 9th June this has dropped to an average of 3700 visits an hour, thats a 37% drop in a matter of 4 days.

Traffic from the US dropped by 41% over those few days, India dropped 60% and the UK dropped 34% in just a few days.
Read the rest of this entry »

For the ten years I have had a web site every server has used a Server Alias in the web server configuration so that domain.com and www.domain.com both point to the same site.

This is great as it means that the site is accessable with or without the www. in front of the domain - I am surprised the amount of sites where this does not happen.

I found out recently though that by having a server set up this way could lead to problems with search engines as they are treated as two seperate sites and could therefor be punished for duplicate content.

There is a way around this problem using a 301 redirect. What a 301 redirect does is tell the visitor/search engine that the url they are trying to access has moved. So you could set up a 301 redirect to redirect all traffic from domain.com to www.domain.com or visa versa. This way the search engines will treat both urls as the same site and duplicate content would not be an issue.

A 301 redirect can be set up in either your servers configuration file or in the .htaccess file.

Here is an example based on a 301 redirect I have set up in my server configuration file:

RewriteEngine on
RewriteCond %{HTTP_HOST} ^www.domain.com
RewriteRule (.*) http://domain.com$1 [R=301,L]

This redirects http://www.domain.com/anything.ext to http://domain.com/anything.ext