As we mentioned in the earlier system message, the site is up and running on the new hardware now.

The transfer went pretty smooth, with one exception: sessions were enabled incorrectly when we first brought the site back up, we noticed a few minutes after and fixed the issue, but anyone who accessed the site at that time then got left with an invalid session and would receive an error when trying to view your guild site.

If you or someone in your guild was affected by this and is seeing an error message when attempting to view your guild site, fully close your web browser and re-open it and the issue will then be fixed.

Other then that, everything looks good and we are back to work on the theming update now and hope to have another preview of it ready by next weekend =)

We will be performing hardware upgrades on Wednesday October 3rd, adding additional CPUs and RAM to our webservers.

Expected downtime is 30-60 minutes. If you attempt to access the site while the upgrade is occurring you will not receive the standard maintenance message, the site will simply not load as it will be offline for the upgrades.

We have asked the technicians at our hosting facility to perform the upgrades as early as possible on Wednesday, and are expecting them to happen around 9-10am PST (12-1pm EST), though they may occur slightly before or after this.

This evening on Wednesday May 23 from approximately 9pm - 11pm EST we had some webserver problems.

Our monitor script responsible for keeping our webservers alive crashed; this then caused a few of our webservers to eventually die and not be revived by the monitor script, then causing the remaining webservers to become overloaded and the site to become very unresponsive.

Everything is now in working order, and we apologize for the slowdown caused by this.

This morning a large update was pushed live to the servers geared towards improving performance.

We began writing MMO Guildsites way back in January 2006 and it was overdue for some house-cleaning of our code base. We trimmed some unnecessary fat, re-factored many core components and generally put a lot of polish on the underlying code for MMO Guildsites.

In reaching for better performance we tried a new strategy also; previously each "block", such as a shoutbox, server status block, text block, etc., was pulled in separately as a component after the site. This meant each page view took longer to load as each block had to be pulled in after the site's main content. We decided to try and remove that delay by pulling everything in as one request; it worked great while testing but when we moved it to the live server.... it failed horribly >_<

Each guild site can be very dynamic, it not only pulls in the content you want to view but the theme and layout for the site, each block and its content, and your user and permissions. Attempting to ball it all up into one request gobbled up oodles of memory and caused our web server to throw a fit.

In the end we had to undo that part of the update, it resulted in a fair amount of server issues this afternoon which was very unfortunate and I hope never put a damper on anyone's weekend.

On Tuesday April 3rd there will be a brief period of downtime as we add additional RAM to our server. The extra RAM is being added to support more ruby webservers for MMO Guildsites. This will allow us to handle more requests and handle them quicker.

The hardware upgrade is scheduled for 2pm EST on April 3rd, expected downtime is to be 30-50 minutes.

Well, we tried really hard to make Amazon's S3 work for all of our static content; but at the end of the day it was too slow. [it is actually very fast, but there can be an initial delay connecting to it; which makes it seem slow for small files]

The lag that was previously seen when you visit your site and see only a styled site, with the images coming in after was due to S3 lag. Creating themes was especially slow as each uploaded file was transferred there as well; and whenever an operation is slow it can clog the whole network; all around it was working out poorly.

We stripped out S3 from most of our service now, user avatars and all of the theme data is now served locally from our webserver. This should result in a noticably faster experience for everyone. We will only be using S3 for the image galleries and for the file uploads section [which we'll be adding soon], which will be larger files where the initial delay to connect to Amazon S3 doesn't matter as much [S3 is very fast once you're connected to it]

*Note: Over 50,000 files had to be transferring from S3 to our local server, we've been checking and haven't noticed any missing files so far but may have missed some; if an image file is missing let us know

Just wanted to update everyone on what has been going over the last few days with regards to our new hardware.

Our move to the new hardware a little over a week ago went great and we had almost no downtime and noticed a great boost in performance; but we started to notice a few problems where performance would dip, and it would often be during low traffic.

We discovered there was a few problems with our new setup; a few odd issues were causing some webservers in our cluster to silently drop off and stop receiving requests (as more webservers drop off performance degrades).

After a few days of little sleep, lots of coffee and searching far too many log files we have fixed these problems and webservers are no longer dropping out of the cluster.

Performance over the last two days in particular may have been much worse then normal as we had to reboot the webserver many times attempting to resolve these problems.

With this issue resolved we now move back to more fun hacking: adding new features

Most users should have propagated to the new DNS entry now for our domains and should be seeing the new site [you will know you are on the new site if you do not see a warning notice on your home page].

The webserver upgrade seems to be successful with no breakage, and things seem much zippier now. We will continue to make a few enhancements to the speed over the next week or two.

There was fortunately little downtime other than a small rough patch early this morning, and future upgrades should require no downtime as we are now in a clustered environment and will just stack on additional webservers as needed [no DNS entry update required]

Now back to checking off items on the feature list! =)

Today we are moving to a much more powerful and robust setup to handle the traffic MMO Guildsites is beginning to receive.

A notice of this has been placed on every guild site. The important part for guild members is to know that as long as they see the notice on their site they are connected to the old site and data entered into their site from now until they are on the new site and the notice disappears will not be transferred.

This is due to how the move process is handled, which I'll attempt to explain. We currently have two webservers online, the old one and the new one. The database has been transferred to the new one already. We have now made a change to the DNS entry for the www.mmoguildsites.com domain name and the www.mmoguildsites.net domain name; this DNS change takes up to 24 hours to propagate through the internet; so some users will be seeing the new site while others may still see the old one. The database will remain current with the new webserver, so users who are slower to propagate will see the old site, which is connected to the old database.

Long story short, until you see the notice message disappear from your guild site do not write long forum posts as they will not be transferred =)

We are moving to a much more complex setup on the new webserver with fancy load balancers managing clusters of web servers and other goodies. Because of this there is the risk of some downtime if some configurations between the old and the new clash.

As some may have noticed, we are nearing the capacities of our current webserver and it is beginning to affect the performance of our service.

We have begun the process of upgrading to a new and more powerful setup to handle the traffic increase; we are expecting to be up and running on the new hardware this weekend.

Downtime should be minimal during the transfer. We will post another blog entry when the transfer is complete.

Update: Unfortunately a configuration error on our new webserver has caused a delay and we will not be able to begin moving to it until Tuesday, January 16th. We've made several changes to increase the speed on the current webserver until we can complete the move. Sorry for the delay.