Paul Marsh Has a Blog - Home

Passenger in Production Take 2

2008-07-23T21:44:00Z

Well after a quick failure with Passenger on my larger production box I reported my problems and low and behold there had been some fixes committed that Friday that would have prevented my problems had I been following the development versions.

This time things went smoother. I compiled Ruby Enterprise 20080709 and then pulled the latest updates to Passenger and recompiled that. After again making sure monit wasn't going to flip out that the mongrels were coming down, a quick restart of Apache and we were off and running.

Now I haven't done any real number collecting, but the site does feel faster when rendering a new page.

We've been up and running on Passenger for almost 2 weeks now and things have been solid. No random memory spikes or Apache alerts.

Keep Internet Memes On The Internet

2008-07-14T20:42:00Z

There's been either a growing trend, or an existing trend I'm finally catching onto, where people use internet-speak in real life conversations. Might as well play the sounds of nails on a chalkboard as far as I'm concerned. A sure sign that I'm getting old.

I remember the time my parents tried to use the Wayne's World slang, "NOT!". I still cringe. I have the same reaction to hearing people say they did something "for the lulz", or "such and such was full of fail." A perfect example would be the guy with purple hair towards the end of this amusing video blog by DavidJr.com.

Now I'm no scholar, I have my own problems with proper speech, but my God does this not bother anyone else?

Are you one of the people who talks this way? I certainly hope you keep it between friends because you genuinely sound like a moron. What's next? Should we just troll around 4chan and start picking up new lame terms to break out in normal conversation.

Internet memes can stay right there on the internet please.

Rails Development Under Windows XP or OS X or any other OS

2008-07-11T21:45:00Z

Just what is the best way to develop Rails applications? Well my answer is a question, what's your production environment?

If you are currently developing your Rails app under Windows XP or OSX and deploying to a Linux production box, why are you not running a copy of whatever Linux flavor your production server is on in a Virtual Machine?

The only reason I could see for not doing so would be because you are lacking sufficient RAM. Which I'd find it hard to believe as it seems everyone is running around with 2GB or more and I can tell you that even with 1GB you can get away with dedicating 128MB to a VM instance. There are plenty of VM programs out there, VMWare Server and VirtualBox if you're on a PC, or for the Mac, VMWare Fusion and Parallels on the Mac .

Plus probably a dozen others that all can do the job.

After you get your favorite flavor installed with the same versions of packages that are running in production turn on samba and share out whatever directories you need. I have symlinks in my home directory to my various apps and simply share out home directories in my smb.conf. Then mount that share on your host OS. In my case my home directory is mapped to H:.

Now you can edit files from that drive with your favorite editor. Or you can do what I do and SSH into the guest OS and edit there. If you need X applications like gitk get an X server running on your host and ssh into the guest with XForwarding turned on.

All in all now you can deal with situations like "I can't get this gem installed on XP/OSX", "For some reason when I deployed my app is segfaulting", "I wonder if my app will run under this fancy new app server". Everything you do and test will be under the same environment as your production saving you time and headaches.

For me at some point along the line I realized that writing my Rapture CMS code to run on both Windows and Linux was a exercise in futility. It ate up way too much time so I decided to focus on just writing for my production environment, which was and is RedHat Enterprise (RHEL). This was back pre-0.13 days and trying to get certain things to work right under Windows XP, wether it was ruby, gems, Apache, whatever was just eating up time. It probably still would.

Using VMWare Server machine of an install of CentOS I was easily able to mimic my production environment. Yes I realize it's not a perfect mirror, but it's close enough. It has 440MB to play with out of the 2GB of RAM I have on my laptop and 12GB of hard disk space dedicated to it. More than enough to run the Mongrels I need to do debugging.

Give it a try next time you can't get that random gem to compile and it's driving you nuts for a day.

Passenger with Ruby Enterprise in Production Take 1

2008-06-30T21:35:00Z

Well Passenger had been running on this VPS nicely so why not try it out on a larger site? My server could use a nice decrease in memory usage, simpler Apache config files, and better concurrency.

Using the 06242008 release of Ruby Enterprise and Passenger 2.0.1 I converted over a site with 8 Mongrels that's been running very steady for almost a year now.

Everything seemed to be going well enough for about a day and a half and then while finishing dinner Saturday night I get an alert message that Apache has crapped out. Getting back in front of a computer I see that the load on the box is fine, but Apache logged an error saying that it reached its MaxClients. Considering I have that set to 400, anything over 256 requires some custom tuning, this was surprising.

Apache had been restarted automatically by monit, but that's not what I wanted on a production box. So it's a quick switch back to Mongrel for now!

See the chart below that illustrates the Apache crap-out.

Passenger (mod_rails) with Mephisto

2008-04-22T16:07:00Z

With what appears at first glance as a solution to the "complicated" problems of hosting Rails apps Passenger could make the Rails world an even happier place to live. Less administration and more coding.

Well that was too easy. Following the simple Passenger instructions on this CentOS 5 box and now the blog is running on mod_rails 1.0.1. Passenger runs using the user/group of your app's config/environment.rb so some changes in permission for my cache and asset directories was all I had to do. Memory use doesn't seem to be any higher than a mongrel and I haven't seem any performance problems. Not exactly a high traffic site.

If this proves to be stable Rails just took a big leap for many sites. Using an app server still offers some nice benefits but at this point I'll be quite happy installing Passenger on most hosts.

Time will tell though.

UPDATE Just FYI for anyone using 1.0.1 if you have Rails' page cache directory set to something other than public/ Passenger currently doesn't honor that. Hongli on #passenger on freenode has informed me that's fixed in the development version. So it should be in the next release.

UPDATE 05/07/08 See graph below to see the difference in memory usage between the site with Mongrel and the site with Passenger. No problems yet to report with using Apache Worker with it. Upgraded to Passenger 1.0.4.

UPDATE 05/23/08 The second graph shows two changes. The first was I dropped the number of MinSpareThreads and MaxSpareThreads to cut down on the Committed memory. For a rundown of what the committed memory on munin means I would refer you to this RedHat Article.

The second drop was when I paid attention to the Passenger Documentation and dropped my RailsMaxPoolSize in my Apache configuration file to 2. Now running passenger-memory-stats I show less than 7MB of dirty memory.

Also a great post by Technoweenie over at NubyOnRails about mod_rails

UPDATE 06/19/08 Upgraded to Ruby Enterprise, also from the Phusion guys and the latest Passenger from Github. In the third chart you can see that, around the 17th the memory usage dropped, although Swap usage bumped up a little bit. I'll see over the next week if committed memory remains in a lower range. That gap of space in the chart is due to a server crash. One which I am not sure of the cause unfortunately.

What is interesting to me is the drop in the Individual Interrupts and Interupts and Context Switches. Not sure what they do yet though. I'll dig up some Google links in a bit.

When you do install Ruby Enterprise though, be sure to install the necessary gems for your app. I forgot this step and missed installing tzinfo and image_science. If you install to /opt/ like the default install recommends just run /opt/ruby-ent/bin/gem install tzinfo image_science.

Oh, and if you're on a VPS I would suggest running gem install with --no-rdoc and --no-ri for a faster install. Unless of course you're developing on it at the same time.

UPDATE 07/08/08 After a couple of weeks things are looking good. The monthly memory graph below shows that my committed memory is now staying under the total available memory which is good. There haven't been any of those random spikes/crashes that occurred early on so I'm happy to keep Passenger running on this VPS server.

OpenX (OpenAds) 2.4.x Performance

2008-03-10T20:38:00Z

Well, if a server is going to turn on you suddenly like a good dog gone bad I guess having something of a clue as to why is better than none. Today we had our largest spike in traffic in several months and everything held up fine, except the OpenX server wigged out shooting load up to 190! Let's see what you can do if this happens to you.

I should have known. Isn't it one of Murphy's laws, you take one day off and that's the day a server decides to wig out? Well that's what ended up happening. From my other post you'll know we recently brought our ad server up-to-date with 2.4.2 of OpenAds/OpenX.

At that time there was a bump up in the CPU usage on the webserver with the MySQL server seeming to take things in stride. Things seemed to be relatively normal since then with just that increase in CPU usages, but the patterns were close to what they were over the course of a day for the past year.

Then on this past Friday we had a huge jump in traffic compared to the norm. Well that bump caused an exponential increase in the load on the box. Started getting notifications that things weren't quite right and spent the morning trying to get things fixed.

First off we've had spikes of this nature in the past while running 0.3.x of Openads without this kind of load increase so this was pretty surprising. After trying Xcache using the source rpms from Jason Litka and enabling it not much changed. Although the amount of RAM being used by the cache went up compared to eAccelerator, which is good and as a side note you can give more memory to eAccel as well, the overall CPU usage stayed way too high.

Time to look more closely at OpenAds' configuration for this box. Everything looked right, caching is turned on and working, file store is set to be in a local directory, check. Wait, there's nothing actually *in* that directory. A frantic couple of questions to #openads on freenode confirmed that ads were being pulled from the database. Not sure if that was the problem, it couldn't hurt to move the images from the DB into the directory. Fortunately Openads under settings has a link to a script under Settings -> Maintenance -> Storage -> Move images stored inside the database to a directory.

Just one small problem it seems that over time there must be some banner records that are 'dirty', meaning although they are in the DB something isn't quite right causing the script to throw an error in lib-storage.inc.php on line 224. It seems that it is trying to delete an image from the DB if the storage type is SQL after moving that image to the filesystem. Problem is, if there isn't a file to delete the script errors out.

Here's a quick ugly fix :

if($doImages) {
$doImages->delete();
} else { echo "ERROR :".$name."";}

With that in place I've noticed only one banner that actually seemed to lose its artwork.

After dumping out the images the load on the server started to drop and settled under 0.75 for the rest of the day and into the next even as traffic remained relatively high.

Next is to upgrade to 2.4.4 and address some other admin interface slowdowns.

UPDATE Bug was reported and tagged as invalid https://developer.openx.org/jira/browse/OX-1832. Not sure how my records got screwy but I think an error message out to the user would be helpful. Anyone out there ever run into this problem and have some info on what caused it for you?

OpenAds 0.3.30-alpha to 2.4.2 Upgrade

2008-02-21T19:38:00Z

There's a nice clear upgrade path from 0.3.31-alpha-pr3 but not from 0.3.31-alpha. Thankfully OpenAds is a well written app and rolling up to 0.3.31-alpha-pr3 wasn't much of a problem.

EDIT : Totally not true any more with the 2.4.4 release.

Well now of course OpenAds has become OpenX since I've been dragging my feet on getting this up. However that doesn't change the underlying code much and the changes from 2.4.2 to 2.4.4 aren't that great.

What's even nicer is in the release notes upgrades from v0.1.29-rc now appear supported. 2.4.2 required v0.1.31-alpha-pr3. So I guess this post is a bit deprecated but might still help some. The install/upgrade procedure on the OpenX site is also recommended reading.

Let me get back to the initial upgrade that I meant to be writing about. The install that I was working on upgrading has been in use since way back when OpenX was known as phpAdsNew and version 2.0 was not released yet. As such sometimes upgrades require just a little more manual massaging.

While v0.3.30 was running well enough there were plenty of change in the move to 2.4.x and it's always good to keep current especially for security. My main goal was to speed up things a bit more on the admin site of the install. Along with the code upgrade I also switched all my tables to MyISAM type instead of InnoDB.

The CPU usage graphs below is from my webserver and the other is a query graph of my DB server. There is an increase in overall CPU usage once the upgrade was in place on the webserver went up a bit. The two query graphs show that my cronjob for maintenance seemed to have stopped working. Once it was enabled the queries went back to normal.

Overall Openads/OpenX continues to support what we need extremely well. It's versitile to handle some interesting ad formats without requiring much tinkering.

As always remeber to back up those databases before preforming upgrades.

CakePHP on Dreamhost Linux

2008-02-14T20:14:00Z

Really not that difficult, but there are a couple of trip-up spots.

Let me say this first, GoDaddy is a miserable experience. I will back off a little and say perhaps it's just been a very long time since I've dealt with a shared host, but that doesn't change the fact that their web interface times out, is slow, and cluttered.

That all aside I needed to get a CakePHP based app up and running on an account there. There are some google results that pointed the way.

First of all if you have a relatively new GoDaddy account hosted on Linux you'll be able to use .htaccess files. Windows hosts seem to be out of luck. Of course the rewrite rules in .htaccess files are needed by CakePHP to function properly.

Before uploading your app edit app/webroot/.htaccess and change :

RewriteRule ^(.*)$ index.php?url=$1 [QSA,L]

RewriteRule ^(.*)$ /index.php?url=$1 [QSA,L]

Just an additional slash. GoDaddy has some of its own rewrites that seem to enjoy getting in the way.

Now the fun part that caught me off guard and doesn't seem to get any mention in GoDaddy's Help Documentation, wait 15 minutes and check your site. So I would deploy your app to a subdomain first then move it to production to make sure everything is actually working properly.

And that's about it. Seems to be all you need to do for a 1.1 and 1.2 Cake apps.