926: resource overutilization with apache/linux

hashi******@gmai***** (Google Code) (Is this you? Claim this profile.)
Sept. 1, 2009 
What's the URL of the page containing the problem?
  All pages are affected

What steps will reproduce the problem?
1. Have a lot of diffs being submitted / reviewed. Bigger changes help
2. Do this for ~6-12 hrs.
3. Apache is locked up, load-avg on the box is ~25. Need to restart (at
times even kill -9) apache for the box to become normal again.

What is the expected output? What do you see instead?
  Expected: System behaviour is normal.
  Actual:   Load average > 20; need to restart apache for things to become
normal

What operating system are you using? What browser?
  Ubuntu 8.04 (hardy), multiple browsers (Firefox, IE, Opera, Konqueror)

Please provide any additional information below.
  It looks like review-board is overutilizing resources someplace, 
  causing load-averages >> 1 (we've seen 25-30), at which point 
  reviewboard becomes unusable, and we need to forcibly restart 
  apache to resolve the problem.
  
  Hardware: 2 GB RAM, Xeon 2.66 GHz. Free disk space 16 GB.
#1 vinay%djgr**********@gtempacc******** (Google Code) (Is this you? Claim this profile.) 
What's the configuration on apache? Is it using mod_python? fcgi? could you provide more information?
chipx86
#2 chipx86 
Also, what kind of MPM are you using in Apache?

We run Review Board at VMware on a machine with similar specs, with a thousand or so
users, without these problems. We're using the prefork MPM.

Also, are you using memcached?
  • +NeedInfo
#3 hashi******@gmai***** (Google Code) (Is this you? Claim this profile.) 
We're using mod_python (no fcgi), with mpm-worker installed but not enabled. Apache's
mem_cache module is disabled, but memcached is running.

Here's the settings / package versions. All are stock ubuntu 8.04 packages.

dpkg --list:
ii  apache2                               2.2.8-1ubuntu0.3
ii  apache2-mpm-worker                    2.2.8-1ubuntu0.3
ii  apache2-utils                         2.2.8-1ubuntu0.3
ii  apache2.2-common                      2.2.8-1ubuntu0.3
ii  libapache2-mod-python                 3.3.1-2build1   
ii  libmemcache-dev                       1.4.0.rc2-1     
ii  libmemcache0                          1.4.0.rc2-1     
ii  memcached                             1.2.2-1         

ps -ef | grep memcached
... /usr/bin/memcached -m 512 -p 11211 -u root -l 127.0.0.1
chipx86
#4 chipx86 
Apache's memcached support isn't really too important for Review Board. It may
actually cause problems, I don't know, but the key thing is that Review Board itself
is configured to use memcached. Otherwise there will be major performance hits. You
may want to verify this in your conf/settings_local.py file in your site directory.

I'd also recommend the prefork MPM, as we had some issues before switching to that.
david
#5 david 
Any updates on this?
#6 hashi******@gmai***** (Google Code) (Is this you? Claim this profile.) 
Sorry - forgot to update this thread :(

We're not seeing the performance degradation that we used to (and we didn't really
change any configuration).
david
#7 david 
Glad to hear it. We made a bunch of improvements to the performance of the dashboard,
so maybe that was it.
  • -NeedInfo
    +Fixed