3022: API requests create a ton of entries in the 'sessions' table.

ee**@plaxi***** (Google Code) (Is this you? Claim this profile.)
What's the URL of the page containing the problem?

Describe the problem and any recommendations below.
I would like to build an index for my site. However, the 'index' command seems to be invalid.

Please provide any additional information below.
I get the following error message:
/usr/local/lib/python2.6/dist-packages/Django-1.4.5-py2.6.egg/django/conf/__init__.py:110: DeprecationWarning: The SECRET_KEY setting must not be empty.
  warnings.warn("The SECRET_KEY setting must not be empty.", DeprecationWarning)
Unknown command: 'index'
Type 'rbsite.pyc help' for usage.

I have ReviewBoard version 1.7.9 installed (upgraded from 1.6.13) and Django version 1.4.5.
#1 ee**@plaxi***** (Google Code) (Is this you? Claim this profile.)
The command I'm using is (copied from search-cron.conf):

sudo "/usr/bin/python" "/usr/local/bin/rb-site" manage "<site-name>" index -- --full
#2 david
I think this is an error in the command you used: where you have "<site-name>", it should be the path to the site installation files on the filesystem.
  • +UserError
#3 ee**@plaxi***** (Google Code) (Is this you? Claim this profile.)
Where I have "<site-name>", I put the full path to my site:


However, this seems to be not working as I get the DeprecationWarning regarding the empty SECRET_KEY.
#4 david
SECRET_KEY should be defined in the settings_local.py file in /var/www/test-site/conf/
#6 ee**@plaxi***** (Google Code) (Is this you? Claim this profile.)
SECRET_KEY was already defined in my settings_local.py at /var/www/test-site/conf. So I don't understand why I got this warning in the first place.

But I think this is unrelated to the actual problem I'm having: not being able to update/create the index (because of the error: 'Unknown command: 'index').
#7 ee**@plaxi***** (Google Code) (Is this you? Claim this profile.)
And if I do 'rb-site help' I get the following (in which I don't see the 'index'-command):

Usage: rb-site command [options] path

  --version             show program's version number and exit
  -h, --help            show this help message and exit
  --console             force the console UI
  -d, --debug           display debug output

  'install' command:
    Installs a new Review Board site tree and generates web server
    configuration files. This will ask several questions about the
    site before performing the installation.

    --copy-media        copy media files instead of symlinking
    --noinput           run non-interactively using configuration provided in
                        command-line options
                        fully-qualified host name of the site, excluding the
                        http://, port or path
                        path to the site relative to the domain name
                        the URL containing the static (shipped) media files
                        the URL containing the uploaded media files
    --db-type=DB_TYPE   database type (mysql, postgresql or sqlite3)
    --db-name=DB_NAME   database name (not for sqlite3)
    --db-host=DB_HOST   database host (not for sqlite3)
    --db-user=DB_USER   database user (not for sqlite3)
    --db-pass=DB_PASS   password for the database user (not for sqlite3)
                        cache server type (memcached or file)
                        cache identifier (memcached connection string or file
                        cache directory)
                        web server (apache or lighttpd)
                        port that the web server should listen on
                        python loader for apache (modpython, fastcgi or wsgi)
                        the site administrator's username
                        the site administrator's password
                        the site administrator's e-mail address

  'upgrade' command:
    Upgrades an existing site installation, synchronizing media trees and
    upgrading the database, unless otherwise specified.

    --no-db-upgrade     don't upgrade the database
#8 david
The 'manage' command just doesn't have help defined, it's still there.

It sounds like something's going wrong with the path to the site, so it's not finding the django management commands, and not finding your settings_local.py file.
#9 david
  • -UserError
#11 ee**@plaxi***** (Google Code) (Is this you? Claim this profile.)
Is there something I could check in my installation that might be causing this problem?
#12 chipx86
This is an error we've never seen before, and frankly, it doesn't make any sense. That is, unless you either have a stray settings_local.py or settings_local.pyc file somewhere on your filesystem.

Does the 'shell' management command work? If so, can you do that and then:

    from django.conf import settings
    print settings.__file__
    print settings.SECRET_KEY

Don't actually show us the SECRET_KEY, but do show the file path, and let us know if the SECRET_KEY matches what is in your settings_local.py file.
  • -New
#13 ee**@plaxi***** (Google Code) (Is this you? Claim this profile.)
When I issued:
sudo "/usr/bin/python" "/usr/local/bin/rb-site" manage <path/to/site> shell

I got the following deprecation warning:
/usr/local/lib/python2.6/dist-packages/Django-1.4.5-py2.6.egg/django/conf/__init__.py:110: DeprecationWarning: The SECRET_KEY setting must not be empty.
  warnings.warn("The SECRET_KEY setting must not be empty.", DeprecationWarning)

Although, the shell did start. However, when I issue following commands:
from django.conf import settings
print settings.__file__

I got:
/usr/local/lib/python2.6/dist-packages/Django-1.4.5-py2.6.egg/django/utils/functional.pyc in inner(self, *args)
    183         if self._wrapped is empty:
    184             self._setup()
--> 185         return func(self._wrapped, *args)
    186     return inner

AttributeError: 'Settings' object has no attribute '__file__'
#14 ee**@plaxi***** (Google Code) (Is this you? Claim this profile.)
I think I have solved part of the problem :D

In our company, we have a 'commit-reader'-script running on the codereview server, which monitors any svn-commits. For each such commit, a draft review-request (or update review, depending on the commit-message) is created. During initialization of this script, it tries to synchronize svn and Review Board, by issuing many webapi-requests onto the Review Board server in order to find out which svn-revisions have already been added to Review Board.

However, each webapi http-request creates a new record in the django_sessions-table. This gave problems, as the hard-disk quickly ran out of memory. In order to fix this, I 'patched' the Sessions-middleware, by creating the following file (about 2 years ago):

# settings_local_sessionmiddleware.py ==========================

from django.contrib.sessions.middleware import SessionMiddleware

def dontsave(must_create=True):

class PatchedSessionMiddleware(SessionMiddleware):
    def process_request(self, request):
        super(PatchedSessionMiddleware, self).process_request(request)

        # Because the commit_reader performs lots of requests to /api/... urls,
        # for each request a session-id would be stored in the database.
        # This would grow the database in size very quickly.
        # This check prevents that.
        if request.path_info[0:5] == '/api/':
            request.session.save = dontsave

# First import the original MIDDLEWARE_CLASSES-list
from reviewboard.settings import MIDDLEWARE_CLASSES

# Now overwrite the SessionMiddleware-class with our patched version
cls = SessionMiddleware.__module__ + '.' + SessionMiddleware.__name__
idx = MIDDLEWARE_CLASSES.index(cls)
MIDDLEWARE_CLASSES[idx] = __name__ + '.' + PatchedSessionMiddleware.__name__


And finally I added the following line to the settings_local.py file:

from settings_local_sessionmiddleware import MIDDLEWARE_CLASSES

Apparently, importing this file was the cause of the current problem (perhaps due to a circular dependency?). By removing this line, it looks like everything works again.

Indexing now works fine, as well as the 'rb-site manage shell' (although issuing 'print settings.__file__' still complains about the 'Settings' object not having this attribute).

The only issue that remains is the Sessions-middleware. If I inspect my database, I see:

                      relation                       |  size
 public.django_session_pkey                          | 1785 MB
 public.django_session_expire_date                   | 1030 MB
 public.django_session                               | 699 MB
 pg_toast.pg_toast_16671                             | 203 MB
 public.diffviewer_filediff                          | 70 MB
 pg_toast.pg_toast_119386                            | 51 MB
 public.diffviewer_filediffdata                      | 19 MB
 public.reviews_reviewrequest                        | 3776 kB
 public.diffviewer_filediff_diff_hash_id             | 3040 kB
 public.diffviewer_filediff_diff_hash_id_like        | 3040 kB
 public.reviews_comment                              | 2752 kB
 pg_toast.pg_toast_16671_index                       | 2600 kB
 public.diffviewer_filediff_diffset_id               | 2088 kB
 public.changedescs_changedescription                | 1912 kB
 public.diffviewer_filediff_parent_diff_hash_id_like | 1848 kB
 public.diffviewer_filediff_parent_diff_hash_id      | 1840 kB
 public.diffviewer_filediffdata_pkey                 | 1776 kB
 public.diffviewer_filediff_pkey                     | 1728 kB
 public.reviews_review                               | 1424 kB
 public.diffviewer_diffset                           | 1136 kB

As can be seen, the session-data uses quite some space. Furthermore, I believe I already emptied the django_session table back then, when I created the patched middleware-class.

Is this a known problem and is there a better way to solve this?
#15 david
There is a 'clearsessions' management command that you can hook up to a cron job to clear out expired sessions.

I'm going to repurpose this bug to see if we can disable session saving for API requests generally.
  • -NeedInfo
  • +Component-API
  • +API requests create a ton of entries in the 'sessions' table.
#16 erikva*******@gmai***** (Google Code) (Is this you? Claim this profile.)
The 'clearsessions' command is not available in rbtools 0.5.1, is that right?
#17 david
It's a management command to be run on the server. So you'd use something like this:

$ rb-site manage /path/to/site clearsessions
#18 ee**@plaxi***** (Google Code) (Is this you? Claim this profile.)
My bad, it's part of reviewboard and not rbtools. However, in our version (1.7.9) the command 'clearsessions' is not available.

What exactly does the command do? Perhaps I can do it manually for the time being, until we upgrade to a newer version.
#19 chipx86
It's a command provided by Django to clear the expired sessions from the database.

Older versions appear to call this "cleanup" instead of "clearsessions", so give that a try.