Welcome to the Splat alpha!

Are you sure it wasn't coincidental timing? We don't have any changes to either rbssh or svn in this release.

• Status:

  +	NeedInfo

I don't know how it could be since it seems to work correctly with version 1.6.9. Is there something I can do to get more information about the failure? Everything online seems to indicate that it is an SSH issue, but I have tested and retested those settings. Could this be due to a change in the version of a dependency in this version of ReviewBoard?

No dependency changes were made. The only SCM code that changed was some file parsing for CVS and some hosting support for Mercurial, neither of which comes anywhere near the Subversion code.

Certainly nothing changed to impact SSH/rb-ssh code.

I believe it's working in 1.6.9, but that traceback is showing some communication error between rb-site and the repository. That's why I'm wondering if there happened to be some downtime as you were trying 1.6.10 that resolved itself (or someone resolved it) after. Nothing else to me is making sense right now.

I don't think there's anything more to get out of that failure. As it said, the connection was closed unexpectedly, presumably because of a server issue or it not liking something that was sent across.

Would you be able to set up a 1.6.10 on a developer box/VM/virtualenv and see if it still happens?

I moved the /var/lib/reviewboard to /var/lib/reviewboard-backup and did a fresh install of 1.6.10 to /var/lib/reviewboard using sqlite as the database. This should ensure that there wasn't anything in my existing database confusing things. I then restarted memcached and apache2.

After going to the install the first time, it informed me that the data directory had the wrong permissions. I ran the command to fix them.

I added a repository and was asked to trust the host. The following error: "Unable to authenticate against this repository using one of the supported authentication types." I copied the old id_rsa over using the following command. I have tested this id_rsa with ssh -i and connected successfully to the svn server.

sudo cp -a /var/lib/reviewboard-backup/data/.ssh/id_rsa /var/lib/reviewboard/data/.ssh/id_rsa

ReviewBoard didn't pick this up, so I downloaded the key from the server and used the ReviewBoard UI to upload the SSH key. This picked up the key. I then tried to add the repository again and saw the following error: "A repository was not found at the specified path."

I went ahead and did the same process described above using ReviewBoard 1.6.9 using "sudo easy_install ReviewBoard==1.6.9" and the rest of the procedure is the same. The result was that no errors were thrown and the repository is added without problems.

By the way, the server I am running on is Ubuntu 10.04.4 LTS -- "Linux reviewboard 2.6.32-30-generic #59-Ubuntu SMP Tue Mar 1 21:30:46 UTC 2011 x86_64 GNU/Linux" with Python 2.6.5

Very odd. Do you have a log file from that attempt? It should cover more what was happening there.

I am currently setting up a VM with CentOS 5 to see if that makes a difference. The log from the previous attempt follows:

13:45:09 	DEBUG 	

SVNTool: Attempting ssh connection with host: ********, username: reviewboard

13:45:09 	DEBUG 	

starting thread (client mode): 0x71514790L

13:45:09 	INFO 	

Connected (version 2.0, client OpenSSH_5.3p1)

13:45:09 	DEBUG 	

kex algos:['diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] server encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] client mac:['hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False

13:45:09 	DEBUG 	

Ciphers agreed: local=aes128-ctr, remote=aes128-ctr

13:45:09 	DEBUG 	

using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none

13:45:09 	DEBUG 	

Switch to new keys ...

13:45:09 	DEBUG 	

Trying SSH key 6d41f76410303cb216b7b8f7c687dab6

13:45:09 	DEBUG 	

userauth is OK

13:45:09 	INFO 	

Authentication (publickey) successful!

13:45:09 	DEBUG 	

EOF in transport thread

13:45:09 	ERROR 	

SVN: Failed to get repository information for svn+ssh://reviewboard@********/home/svn/********: Network connection closed unexpectedly

13:45:09 	DEBUG 	

EOF in transport thread

Try setting the DEBUG_RBSSH to 1 in the environment for Apache (SetEnv), and set DEBUG_LOGDIR to the directory you're using for reviewboard.log. There may be some more info there.

I added both of those to my apache configuration in the VirtualHost associated with ReviewBoard at the beginning as follows, but I didn't see any new information appear in the log and I don't see any new log files.

<VirtualHost *:80>
        SetEnv DEBUG_RBSSH 1
        SetEnv DEBUG_LOGDIR /var/lib/reviewboard/logs/

        ServerName reviewboard
        DocumentRoot "/var/lib/reviewboard/htdocs"
.
.
.

Also, I have run into issues using CentOS as a base since the default yum repos don't have all the needed packages. It appears from the documentation that the Debian-based distros are the preferred bases anyhow so I am probably giving up on that avenue.

Debian-based distros are definitely a lot easier. Fedora's not bad either, since they maintain up-to-date RB packages.

Sorry, I was in a rush and gave you the wrong environment variable for the log directory. It should be RBSSH_LOG_DIR.

I made that change, but it still isn't giving me the additional output. I made sure to restart apache2 and memcached. I tried moving the SetEnv options into the Location tag with the other ones in case that made a difference too. It didn't. I noticed in the apache config, there is also a reference to PythonDebug Off. Is that something I can turn on to get more data or is that more for gdb-style stepping?

<VirtualHost *:80>
        ServerName reviewboard
        DocumentRoot "/var/lib/reviewboard/htdocs"

        # Error handlers
        ErrorDocument 500 /errordocs/500.html

        # Serve django pages
        <Location "/">
                PythonPath "['/var/lib/reviewboard/conf'] + sys.path"
                SetEnv DEBUG_RBSSH 1
                SetEnv RBSSH_LOG_DIR "/var/lib/reviewboard/logs/"
                SetEnv DJANGO_SETTINGS_MODULE reviewboard.settings
                SetEnv PYTHON_EGG_CACHE "/var/lib/reviewboard/tmp/egg_cache"
                SetEnv HOME "/var/lib/reviewboard/data"
                SetHandler mod_python
                PythonHandler django.core.handlers.modpython
                PythonAutoReload Off
                PythonDebug Off
                # Used to run multiple mod_python sites in the same apache
                PythonInterpreter reviewboard_reviewboard
        </Location>
.
.
.

(I am heading out for the day, but I will check back tomorrow morning to see if you have any other suggestions for how to get more debug output and help solve this problem. I really appreciate your help so far, thanks!)

I have reverted back to 1.6.9 on Ubuntu and that is working just fine. If others are running into the same issue described in this bug report, please post here and maybe we can gather enough information to see what is causing this problem. Thanks!

PythonDebug Off shouldn't be related.

I'm surprised the SetEnv calls didn't work.

I'm trying to reproduce this bug here, but haven't succeeded yet. We run unit tests nightly that cover SSH-backed SVN repositories, and haven't hit anything, and as I said, there's nothing in 1.6.10 that relates to this in any form. So, I'm really unsure of what's going on.

Just to be completely sure, the log files should be rbssh-<pid>.log in that directory.

One thing you could try (I know you reverted back now) is to run this manually, as Apache.


    $ sudo -s
    $ sudo -u apache2 -s
    $ DEBUG_RBSSH=1 rbssh <url from the log file>

And just see what's outputted to the terminal.

Forgot a step. You'll need to set HOME to be /var/lib/reviewboard/data as well.

I still have the virtual machine that is running 1.6.10 and is broken so I was able to generate the following output using the method you described.

/usr/local/lib/python2.6/dist-packages/pycrypto-2.3-py2.6-linux-x86_64.egg/Crypto/Util/randpool.py:40: RandomPool_DeprecationWarning: This application uses RandomPool, which is BROKEN in older releases.  See http://www.pycrypto.org/randpool-broken
  RandomPool_DeprecationWarning)
Traceback (most recent call last):
  File "/usr/local/bin/rbssh", line 9, in <module>
    load_entry_point('ReviewBoard==1.6.10', 'console_scripts', 'rbssh')()
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 299, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2229, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 1948, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/local/lib/python2.6/dist-packages/ReviewBoard-1.6.10-py2.6.egg/reviewboard/cmdline/rbssh.py", line 42, in <module>
    from reviewboard.scmtools import sshutils
  File "/usr/local/lib/python2.6/dist-packages/ReviewBoard-1.6.10-py2.6.egg/reviewboard/scmtools/sshutils.py", line 8, in <module>
    from reviewboard.scmtools.errors import AuthenticationError, \
  File "/usr/local/lib/python2.6/dist-packages/ReviewBoard-1.6.10-py2.6.egg/reviewboard/scmtools/errors.py", line 4, in <module>
    from djblets.util.templatetags.djblets_utils import humanize_list
  File "/usr/local/lib/python2.6/dist-packages/Djblets-0.6.21-py2.6.egg/djblets/util/templatetags/djblets_utils.py", line 31, in <module>
    from django.contrib.auth.models import AnonymousUser, User
  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/contrib/auth/models.py", line 7, in <module>
    from django.db import models
  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/db/__init__.py", line 14, in <module>
    if not settings.DATABASES:
  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/utils/functional.py", line 276, in __getattr__
    self._setup()
  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/conf/__init__.py", line 40, in _setup
    raise ImportError("Settings cannot be imported, because environment variable %s is undefined." % ENVIRONMENT_VARIABLE)
ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.

Now that is interesting, and gets us much further. While I don't know why I'm not hitting it, I think I can come up with a repro case for 1.6.11 now.

• Status:

  -	NeedInfo
  +	Confirmed

• Tags:

  +	Component-SCMTools

• Assigned to:

  +	chipx86

This has been fixed on the release-0.6.x branch of Djblets (e940ca0). You can manually download and install this, and hopefully it will fix the problem. RB 1.6.11 (coming this week) will require this version, so it'll be fixed along with that release.

Thanks for all the debugging info and being patient with this. Hopefully this fixes the problem for you! I'd definitely be interested in knowing if a custom-built Djblets from this branch fixes the problem for you.

• Status:

  -	Confirmed
  +	Fixed

• Tags:

  +	Djblets

That sounds great! I would be happy to test this ahead of the 1.6.11 release. I'm not familiar with the process of the manual install you mentioned. If you could direct me to some docs for that, I will go ahead and test. Thanks!

You would have to check out Djblets from git (http://github.com/djblets/djblets/), check out the release-1.6.x branch, and then do:

  $ sudo ./setup.py install

Then just restart/reload your web server.

I did the following:

$ git clone -b release-0.6.x https://github.com/djblets/djblets.git
$ cd djblets/
$ sudo ./setup.py install
$ sudo service memcached restart
$ sudo service apache2 restart

Unfortunately, I got the same error through the ReviewBoard UI. I ran the same rbssh command you had me run before as the www-data user and got the following output:

/usr/local/lib/python2.6/dist-packages/pycrypto-2.3-py2.6-linux-x86_64.egg/Crypto/Util/randpool.py:40: RandomPool_DeprecationWarning: This application uses RandomPool, which is BROKEN in older releases.  See http://www.pycrypto.org/randpool-broken
  RandomPool_DeprecationWarning)
Traceback (most recent call last):
  File "/usr/local/bin/rbssh", line 9, in <module>
    load_entry_point('ReviewBoard==1.6.10', 'console_scripts', 'rbssh')()
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 299, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2229, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 1948, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/local/lib/python2.6/dist-packages/ReviewBoard-1.6.10-py2.6.egg/reviewboard/cmdline/rbssh.py", line 42, in <module>
    from reviewboard.scmtools import sshutils
  File "/usr/local/lib/python2.6/dist-packages/ReviewBoard-1.6.10-py2.6.egg/reviewboard/scmtools/sshutils.py", line 8, in <module>
    from reviewboard.scmtools.errors import AuthenticationError, \
  File "/usr/local/lib/python2.6/dist-packages/ReviewBoard-1.6.10-py2.6.egg/reviewboard/scmtools/errors.py", line 4, in <module>
    from djblets.util.templatetags.djblets_utils import humanize_list
  File "/usr/local/lib/python2.6/dist-packages/Djblets-0.6.21-py2.6.egg/djblets/util/templatetags/djblets_utils.py", line 31, in <module>
    from django.contrib.auth.models import AnonymousUser, User
  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/contrib/auth/models.py", line 7, in <module>
    from django.db import models
  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/db/__init__.py", line 14, in <module>
    if not settings.DATABASES:
  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/utils/functional.py", line 276, in __getattr__
    self._setup()
  File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/conf/__init__.py", line 40, in _setup
    raise ImportError("Settings cannot be imported, because environment variable %s is undefined." % ENVIRONMENT_VARIABLE)
ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.

Upgrade to 1.6.11 and see if this is fixed for you.

It looks like ReviewBoard 1.6.11 has resolved the issue for me. :) Thanks so much for working with me on this!

I just upgraded from 1.6.9 to 1.6.14. My setup also used svn+ssh, and had some diff files uploaded. After the upgrade, I am facing the same problem. The reviewboard server log says:
DEBUG	userauth is OK
INFO	Authentication (publickey) successful!
DEBUG	EOF in transport thread
ERROR	SVN: Failed to get repository information for svn+ssh://reviewboard@***: To better debug SSH connection problems, remove the -q option from 'ssh' in the [tunnels] section of your Subversion configuration file. Network connection closed unexpectedly

As suggested above, I ran:
$ sudo -s
$ sudo -u apache2 -s
$ DEBUG_RBSSH=1 rbssh <url from the log file>
It got me the following output:
/usr/lib/python2.4/site-packages/paramiko-1.7.7.2-py2.4.egg/paramiko/client.py:94: UserWarning: Unknown ssh-rsa host key for ***
Error connecting to server: No authentication methods available

Additional "pip freeze" info:
Django==1.3.4
Djblets==0.6.27
MySQL-python==1.2.3
PIL==1.1.6
Pygments==1.5
RBTools==0.4.1
ReviewBoard==1.6.14
django-evolution==0.6.7
feedparser==5.1.2
flup==1.0.3.dev-20110405
paramiko==1.7.7.2
pycrypto==2.6
python-dateutil==1.5
python-memcached==1.48
pytz==2012c
recaptcha-client==1.0.6
simplejson==2.0.9

Any ideas as to why this is happening? I should point out that running "ssh -i /var/www/review-site/data/.ssh/id_rsa reviewboard@***" is establishing the connection just fine.

Same problem with newest version 1.7.5 
After upgrading from 1.6.1 the repository access is no more possible


rbssh gives the following:

SSH: Unknown error accessing user key: Private key file is encrypted
Traceback (most recent call last):
  File "/usr/local/bin/rbssh", line 8, in <module>
    load_entry_point('ReviewBoard==1.7.5', 'console_scripts', 'rbssh')()
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.5-py2.7.egg/reviewboard/cmdline/rbssh.py", line 330, in main
    channel.get_pty()
  File "build/bdist.linux-x86_64/egg/paramiko/channel.py", line 158, in get_pty
  File "build/bdist.linux-x86_64/egg/paramiko/channel.py", line 1114, in _wait_for_event
paramiko.SSHException: Channel closed.

After downgrading to 1.6.9 everything works fine again