4856: rbt post fails if svn info contains non utf-8 characters

levine

What version are you running?

rbtools 2.0 alpha 0 (dev) (installed using setup.py latest from github as of today 2020-01-08)
same error also occurs with rbtools 1.0.2 installed from Windows installer and pip installer

Python 3.8.1

What steps will reproduce the problem?

  1. Using Subversion as CVS, I have a locally changed file whose "svn info" has a character encoded in non-utf-8. This is because our Subversion server is running on a Swedish locale and the file was changed on a Monday, so svn info contains an "å" character in "Last changed date".
  2. Do "rbt post --svn-show-copies-as-adds=y" (I assume the latter argument was not relevant to the error, but was required in my case).

What is the expected output? What do you see instead?

Expected: A new review request should be created.

Actual result: Failure with "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 721: invalid continuation byte"

What operating system are you using?

Windows 7

Attach the debug out from the command.

With URLs and project names masked:

$ rbt post --svn-show-copies-as-adds=y -d

RBTools 2.0 alpha 0 (dev)
Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)]
Running on Windows-7-6.1.7601-SP1
Home = H:\
Current directory = C:\Development\trunk
Command line: rbt post --svn-show-copies-as-adds=y -d
Running: tf vc help
Checking for a Subversion repository...
Running: svn --non-interactive info
Running: diff --version
Running: svn --non-interactive --version -q
Repository info: Path: https://svn.our.repo.address.com/svn/MainRepo, Base path: /trunk, Supports changesets: False
Making HTTP GET request to http://our.review.board.url.com/api/
Making HTTP GET request to http://our.review.board.url.com/api/repositories/?tool=Subversion
Cached response for HTTP GET http://our.review.board.url.com/api/repositories/?tool=Subversion expired and was modified
Making HTTP GET request to http://our.review.board.url.com/api/repositories/2/info/
Cached response for HTTP GET http://our.review.board.url.com/api/repositories/2/info/ expired and was modified
Repository info: Path: http://svn.our.repo.address.com/svn/MainRepo, Base path: //t/r/u/n/k, Supports changesets: False
Running: svn --non-interactive diff --diff-cmd=diff --notice-ancestry -r BASE --show-copies-as-adds
Running: svn --non-interactive diff --diff-cmd=diff --notice-ancestry -r BASE --show-copies-as-adds --no-diff-deleted
Running: svn --non-interactive info components/Foo/Build/FooAPI/FooAPI.vcproj
Command exited with rc 1: ['svn', '--non-interactive', 'info', 'components/Foo/Build/FooAPI/FooAPI.vcproj\r']
["svn: E020024: Error resolving case of 'components\Foo\Build\FooAPI\FooAPI.vcproj\n", "'\n"]---
Running: svn --non-interactive info components/Foo/Build/FooAPI/FooAPI.vcproj
Running: svn --non-interactive info components/Foo/Build/FooAPI/FooAPI.vcxproj
Command exited with rc 1: ['svn', '--non-interactive', 'info', 'components/Foo/Build/FooAPI/FooAPI.vcxproj\r']
["svn: E020024: Error resolving case of 'components\Foo\Build\FooAPI\FooAPI.vcxproj\n", "'\n"]---
Running: svn --non-interactive info components/Foo/Build/FooAPI/FooAPI.vcxproj
Exception in thread Thread-9:
Traceback (most recent call last):
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\threading.py", line 870, in run
self._target(*self._args, self._kwargs)
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1366, in _readerthread
buffer.append(fh.read())
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 721: invalid continuation byte
Traceback (most recent call last):
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\Scripts\rbt-script.py", line 11, in <module>
load_entry_point('RBTools==2.0a0.dev0', 'console_scripts', 'rbt')()
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\site-packages\rbtools-2.0a0.dev0-py3.8.egg\rbtools\commands\main.py", line 120, in main
command.run_from_argv([RB_MAIN, command_name] + args)
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\site-packages\rbtools-2.0a0.dev0-py3.8.egg\rbtools\commands__init__.py", line 740, in run_from_argv
exit_code = self.main(args) or 0
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\site-packages\rbtools-2.0a0.dev0-py3.8.egg\rbtools\commands\post.py", line 1055, in main
squashed_diff = self._get_squashed_diff(repository_info,
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\site-packages\rbtools-2.0a0.dev0-py3.8.egg\rbtools\commands\post.py", line 1337, in _get_squashed_diff
diff_info = self.tool.diff(
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\site-packages\rbtools-2.0a0.dev0-py3.8.egg\rbtools\clients\svn.py", line 523, in diff
diff = self.convert_to_absolute_paths(diff, repository_info)
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\site-packages\rbtools-2.0a0.dev0-py3.8.egg\rbtools\clients\svn.py", line 848, in convert_to_absolute_paths
info = self.svn_info(file, True)
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\site-packages\rbtools-2.0a0.dev0-py3.8.egg\rbtools\clients\svn.py", line 887, in svn_info
result = self._run_svn(['info', path],
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\site-packages\rbtools-2.0a0.dev0-py3.8.egg\rbtools\clients\svn.py", line 1191, in _run_svn
return execute(cmdline,
args,
kwargs)
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\site-packages\rbtools-2.0a0.dev0-py3.8.egg\rbtools\utils\process.py", line 167, in execute
data, errors = p.communicate()
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1024, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "C:\Users\u002257\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1416, in _communicate
stdout = stdout[0]
IndexError: list index out of range

Please provide any additional information below.

If I change the execute function in process.py, to always use
popen_encoding_args['universal_newlines'] = True
and never
popen_encoding_args['encoding'] = 'utf-8'
then it starts working.

(Have not ran any unit tests or considered in any capacity if this breaks some other platform or use case. Just observing that it fixed the problem for me in this particular instance)

levine
#1 levine

Pasting the log with formatting into the description was a mistake... Attaching the log as text file instead.

  • +
    $ rbt post --svn-show-copies-as-adds=y -d
    >>> RBTools 2.0 alpha 0 (dev)
    >>> Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)]
    >>> Running on Windows-7-6.1.7601-SP1
    >>> Home = H:\
    >>> Current directory = C:\Development\trunk
    >>> Command line: rbt post --svn-show-copies-as-adds=y -d
    >>> Running: tf vc help
    >>> Checking for a Subversion repository...
    >>> Running: svn --non-interactive info
    >>> Running: diff --version
    >>> Running: svn --non-interactive --version -q
    >>> Repository info: Path: https://svn.our.repo.address.com/svn/MainRepo, Base path: /trunk, Supports changesets: False
    >>> Making HTTP GET request to http://our.review.board.url.com/api/
    >>> Making HTTP GET request to http://our.review.board.url.com/api/repositories/?tool=Subversion
    >>> Cached response for HTTP GET http://our.review.board.url.com/api/repositories/?tool=Subversion expired and was modified
    >>> Making HTTP GET request to http://our.review.board.url.com/api/reposi
david
#2 david
  • +EasyFix
keedav
#3 keedav

I experience the same issue on Windows with Hungarian locale.
Executing "svn info" gives the following output (notice the 'ú' character in júl.)
Last Changed Date: 2021-07-26 14:04:28 +0200 (H, 26 júl. 2021)

  • +
    c:\Dev\reviewtest>svn info
    Path: .
    Working Copy Root Path: C:\Dev\reviewtest
    URL: http://gamer_pc/svn/reviewtest/trunk
    Relative URL: ^/trunk
    Repository Root: http://gamer_pc/svn/reviewtest
    Repository UUID: 16be962d-fa22-b14c-9f5e-b4c5e8d4f088
    Revision: 1
    Node Kind: directory
    Schedule: normal
    Last Changed Author: VisualSVN Server
    Last Changed Rev: 1
    Last Changed Date: 2021-07-26 14:04:28 +0200 (H, 26 júl. 2021)
    c:\Dev\reviewtest>rbt status
    Exception in thread Thread-2:
    Traceback (most recent call last):
      File "C:\Program Files\Python39\lib\threading.py", line 973, in _bootstrap_inner
        self.run()
      File "C:\Program Files\Python39\lib\threading.py", line 910, in run
        self._target(*self._args, **self._kwargs)
      File "C:\Program Files\Python39\lib\subprocess.py", line 1479, in _readerthread
        buffer.append(fh.read())
      File "C:\Program Files\Python39\lib\codecs.py", line 322, in decode
        (result, consumed) = self._buffer_decode(data, self.errors, final)