4825: rbt diff fails for git-p4 repo with Python 3.7.2 on Windows.

alebastr

What version are you running?

RBTools 1.0.2 + a32dadb

What steps will reproduce the problem?

  1. Install Python 3.7.x and rbtools 1.0.2 on Windows
  2. Apply patch from https://github.com/reviewboard/rbtools/commit/a32dadbbd5b071cce5a12128240b9ff6ce7e83a2
  3. Clone perforce repo with git-p4
  4. Commit some changes
  5. Run rbt diff

What is the expected output? What do you see instead?

Expected:
diff output

Actual:
CRITICAL: cannot use a bytes pattern on a string-like object
CRITICAL: can't concat str to bytes
CRITICAL: a bytes-like object is required, not 'str'
...

What operating system are you using?

Windows 10 x64

Attach the debug out from the command.

PS % rbt diff -d
>>> RBTools 1.0.2
>>> Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)]
>>> Running on Windows-10-10.0.16299-SP0
>>> Home = C:\Users\bavshin\AppData\Roaming
>>> Current directory = C:\Users\bavshin\Source\Repos\<...>
>>> Command line: rbt diff -d
>>> Running: tf vc help
>>> Checking for a Bazaar repository...
>>> Unable to execute "bzr help": skipping Bazaar
>>> Checking for a ClearCase repository...
>>> Unable to execute "cleartool help": skipping ClearCase
>>> Checking for a CVS repository...
>>> Unable to execute "cvs": skipping CVS
>>> Checking for a Git repository...
>>> Running: git rev-parse --git-dir
>>> Running: git config core.bare
>>> Running: git rev-parse --show-toplevel
>>> Running: git symbolic-ref -q HEAD
>>> Running: git config --get git-p4.port
>>> Repository info: Path: perforce.example.com:1666, Base path: , Supports changesets: False
>>> Checking for a Mercurial repository...
>>> Running: hg root
>>> Command exited with rc 255: ['hg', 'root']
abort: no repository found in 'C:\Users\bavshin\Source\Repos\<...>' (.hg not found)!
---
>>> Checking for a Perforce repository...
>>> Running: p4 info
>>> Checking for a Plastic repository...
>>> Unable to execute "cm version": skipping Plastic
>>> Checking for a Subversion repository...
>>> Unable to execute "svn help": skipping SVN
>>> Checking for a Team Foundation Server repository...
>>> Unable to execute "tf help": skipping TFS
>>> Running: git config --get reviewboard.url
>>> Making HTTP GET request to https://reviewboard.example.com/api/
>>> Making HTTP GET request to https://reviewboard.example.com/api/info/
>>> Running: git rev-parse refs/heads/current-branch
>>> Running: git branch --remotes
>>> Running: git rev-parse p4/master
>>> Running: git rev-list 8f934ac10ec3fcba20b9bb4359a7240ce06f3295 --not --remotes=p4
>>> Running: git status --porcelain --untracked-files=no --ignore-submodules=dirty
>>> Running: git version
>>> Running: git -c core.quotepath=false diff --no-color --no-prefix -r -u --no-ext-diff 8f934ac10ec3fcba20b9bb4359a7240ce06f3295..7c0084ebef48a82a27ac5c23baa6df762e5ef2e5
>>> Running: git log 8f934ac10ec3fcba20b9bb4359a7240ce06f3295
Traceback (most recent call last):
  File "c:\python3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python3\Scripts\rbt.exe\__main__.py", line 9, in <module>
  File "c:\python3\lib\site-packages\rbtools\commands\main.py", line 120, in main
    command.run_from_argv([RB_MAIN, command_name] + args)
  File "c:\python3\lib\site-packages\rbtools\commands\__init__.py", line 725, in run_from_argv
    exit_code = self.main(*args) or 0
  File "c:\python3\lib\site-packages\rbtools\commands\diff.py", line 77, in main
    extra_args=extra_args)
  File "c:\python3\lib\site-packages\rbtools\clients\git.py", line 636, in diff
    no_renames)
  File "c:\python3\lib\site-packages\rbtools\clients\git.py", line 784, in make_diff
    return self.make_perforce_diff(merge_base, diff_lines)
  File "c:\python3\lib\site-packages\rbtools\clients\git.py", line 890, in make_perforce_diff
    log, re.M)
  File "c:\python3\lib\re.py", line 183, in search
    return _compile(pattern, flags).search(string)
TypeError: cannot use a bytes pattern on a string-like object

Please provide any additional information below.

alebastr
#1 alebastr

Two more tracebacks from rbt diff -d command:

>>> Running: git log 8f934ac10ec3fcba20b9bb4359a7240ce06f3295
Traceback (most recent call last):
  File "c:\python3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python3\Scripts\rbt.exe\__main__.py", line 9, in <module>
  File "c:\python3\lib\site-packages\rbtools\commands\main.py", line 120, in main
    command.run_from_argv([RB_MAIN, command_name] + args)
  File "c:\python3\lib\site-packages\rbtools\commands\__init__.py", line 725, in run_from_argv
    exit_code = self.main(*args) or 0
  File "c:\python3\lib\site-packages\rbtools\commands\diff.py", line 77, in main
    extra_args=extra_args)
  File "c:\python3\lib\site-packages\rbtools\clients\git.py", line 636, in diff
    no_renames)
  File "c:\python3\lib\site-packages\rbtools\clients\git.py", line 784, in make_diff
    return self.make_perforce_diff(merge_base, diff_lines)
  File "c:\python3\lib\site-packages\rbtools\clients\git.py", line 913, in make_perforce_diff
    ['p4', 'files', base_path + filename + '@' + p4rev],
TypeError: can't concat str to bytes

>>> Running: git log 8f934ac10ec3fcba20b9bb4359a7240ce06f3295
>>> Running: p4 files //depot/<...>@3037715
Traceback (most recent call last):
  File "c:\python3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python3\Scripts\rbt.exe\__main__.py", line 9, in <module>
  File "c:\python3\lib\site-packages\rbtools\commands\main.py", line 120, in main
    command.run_from_argv([RB_MAIN, command_name] + args)
  File "c:\python3\lib\site-packages\rbtools\commands\__init__.py", line 725, in run_from_argv
    exit_code = self.main(*args) or 0
  File "c:\python3\lib\site-packages\rbtools\commands\diff.py", line 77, in main
    extra_args=extra_args)
  File "c:\python3\lib\site-packages\rbtools\clients\git.py", line 636, in diff
    no_renames)
  File "c:\python3\lib\site-packages\rbtools\clients\git.py", line 784, in make_diff
    return self.make_perforce_diff(merge_base, diff_lines)
  File "c:\python3\lib\site-packages\rbtools\clients\git.py", line 914, in make_perforce_diff
    ignore_errors=True, results_unicode=False)
  File "c:\python3\lib\site-packages\rbtools\clients\git.py", line 1225, in _execute
    return execute(cmdline, cwd=self._git_toplevel, *args, **kwargs)
  File "c:\python3\lib\site-packages\rbtools\utils\process.py", line 155, in execute
    **popen_encoding_args)
  File "c:\python3\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "c:\python3\lib\subprocess.py", line 1119, in _execute_child
    args = list2cmdline(args)
  File "c:\python3\lib\subprocess.py", line 530, in list2cmdline
    needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: a bytes-like object is required, not 'str'
puremourning
#2 puremourning

How about this patch ?

commit ad51dadc52685520293a0ed226b95649b4104298
Author: Ben Jackson <ben.jackson@fidessa.com>
Date:   Wed Apr 24 10:16:37 2019 +0100

    Fix unicode errors

diff --git a/rbtools/clients/git.py b/rbtools/clients/git.py
index 142e547..cd4d459 100644
--- a/rbtools/clients/git.py
+++ b/rbtools/clients/git.py
@@ -925,7 +925,9 @@ class GitClient(SCMClient):
         p4rev = b''

         # Find which depot changelist we're based on
-        log = self._execute([self.git, 'log', merge_base], ignore_errors=True)
+        log = self._execute([self.git, 'log', merge_base],
+                            ignore_errors=True,
+                            results_unicode=False)

         for line in log:
             m = re.search(br'[rd]epo.-paths = "(.+)": change = (\d+).*\]',
@@ -952,23 +954,24 @@ class GitClient(SCMClient):
             elif (line.startswith(b'--- ') and i + 1 < len(diff_lines) and
                   diff_lines[i + 1].startswith(b'+++ ')):
                 data = self._execute(
-                    ['p4', 'files', base_path + filename + '@' + p4rev],
-                    ignore_errors=True, results_unicode=False)
+                    [b'p4', b'files', base_path + filename + b'@' + p4rev],
+                    ignore_errors=True,
+                    results_unicode=False)
                 m = re.search(br'^%s%s#(\d+).*$' % (re.escape(base_path),
                                                     re.escape(filename)),
                               data, re.M)
                 if m:
                     file_version = m.group(1).strip()
                 else:
-                    file_version = 1
+                    file_version = b'1'

-                diff_data += b'--- %s%s\t%s%s#%s\n' % (base_path, filename,
+                diff_data += ( b'--- %s%s\t%s%s#%s\n' % (base_path, filename,
                                                        base_path, filename,
-                                                       file_version)
+                                                       file_version) )
             elif line.startswith(b'+++ '):
                 # TODO: add a real timestamp
-                diff_data += b'+++ %s%s\t%s\n' % (base_path, filename,
-                                                  b'TIMESTAMP')
+                diff_data += ( b'+++ %s%s\t%s\n' % (base_path, filename,
+                                                  b'TIMESTAMP') )
             else:
                 diff_data += line

works for me...

david
#3 david

The bug tracker is not a good place to review patches. Can you please post this on https://reviews.reviewboard.org?

Thanks!

puremourning
#4 puremourning

i posted it really in the hope that it would be useful. apologies, but i don't have the bandwidth right now to do a proper 9 yards on it, not least because it was many months ago that i made the change and i no longer have the code paged in :)