4890: Cleanly handle unicode characters in review bots

almt 
What version of Review Board are you running?


3.0.18


What version of Review Bot are you running?


1.0.1


What tool(s) is Review Bot running?


custom, similar to flake8


What steps will reproduce the problem?


    

  1. Submit code changes that adds unicode characters in a string
    2. 





What is the expected output? What do you see instead?


Internal error in the code review result


Please provide any additional information below.


Sample stack trace:


[2020-09-22 12:15:44,619: ERROR/Worker-3] reviewbot.tasks.RunTool: Failed to in

itialize review: HTTP 500 (server=https://reviews.com/, review_request_id=110141, diff_revis

ion=1)

Traceback (most recent call last):

 File "/usr/local/lib/python3.6/dist-packages/rbtools/api/request.py", line 652, in process_error


[2020-09-22 12:15:44,619: ERROR/Worker-3] reviewbot.tasks.RunTool: Failed to in

itialize review: HTTP 500 (server=https://reviews.com/, review_request_id=110141, diff_revis

ion=1)

Traceback (most recent call last):

 File "/usr/local/lib/python3.6/dist-packages/rbtools/api/request.py", line 652, in process_error

   rsp = json_loads(data)

 File "/usr/lib/python3.6/json/init.py", line 354, in loads

   return _default_decoder.decode(s)

 File "/usr/lib/python3.6/json/decoder.py", line 339, in decode

   obj, end = self.raw_decode(s, idx=_w(s, 0).end())

 File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode

   raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

 File "/usr/local/lib/python3.6/dist-packages/reviewbot/tasks.py", line 165, in RunTool

   review_settings)

 File "/usr/local/lib/python3.6/dist-packages/reviewbot/processing/review.py", line 287, in init

   self.files = [File(self, f) for f in files]

 File "/usr/local/lib/python3.6/dist-packages/reviewbot/processing/review.py", line 287, in <listcomp>

   self.files = [File(self, f) for f in files]

 File "/usr/local/lib/python3.6/dist-packages/reviewbot/processing/review.py", line 42, in init

   self.diff_data = api_filediff.get_diff_data()

 File "/usr/local/lib/python3.6/dist-packages/rbtools/api/decorators.py", line 27, in request_method

args, *kwargs)

 File "/usr/local/lib/python3.6/dist-packages/rbtools/api/transport/sync.py", line 77, in execute_request_method

   return self._execute_request(request)

 File "/usr/local/lib/python3.6/dist-packages/rbtools/api/transport/sync.py", line 86, in _execute_request

   rsp = self.server.make_request(request)

 File "/usr/local/lib/python3.6/dist-packages/rbtools/api/request.py", line 687, in make_request

   self.process_error(e.code, e.read())

 File "/usr/local/lib/python3.6/dist-packages/rbtools/api/request.py", line 664, in process_error

   raise APIError(http_status, None, None, data)

rbtools.api.errors.APIError: HTTP 500
david
#1 david 
Given that you're getting an HTTP 500 from the API, there's going to be an error listed on the server side. Can you check the Review Board and/or Apache log files for that?
  • -New
    +NeedInfo
#2 almt 
Server side log




'ascii' codec can't decode byte 0xf0 in position 3220: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/cache.py", line 52, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/vary.py", line 19, in inner_func
    response = func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djblets/webapi/resources/base.py", line 244, in __call__
    request, method, view, api_format=api_format, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/webapi/base.py", line 338, in call_method_view
    request, method, view, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djblets/webapi/resources/mixins/api_tokens.py", line 66, in call_method_view
    request, method, view, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djblets/webapi/resources/mixins/oauth2_tokens.py", line 102, in call_method_view
    request, method, view, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djblets/webapi/resources/base.py", line 369, in call_method_view
    return view(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djblets/webapi/decorators.py", line 125, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/webapi/decorators.py", line 37, in _check
    return webapi_login_required(view_func)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djblets/webapi/decorators.py", line 125, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djblets/webapi/decorators.py", line 146, in _checklogin
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djblets/webapi/decorators.py", line 125, in _call
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/webapi/decorators.py", line 169, in _check
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/webapi/resources/filediff.py", line 389, in get
    return self._get_diff_data(request, mimetype, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/webapi/resources/filediff.py", line 457, in _get_diff_data
    populate_diff_chunks(files, highlighting, request=request)
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/diffviewer/diffutils.py", line 784, in populate_diff_chunks
    chunks = list(generator.get_chunks())
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/diffviewer/chunk_generator.py", line 786, in get_chunks
    for chunk in super(DiffChunkGenerator, self).get_chunks(cache_key):
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/diffviewer/chunk_generator.py", line 107, in get_chunks
    large_data=True)
  File "/usr/local/lib/python2.7/dist-packages/djblets/cache/backend.py", line 299, in cache_memoize
    compress_large_data))
  File "/usr/local/lib/python2.7/dist-packages/djblets/cache/backend.py", line 253, in cache_memoize_iter
    items = items_or_callable()
  File "/usr/local/lib/python2.7/dist-packages/djblets/cache/backend.py", line 296, in <lambda>
    lambda: [lookup_callable()],
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/diffviewer/chunk_generator.py", line 106, in <lambda>
    lambda: list(self.get_chunks_uncached()),
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/diffviewer/chunk_generator.py", line 793, in get_chunks_uncached
    new = get_patched_file(old, self.filediff, self.request)
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/diffviewer/diffutils.py", line 307, in get_patched_file
    request=request)
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/diffviewer/diffutils.py", line 185, in patch
    if empty_diff(diff):
  File "/usr/local/lib/python2.7/dist-packages/reviewboard/diffviewer/diffutils.py", line 126, in empty_diff
    arr = diff.split("\n",7)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 3220: ordinal not in range(128)
david
#3 david 
It looks like you have some custom changes in your Review Board deployment which are causing the issue. There's no empty_diff method in the upstream code.
#4 almt 
Thanks, I failed to look at the server log stacktrace. You can close this ticket as not a bug. That custom function was a Q&D fix to address issue 4889.
david
#5 david
  • -NeedInfo
    +NotABug