5016: perforce rbt post fails with 500 internal error

shubhar
chipx86
chipx86

What version are you running?

rbtools-4.1
ReviewBoard 6.0.1

What's the URL of the page containing the problem?

What steps will reproduce the problem?

  1. rbt post -g --target-people <userid> --summary "hi this is a summary" --description "hi this is a description" --debug

What is the expected output? What do you see instead?

I would expect rbt to succeed with diff URL

What operating system are you using? What browser?

Chrome, RHEL 8.x

Please provide any additional information below.

Got HTTP error: 500: <!DOCTYPE html>

<html>
<head>
<title>500 - Internal Server Error | Review Board</title>
</head>
<body>
<h1>Something broke! (Error 500)</h1>
<p>
It appears something broke when you tried to go to here. This is either
a bug in Review Board or a server configuration error. Please report
this to your administrator.
</p>
</body>
</title>

Traceback (most recent call last):
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/site-packages/rbtools/api/request.py", line 1222, in process_error
rsp = json_loads(data_str)
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/site-packages/rbtools/commands/post.py", line 756, in post_request
draft = draft.update(update_fields)
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/site-packages/rbtools/api/resource.py", line 319, in <lambda>
meth(resource,
kwargs)))
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/site-packages/rbtools/api/decorators.py", line 23, in request_method
return self._transport.execute_request_method(method_wrapper,
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/site-packages/rbtools/api/transport/sync.py", line 289, in execute_request_method
return self._execute_request(request)
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/site-packages/rbtools/api/transport/sync.py", line 310, in _execute_request
rsp = self.server.make_request(request)
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/site-packages/rbtools/api/request.py", line 1267, in make_request
self.process_error(e.code, e.read())
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/site-packages/rbtools/api/request.py", line 1234, in process_error
raise APIError(http_status, None, None, data_str)
rbtools.api.errors.APIError: <!DOCTYPE html>

<html>
<head>
<title>500 - Internal Server Error | Review Board</title>
</head>
<body>
<h1>Something broke! (Error 500)</h1>
<p>
It appears something broke when you tried to go to here. This is either
a bug in Review Board or a server configuration error. Please report
this to your administrator.
</p>
</body>
</title>
(HTTP 500: Internal Server Error)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/proj/pt_verif_dev4/users/shramani/pt_trunk/env_squash/default/squash_full/7AA3CBADF04B4FEE46E0C01EC37EFFFC-rhel7/PATH-2/rbt", line 8, in <module>
sys.exit(main())
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/site-packages/rbtools/commands/main.py", line 207, in main
command.run_from_argv([RB_MAIN, command_name] + args)
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/site-packages/rbtools/commands/init.py", line 1118, in run_from_argv
exit_code = self.main(*args) or 0
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/site-packages/rbtools/commands/post.py", line 968, in main
review_request_id, review_request_url = self.post_request(
File "/tool/pandora64/.package/python-3.10.1/lib/python3.10/site-packages/rbtools/commands/post.py", line 758, in post_request
raise CommandError(
rbtools.commands.CommandError: Error updating review request draft: <!DOCTYPE html>

<html>
<head>
<title>500 - Internal Server Error | Review Board</title>
</head>
<body>
<h1>Something broke! (Error 500)</h1>
<p>
It appears something broke when you tried to go to here. This is either
a bug in Review Board or a server configuration error. Please report
this to your administrator.
</p>
</body>
</title>
(HTTP 500: Internal Server Error)

Your review request still exists, but the diff is not attached.

chipx86
#1 chipx86

Thanks for the report! There should be a server-side error somewhere as well, perhaps e-mailed to the administrator, or found in the reviewboard.log file. Would you be able to locate the backtrace there and attach it?

#2 shubhar

look at the end of the log and you will see the related problem

  • +
    starter
    2023-12-07 17:58:09,437 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:03:09,445 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:08:25,491 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:18:35,520 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:18:51,274 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:29:44,809 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:29:45,528 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:29:45,623 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:32:20,787 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:37:00,389 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:39:04,915 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:39:05,869 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:40:27,103 - DEBUG -  - root - Reloading logging settings
    2023-12-07 18:40:29,898 - DEBUG - None - AnonymousUser - /pt_www/pt/r
chipx86
#3 chipx86

Hmm, interesting. Looks like an issue in a third-party module we use. We'll look into what's going on there. Thank you for the log!

Can you check the result of /webapp/reviewboard/site_prob-6.0.1/venv/bin/pip show publicsuffix (path based on what's in the logs -- feel free to adjust if needed) and show me what that says?

chipx86
#4 chipx86

We have a fix pending for this, switching to a modern, maintained alternative to publicsuffix. We'll ship this in Djblets 4.1 soon. If you have a support contract with us, we can get you tailored builds, or you can try backporting the patch internally and seeing if it resolves your problem.

https://reviews.reviewboard.org/r/13454/

  • -New
    +PendingReview
  • -reviewboard
    +djblets
  • +Component:EMail
  • +chipx86
#5 shubhar

Sure, here you go:
atlvptwebapp01:/webapp/PYTHON_VENVS/ReviewBoard-6.0.1-venv/bin[ 66 ] --> ./pip show publicsuffix
Name: publicsuffix
Version: 1.1.1
Summary: Get a public suffix for a domain name using the Public Suffix List.
Home-page:
Author: Tomaz Solc
Author-email: tomaz.solc@tablix.org
License: MIT
Location: /webapp/PYTHON_VENVS/ReviewBoard-6.0.1-venv/lib/python3.11/site-packages
Requires:
Required-by: Djblets
atlvptwebapp01:/webapp/PYTHON_VENVS/ReviewBoard-6.0.1-venv/bin[ 67 ] --> ./pip3 show publicsuffix
Name: publicsuffix
Version: 1.1.1
Summary: Get a public suffix for a domain name using the Public Suffix List.
Home-page:
Author: Tomaz Solc
Author-email: tomaz.solc@tablix.org
License: MIT
Location: /webapp/PYTHON_VENVS/ReviewBoard-6.0.1-venv/lib/python3.11/site-packages
Requires:
Required-by: Djblets
atlvptwebapp01:/webapp/PYTHON_VENVS/ReviewBoard-6.0.1-venv/bin[ 68 ] -->

#6 shubhar

" We'll ship this in Djblets 4.1 soon. If you have a support contract with us, we can get you tailored builds, or you can try backporting the patch internally and seeing if it resolves your problem."

1) How soon ? Like in 2 weeks ? A month ?
2) Sure. I can try backporting the patch internally. How can I get this patch ?

Thanks kindly

Shubha

chipx86
#7 chipx86
  1. We're still trying to figure that out. There's a few things we're coordinating for a release right now, and we have to wait until things are in place.

The thing about this failure is that, provided you're using the same version of Python between your old RB5 install and RB6, you honestly should have had the same problem. This was an issue in the third-party module, and the same version would have been used in both.

I haven't been able to reproduce this yet, but I'll have a couple more follow-up questions on that in a minute.

  1. You can use the version of pip you used above and install the publicsuffixlist package, and then apply this patch to your <venv_path>/lib/python3.11/site-packages/djblets/:

https://reviews.reviewboard.org/r/13454/diff/raw/

That should get you the patch file. You'd want to back up your djblets directory before proceeding. If all goes well, you can restart your web server and try to reproduce the problem.

So those questions:

  1. Is your /webapp path and everything in it local to the file system, or mounted over something like NFS?
  2. Was Review Board 5 installed and managed on Python 3.11, or was it using an older version?
  3. Can you think of any differences in the deployment between your RB5 and RB6 that might be useful for me to know?

Thanks!

chipx86
#8 chipx86

Oh, and part of that patch (the part to setup.py) won't be able to apply to an installed copy. It's meant for the source tree. If it complains, you can skip that file.

chipx86
#9 chipx86

How was your Python installed? What built that environment? I'm looking at the details and flow behind the error and the default encoding for streams isn't what I'd expect.

Can you run this on the server:

$ /webapp/PYTHON_VENVS/ReviewBoard-6.0.1-venv/bin/python
import locale
print(locale.getencoding())
#10 shubhar

Python 3.11.1 (main, Jan 4 2023, 14:43:23) [GCC 9.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import locale
print(locale.getencoding())
UTF-8