What version are you running?
4.0.0-rc1
What's the URL of the page containing the problem?
N/A
What steps will reproduce the problem?
'rb-site manage <path> dumpdb'
What is the expected output? What do you see instead?
I expect the command to progress to 100% and exit successfully. Instead, it errors out at about 43%.
Output (filepaths changed because reasons):
Traceback (most recent call last):
File "/foobar/python3_user_packages/bin/rb-site", line 8, in <module>
sys.exit(main())
File "/foobar/python3_user_packages/lib/python3.9/site-packages/reviewboard/cmdline/rbsite.py", line 2706, in main
command.run(site, options)
File "/foobar/python3_user_packages/lib/python3.9/site-packages/reviewboard/cmdline/rbsite.py", line 2476, in run
site.run_manage_command(manage_command, manage_args)
File "/foobar/python3_user_packages/lib/python3.9/site-packages/reviewboard/cmdline/rbsite.py", line 1047, in run_manage_command
manage_util.execute()
File "/foobar/python3_user_packages/lib/python3.9/site-packages/django/core/management/init.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/foobar/python3_user_packages/lib/python3.9/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(args, cmd_options)
File "/foobar/python3_user_packages/lib/python3.9/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, options)
File "/foobar/python3_user_packages/lib/python3.9/site-packages/reviewboard/admin/management/commands/dumpdb.py", line 52, in handle
value = serializer.serialize([obj])
File "/foobar/python3_user_packages/lib/python3.9/site-packages/django/core/serializers/base.py", line 97, in serialize
self.end_object(obj)
File "/foobar/python3_user_packages/lib/python3.9/site-packages/django/core/serializers/json.py", line 64, in end_object
json.dump(self.get_dump_object(obj), self.stream, *self.json_kwargs)
File "/foobar/lib/python3.9/json/init.py", line 179, in dump
for chunk in iterable:
File "/foobar/lib/python3.9/json/encoder.py", line 431, in _iterencode
yield from _iterencode_dict(o, _current_indent_level)
File "/foobar/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/foobar/lib/python3.9/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/foobar/lib/python3.9/json/encoder.py", line 438, in _iterencode
o = _default(o)
File "/foobar/python3_user_packages/lib/python3.9/site-packages/django/core/serializers/json.py", line 124, in default
return super(DjangoJSONEncoder, self).default(o)
File "/foobar/lib/python3.9/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.class.name} '
TypeError: Object of type bytes is not JSON serializableWhat operating system are you using? What browser?
Red Hat Enterprise Linux 7
Please provide any additional information below.
Using a postgresql 11.5 DB backend. Using python 3.9.1. Attempted the dump immediately after updating from ReviewBoard 3.0.20
Hmm, that's strange. Looks more like an issue in Django's code. I'll see if I can reproduce.
As mentioned on issue #4921, we're probably going to have to phase out these tools at some point, due to issues like this in Django. Native database dump/restore tools are more reliable. We do have a successor to dumpdb/loaddb that's in the works that avoids all the Django compatibility issues that these tools have (as they're based on Django's own loaddata/dumpdata, and aren't necessarily cross-version-compatible). That'll be coming as a free component to Power Pack in the future.
-
+ Release-4.0 -
+ Component:RB-Site -
+ chipx86
After much analysis, and making a real attempt and keeping things compatible, the unfortunate conclusion is that we're removing these tools for 4.0.
As mentioned,
dumpdb/loaddbare fairly thin wrappers around Django'sdumpdata/loaddata, and those are absolutely not forwards/backwards-compatible between the versions of Django used for Review Board 3.0/4.0. These were never really fully built for production use anyway, more for development (in fact, loading requires a development setup).There are plenty of issues deep within Django for compatible serialization, and we'd have to rewrite much of that stack, which isn't worth the time, given the work going into Power Pack.
We'll be adding a deprecation notice and warning when running these tools in 3.0.21, and we'll be stubbing them out entirely in 4.0, showing just an error message with guidance on alternatives.
absolutely not forwards/backwards-compatible between the versions of Django used for Review Board 3.0/4.0.
Yeah, it was pretty clear to me (from forum posts or documentation, I don't remember) that dumbdb/loaddb were only suitable for changing DB backends within the same version of Django and Review Board. And that was my use case - I wanted to migrate from postgresql to MySQL.
guidance on alternatives
At least for postgresql->MySQL migration, I had good luck with this process:
1) use
pg_dumpto snapshot the existing DB (data only, not table structures)
2) use https://github.com/ChrisLundquist/pg2mysql to convert it into a MySQL-compatible format (this required a bit of manual tweaking as well, mostly to the table names in the INSERT statements, and the syntax for binary values (i.e. '\x0123456789abcdef' to X'0123456789abcdef') )
3) use 'rb-site install' to set up the new MySQL database
4) truncate the MySQL tables & load the data from the MySQL-compatible snapshot (need to disable foreign key checks during this step)I can understand you might be reluctant to 'bless' this flow by adding it to the documentation, but it would be nice if this were documented somewhere.
Thanks. Yeah, documentation would be good here. We're working to get the new tool in shape, which will make this a lot easier.
Given that we had to nuke the old tool, though, I have to close as WontFix.
-
- New + WontFix