1481: Review Board 500 error if testing done is over 65535 characters (or a bit smaller)
- Fixed
- Review Board
| lon***@gmai***** (Google Code) (Is this you? Claim this profile.) | |
| Aug. 29, 2011 | |
| 1648 |
What version are you running?
1.5.0.1
What's the URL of the page containing the problem?
private, but can be reproduced easily, by submitting a new request
see: http://demo.reviewboard.org/r/3217/
(or http://demo.reviewboard.org/r/3217/diff as the above reports a 500)
What steps will reproduce the problem?
1. Create a new review request, publish it.
2. Update the review request, filling the testing done field with over
67000 characters (for instance: "".join(map(str,range(1,16000))) )
3. Save
4. Review Board reports a 500 error.
Traceback (most recent call last):
File
"/opt/software/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/handlers/base.py",
line 92, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File
"/opt/software/lib/python2.5/site-packages/ReviewBoard-1.0.5.1-py2.5.egg/reviewboard/accounts/decorators.py",
line 25, in _check
return login_required(view_func)(*args, **kwargs)
File
"/opt/software/lib/python2.5/site-packages/Djblets-0.5.5-py2.5.egg/djblets/auth/util.py",
line 46, in _checklogin
return view_func(request, *args, **kwargs)
File
"/opt/software/lib/python2.5/site-packages/ReviewBoard-1.0.5.1-py2.5.egg/reviewboard/reviews/views.py",
line 188, in review_detail
for changedesc in changedescs:
File
"/opt/software/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/db/models/query.py",
line 106, in _result_iter
self._fill_cache()
File
"/opt/software/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/db/models/query.py",
line 692, in _fill_cache
self._result_cache.append(self._iter.next())
File
"/opt/software/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/db/models/query.py",
line 251, in iterator
obj = self.model(*row[index_start:aggregate_start])
File
"/opt/software/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/db/models/base.py",
line 324, in __init__
signals.post_init.send(sender=self.__class__, instance=self)
File
"/opt/software/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/dispatch/dispatcher.py",
line 166, in send
response = receiver(signal=self, sender=sender, **named)
File
"/opt/software/lib/python2.5/site-packages/Djblets-0.5.5-py2.5.egg/djblets/util/fields.py",
line 174, in post_init
value = self.loads(value)
File
"/opt/software/lib/python2.5/site-packages/Djblets-0.5.5-py2.5.egg/djblets/util/fields.py",
line 206, in loads
val = eval(val)
File "<string>", line 1
{"testing_done": {"new":
["12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814....
5264152651526615267152681526915270152711527215273152741527515276152771527815279152801528115282152831528415285152861528715288152891529015291152921529315294152951529615297152981529915300153011530215303153041530515306153071530815309153101531115312153131531415315153161531715318153191532015321153221532
^
SyntaxError: EOL while scanning single-quoted string
What is the expected output? What do you see instead?
The review should be published. A 500 error is displayed instead.
What operating system are you using? What browser?
Linux for the server. Firefox/windows for the browser.
Please provide any additional information below.
The issue is that the fields_changed field in changedescs_changedescription
is truncated to 65535 characters as shown below, and the python object is
missing the ']}} termination characters to have a well formed string that
JSON could parse (I'm using simplejson).
mysql> select id, right(fields_changed,100) from
changedescs_changedescription where length(fields_changed)>65000 ;
+-------+------------------------------------------------------------------------------------------------------+
| id | right(fields_changed,100)
|
+-------+------------------------------------------------------------------------------------------------------+
| 10043 |
3153041530515306153071530815309153101531115312153131531415315153161531715318153191532015321153221532
|
+-------+------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql> select id, left(fields_changed,100) from
changedescs_changedescription where length(fields_changed)>65000 ;
+-------+------------------------------------------------------------------------------------------------------+
| id | left(fields_changed,100)
|
+-------+------------------------------------------------------------------------------------------------------+
| 10043 | {"testing_done": {"new":
["1234567891011121314151617181920212223242526272829303132333435363738394041 |
+-------+------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql> select length(fields_changed) from changedescs_changedescription
where length(fields_changed)>65000 ;
+------------------------+
| length(fields_changed) |
+------------------------+
| 65535 |
+------------------------+
1 row in set (0.00 sec)
mysql> select id, right(testing_done, 100) from reviews_reviewrequest where
length(testing_done)>65000 ;
+------+------------------------------------------------------------------------------------------------------+
| id | right(testing_done, 100)
|
+------+------------------------------------------------------------------------------------------------------+
| 4246 |
091531015311153121531315314153151531615317153181531915320153211532215323153241532515326153271532815A
|
+------+------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql> select id, length(testing_done) from reviews_reviewrequest where
length(testing_done)>65000 ;
+------+----------------------+
| id | length(testing_done) |
+------+----------------------+
| 4246 | 65537 |
+------+----------------------+
1 row in set (0.01 sec)
Clearly, it's not very reasonable to have such a large data set, but I
think we need to protect from it. As the fields_changed can contain
several fields, the problem could be recreated using different combination
of old and new fields as well.
I reproduced the issue on the demo server 1.1 alpha 3 (dev), but I'm getting an
additional error at submission time. It's a bit better, but the review is still not
accessible:
http://demo.reviewboard.org/r/3217/
Warning at /api/json/reviewrequests/3217/publish/
Data truncated for column 'fields_changed' at row 1
Request Method: POST
Request URL: http://demo.reviewboard.org/api/json/reviewrequests/3217/publish/
Exception Type: Warning
Exception Value:
Data truncated for column 'fields_changed' at row 1
Exception Location: /usr/lib/python2.5/warnings.py in warn_explicit, line 102
Python Executable: /usr/bin/python
Python Version: 2.5.2
Python Path: ['/var/www/demo.reviewboard.org/conf',
'/usr/lib/python2.5/site-packages/python_memcached-1.40-py2.5.egg',
'/usr/lib/python2.5/site-packages/BuildBatter-0.1-py2.5.egg',
'/usr/lib/python2.5/site-packages/virtualenv-1.3-py2.5.egg',
'/usr/lib/python2.5/site-packages/django_evolution-0.0.0-py2.5.egg',
'/usr/lib/python2.5/site-packages/flup-1.0.1-py2.5.egg',
'/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg',
'/usr/lib/python2.5/site-packages/Jinja2-2.1.1-py2.5-linux-i686.egg',
'/usr/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg',
'/usr/lib/python2.5/site-packages/JCC-2.1-py2.5-linux-i686.egg',
'/usr/lib/python2.5/site-packages/pytz-2009f-py2.5.egg',
'/usr/lib/python2.5/site-packages/gitosis-0.2-py2.5.egg',
'/usr/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg',
'/usr/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg',
'/usr/lib/python2.5/site-packages/Sphinx-0.6.3-py2.5.egg',
'/usr/lib/python2.5/site-packages/recaptcha_client-1.0.5-py2.5.egg',
'/usr/lib/python2.5/site-packages/paramiko-1.7.6-py2.5.egg',
'/usr/lib/python2.5/site-packages/Pygments-1.1.1-py2.5.egg',
'/usr/lib/python2.5/site-packages/pycrypto-2.0.1-py2.5-linux-i686.egg',
'/usr/lib/python2.5/site-packages/ReviewBoard-1.1alpha3.dev_20100113-py2.5.egg',
'/usr/lib/python2.5/site-packages/Djblets-0.5.7alpha0.dev_20100113-py2.5.egg',
'/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2',
'/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload',
'/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages',
'/usr/lib/python2.5/site-packages/PIL', '/var/lib/python-support/python2.5']
Server time: Mon, 1 Feb 2010 12:05:04 -0800
Traceback Switch to copy-and-paste view
*
/usr/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/handlers/base.py
in get_response
85. # Apply view middleware
86. for middleware_method in self._view_middleware:
87. response = middleware_method(request, callback, callback_args,
callback_kwargs)
88. if response:
89. return response
90.
91. try:
92. response = callback(request, *callback_args, **callback_kwargs) ...
93. except Exception, e:
94. # If the view raised an exception, run it through exception
95. # middleware, and if the exception middleware returns a
96. # response, use that. Otherwise, reraise the exception.
97. for middleware_method in self._exception_middleware:
98. response = middleware_method(request, e)
? Local vars
Variable Value
callback
<function review_request_publish at 0x9733a04>
callback_args
()
callback_kwargs
{'api_format': u'json', 'review_request_id': u'3217'}
e
Warning(u"Data truncated for column 'fields_changed' at row 1",)
exc_info
(<class '_mysql_exceptions.Warning'>, Warning(u"Data truncated for column
'fields_changed' at row 1",), <traceback object at 0xa4f0d24>)
exceptions
<module 'django.core.exceptions' from
'/usr/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/exceptions.pyc'>
middleware_method
<bound method LoggingMiddleware.process_exception of
<djblets.log.middleware.LoggingMiddleware object at 0xa16718c>>
receivers
[(<function _rollback_on_exception at 0x8f59a04>, None)]
request
<ModPythonRequest path:/api/json/reviewrequests/3217/publish/, GET:<QueryDict:
{}>, POST:<QueryDict: {u'dummy': [u'']}>, COOKIES:{'__utma':
'78002967.1485541389.1255497952.1264623807.1265052397.42', '__utmb':
'78002967.3.10.1265052397', '__utmc': '78002967', '__utmz':
'78002967.1264623807.41.24.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=reviewboard%20admin',
'collapsediffs': 'True', 'rbsessionid': '15e22e0f7022de794508d3159bedfbb9'},
META:{'AUTH_TYPE': None, 'CONTENT_LENGTH': '6', 'CONTENT_TYPE':
'application/x-www-form-urlencoded; charset=UTF-8', 'GATEWAY_INTERFACE': 'CGI/1.1',
'HTTP_ACCEPT': 'application/json, text/javascript, */*', 'HTTP_ACCEPT_CHARSET':
'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'HTTP_CACHE_CONTROL': 'no-cache',
'HTTP_CONNECTION': 'keep-alive', 'HTTP_CONTENT_LENGTH': '6', 'HTTP_CONTENT_TYPE':
'application/x-www-form-urlencoded; charset=UTF-8', 'HTTP_COOKIE':
'__utma=78002967.1485541389.1255497952.1264623807.1265052397.42;
__utmz=78002967.1264623807.41.24.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=reviewboard%20admin;
__utmc=78002967; __utmb=78002967.3.10.1265052397;
rbsessionid=15e22e0f7022de794508d3159bedfbb9; collapsediffs=True', 'HTTP_HOST':
'demo.reviewboard.org', 'HTTP_PRAGMA': 'no-cache', 'HTTP_REFERER':
'http://demo.reviewboard.org/r/3217/', 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U;
Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7', 'HTTP_VIA': '1.1
nc-corpse (NetCache NetApp/6.0.3)', 'HTTP_X_FORWARDED_FOR': '10.34.25.33',
'HTTP_X_REQUESTED_WITH': 'XMLHttpRequest', 'PATH_INFO':
u'/api/json/reviewrequests/3217/publish/', 'PATH_TRANSLATED': None, 'QUERY_STRING':
None, 'REMOTE_ADDR': '198.95.226.224', 'REMOTE_HOST': None, 'REMOTE_IDENT': None,
'REMOTE_USER': None, 'REQUEST_METHOD': 'POST', 'SCRIPT_NAME': '', 'SERVER_NAME':
'demo.reviewboard.org', 'SERVER_PORT': 80, 'SERVER_PROTOCOL': 'HTTP/1.1',
'SERVER_SOFTWARE': 'mod_python'}>
resolver
<RegexURLResolver djblets.util.rooturl (None:None) ^/>
response
None
self
<django.core.handlers.modpython.ModPythonHandler object at 0xa21eecc>
settings
<django.conf.LazySettings object at 0x8d1386c>
urlconf
'djblets.util.rooturl'
urlresolvers
<module 'django.core.urlresolvers' from
'/usr/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/urlresolvers.pyc'>
*
Attaching the complete traceback on submission.
-
+
We've experienced this recently on a couple of reviews. Contemplating changing fields_changed in relation changedescs_changedescription from MySQL 'text' type to 'mediumtext' or 'longtext' but curious what others have done or if there is recommended solution besides telling users not to post so much content in description.
It's a problem of djblets' JSONField.
Please remove 'db_type' method from JSONField, and everything goes well, because TextField uses 'longtext' in default.
class JSONField(models.TextField):
"""
A field for storing JSON-encoded data. The data is accessible as standard
Python data types and is transparently encoded/decoded to/from a JSON
string in the database.
"""
serialize_to_string = True
def __init__(self, verbose_name=None, name=None,
encoder=DjangoJSONEncoder(), **kwargs):
models.TextField.__init__(self, verbose_name, name, blank=True,
**kwargs)
self.encoder = encoder
- def db_type(self, connection=None):
- return "text"
Django TextField's mysql type is defined at django.db.backends.mysql.creation
Related issue (Django): https://code.djangoproject.com/ticket/489
sorry, diff above is invalid. new one here:
diff --git a/djblets/util/fields.py b/djblets/util/fields.py
index 33b901b..30671fc 100644
--- a/djblets/util/fields.py
+++ b/djblets/util/fields.py
@@ -149,8 +149,8 @@ class JSONField(models.TextField):
**kwargs)
self.encoder = encoder
- def db_type(self, connection=None):
- return "text"
+ def get_internal_type(self):
+ return "TextField"
def contribute_to_class(self, cls, name):
def get_json(model_instance):
Why is the get_internal_type needed? We should automatically get this from TextField.get_internal_type.
You are right. I misunderstood something. We don't need get_internal_type here. Related changeset: https://code.djangoproject.com/changeset/7133 Sorry for confusing post.
Fixed on Djblets 0.6.10 (a04eae9).
-
+ Fixed