3744: Unicode in the "Ship It' comments isn't viewable in the review page

jaredgrubb
May 17, 2015 
What version are you running?
2.0.11

What's the URL of the page containing the problem?
private.

What steps will reproduce the problem?
1. Click the "ship it" button but add the text "👍👍👍" to the comment field.

What is the expected output? What do you see instead?
The email sends the comment just fine.
The web page shows no commend at all.

What operating system are you using? What browser?
OSX 10.10. Safari.
david
#1 david 
Can you check to make sure that your database tables are utf-8? By default, mysql tables use iso-latin-1, which will squash any unicode characters.
  • +NeedInfo
#2 jaredgrubb 
Yes... the fact that the email notification contained the characters would suggest that the DB was ok and that it's a web-ui issue?

> SHOW CREATE DATABASE reviewboard;
+-------------+----------------------------------------------------------------------+
| Database    | Create Database                                                      |
+-------------+----------------------------------------------------------------------+
| reviewboard | CREATE DATABASE `reviewboard` /*!40100 DEFAULT CHARACTER SET utf8 */ |
+-------------+----------------------------------------------------------------------+
david
#3 david 
Would you mind checking the contents of the reviews table for your review and verify that it's stored correctly?

The e-mail might look correct because the e-mail code is operating on the same python object as when it was saved. It's only when the data is re-loaded from the database that you'd see the problem.
#4 jaredgrubb 
I'm having trouble finding the comment ... here is what I tried:

select * from reviews_review_comments, reviews_comment where reviews_comment.id=reviews_review_comments.comment_id and reviews_review_comments.review_id=4371;

But the comments dont seem to match the review id (which appears to be right based on the reviewrequest table: select * from reviews_reviewrequest where id=4371;).
david
#5 david 
If you just put the text in the text box that you get in the "Review" dialog, it's part of the review, not in a comment (comments are when you add a comment to the diff)

select * from reviews_review where review_request_id=4371;
#6 jaredgrubb 
Ah, ok... yes that worked. Here is the row. I assume that this means it never got in the DB?

+------+-------------------+---------+---------------------+--------+---------+------------------+-------------------------------------------------+---------------------+----------+-------------+----------------------+-------------------------+---------------------+-----------+------------+
| id   | review_request_id | user_id | timestamp           | public | ship_it | base_reply_to_id | email_message_id                                | time_emailed        | body_top | body_bottom | body_top_reply_to_id | body_bottom_reply_to_id | reviewed_diffset_id | rich_text | extra_data |
+------+-------------------+---------+---------------------+--------+---------+------------------+-------------------------------------------------+---------------------+----------+-------------+----------------------+-------------------------+---------------------+-----------+------------+
| 9242 |              4371 |       9 | 2015-01-29 20:50:21 |      1 |       1 |             NULL | <20150129205021.11594.12299@xxxxxxxx.xxxxx.xxx> | 2015-01-29 20:50:21 |          |             |                 NULL |                    NULL |                NULL |         1 | {}         |
+------+-------------------+---------+---------------------+--------+---------+------------------+-------------------------------------------------+---------------------+----------+-------------+----------------------+-------------------------+---------------------+-----------+------------+

Via "show create table reviews_review;" I can see that the default charset for that table is utf8:

CREATE TABLE `reviews_review` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `review_request_id` int(11) NOT NULL,
  `user_id` int(11) NOT NULL,
  `timestamp` datetime NOT NULL,
  `public` tinyint(1) NOT NULL,
  `ship_it` tinyint(1) NOT NULL,
  `base_reply_to_id` int(11) DEFAULT NULL,
  `email_message_id` varchar(255) DEFAULT NULL,
  `time_emailed` datetime DEFAULT NULL,
  `body_top` longtext NOT NULL,
  `body_bottom` longtext NOT NULL,
  `body_top_reply_to_id` int(11) DEFAULT NULL,
  `body_bottom_reply_to_id` int(11) DEFAULT NULL,
  `reviewed_diffset_id` int(11) DEFAULT NULL,
  `rich_text` tinyint(1) NOT NULL,
  `extra_data` longtext,
  PRIMARY KEY (`id`),
  KEY `reviews_review_bf7a5334` (`review_request_id`),
  KEY `reviews_review_fbfc09f1` (`user_id`),
  KEY `reviews_review_3534312d` (`base_reply_to_id`),
  KEY `reviews_review_9faae2ee` (`body_top_reply_to_id`),
  KEY `reviews_review_4a7afee7` (`body_bottom_reply_to_id`),
  KEY `reviews_review_54af9375` (`reviewed_diffset_id`),
  CONSTRAINT `base_reply_to_id_refs_id_e4301b6b` FOREIGN KEY (`base_reply_to_id`) REFERENCES `reviews_review` (`id`),
  CONSTRAINT `body_bottom_reply_to_id_refs_id_e4301b6b` FOREIGN KEY (`body_bottom_reply_to_id`) REFERENCES `reviews_review` (`id`),
  CONSTRAINT `body_top_reply_to_id_refs_id_e4301b6b` FOREIGN KEY (`body_top_reply_to_id`) REFERENCES `reviews_review` (`id`),
  CONSTRAINT `reviewed_diffset_id_refs_id_ab2fc2f6` FOREIGN KEY (`reviewed_diffset_id`) REFERENCES `diffviewer_diffset` (`id`),
  CONSTRAINT `review_request_id_refs_id_f4615875` FOREIGN KEY (`review_request_id`) REFERENCES `reviews_reviewrequest` (`id`),
  CONSTRAINT `user_id_refs_id_b356281c` FOREIGN KEY (`user_id`) REFERENCES `auth_user` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=9333 DEFAULT CHARSET=utf8 |
david
#7 david 
Yeah, the text should be present in the body_top column. Perhaps there was some error in the server logs when you published this?
david
#8 david
  • -NeedInfo
    +Incomplete