3915: Resolve HG Commit IDs before posting

Misery
What version are you running?
2.0.16 - 2.0.18: broken
2.0.15: works


What steps will reproduce the problem?
1. Create a patch from mercurial (hg diff)
2. Create a review request with that patch
3. Show the diff

Sometimes I have the problem that I cannot upload the patch with the "same problem".


What is the expected output? What do you see instead?
Expected: Show the diff like in 2.0.15.
Actual: Shows an error:

The file '.hgtags' (rf9e2887948e8a1632f07ab179945027884e1f6f4^1) could not be found in the repository

  
   
    Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/diffviewer/views.py", line 265, in get
    response = renderer.render_to_response(request)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/diffviewer/renderers.py", line 56, in render_to_response
    return HttpResponse(self.render_to_string(request))
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/diffviewer/renderers.py", line 74, in render_to_string
    large_data=True)
  File "/usr/local/lib/python2.7/dist-packages/Djblets-0.8.21-py2.7.egg/djblets/cache/backend.py", line 111, in cache_memoize
    data = lookup_callable()
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/diffviewer/renderers.py", line 73, in <lambda>
    lambda: self.render_to_string_uncached(request),
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/diffviewer/renderers.py", line 87, in render_to_string_uncached
    request=request)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/diffviewer/diffutils.py", line 422, in populate_diff_chunks
    chunks = generator.get_chunks()
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/diffviewer/chunk_generator.py", line 156, in get_chunks
    large_data=True)
  File "/usr/local/lib/python2.7/dist-packages/Djblets-0.8.21-py2.7.egg/djblets/cache/backend.py", line 111, in cache_memoize
    data = lookup_callable()
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/diffviewer/chunk_generator.py", line 155, in <lambda>
    lambda: list(self._get_chunks_uncached()),
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/diffviewer/chunk_generator.py", line 162, in _get_chunks_uncached
    old = get_original_file(self.filediff, self.request, encoding_list)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/diffviewer/diffutils.py", line 198, in get_original_file
    request=request)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/scmtools/models.py", line 343, in get_file
    large_data=True)[0]
  File "/usr/local/lib/python2.7/dist-packages/Djblets-0.8.21-py2.7.egg/djblets/cache/backend.py", line 111, in cache_memoize
    data = lookup_callable()
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/scmtools/models.py", line 342, in <lambda>
    request)],
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/scmtools/models.py", line 530, in _get_file_uncached
    base_commit_id=base_commit_id)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/scmtools/hg.py", line 48, in get_file
    base_commit_id=base_commit_id)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.18-py2.7.egg/reviewboard/scmtools/hg.py", line 234, in cat_file
    raise FileNotFoundError(path, rev)
FileNotFoundError: The file '.hgtags' (rf9e2887948e8a1632f07ab179945027884e1f6f4^1) could not be found in the repository

What operating system are you using? What browser?
Linux, Firefox 39
brennie
#1 brennie
The string rf9e2887948e8a1632f07ab179945027884e1f6f4^1 does not look like a valid SHA. Could you show us the relevant diff lines for the .hftags file?
  • +NeedInfo
Misery
#2 Misery
I added another review with same problem.

The file '.hgtags' (r2b2ebdd19c379afb4cf48618ee4f319d93c7e7cf^1) could not be found in the repository

Patch is attached.
Misery
#3 Misery
Got this in logfile:

Reviewboard_1 | 2015-07-16 19:44:18,074 - ERROR -  - 404
Reviewboard_1 | 2015-07-16 19:44:18,759 - ERROR -  - HTTP error code 400 when fetching file from http://scmmanager/scm/hg/REPO/raw/2b2ebdd19c379afb4cf48618ee4f319d93c7e7cf^1/.hgtags: HTTP Error 400: Bad Request
Reviewboard_1 | 2015-07-16 19:44:19,461 - ERROR -  - HTTP error code 400 when fetching file from http://scmmanager/scm/hg/REPO/hg-history/2b2ebdd19c379afb4cf48618ee4f319d93c7e7cf^1/.hgtags: HTTP Error 400: Bad Request
Misery
#4 Misery
Ping! Do you need some more info?
brennie
#5 brennie
Sorry I haven't had time to look at this bug.

What are you using to serve your repo, hg web?
Misery
#6 Misery
We use https://www.scm-manager.org for this. It will provide a hgweb, yes.
brennie
#7 brennie
SCM Manager is a hosting service that Review Board currently does not support.
  • -NeedInfo
    +Confirmed
  • -Type-Defect
    +Type-Enhancement
  • +Add support for SCM Manager as a HostingService
Misery
#8 Misery
I dont' think that SCM-Manager is the problem! It spawns a python shell with hgweb of Mercurial.

Nevertheless... I tried "hg serve" with the repository and Reviewboard 2.0.18. It is still broken.

Access Log of "hg serve":
10.21.253.22 - - [14/Aug/2015 12:52:43] "GET /raw-file/f9e2887948e8a1632f07ab179945027884e1f6f4^1/.hgtags HTTP/1.1" 404 -
10.21.253.22 - - [14/Aug/2015 12:52:44] "GET /raw/f9e2887948e8a1632f07ab179945027884e1f6f4^1/.hgtags HTTP/1.1" 400 -
10.21.253.22 - - [14/Aug/2015 12:52:44] "GET /hg-history/f9e2887948e8a1632f07ab179945027884e1f6f4^1/.hgtags HTTP/1.1" 400 -
chipx86
#9 chipx86
The "^1" is the problem. That's not a valid HG revision, when it comes to fetching files remotely. 

The diff file you posted doesn't match the revisions being referred to. Can you show an up-to-date diff with the accompanying errors, along with the command line used to generate the diff, and any steps you used to post it?
  • -Confirmed
    +NeedInfo
  • +SCM-Mercurial
Misery
#10 Misery
Ok, I found the problem.

I'm using a script on server side [1] to create the diff, parent diff and the base commit id using MercurialClient class from rbtools 0.7.4 (tried 5985bfc7145fc240c9e23a99d80f1e149bec355b, too).

So the reference is a NODEID^1 that will produce correct diff and parent diff (without ^1) but MercurialClient returns the NODEID^1 as base_commit_id.


[1] https://reviews.reviewboard.org/r/7100/
Misery
#11 Misery
Work-around:

diff --git a/rbtools/clients/mercurial.py b/rbtools/clients/mercurial.py
index ad0d1f9..2f6edc9 100644
--- a/rbtools/clients/mercurial.py
+++ b/rbtools/clients/mercurial.py
@@ -412,6 +412,9 @@ class MercurialClient(SCMClient):
             base_commit_id = revisions['base']
             parent_diff = None
 
+        base_commit_id = self._execute(
+             ['hg', 'log', '-r', base_commit_id, '-T {node}'], env=self._hg_env, results_unicode=False)
+
         return {
             'diff': diff,
             'parent_diff': parent_diff,

Misery
#12 Misery
This introduces it: https://github.com/reviewboard/reviewboard/commit/6b34af11397d92ae5c396ed911840ae6b0392cfd

If in table "diffviewer_diffset" in column "base_commit_id" is already a NODEID^1 it is, of course, still broken for old reviews.
Misery
#14 Misery
update diffviewer_diffset set base_commit_id=NULL where base_commit_id like '%^1';
brennie
#15 brennie
How exactly are you generating the diff with MercurialClient? You may be generating it in an incorrect way. If you generate the diff through rbt, does it work as intended?
Misery
#16 Misery
You can see that in "rbtools/hooks/mercurial.py" with "class MercurialDiffer" at that review request [1] from above.

It calls MercurialClient with "NODEID^1" as it is allowed on local hg clients like with local git clients.

hg log -r tip^1
hg log -r tip^2
hg log -r tip^3

git log -r HEAD~1
git log -r HEAD~2
git log -r HEAD~3

[1] https://reviews.reviewboard.org/r/7100/
misery
#17 misery

This issue should be renamed since it's a bug in rbtools and not a feature request. :-)

#18 ixolt

Please add SCM-manager as a Hosting service for local scm-manager repositories!

misery
#19 misery

This issue is not to add SCM-Manager support!! It's just an incorrect name!

brennie
#20 brennie

A patch is under review at https://reviews.reviewboard.org/r/7823/

  • -NeedInfo
    +PendingReview
  • -Add support for SCM Manager as a HostingService
    +Resolve HG Commit IDs before posting
david
#21 david

Fixed in release-0.7.x (d66f628). This will ship in 0.7.6. Thanks!

  • -PendingReview
    +Fixed