5012: requests to /api/... do not use public server name but private IP address

HeroicAlbeit

What version are you running?

6.0

in docker image 45ada0a9f402

this is a new setup with the nginx+gunicorn setup method and a "API Gateway" on Oracle Cloud in front of the nginx port.

The "API Gateway" is setup to route https://<public server name>/<everything> to http://<private instance ip>:8080/<everything>, where 8080 is the exposed nginx port. This works, as I can login.

What's the URL of the page containing the problem?

https://<public server name>/r/3/

this page shows up, but the "Diff" tab is missing and I am unable to change fields of this request, such as Summary or Description.

Using Debug Console of the browser reveals an error, see below.

What steps will reproduce the problem?

  1. create a new review request, ie. by uploading a patch
  2. browse the request
  3. not the missing Diff tab
  4. inspect browser debug console

What is the expected output? What do you see instead?

the Diff tab would be there

editing Fields such as Summary would work

no errors in browser console

What operating system are you using? What browser?

the instance uses Ubuntu 22.04.3 LTS on ARM processor

however, Reviewboard itself runs as above mentioned Docker image, pulling the ARM sha256.

Please provide any additional information below.

the error on debug console is this:

3rdparty-base.min.js:1 Mixed Content: The page at 'https://<public server name>/r/3/' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://<private instance ip>/api/review-requests/3/draft/?api_format=json&force-text-type=html&include-text-types=raw&expand=depends_on%2Ctarget_people%2Ctarget_groups'. This request has been blocked; the content must be served over HTTPS.

this is absolutely correct and can not work, even if the browser would not block it, since <private instance ip> is not routed on the internet.

also note that port 8080 is missing in <private instance ip>; this tells me the API Gateway is not involved as it is setup to always send to this port.

looking at the Network tab in the browser debug tool shows a Request Initiator chain looking like this:

  1. https://<public server name>/r/3/
  2. https://<public server name>/static/lib/js/3rdparty-base.min.js
  3. http://<private instance ip>/api/review-requests/3/draft/?api_format=json&force-text-type=html&include-text-types=raw&expand=depends_on%2Ctarget_people%2Ctarget_groups

the Request call stack for this is rather long and I can't copy-paste it.

the nginx.conf setup follows the Admin Manual, with the essential part being:

    location / {
        proxy_pass http://reviewboard;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Ssl off;
        proxy_set_header X-Real-IP $remote_addr;

        client_max_body_size        10m;
        client_body_buffer_size     128k;
        proxy_connect_timeout       90;
        proxy_send_timeout          90;
        proxy_read_timeout          90;
        proxy_headers_hash_max_size 512;
        proxy_buffer_size           4k;
        proxy_buffers               4 32k;
        proxy_busy_buffers_size     64k;
        proxy_temp_file_write_size  64k;
    }

This handles login/logout and many other things such as configuring, while some (?) /api/ requests dont ever reach this nginx since the browser gets told to send these using the <private instance ip>.

The Server name in General Settings is correctly set to <public server name> - I guess login would not be impossible otherwise.

#1 HeroicAlbeit

to get some more insight and rule out the browser I use curl to do the above failed request with correct <public server name>.

what I get is a JSON body that contains (many) wrong URLs with http://<private instance ip> like so:

$ curl -H 'cookie: csrftoken=<token>; rbsessionid<session>;' 'https://<public server name>/api/review-requests/3/draft/?api_format=json&force-text-type=html&include-text-types=raw&expand=depends_on%2Ctarget_people%2Ctarget_groups'
{"draft": {"branch": "", "bugs_closed": [], "changedescription": "", "changedescription_text_type": "html", "commit_id": null, "depends_on": [], "description": "", "description_text_type": "html", "extra_data": {}, "id": 3, "last_updated": "2023-11-02T07:22:25Z", "links": {"delete": {"href": "http://<private instance ip>/api/review-requests/3/draft/", "method": "DELETE"} ... "submitter": {"href": "http://<private instance ip>/api/users/admin/", "method": "GET", "title": "admin"} ...

looks like the /api/ part is not using the configured server name?

chipx86
#2 chipx86

Thanks for the detailed information!

Our default Nginx setup forwards over the requested hostname to Review Board, and Review Board then uses that hostname to construct URLs for the APIs. In this case, Review Board is seeing its host as being <private instance ip>, which probably makes sense in this case, though we'll need to verify.

Since your Nginx isn't serving up as <public server name>, the host it's sending along to Django isn't going to be <public server name>, so one of two options will need to be explored:

  1. Changing proxy_set_header Host $host to proxy_set_header Host your_public_server_name.
  2. If API Gateway forwards on a host using X-Forwarded-Host in the request, then you can set USE_X_FORWARDED_HOST = True in Review Board's /site/conf/settings_local.py. (This would need to be tested for sure).

The latter is off by default for security reasons, and should only be turned on if in this exact situation.

We'll need to do some work on our end to add documentation and comments throughout these files to cover these scenarios.

Can you see if one or both of those work, and let me know? We'll then get this documented on our end.

#3 HeroicAlbeit

ah, yes. thanks for responding.

meanwhile I came cross your point 1. of rewriting the Host header to <public server name>, like you suggest.

This, however, ran me into problems with Djangos' CSRF checks. To get these working I endet up with this in settings_local.py:

   ALLOWED_HOSTS = [
       "127.0.0.1",
       "<private instance ip>",
   ]
   CSRF_TRUSTED_ORIGINS = [
       "http://<private instance ip>",
       "https://<public server name>"
   ]
   SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

plus proxy_set_header X-Forwarded-Proto https in nginx.conf. The tweak here is to get it to send the correct protocoll to the outside, ie. https://<public server name>/... instead of http://<public server name>/.... This is because the API Gateway terminates the SSL.

I have not done any header (re)writings on the API Gateway.

So far I consider this a workaround, since - as far as I understand - it makes the CSRF checks pretty much useless. (please tell me if I get that wrong, I have no understanding of how such attacks actually work and therefore what the risk is)

oh, another side observation: my Server name (in General Settings on the WebUI) was set to http://<public server name> by the installer and that still is so, despite me using https://... only. If I change this to https://... the CSRF checks break again.