Fetching images

[jgaehring]

I’m trying to fetch images in Field Kit, but getting 403 errors, even if I use my OAuth token in an AJAX request.

Here are my request headers:

GET /system/files/farm/log/2023-01/IMG_1372.JPG HTTP/1.1
Accept: image/jpeg
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImp0aSI6IjEyMDlmNWM1MDczNWNiZGYxNDUxNGYyZjk3ZmY2Mjc2MGYwMGFiZDM5ODEwNGQxOTQxMGRjZTc1MTU0NDViMzFmM2MyN2QzZmE4ZTJkZmQzIn0.eyJhdWQiOiJmaWVsZGtpdCIsImp0aSI6IjEyMDlmNWM1MDczNWNiZGYxNDUxNGYyZjk3ZmY2Mjc2MGYwMGFiZDM5ODEwNGQxOTQxMGRjZTc1MTU0NDViMzFmM2MyN2QzZmE4ZTJkZmQzIiwiaWF0IjoxNjc3MTc5OTEzLCJuYmYiOjE2NzcxNzk5MTMsImV4cCI6MTY3NzE4MzUxMy45MjE5MTcsInN1YiI6IjEiLCJzY29wZSI6WyJhbm9ueW1vdXMiLCJhdXRoZW50aWNhdGVkIiwiZmFybV9tYW5hZ2VyIiwiZmFybV92aWV3ZXIiLCJmYXJtX3dvcmtlciJdfQ.uWnXp-wBuuSwpUz3viU5-uAtOs6mzJGBupPDaMTX1VM_SyqdVULbKyGxz8aE_S-czV-HIohesHztWNPsNkBGHRlAORuJlurs2-m0lSm1f2W71oTTEfwgS-f68SJYw-hmaUqLRmb7GIVxuFFk9sizteNOHhN3Cx_DJsOk1I0lLm_9LfcLbbiVQan62w1IQ9InFBq-aFCeoxg0tMa8q6s-wUIxbE2lJa6pMqlo2vxgaQ2YYBxZz1lTWIyyFnmUYT-n9yHVHOibBcofmuTGEoprJ6kalRoP-EorELJ9mFVWPotrtAIZq1mTpoQZFtuDSHbZmbh7KlAPDLE6rsx2lm1IYQ_jjtgEzfoOXdwef02dcQZrrVlCsDYyC0nVsJU-RvKJMQRrMppOBDklwaDbds3oqe9r_xD1qSfhdSiMb59BW4XBaaHeeqvn4enWnBgn5GV8zFHjxRC7KSoWUryVwvVgwcJm_TzAoRAuOLHnth3VGE6jLb-9ispKkz6NNjztHGjnSlMqaTemE9r4kqIu6KD_LdOXb0D2_Y0uZOy6aZ5avOCIYP4UQXZLcL0BUOFXKn-vRVit_ErC1HePipdWjkxj76rs3bAgRzGS3vmvpTC508fm8yYvAp_rkhlz8deVe_aDBuE6H886f-qQN-1sionTJ6019k8ld8bu9J_26btZ2_o
Connection: keep-alive
Host: localhost:8080
Referer: http://localhost:8080/observations
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36
sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Linux"

And my response headers:

HTTP/1.1 403 Forbidden
Access-Control-Allow-Origin: *
date: Thu, 23 Feb 2023 19:24:46 GMT
server: Apache/2.4.54 (Debian)
cache-control: must-revalidate, no-cache, private
x-ua-compatible: IE=edge
content-language: en
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
expires: Sun, 19 Nov 1978 05:00:00 GMT
x-generator: Drupal 9 (https://www.drupal.org)
connection: close
transfer-encoding: chunked
content-type: text/html; charset=UTF-8

Strangely, I’ve tried the same thing in Node.js and it works there. I’m wondering, do I need to configure the server’s authorization and/or CORS settings somehow to specifically allow images to be sent, or is this just a problem with my request?

[Symbioquine]

It’s working for me via the fetch API;

Setup is more or less the same as this docker-compose.yml file: farmOS_asset_link/docker-compose.yml at c37d405a302c638d18906c28bf9a36459c6df883 · symbioquine/farmOS_asset_link · GitHub (I’m running it through an NGINX proxy for https because I was testing a web worker detail that doesn’t work without https, but I don’t think that should matter for this.)

[Symbioquine]

If you check out this directory: farmOS_asset_link/docker at 1c6af6b2a06036a0f59b4d9d02cf5d581206b486 · symbioquine/farmOS_asset_link · GitHub and do docker-compose up in it, it should spin up a farmOS instance with that tractor asset/image already available in case you want to try reproducing my request/environment.

[mstenta]

@Symbioquine Does Asset Link leverage the existing cookie-based authentication that is present when you are logged into farmOS normally? The /system/files/.. endpoint that is used for “private files” is a Drupal endpoint (not a direct URL to the file itself through the webserver), so it would make sense that sending a valid Drupal session cookie with a request to that endpoint would work. I see a session cookie in the headers from your screenshot.

@jgaehring It’s interesting that you said this works through node.js, but not through client-side requests. Are all the headers the same otherwise?

[Symbioquine]

@mstenta

Does Asset Link leverage the existing cookie-based authentication that is present when you are logged into farmOS normally?

Yes

@mstenta

The /system/files/.. endpoint that is used for “private files” is a Drupal endpoint (not a direct URL to the file itself through the webserver), so it would make sense that sending a valid Drupal session cookie with a request to that endpoint would work. I see a session cookie in the headers from your screenshot.

Oh yeah, that could certainly be a difference between how Asset Link and Field Kit would be authorized to access images.

[Symbioquine]

It looks like it’s working via a bearer token (via curl) too;

bash$ OAUTH2_ACCESS_TOKEN=`curl -X POST -d "grant_type=password&username=root&password=test&client_id=farm&scope=farm_manager" https://farmos.test/oauth/token | grep -Po 'access_token":"\K[^"]+'`
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2191    0  2110  100    81   5538    212 --:--:-- --:--:-- --:--:--  5765
bash$ curl -i -v --header "Authorization: Bearer $OAUTH2_ACCESS_TOKEN" 'https://farmos.test/system/files/farm/asset/2022-06/MF_698_tractor_24f07.JPG'
*   Trying 192.168.88.176:443...
* Connected to farmos.test (192.168.88.176) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* [CONN-0-0][CF-SSL] TLSv1.0 (OUT), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS handshake, Server finished (14):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Finished (20):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS handshake, Finished (20):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Finished (20):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: O=mkcert development certificate; OU=morgan@scorpion
*  start date: Jun  1 00:00:00 2019 GMT
*  expire date: Aug 29 15:13:13 2030 GMT
*  subjectAltName: host "farmos.test" matched cert's "farmos.test"
*  issuer: O=mkcert development CA; OU=morgan@scorpion; CN=mkcert morgan@scorpion
*  SSL certificate verify ok.
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /system/files/farm/asset/2022-06/MF_698_tractor_24f07.JPG HTTP/1.1
> Host: farmos.test
> User-Agent: curl/7.87.0
> Accept: */*
> Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImp0aSI6ImNmMmYxMjcwNDU0MmIxYmQ5ZGVjYzZhMjAzNjEyODlhOGJjMWY2OTc2OWVlOTA4ZTg3ZDE4NjdmOWVlYjBkNjU4YjMyOGJhMmQ1NzMzNTY4In0.eyJhdWQiOiJmYXJtIiwianRpIjoiY2YyZjEyNzA0NTQyYjFiZDlkZWNjNmEyMDM2MTI4OWE4YmMxZjY5NzY5ZWU5MDhlODdkMTg2N2Y5ZWViMGQ2NThiMzI4YmEyZDU3MzM1NjgiLCJpYXQiOjE2NzcxODg1NjgsIm5iZiI6MTY3NzE4ODU2OCwiZXhwIjoxNjc3MTkyMTY4LjUyNzEyOSwic3ViIjoiMSIsInNjb3BlIjpbImFub255bW91cyIsImF1dGhlbnRpY2F0ZWQiLCJmYXJtX21hbmFnZXIiLCJmYXJtX3ZpZXdlciIsImZhcm1fd29ya2VyIl19.kfVPOCY8NrJJlHFDUcCiMjpL8H6yHhDg67s55x2wy7RL8SMSt3unvqsuommy1PSJ2_zFltSnXfpUOwEl--Z4VVGG4ETw1DOxIrmxS_8LNW--ncIsdd9wRDl2yqMiibpo7vbSsNZD0JZUKQQIjdb0l2zIxXpTzIB7nCHfzh-3G_Tn-NwgxWahjAP-xBhQ0PZF7WaH0891vbReyYzlE-OcmNOzDTLqoN9eo8i121YQpF1BMTiySjcMqwRaUoFOvCBPxfJO2_Kfly1MTrw8qJVmOhrHRz3h92F0A9s8mHWBNpCZaH6pqYK6R5Ox57vHvMKttGUKWc1T0LpBV0QNlPHRc_5c6jWfdvz2-EhECB4-CqnEDsAoblLIw6zt5iplnvQuqaMMuCodkN1kXiLGLQrBHF_Ib8K_edVKroiJPJtnC7euhN_7unSszqhWQVKdHWilQk4PH1NJfX-mWGHPnFQ4lLa_KaC6Fh4VNyxLfGUQn91v70s6DgyGxjFS2YMA5mc4hc8DfH4KRScOvlts-zyRl61jfnlm3E5-lJjh6xXniW0qYkDIdcEmFoVe0oRrLpLZEFtniZo9LU0sq73932e-8K8zHeXhXA35bAA-o7_H4WCsoP9kLZA6E4_-og9krdpMsbgh_GJINHPNHkiw9ccXNkCc2wytgAgOShwix2VX3KY
> 
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: nginx/1.21.5
Server: nginx/1.21.5
< Date: Thu, 23 Feb 2023 21:42:52 GMT
Date: Thu, 23 Feb 2023 21:42:52 GMT
< Content-Type: image/jpeg
Content-Type: image/jpeg
< Content-Length: 463146
Content-Length: 463146
< Connection: keep-alive
Connection: keep-alive
< Cache-Control: private
Cache-Control: private
< Last-Modified: Mon, 20 Feb 2023 22:20:39 GMT
Last-Modified: Mon, 20 Feb 2023 22:20:39 GMT
< X-UA-Compatible: IE=edge
X-UA-Compatible: IE=edge
< Content-language: en
Content-language: en
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
X-Frame-Options: SAMEORIGIN
< Expires: Sun, 19 Nov 1978 05:00:00 GMT
Expires: Sun, 19 Nov 1978 05:00:00 GMT
< X-Generator: Drupal 9 (https://www.drupal.org)
X-Generator: Drupal 9 (https://www.drupal.org)
< Accept-Ranges: bytes
Accept-Ranges: bytes

< 
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* Closing connection 0
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Unknown (21):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS alert, close notify (256):

[mstenta]

@Symbioquine

It looks like it’s working via a bearer token (via curl) too;

Oh excellent! Thanks for confirming that works! I was worried for a minute there that we were discovering a new bug.

[Symbioquine]

@mstenta

Oh excellent! Thanks for confirming that works! I was worried for a minute there that we were discovering a new bug.

Could still be a bug, but at least not all bearer token file access is broken…

Worth trying via a non-root user and trying roughly the same access, but in the browser rather than via curl, next.

[mstenta]

Yea when @jgaehring first mentioned it I was reminded of this open PR (for POSTing files, not GETing them): Fix JSON:API permission issue so non-admin users can upload files by symbioquine · Pull Request #563 · farmOS/farmOS · GitHub

We should dust that one off…

[jgaehring]

@mstenta

@jgaehring It’s interesting that you said this works through node.js, but not through client-side requests. Are all the headers the same otherwise?

Sorry, it’s always harder than I think to get the actual request headers from axios/Node. Here are the request headers:

GET /system/files/farm/log/2023-02/00_TEST_PLUM_0.JPG HTTP/1.1
Accept: application/vnd.api+json
Content-Type: application/vnd.api+json
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImp0aSI6ImU2YTc5OTZmYjZlYjU0OGUxMTcyNWM1NWZkN2Q3NGEwZjEwM2E1YjdkODIxZDNiNjEzYzk2NGIzYjMwNjExYzQzNWU4ZmE3N2MyZGU4YTU5In0.eyJhdWQiOiJmaWVsZGtpdCIsImp0aSI6ImU2YTc5OTZmYjZlYjU0OGUxMTcyNWM1NWZkN2Q3NGEwZjEwM2E1YjdkODIxZDNiNjEzYzk2NGIzYjMwNjExYzQzNWU4ZmE3N2MyZGU4YTU5IiwiaWF0IjoxNjc3MTg4MTMzLCJuYmYiOjE2NzcxODgxMzMsImV4cCI6MTY3NzE5MTczMy40NTM4NzQsInN1YiI6IjEiLCJzY29wZSI6WyJhbm9ueW1vdXMiLCJhdXRoZW50aWNhdGVkIiwiZmFybV9tYW5hZ2VyIiwiZmFybV92aWV3ZXIiLCJmYXJtX3dvcmtlciJdfQ.ZfZJhgjuvwM3_AqohPF4gMuKtmeGoDEzo7--GCWhJD2NUXJ_1YHnOMfuS9nTeAQeIK189fdYUP6QzdEhOk2iTrN11OJz4Kj8UBdC_ThnpekeZfaAlGoOypBGHrjVVPVWu2cYK4iZ-MI9bIuf_3VyNxsL6aV1MrOQWCmQXuh34WYckTIb5Vy4idDbbjyM1e3Nbwo0hCH7zb-fl8oN77HFfTTffKgigGnmpkLadAY_E2y0eIMNcVO0POHyoao1666RT6AzokH0zK-sTrO4WBuGIVS0Zg34vfpaaqq3Sb9v9ZnMtrIc3y65lfgZ-qiDT0hemYPsh0nWIVlJPhTkYIgxpqVliElv5n-012I0KxmGVpL_YT2myS32L4rckUdAxgm2srmlmkJB-LZKHCBttR77xD4equGIuw1xcZeEcuUmTaiZuLGW74E66_DOPrhpmaAD5bhP5Dl-BULGygBjtGg0iesfEhlV8ffWW010-67ZMx3NBZimsw-KEZjRkVcOkiPvV2OYy7Wvk5sPogExEPkKEiWuvSr5zVdIc1H-HI7qP0FQQCmCa0US48VjxI-EoBcj_S1ivJsx8lxA2tF7bAxxcmpQSfgHpRFgMINr--XvnO5gQAbQvqMeoP8O_GDGR652Th2jrrlM2PpnuD7XDYNSABFedrUf0lzpR23kQs93Q6U
User-Agent: axios/0.25.0
Host: localhost
Connection: close

And the response headers (as JSON):

{
      date: 'Thu, 23 Feb 2023 21:35:36 GMT',
      server: 'Apache/2.4.54 (Debian)',
      'content-length': '76498',
      'cache-control': 'private',
      'last-modified': 'Thu, 16 Feb 2023 14:42:16 GMT',
      'x-ua-compatible': 'IE=edge',
      'content-language': 'en',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'SAMEORIGIN',
      expires: 'Sun, 19 Nov 1978 05:00:00 GMT',
      'x-generator': 'Drupal 9 (https://www.drupal.org)',
      'accept-ranges': 'bytes',
      connection: 'close',
      'content-type': 'image/jpeg'
    }

Would having a keep-alive connection make the difference?

[Symbioquine]

I got it working via a non-admin user in the browser too…

OAUTH2_ACCESS_TOKEN = (await fetch("https://farmos.test/oauth/token",
      { body: "grant_type=password&username=notadmin&password=test&client_id=farm&scope=farm_manager",
        credentials: 'omit',
        method: 'POST',
        headers: { 'Content-Type': 'application/x-www-form-urlencoded' } })
    .then(r => r.json())).access_token;
await fetch('https://farmos.test/system/files/farm/asset/2022-06/MF_698_tractor_24f07.JPG',
     { credentials: 'omit', headers: { 'Authorization': `Bearer ${OAUTH2_ACCESS_TOKEN}` } });

(pardon the weird manual code formatting)

[Symbioquine]

@Symbioquine

I got it working via a non-admin user in the browser too…

I think that means the main thing left to test (other than some sort of weird header issue) is CORS. My most recent tests in the browser were from the same domain.

[mstenta]

@jgaehring

Accept: application/vnd.api+json
Content-Type: application/vnd.api+json

I wonder if this is the problem? The files won’t be JSON. I think Drupal varies it’s routing logic based on these headers.

[Symbioquine]

@mstenta

I wonder if this is the problem? The files won’t be JSON. I think Drupal varies it’s routing logic based on these headers.

Adding those headers doesn’t break the request the way I’m sending it;

[jgaehring]

@mstenta

wonder if this is the problem? The files won’t be JSON.

That was my first thought, but that content type worked in Node, and yet, even when I corrected it to image/jpeg in the browser, it still didn’t work.

@Symbioquine

I think that means the main thing left to test (other than some sort of weird header issue) is CORS. My most recent tests in the browser were from the same domain.

Yea, I’m wondering if it could be the way I have my proxy set up.

[Symbioquine]

@Symbioquine

I think that means the main thing left to test (other than some sort of weird header issue) is CORS. My most recent tests in the browser were from the same domain.

Well, it’s not CORS (at least with the default Field Kit module and the https://farmOS.app domain… (request otherwise executed the same as in my last test via the browser console)

[Symbioquine]

@jgaehring

Yea, I’m wondering if it could be the way I have my proxy set up.

Yeah, maybe something specific to your development setup?

[jgaehring]

@Symbioquine

I got it working via a non-admin user in the browser too…

Interesting… It appears it’s only working for me with non-admin users (so far). My initial posts errors were using the admin user credentials to sign into Field Kit.

[Symbioquine]

Weird! I just did my same test from above from https://farmOS.app, but as my “root” (admin) user and it also worked.

[Symbioquine]

Does it work for you from the browser console if you copy/adapt my testing strategy?

OAUTH2_ACCESS_TOKEN = (await fetch("https://farmos.test/oauth/token",
      { body: "grant_type=password&username=notadmin&password=test&client_id=farm&scope=farm_manager",
        credentials: 'omit',
        method: 'POST',
        headers: { 'Content-Type': 'application/x-www-form-urlencoded' } })
    .then(r => r.json())).access_token;
await fetch('https://farmos.test/system/files/farm/asset/2022-06/MF_698_tractor_24f07.JPG',
     { credentials: 'omit', headers: { 'Authorization': `Bearer ${OAUTH2_ACCESS_TOKEN}` } });