| 16 Sep 2024 |
 Sam Bull | In your examples, you are getting duplicate values right? i.e. The actual numbers are the same. | 21:51:23 |
 Om Thorat | In reply to @sam:sambull.org
In your examples, you are getting duplicate values right? i.e. The actual numbers are the same. In my example i get two headers with the same value and the total content length actually is the total of that. | 21:52:07 |
| Sam Bull | Sorry, you get 2 headers with something like '42', and the message length is actually 84? | 21:53:06 |
| Om Thorat | ie Content-Length: 1570\r\nContent-Length: 1570
request turns it into Content-Length: 1570,1570
meanwhile curl turns it into Content-Length: 3654 | 21:53:22 |
| Om Thorat | In reply to @sam:sambull.org
Sorry, you get 2 headers with something like '42', and the message length is actually 84? not exactly total that's a misleading term my bad | 21:53:57 |
| Sam Bull | The spec you just referenced said to treat it as one of the numbers, i.e. the length should be 1570. If the message is not actually 1570, we wouldn't be able to parse it.. | 21:55:06 |
 @webknjaz 🇺🇦 #StandWithUkraine | https://stand-with-ukraine.pp.ua | #russiaIsANaziState | In reply to @omthorat:matrix.org
I don't think it's useful per sey but for my usecase that is vulnerability scanning there are some misbehaving apps that'll use that and I am unable to process my request with aiohttp because the html gets declared invalid. Other tools and frameworks such as curl and requests seem to parse it. I understand that isn't a reason for aiohttp to allow it too, I am asking if it'll possible to do so with the projects current opinions.
a recipient of a Content-Length header field value consisting of the same decimal value repeated as a comma-separated list (e.g, "Content-Length: 42, 42") MAY either reject the message as invalid or replace that invalid field value with a single instance of the decimal value, since this likely indicates that a duplicate was generated or combined by an upstream message processor.
src - https://httpwg.org/specs/rfc9110.html#field.content-length
Section 8.6 last paragraph
FWIW curl does not validate a lot of things, letting you send slightly invalid payload. But I don't think invalid queries are within our scope. | 21:56:17 |
| Om Thorat | In reply to @sam:sambull.org
The spec you just referenced said to treat it as one of the numbers, i.e. the length should be 1570. If the message is not actually 1570, we wouldn't be able to parse it.. Let me confirm the total content length | 21:56:40 |
| Sam Bull | The RFC also doesn't allow duplicate headers, only a comma-separated list in a header:
https://www.rfc-editor.org/rfc/rfc9110#section-8.6-13 | 21:57:59 |
| @webknjaz 🇺🇦 #StandWithUkraine | https://stand-with-ukraine.pp.ua | #russiaIsANaziState | In reply to @sam:sambull.org
Generally more important on the server-side though. So, if you're talking about client-side, then it could be considered as acceptable in lax mode (the default). But, we generally want the parsers to behave the same, so ideally we'd want llhttp to support this too. Technically, the client side may be vulnerable too when the input is coming from untrusted sources.. | 21:58:13 |
| Om Thorat | In reply to @webknjaz:matrix.org
FWIW curl does not validate a lot of things, letting you send slightly invalid payload. But I don't think invalid queries are within our scope. Hmm well I understand this errors on the homepage it's a fault of the server not because of a certain query. | 21:59:28 |
| Om Thorat | In reply to @webknjaz:matrix.org
FWIW curl does not validate a lot of things, letting you send slightly invalid payload. But I don't think invalid queries are within our scope. * Hmm well I understand though this errors on the homepage it's a fault of the server not because of a certain query. | 21:59:34 |
| Sam Bull | In reply to @webknjaz:matrix.org
Technically, the client side may be vulnerable too when the input is coming from untrusted sources.. Client side generally only matters when the server-side application is proxying requests or similar. This is why we have the lax mode which allows several common mistakes, even though they can present request smuggling issues. | 22:00:24 |
| @webknjaz 🇺🇦 #StandWithUkraine | https://stand-with-ukraine.pp.ua | #russiaIsANaziState | In reply to @omthorat:matrix.org
Hmm well I understand though this errors on the homepage it's a fault of the server not because of a certain query. But you said you're sending a broken request, right? Or is it the response that your client is getting that's invalid? | 22:00:35 |
| Om Thorat | In reply to @sam:sambull.org
The RFC also doesn't allow duplicate headers, only a comma-separated list in a header:
https://www.rfc-editor.org/rfc/rfc9110#section-8.6-13 I interpreted the last line to mean a duplicate header might also be generated, Is that not it? | 22:00:41 |
| Sam Bull |
a recipient of a Content-Length header field value consisting of the same decimal value repeated as a comma-separated list (e.g, "Content-Length: 42, 42") MAY either reject the message as invalid or replace that invalid field value with a single instance of the decimal value, since this likely indicates that a duplicate was generated or combined by an upstream message processor.
| 22:02:17 |
| Om Thorat | In reply to @webknjaz:matrix.org
But you said you're sending a broken request, right? Or is it the response that your client is getting that's invalid? sending a broken request is my goal, I face the same problem even on just hitting '/' it's an invalid response thay my client is getting | 22:02:30 |
| Sam Bull | It's pretty explicit for some reason, even if I don't see a risk with duplicate headers. If llhttp are happy to allow it (or already do), then I don't mind allowing it on the Python side too. But, first, you need to confirm the content length is the length of the number, and not the total of the numbers added together. | 22:05:52 |
| Om Thorat | It's certainly not added i see that now probably should discard one and just consider one. | 22:07:04 |
| Om Thorat | So, I should go consult with the llhttp people? | 22:07:26 |
| Sam Bull | I would note that vulnerability scanning should probably be done with a lower-level tool, as you'll probably have more issues in future. | 22:08:07 |
| Sam Bull | But, if you want to try building llhttp locally, you can test it out. | 22:08:24 |
| Om Thorat | In reply to @sam:sambull.org
I would note that vulnerability scanning should probably be done with a lower-level tool, as you'll probably have more issues in future. Understood thanks a lot 😄 will communicate that | 22:08:43 |
| Om Thorat | Seems like llhttp flags it as well currently. i'll ask forwards thanks a lot : ) | 22:10:06 |
| Sam Bull | Instructions are at: https://github.com/aio-libs/aiohttp/tree/master/vendor
We enable the lenient flags here (this is what you'll need to try changing): https://github.com/aio-libs/aiohttp/blob/fa98921e04a0c108af77af275dcbe1cc54311165/aiohttp/_http_parser.pyx#L658-L661
This flag may do what you want (if not, then you'll probably need to open an issue with llhttp): https://github.com/nodejs/llhttp/?tab=readme-ov-file#void-llhttp_set_lenient_chunked_lengthllhttp_t-parser-int-enabled | 22:11:20 |
| Om Thorat | Thank you : ) | 22:13:43 |
| 17 Sep 2024 |
|  lassulus changed their profile picture. | 14:38:27 |
|  Parth Shah joined the room. | 23:57:31 |
| 18 Sep 2024 |
| Parth Shah | I have requirement where based on incoming request, triggers the fire and forget background task. I have tried using aiojobs spawn however seeing below issues.
I) without await not able to spawn the background task
ii) if I provide await spawn(background_task()) then its starting the task background successfully however still this task is going on, next request is getting blocked.
What am I missing? | 00:01:44 |
| Sam Bull | No idea without some actual code. spawn() only blocks if the queue is full (i.e. you have too many jobs already running) | 12:54:15 |
