| 19 Sep 2024 |
 Olivier 'reivilibre' | is it considered acceptable that an old server that doesn't know about the key yet, would be able to join the room, but a newer one cannot? | 09:33:41 |
| Olivier 'reivilibre' | I oscillate in what I think on that | 09:33:49 |
| Olivier 'reivilibre' | but at least least, it seems like it would be a minimal effort thing to include a JSON schema of known allowed events in each room version for example (though I guess that wouldn't help how servers have to fall back to ugly handling for older room verisons) | 09:34:33 |
 neilalexander | In reply to @reivilibre.element:librepush.net
is it considered acceptable that an old server that doesn't know about the key yet, would be able to join the room, but a newer one cannot? I'm not really sure this matters, a past server might not have known about that key but that's unavoidable, the fact is that servers today can know about it because it's in the spec | 09:35:41 |
 Cat | Isnt room versions literally how we are supposed to break validation compat? | 09:35:54 |
| Olivier 'reivilibre' | alternatively, the spec should define semantics in 'layers' — each layer is a JSONSchema that applies to portion of an event, e.g. there would be a 'm.room.create predecessor layer', which defines exactly how to validate the predecessor portion of the event, and every server agrees to only apply the semantic of a room predecessor if that layer validates | 09:36:03 |
| Olivier 'reivilibre' | currently you just wind up with one table of what fields and expected types you have on an event, but it's not crystal clear what semantics apply if some of those are valid and others are not | 09:37:41 |
 NotHSDevNico | At the very least the client api should validate that known keys of known events have the right type. | 09:38:05 |
| Olivier 'reivilibre' | yes, 100% on this, it would be very reasonable to do some JSONSchema validation on new events originating at the server | 09:38:43 |
 Kegan | I agree, which effectively means once a key name's type is decided it cannot change. | 09:39:31 |
| NotHSDevNico | And if people turn off that validation, it is fair if servers refuse to join those rooms, because nobody needs to respect tulir's experiments! :p | 09:39:46 |
| neilalexander | In reply to @reivilibre.element:librepush.net
alternatively, the spec should define semantics in 'layers' — each layer is a JSONSchema that applies to portion of an event, e.g. there would be a 'm.room.create predecessor layer', which defines exactly how to validate the predecessor portion of the event, and every server agrees to only apply the semantic of a room predecessor if that layer validates Layering like this is also a bit slippery. Take for example the difference in how Synapse decodes JSON (decoding into what is effectively a hashmap and tolerating whatever) vs how Dendrite does (decoding into a typed structure and erroring on type mismatches). Unless we're prescribing that JSON has to be decoded in multiple stages for specific event types, which might not be very feasible in some languages, we're not really getting to the root of the issue. The spec has ultimately told us that certain keys with certain names should be certain types, the bottom line is that is the rule everyone should be playing by | 09:44:06 |
| neilalexander | Every time we grandfather these bugs into the spec, or we add in subtle behaviours to work around them, we make the implementation of the next Matrix homeserver considerably harder, and we introduce more potential for interoperability issues | 09:45:03 |
| Olivier 'reivilibre' | In reply to @neilalexander:dendrite.matrix.org
Layering like this is also a bit slippery. Take for example the difference in how Synapse decodes JSON (decoding into what is effectively a hashmap and tolerating whatever) vs how Dendrite does (decoding into a typed structure and erroring on type mismatches). Unless we're prescribing that JSON has to be decoded in multiple stages for specific event types, which might not be very feasible in some languages, we're not really getting to the root of the issue. The spec has ultimately told us that certain keys with certain names should be certain types, the bottom line is that is the rule everyone should be playing by TBH I don't think decoding in multiple steps/layers is necessarily the worst idea. We already have to be able to 'decode JSON whilst ignoring unknown keys', you'd conceptually just sort of do that several times.
I agree that enforcing these things harder is the right way to do it, but you're still left with the question of how you apply semantics to mishmashes from before that validation was possible (and this sort of stuff also applies client-side in encrypted payloads I'd say) | 09:46:06 |
| Cat | In reply to @deepbluev7:neko.dev
And if people turn off that validation, it is fair if servers refuse to join those rooms, because nobody needs to respect tulir's experiments! :p Doesnt Tulir only turn of CS API level validation? | 09:46:15 |
 tulir | In reply to @cat:feline.support
Doesnt Tulir only turn of CS API level validation? yes, I follow the mandatory parts of the spec, I only skip the unnecessary checks | 09:47:06 |
| NotHSDevNico | Read my previous message and then explain me, where you think that message contradicts the one I sent before it? | 09:47:49 |
| Cat | Essentially Tulir is not supposed to break your stuff. | 09:49:16 |
| Cat | if we are doing validation properly | 09:49:22 |
| Cat | Because the only validation that matters at the end of the day is the validation that is done at the federation level. | 09:49:54 |
| NotHSDevNico | Nope, I am free to ignore tulir's events as not useful and possibly malicious. And if tulir messes with the create event, I will just say the room is not joinable | 09:50:13 |
| NotHSDevNico | I don't have to accept people messing with events for fun (as long as it doesn't cause a room split and not joining a room is not a room split) | 09:51:00 |
| NotHSDevNico | I spent enough time dealing with sloppy events, it makes my hair go grey and I hate it | 09:52:12 |
| neilalexander | In reply to @reivilibre.element:librepush.net
TBH I don't think decoding in multiple steps/layers is necessarily the worst idea. We already have to be able to 'decode JSON whilst ignoring unknown keys', you'd conceptually just sort of do that several times.
I agree that enforcing these things harder is the right way to do it, but you're still left with the question of how you apply semantics to mishmashes from before that validation was possible (and this sort of stuff also applies client-side in encrypted payloads I'd say) People are being subtly affected by these mismatches anyway, so they're already "broken" in the sense that all guarantees went out the window the moment they interacted with someone on a different implementation. If it works without validation then it's basically lucky, but all we've done is create an environment where people expect to be able to break rooms and have them still work | 09:57:04 |
| Olivier 'reivilibre' | mrmm, I'm thinking more in the general sense here, for future extensions | 09:57:38 |
| Olivier 'reivilibre' | and just that in general the spec is pretty underspecified at how you deal with partial invalidness | 09:58:27 |
| neilalexander | Well, a future extension is not an issue so long as the type of an existing field never changes without it being a known change in a known room version. We can decorate events with all of the new fields that we like and that's not an issue | 09:59:01 |
| Olivier 'reivilibre' | just scratching my head on whether it's really fine for 'old innocent HSes' to accept something and then new ones not | 10:00:14 |
| NotHSDevNico | In reply to @reivilibre.element:librepush.net
just scratching my head on whether it's really fine for 'old innocent HSes' to accept something and then new ones not You just bump the schema to validate in a new room version and if an old HS accepts it, it is a bug | 10:00:42 |
| Olivier 'reivilibre' | well yes, schemas in room versions would be great, but assuming you still have to deal with old versions somehow, ... what are you left with? | 10:01:09 |
