| 18 Feb 2020 |
| 18:07:43 |  Github [@travis:t2l.io] | [matrix-org/synapse] clokep synchronize pull request #6941: Do not send alias events when creating / upgrading a room [open] - https://github.com/matrix-org/synapse/pull/6941 |
| 17:36:29 |  Erik | (I think where we're falling over here is that we have rooms that we join large rooms and get 10000s of new events at once, causing us to trip over the limit) |
| 17:36:55 | Erik | another option is to add a timestamp and say we're only allowed to e.g. get 15mins behind |
| 18:07:56 |  TravisR | In reply to @erikj:jki.re
another option is to add a timestamp and say we're only allowed to e.g. get 15mins behind This would be better, as shoving tens of thousands of events down a queue isn't an error until the thing can't keep up. |
| 18:08:13 | TravisR | Just means your queue size is massive for a period of time |
| 18:33:29 | Github [@travis:t2l.io] | [matrix-org/synapse] clokep closed issue #6942: SyTest test "Device deletion propagates over federation" is flaky [closed] - https://github.com/matrix-org/synapse/issues/6942 |
| 18:08:25 | Erik | yeah |
| 18:33:32 | Github [@travis:t2l.io] | [matrix-org/synapse] clokep reopened issue #6942: SyTest test "Device deletion propagates over federation" is flaky [open] - https://github.com/matrix-org/synapse/issues/6942 |
| 18:08:47 | Erik | the idea that queue size ~ how fare behind you are isn't really true for us |
| 21:10:34 |  richvdh | I think a timelimit would be an improvement and less prone to false positives, but still, unless there's something that the process can do to try and fix itself, I'm not sure what the point of disconnecting the worker is |
| 21:11:05 | richvdh | ultimately it just becomes a way of flagging that something is wrong, and kicking off the worker isn't something that gets noticed unless you have monitoring on reconnection rates |
| 21:12:07 | richvdh | so you might as well alert directly on the replication delay rather than using "how many disconnections" as an obscure proxy for it |
| 21:12:50 | richvdh | or maybe better: monitor on "size of transmission buffer for the worker stream" |
| 21:15:09 | TravisR | yea, a metric for how far behind (timewise) it is would be better |
| 21:15:29 | TravisR | so that admins can decide if they want to fix it after 30 seconds or 30 days. |
| 21:16:47 | richvdh | I think the reason it all falls apart is that if the master ends up buffering up an hour's worth of replication updates for each worker, that is huge amounts of memory usage with few clues as to what is using the memory |
| 21:19:45 | TravisR | more replication metrics would help with that though, stuff like also recording the number of backlogged events/updates and which workers are falling behind. |
| 21:20:24 | TravisR | High memory usage isn't really a problem until you run out, in which case the planet has been restarted and the backlog goes away |
| 22:39:49 | Github [@travis:t2l.io] | [matrix-org/synapse] richvdh opened pull request #6948: Move MSC2432 stuff onto unstable prefix [open] - https://github.com/matrix-org/synapse/pull/6948 |
| 22:40:16 | Github [@travis:t2l.io] | [matrix-org/synapse] richvdh edited pull request #6948: Move MSC2432 stuff onto unstable prefix [open] - https://github.com/matrix-org/synapse/pull/6948 |
| 22:40:25 | Github [@travis:t2l.io] | [matrix-org/synapse] richvdh review_requested pull request #6948: Move MSC2432 stuff onto unstable prefix [open] - https://github.com/matrix-org/synapse/pull/6948 |
| 22:45:51 | Github [@travis:t2l.io] | [matrix-org/synapse] richvdh opened pull request #6949: Make room alias lists peekable [open] - https://github.com/matrix-org/synapse/pull/6949 |
| 22:52:33 | Github [@travis:t2l.io] | [matrix-org/synapse] richvdh synchronize pull request #6949: Make room alias lists peekable [open] - https://github.com/matrix-org/synapse/pull/6949 |
| 22:53:55 | Github [@travis:t2l.io] | [matrix-org/synapse] richvdh edited pull request #6949: Make room alias lists peekable [open] - https://github.com/matrix-org/synapse/pull/6949 |
| 23:16:59 | Github [@travis:t2l.io] | [matrix-org/synapse] richvdh synchronize pull request #6949: Make room alias lists peekable [open] - https://github.com/matrix-org/synapse/pull/6949 |
| 23:19:41 | Github [@travis:t2l.io] | [matrix-org/synapse] richvdh edited pull request #6949: Make room alias lists peekable [open] - https://github.com/matrix-org/synapse/pull/6949 |
| 23:19:52 | Github [@travis:t2l.io] | [matrix-org/synapse] richvdh review_requested pull request #6949: Make room alias lists peekable [open] - https://github.com/matrix-org/synapse/pull/6949 |
| 23:21:56 | Github [@travis:t2l.io] | [matrix-org/synapse] richvdh synchronize pull request #6949: Make room alias lists peekable [open] - https://github.com/matrix-org/synapse/pull/6949 |
| 23:26:56 | Github [@travis:t2l.io] | [matrix-org/synapse] richvdh opened pull request #6950: Tiny optimisation for _get_handler_for_request [open] - https://github.com/matrix-org/synapse/pull/6950 |
| 23:27:06 | Github [@travis:t2l.io] | [matrix-org/synapse] richvdh review_requested pull request #6950: Tiny optimisation for _get_handler_for_request [open] - https://github.com/matrix-org/synapse/pull/6950 |
