| 6 Oct 2022 |
 bblack | we have a lot of tooling vs problem-space mismatches :/ | 16:54:01 |
| bblack | (tooling at the lowest level I mean, e.g. vsthrottle as a universal solution) | 16:54:37 |
 cdanis | bblack: yeah, that's one of the reason why I was proposing haproxy, it's quite a bit more flexible in what it can track | 16:56:40 |
| bblack | yeah, it makes sense in general for that reason, and it's the frontmost layer in terms of defending the rest of the stack | 16:57:18 |
| cdanis | re: the research _joe_ mentioned, research is a bit generous but there is some anecdata comparison at https://phabricator.wikimedia.org/F35546836 | 16:57:23 |
| cdanis | https://gerrit.wikimedia.org/r/c/operations/puppet/+/768723/31#message-e727b6f1ba658c0d9623084bda2ddf18ba21b02f | 16:58:40 |
| cdanis | very long morning and I need to step away for a bit, back in about an hour | 16:59:07 |
| bblack | back on the kernel stuff, in modules/cacheproxy/manifests/performance.pp we currently define (for the cp nodes) | 17:04:00 |
| bblack | qdisc => 'fq flow_limit 300 buckets 8192 maxrate 256mbit', | 17:04:05 |
| bblack | which the interface-rps script applies at the per-queue level for our multiq cards, so for example cp1075 in "tc qdisc" will show 12 queues each running that set of fq params. | 17:04:42 |
| bblack | very little has been done in this area, those parameter are very rough guesses | 17:05:03 |
| bblack | there's other qdisc we could use, other fairness policies, the bucketing could be wildly wrong, there's no overall shape limit, etc | 17:05:29 |
| bblack | this is basically attacking one edge of the problem, from down there. how do you ensure all the various tcp flows coming out of this machines are fair to each other and the network, basically (which more targets the "few IPs downloading lots of stuff" problem than the hotlinking/mobileapp sort of thing) | 17:07:44 |
| bblack | we could even shape the whole 10G card down to a reasonable traffic level | 17:08:50 |
| bblack | (given the total outbound capacity of all the cp nodes at a site is larger than the total possible transit+peering out) | 17:09:16 |
| bblack | by doing that you're still "inducing failure" by dropping packets, but by doing it proactively with an aim towards per-flow fairness, more of the good stuff is getting through and less of the bad stuff. | 17:10:27 |
| bblack | (unfortunately, tc can't see L7, so we can't fix a widely-hotlinked/embedded image this way :/) | 17:12:14 |
 topranks | bblack: late to this, but would be interested to discuss. We are looking to introduce some QoS/categorisation on the network level. So for instance instead of dropping traffic below the 10G, it could instead be marked on the host as drop eligible, and the network could be left to decide if it ultimately needs to be dropped upstream. Lots of moving parts here, and none of it is a cure for insufficient bandwidth. But be interested to get | 20:35:43 |
| topranks | your thoughts. | 20:35:46 |
| cdanis | topranks: that does give us some more options, cool! | 20:41:34 |
| cdanis | at some point I also want to pick yours/ XioNoX‘s brains about egress traffic management options too (mostly out of curiosity, since it seems hard to get right at our scale with anything floss) | 20:43:02 |
| topranks | cdanis: yep, absolutely. | 20:44:01 |
| topranks | bu egress traffic management you mean right out at the edge towards the internet? | 20:44:13 |
| topranks | s/bu/by | 20:44:21 |
| topranks | this stuff isn't easy, but I think we could pull of a better integration of host-level and network-level control than many manage to do | 20:45:39 |
| cdanis | topranks: yeah, either by knowing to limit traffic at the host level, or, by knowing to send one AS’s egress split between peering and transit | 20:54:27 |
| topranks | The last one is the real trick. Some of the SDN / SD-WAN stuff is built to tackle that, and also feed back performance info to the path selection. Very difficult to achieve at our scale, and for so many different destinations though. | 20:59:24 |
| topranks | On a manual level, for AS X, you can do BGP things to make it all seem equal and do ECMP. But weighted between one and the other, and adapting dynamically, is where it really gets hard. | 21:00:39 |
|  @Jeff_Green:libera.chat left the room. | 21:45:13 |
| cdanis | yeah, it seems quite hard to do at all, and I’m sure I don’t even realize the half of it :) | 22:32:45 |
