| 3 Apr 2024 |
 heaven_devil | In reply to @effendy:matrix.org
So only from one router to another? I have trunk between firewall and switch in each site | 17:47:14 |
| heaven_devil | In reply to @effendy:matrix.org
Meaning only between the VPN endpoints? For example I am login in to A site firewall and execute traceroute to B site firewall's IPsec tunnel address.
I will see it went through A site switch and B site switch. | 17:48:17 |
 David | How are routes configured on the firewall? | 20:49:23 |
| 4 Apr 2024 |
|  colgano joined the room. | 00:09:37 |
| 5 Apr 2024 |
|  WirelesslyWired changed their profile picture. | 14:28:11 |
|  cluethulhu changed their profile picture. | 22:02:46 |
| 6 Apr 2024 |
|  bbthegeek set a profile picture. | 00:03:42 |
|  Nick Bouwhuis joined the room. | 09:33:14 |
|  Melvin Douglas joined the room. | 15:31:00 |
| 7 Apr 2024 |
|  n joined the room. | 18:08:50 |
| 8 Apr 2024 |
 effendy | Hello. I'm trying to understand why I cannot access my Proxmox interface from an LXC container when the Proxmox firewall is on. I can't for the life of me understand where it's actually dropping the packet. For me, this doesn't make any sense.
Proxmox public_ip (vmbr0) -> 100.100.100.1 (made up)
Proxmox private ip (vmbr1) -> 10.99.99.01
LXC-Container ip (vmbr1) -> 10.99.0.10
INPUT chain references PVEFW-INPUT which references PVEFW-HOST-IN.
I'm trying not to spam a lot, but this is what the PVEFW-HOST-IN chain looks like:
Chain PVEFW-HOST-IN (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 ACCEPT 0 -- lo * 0.0.0.0/0 0.0.0.0/0
2 0 0 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
3 2140 715K ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
4 99 5401 PVEFW-smurfs 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID,NEW
5 0 0 RETURN 2 -- * * 0.0.0.0/0 0.0.0.0/0
6 87 4681 GROUP-node-mgmt-IN 0 -- vmbr0 * 0.0.0.0/0 0.0.0.0/0
7 39 2496 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x80000000/0x80000000
8 0 0 RETURN 6 -- * * 0.0.0.0/0 0.0.0.0/0 match-set PVEFW-0-management-v4 src tcp dpt:8006
9 0 0 RETURN 6 -- * * 0.0.0.0/0 0.0.0.0/0 match-set PVEFW-0-management-v4 src tcp dpts:5900:5999
10 0 0 RETURN 6 -- * * 0.0.0.0/0 0.0.0.0/0 match-set PVEFW-0-management-v4 src tcp dpt:3128
11 0 0 RETURN 6 -- * * 0.0.0.0/0 0.0.0.0/0 match-set PVEFW-0-management-v4 src tcp dpt:22
12 0 0 RETURN 6 -- * * 0.0.0.0/0 0.0.0.0/0 match-set PVEFW-0-management-v4 src tcp dpts:60000:60050
13 60 2905 PVEFW-Drop 0 -- * * 0.0.0.0/0 0.0.0.0/0
14 60 2905 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0
15 0 0 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:L8IWPwzv6zwDkf/N0Ra//G5+sPU */
I've added some tracing to understand what's going on, so it all seems to be ending here (I'm adding just the few last lines):
Apr 08 17:13:05 amoklaufen kernel: TRACE: filter:PVEFW-Drop:rule:12 IN=vmbr1 OUT= PHYSIN=veth2000i0 MAC=fe:23:f3:fa:b0:89:bc:24:11:b3:14:a8:08:00 SRC=10.99.0.10 DST=100.100.100.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43655 DF PROTO=TCP SPT=43010 DPT=8006 SEQ=498821507 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A2DD691970000000001030307)
Apr 08 17:13:05 amoklaufen kernel: TRACE: filter:PVEFW-Drop:return:13 IN=vmbr1 OUT= PHYSIN=veth2000i0 MAC=fe:23:f3:fa:b0:89:bc:24:11:b3:14:a8:08:00 SRC=10.99.0.10 DST=100.100.100.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43655 DF PROTO=TCP SPT=43010 DPT=8006 SEQ=498821507 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A2DD691970000000001030307)
Apr 08 17:13:05 amoklaufen kernel: TRACE: filter:PVEFW-HOST-IN:return:14 IN=vmbr1 OUT= PHYSIN=veth2000i0 MAC=fe:23:f3:fa:b0:89:bc:24:11:b3:14:a8:08:00 SRC=10.99.0.10 DST=100.100.100.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43655 DF PROTO=TCP SPT=43010 DPT=8006 SEQ=498821507 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A2DD691970000000001030307)
One of things that I don't understand is what 14 refers to in this context "PVEFW-HOST-IN:return:14".
I often saw RETURN with the number of RULES + 1 (so the packet would eventually return after all the rules in the chain are tested for)
But here (in PVEFW-HOST-IN) there are 15 rules, as you can see. So I'm kind of misled, by that.
I will happily share more, if there's interest in solving this issue, but I also didn't want to spam and overwhelm people :)
| 14:23:52 |
 iam_tj | effendy: is there any NAT going on or is it 'just' routing rules? if only routing, is there any policy routing involved? | 14:34:09 |
| effendy | it might be related to this rule in PVEFW-HOST-IN:
num pkts bytes target prot opt in out source destination
6 185 9915 GROUP-node-mgmt-IN 0 -- vmbr0 * 0.0.0.0/0 0.0.0.0/0
This isn't matched, and the reason is that vmbr0 is set here as a inbound interface, whereas the packet uses vmbr1 as an inbound interface. Which kind of confuses me, because I expected the packet to go out of vmbr1 interface and into vmbr0 (given the destination IP of 100.100.100.1, set on vmbr0)
| 14:35:20 |
| effendy | Yeah, there's a SNAT for the container. But just to be clear again, if I disable the proxmox firewall, it works (disabling the firewall doesn't disable the SNAT). | 14:36:11 |
| effendy | Chain POSTROUTING (policy ACCEPT 9636 packets, 658K bytes)
pkts bytes target prot opt in out source destination
33 2235 MASQUERADE 0 -- * vmbr0 10.99.0.0/24 0.0.0.0/0
| 14:36:27 |
| iam_tj | the path is directly in on vmbr1, via the local table since the interfaces are both on the host so yes, you'd need an IN vmbr1 for that | 14:38:22 |
| effendy | Ok, so adding this rule in PVEFW-HOST-IN seems to be make it work:
iptables -I PVEFW-HOST-IN 6 -p tcp -m tcp --dport 8006 -i vmbr1 -j ACCEPT
I'm now trying to understanding exactly how this is happening, given that I'm accessing the interface on the public IP (assigned to vmbr0).
| 14:38:26 |
| effendy | Oh, I see. | 14:38:41 |
| iam_tj | if the destination is on the host there is no out and in again 🙂 | 14:39:07 |
| effendy | Shit, I need to ingest this :) | 14:39:12 |
| iam_tj | It's quite easy if you do not forget to consider policy routing. Take a look at ip route show and note the local table has highest priority | 14:40:19 |
| effendy | In reply to @iam_tj:matrix.org
if the destination is on the host there is no out and in again 🙂 I'm not sure what you mean by that. There's clearly an IN, but then... you mean to say, there's no OUT and IN and so on like there would be between two different hosts communicating over the network. | 14:40:21 |
| iam_tj | then look at ip route show table local | 14:40:33 |
| iam_tj | * It's quite easy if you do not forget to consider policy routing. Take a look at ip rule show and note the local table has highest priority | 14:41:04 |
| iam_tj | Edited the first command! s/route/rule/ | 14:41:15 |
| effendy | I'm not sure how ip route show table local enlightens me :) | 14:41:33 |
| effendy | yes, I see now the output of ip rule show. | 14:41:59 |
| effendy | I remember tinkering with it when I tried using multiple routing tables. | 14:42:10 |
| iam_tj | If both interfaces are on the current host, and same network namespace, and therefore traversing the same network stack, then the packet arrives directly via the interface it is sent on... it does NOT do vmbr1 --> out ---> in --> vmbr0 --> local | 14:42:57 |
| effendy | Yes. This is what I initially thought it would do. | 14:43:46 |
