30 Jun 2023 |
dch [he/them] | - full config https://termbin.com/73uc
- GELF output via tcpdump
``` T 100.64.0.5:34626 -> 100.64.0.5:1514 [AP] {"version":"1.1","timestamp":1688112721,"short_message":"2023-06-30T08:12:01.712539+00:00 indie postgres 52235 - - [394-1] 2023-06-30 0 8:12:01.712 UTC [52235] LOG: checkpoint starting: time","level":6,"host":"f01","_program":"1","_facility":"local0"}. ``` this one's from postgres, see "_program":"1" there? | 08:20:14 |
dch [he/them] | I fixed haproxy by moving from RFC5424 format to the older one. bird daemon is fine. | 08:20:44 |
dch [he/them] | do I enable trace via syslog-ng-ctl debug --set=on ? just never done this before | 08:22:59 |
bazsi77 | I'd need the source side, that syslog-ng receives. On the debug level you should see "Incoming message" that's the input we are parsing. | 08:22:59 |
bazsi77 | Newer syslog-ng's have an easier to use log-level command in syslog-ng-ctl which is easier to use. | 08:23:44 |
bazsi77 | But yeah, that should do too | 08:23:52 |
dch [he/them] | this is 4.2. I can't find the online docs btw for this, and the manpage doesn't mention log-level anywhere. | 08:26:10 |
bazsi77 | It's coming. One Identity published the docs under a more permissive license so it's only a matter of weeks now. Still need to make sure that we remove branding. | 08:27:46 |
dch [he/them] | good to know :-) thank-you! | 08:28:39 |
dch [he/them] | OK looking (again) at tcpdump I see this | 08:48:10 |
dch [he/them] | UTC [21688] LOG: PID 34364 in cancel request did not match any process","level\
":6,"host":"f01","_program":"1","_facility":"local0"}'","level":7,"host":"f01","_program":"syslog-ng","_pid":94430,"_facili
ty":"syslog"}.{"version":"1.1","timestamp":1688114805,"short_message":"2023-06-30T08:46:45.195862+00:00 indie postgres 39657 - - [7-1]
2023-06-30 08:46:45.195 UTC [39657] | 08:49:37 |
dch [he/them] | so this makes more sense now | 08:49:42 |
dch [he/them] | haproxy was running in the host OS, not a container, it just needed a format nudge | 08:50:00 |
dch [he/them] | postgres is running in a jail | 08:50:09 |
dch [he/them] | but syslog-ng should still have no trouble fetching the process name, its on the host OS | 08:50:36 |
dch [he/them] | or maybe I'm misreading that. So, re-running, with syslog-ng in foreground | 08:58:25 |
dch [he/them] | [2023-06-30T08:57:45.693028] Incoming log entry; input='<134>1 2023-06-30T08:57:45.692964+00:00 indie postgres 58111 - - [3-1] 2023-06-30 08:57:45.692 UTC [58111] LOG: listening on IPv6 address "::1", port 5432', msg='0x83fcbef00', rcptid='0' | 08:58:42 |
dch [he/them] | [2023-06-30T08:57:45.693128] Outgoing message; message='{"version":"1.1","timestamp":1688115465,"short_message":"2023-06-30T08:57:45.692964+00:00 indie postgres 58111 - - [3-1] 2023-06-30 08:57:45.692 UTC [58111] LOG: listening on IPv6 address "::1", port 5432","level":6,"host":"f01","_program":"1","_facility":"local0"}' | 08:59:07 |
dch [he/them] | ok, that's useful. | 08:59:31 |
bazsi77 | This probably means that your input does not parse rfc5424 style | 09:02:37 |
bazsi77 | Add a flags(syslog-format) | 09:02:53 |
dch [he/them] | its the "standard" FreeBSD syslog /var/run/log here | 09:03:29 |
dch [he/them] | I will check to see if the other app in the jail is also sending the same unexpected format | 09:04:23 |
dch [he/them] | is there any way to confirm if its parsing correctly? | 09:04:36 |
bazsi77 | Syslog is tricky to parse so by default syslog-ng tries to make the most of the message, instead of failing | 09:05:10 |
bazsi77 | The program being set to 1 actually parses the rfc5424 version string into PROGRAM | 09:05:54 |
dch [he/them] | bazsi77, perfect thanks! I add flags(syslog-protocol) and we're all happy | 09:18:52 |
dch [he/them] | I should see why we are not sending RFC5424 from postgres already though | 09:19:07 |
dch [he/them] | looks like postgres doesn't provide an alternative https://support.oneidentity.com/syslog-ng-premium-edition/kb/4273418/how-to-configure-bsd-syslog-and-ietf-syslog-message-formats-in-syslog-ng | 09:22:56 |
4 Jul 2023 |
| lazarimre78 joined the room. | 07:17:05 |