Rachit Chauhan | Hi all, I am installing knative-serving in namespace other than knative-serving and have enabled istio-injection (for that NS too). All other control plane components seems to be working fine except activator. Seeing this in activator’s logs:
{"severity":"WARNING","timestamp":"2022-05-31T08:49:23.436527148Z","logger":"activator","caller":"handler/healthz_handler.go:36","message":"Healthcheck failed: received SIGTERM from kubelet","commit":"6ec4509","knative.dev/controller":"activator","knative.dev/pod":"activator-848f9bfddf-txhbz"}
{"severity":"ERROR","timestamp":"2022-05-31T08:49:25.693721434Z","logger":"activator","caller":"websocket/connection.go:142","message":"Websocket connection could not be established","commit":"6ec4509","knative.dev/controller":"activator","knative.dev/pod":"activator-848f9bfddf-txhbz","error":"websocket: bad handshake","request":"HTTP/1.1 503 Service Unavailable\r\nConnection: close\r\nContent-Length: 195\r\nContent-Type: text/plain\r\nDate: Tue, 31 May 2022 08:49:25 GMT\r\nServer: envoy\r\n\r\nupstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func1\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:142\nknative.dev/pkg/websocket.(*ManagedConnection).connect.func1\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:226\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\tk8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\tk8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\tk8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:226\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\tk8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:421\nknative.dev/pkg/websocket.(*ManagedConnection).connect\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:223\nknative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:163"}
{"severity":"ERROR","timestamp":"2022-05-31T08:49:26.692820496Z","logger":"activator","caller":"websocket/connection.go:192","message":"Failed to send ping message to ws://autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local:8080 ","commit":"6ec4509","knative.dev/controller":"activator","knative.dev/pod":"activator-848f9bfddf-txhbz","error":"connection has not yet been established","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func3\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:192"}
And this in istio-proxy (envoy) logs:
{ "time":"2022-05-31T20:02:02.586Z", "hostname":"activator-848f9bfddf-srghq", "txId":"e0bc1498-1bee-4c95-82cf-5a08526d92ab", "sourceIP":"-", "xFor":"-", "originatingIp":"-", "upstream_host":"10.199.173.222:8080", "user-agent":"Go-http-client/1.1", "downstreamRemoteAddress":"10.199.163.212:58164", "req":"/", "method":"GET", "protocol":"HTTP/1.1", "xHost":"autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local:8080", "status":"503", "response_flags":"UF,URX", "msg":"upstream_reset_before_response_started{connection_failure,TLS_error:_268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED}", "authority":"autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local:8080", "reqSize":"0", "respSize":"195", "upstreamTime":"-", "requestDuration":"-", "responseDuration":"-", "envoyTime":"-", "txTime":"11", "routeName":"default", "upstreamCluster":"outbound,8080,,autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local", "upstreamTransportFailureReason":"TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED", "downstreamLocalSAN":"-", "downstreamPeerSAN":"-", "downstreamLocalSubject":"-", "downstreampeerSubject":"-", "event":"source", "destinationAsset":"autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local:8080", "sourceAsset":"Intuit.data.mlplatform.mlpinfrastructure", "app":"knative-serving" }
| 20:02:44 |