!LuUSGaeArTeoOgUpwk:matrix.org

kubeflow-kfserving

432 Members
2 Servers

Load older messages


SenderMessageTime
31 May 2022
@_slack_kubeflow_U03H9RTMF3R:matrix.orgThomas Watkin joined the room.19:11:04
@_slack_kubeflow_U03H9RTMF3R:matrix.orgThomas Watkin changed their display name from _slack_kubeflow_U03H9RTMF3R to Thomas Watkin.19:16:44
@_slack_kubeflow_U03H9RTMF3R:matrix.orgThomas Watkin set a profile picture.19:16:45
@_slack_kubeflow_U01JLCXHSJY:matrix.orgJuergen Stary joined the room.19:24:14
@_slack_kubeflow_U01JLCXHSJY:matrix.orgJuergen Stary changed their display name from _slack_kubeflow_U01JLCXHSJY to Juergen Stary.19:25:37
@_slack_kubeflow_U01JLCXHSJY:matrix.orgJuergen Stary set a profile picture.19:25:38
@_slack_kubeflow_U01JLCXHSJY:matrix.orgJuergen Stary Hi there, i wanted to join your community meeting from time to time but seems like there is not fixed schedule or at least no day given in the Group Sync log? Cheers 19:25:38
@_slack_kubeflow_U022U7KG24W:matrix.orgRachit Chauhan Hi all, I am installing knative-serving in namespace other than knative-serving and have enabled istio-injection (for that NS too). All other control plane components seems to be working fine except activator. Seeing this in activator’s logs:
{"severity":"WARNING","timestamp":"2022-05-31T08:49:23.436527148Z","logger":"activator","caller":"handler/healthz_handler.go:36","message":"Healthcheck failed: received SIGTERM from kubelet","commit":"6ec4509","knative.dev/controller":"activator","knative.dev/pod":"activator-848f9bfddf-txhbz"}
{"severity":"ERROR","timestamp":"2022-05-31T08:49:25.693721434Z","logger":"activator","caller":"websocket/connection.go:142","message":"Websocket connection could not be established","commit":"6ec4509","knative.dev/controller":"activator","knative.dev/pod":"activator-848f9bfddf-txhbz","error":"websocket: bad handshake","request":"HTTP/1.1 503 Service Unavailable\r\nConnection: close\r\nContent-Length: 195\r\nContent-Type: text/plain\r\nDate: Tue, 31 May 2022 08:49:25 GMT\r\nServer: envoy\r\n\r\nupstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func1\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:142\nknative.dev/pkg/websocket.(*ManagedConnection).connect.func1\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:226\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\tk8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\tk8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\tk8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:226\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\tk8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:421\nknative.dev/pkg/websocket.(*ManagedConnection).connect\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:223\nknative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:163"}
{"severity":"ERROR","timestamp":"2022-05-31T08:49:26.692820496Z","logger":"activator","caller":"websocket/connection.go:192","message":"Failed to send ping message to  ws://autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local:8080 ","commit":"6ec4509","knative.dev/controller":"activator","knative.dev/pod":"activator-848f9bfddf-txhbz","error":"connection has not yet been established","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func3\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:192"}
And this in istio-proxy (envoy) logs:
{ "time":"2022-05-31T20:02:02.586Z", "hostname":"activator-848f9bfddf-srghq", "txId":"e0bc1498-1bee-4c95-82cf-5a08526d92ab", "sourceIP":"-", "xFor":"-", "originatingIp":"-", "upstream_host":"10.199.173.222:8080", "user-agent":"Go-http-client/1.1", "downstreamRemoteAddress":"10.199.163.212:58164", "req":"/", "method":"GET", "protocol":"HTTP/1.1", "xHost":"autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local:8080", "status":"503", "response_flags":"UF,URX", "msg":"upstream_reset_before_response_started{connection_failure,TLS_error:_268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED}", "authority":"autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local:8080", "reqSize":"0", "respSize":"195", "upstreamTime":"-", "requestDuration":"-", "responseDuration":"-", "envoyTime":"-", "txTime":"11", "routeName":"default", "upstreamCluster":"outbound,8080,,autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local", "upstreamTransportFailureReason":"TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED", "downstreamLocalSAN":"-", "downstreamPeerSAN":"-", "downstreamLocalSubject":"-", "downstreampeerSubject":"-", "event":"source", "destinationAsset":"autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local:8080", "sourceAsset":"Intuit.data.mlplatform.mlpinfrastructure", "app":"knative-serving" }
20:02:44
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun I guess mTLS is turned on ? 23:48:57
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun seems like activtor is not able to connect to autoscaler 23:49:11
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED 23:49:20
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun some certificate issue I assume 23:49:27
@_slack_kubeflow_U022U7KG24W:matrix.orgRachit Chauhan yeah activtor is not able to connect to autoscaler 23:49:42
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun not sure about your istio setup 23:50:35
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun it is fixed schedule, we do biweekly cadence on Wednesday 23:51:22
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan SunRedacted or Malformed Event23:55:02
1 Jun 2022
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun Rachit Chauhan the release is delayed because of the deprecated testing infra which is blocking a few PRs. We are actively resolving that. https://github.com/kubeflow/testing/issues/1006#issuecomment-1142613291 00:18:46
@_slack_kubeflow_U022U7KG24W:matrix.orgRachit Chauhan Thanks for the info Dan Sun 02:49:48
@_slack_kubeflow_URY59MKC7:matrix.org_slack_kubeflow_URY59MKC7 joined the room.07:13:41
@_slack_kubeflow_U01JLCXHSJY:matrix.orgJuergen Stary Thanks for letting me know. 07:16:06
@_slack_kubeflow_U03JHCGN0L8:matrix.org_slack_kubeflow_U03JHCGN0L8 joined the room.10:51:51
@_slack_kubeflow_U03HM661ZPY:matrix.org_slack_kubeflow_U03HM661ZPY joined the room.11:33:42
@_slack_kubeflow_U9UFLSBM4:matrix.orgcroberts I'm working with modelmesh-serving and I'm looking to add some labels/annotations to the runtimes. Per the docs, it looks like I can set runtimePodLabels and runtimePodAnnotations in the model-serving-config-defaults configmap, which I have done, but I'm not seeing those labels/annotations show-up on the runtime pods yet. Any ideas of where I've gone wrong? 14:06:08
@_slack_kubeflow_U9UFLSBM4:matrix.orgcroberts Oh wait....that feature might be too new for me, I'm still using the 0.8 release. 14:16:27
@_slack_kubeflow_U03CN7QAHN3:matrix.orgzorba(손주형) when 0.9.0 version may be release? is there any timeline? 15:23:48
@_slack_kubeflow_U03CM40MZQB:matrix.orgRaphael Chazelle joined the room.17:10:50
2 Jun 2022
@_slack_kubeflow_U02L5MAN5NU:matrix.org_slack_kubeflow_U02L5MAN5NU joined the room.03:32:36
@_slack_kubeflow_U03J0DGHKFD:matrix.org_slack_kubeflow_U03J0DGHKFD joined the room.13:21:10
@_slack_kubeflow_U03CN7QAHN3:matrix.orgzorba(손주형) but it is not possible..
v1beta1Controllers  fails to reconcile predictor: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: must not set the field(s): spec.template.spec.
│ terminationGracePeriodSeconds
14:56:54
@_slack_kubeflow_U03HYSZAD45:matrix.org_slack_kubeflow_U03HYSZAD45 joined the room.16:35:52

Show newer messages


Back to Room ListRoom Version: 6