!LuUSGaeArTeoOgUpwk:matrix.org

kubeflow-kfserving

434 Members
3 Servers

Load older messages


SenderMessageTime
31 May 2022
@_slack_kubeflow_U03CN7QAHN3:matrix.orgzorba(손주형)image.png
Download image.png
05:12:36
@_slack_kubeflow_U03CN7QAHN3:matrix.orgzorba(손주형) kserve user survey is not available 05:12:36
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun yes the survey is closed, just made a PR to update that https://github.com/kserve/website/pull/141/files 06:14:50
@_slack_kubeflow_U022U7KG24W:matrix.orgRachit Chauhan I missed last bi-weekly meeting but I do see KServe 0.9 release tracking item in the notes. Can someone summarize what was discussed around this ? OR do we have issue to track it’s timeline ? 17:27:08
@_slack_kubeflow_U03H9RTMF3R:matrix.orgThomas Watkin joined the room.19:11:04
@_slack_kubeflow_U03H9RTMF3R:matrix.orgThomas Watkin changed their display name from _slack_kubeflow_U03H9RTMF3R to Thomas Watkin.19:16:44
@_slack_kubeflow_U03H9RTMF3R:matrix.orgThomas Watkin set a profile picture.19:16:45
@_slack_kubeflow_U01JLCXHSJY:matrix.orgJuergen Stary joined the room.19:24:14
@_slack_kubeflow_U01JLCXHSJY:matrix.orgJuergen Stary changed their display name from _slack_kubeflow_U01JLCXHSJY to Juergen Stary.19:25:37
@_slack_kubeflow_U01JLCXHSJY:matrix.orgJuergen Stary set a profile picture.19:25:38
@_slack_kubeflow_U01JLCXHSJY:matrix.orgJuergen Stary Hi there, i wanted to join your community meeting from time to time but seems like there is not fixed schedule or at least no day given in the Group Sync log? Cheers 19:25:38
@_slack_kubeflow_U022U7KG24W:matrix.orgRachit Chauhan Hi all, I am installing knative-serving in namespace other than knative-serving and have enabled istio-injection (for that NS too). All other control plane components seems to be working fine except activator. Seeing this in activator’s logs:
{"severity":"WARNING","timestamp":"2022-05-31T08:49:23.436527148Z","logger":"activator","caller":"handler/healthz_handler.go:36","message":"Healthcheck failed: received SIGTERM from kubelet","commit":"6ec4509","knative.dev/controller":"activator","knative.dev/pod":"activator-848f9bfddf-txhbz"}
{"severity":"ERROR","timestamp":"2022-05-31T08:49:25.693721434Z","logger":"activator","caller":"websocket/connection.go:142","message":"Websocket connection could not be established","commit":"6ec4509","knative.dev/controller":"activator","knative.dev/pod":"activator-848f9bfddf-txhbz","error":"websocket: bad handshake","request":"HTTP/1.1 503 Service Unavailable\r\nConnection: close\r\nContent-Length: 195\r\nContent-Type: text/plain\r\nDate: Tue, 31 May 2022 08:49:25 GMT\r\nServer: envoy\r\n\r\nupstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func1\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:142\nknative.dev/pkg/websocket.(*ManagedConnection).connect.func1\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:226\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\tk8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\tk8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\tk8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:226\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\tk8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:421\nknative.dev/pkg/websocket.(*ManagedConnection).connect\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:223\nknative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:163"}
{"severity":"ERROR","timestamp":"2022-05-31T08:49:26.692820496Z","logger":"activator","caller":"websocket/connection.go:192","message":"Failed to send ping message to  ws://autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local:8080 ","commit":"6ec4509","knative.dev/controller":"activator","knative.dev/pod":"activator-848f9bfddf-txhbz","error":"connection has not yet been established","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func3\n\tknative.dev/pkg@v0.0.0-20220412134708-e325df66cb51/websocket/connection.go:192"}
And this in istio-proxy (envoy) logs:
{ "time":"2022-05-31T20:02:02.586Z", "hostname":"activator-848f9bfddf-srghq", "txId":"e0bc1498-1bee-4c95-82cf-5a08526d92ab", "sourceIP":"-", "xFor":"-", "originatingIp":"-", "upstream_host":"10.199.173.222:8080", "user-agent":"Go-http-client/1.1", "downstreamRemoteAddress":"10.199.163.212:58164", "req":"/", "method":"GET", "protocol":"HTTP/1.1", "xHost":"autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local:8080", "status":"503", "response_flags":"UF,URX", "msg":"upstream_reset_before_response_started{connection_failure,TLS_error:_268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED}", "authority":"autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local:8080", "reqSize":"0", "respSize":"195", "upstreamTime":"-", "requestDuration":"-", "responseDuration":"-", "envoyTime":"-", "txTime":"11", "routeName":"default", "upstreamCluster":"outbound,8080,,autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local", "upstreamTransportFailureReason":"TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED", "downstreamLocalSAN":"-", "downstreamPeerSAN":"-", "downstreamLocalSubject":"-", "downstreampeerSubject":"-", "event":"source", "destinationAsset":"autoscaler.data-mlplatform-knativeserving3-usw2-dev.svc.cluster.local:8080", "sourceAsset":"Intuit.data.mlplatform.mlpinfrastructure", "app":"knative-serving" }
20:02:44
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun I guess mTLS is turned on ? 23:48:57
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun seems like activtor is not able to connect to autoscaler 23:49:11
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED 23:49:20
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun some certificate issue I assume 23:49:27
@_slack_kubeflow_U022U7KG24W:matrix.orgRachit Chauhan yeah activtor is not able to connect to autoscaler 23:49:42
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun not sure about your istio setup 23:50:35
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun it is fixed schedule, we do biweekly cadence on Wednesday 23:51:22
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan SunRedacted or Malformed Event23:55:02
1 Jun 2022
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun Rachit Chauhan the release is delayed because of the deprecated testing infra which is blocking a few PRs. We are actively resolving that. https://github.com/kubeflow/testing/issues/1006#issuecomment-1142613291 00:18:46
@_slack_kubeflow_U022U7KG24W:matrix.orgRachit Chauhan Thanks for the info Dan Sun 02:49:48
@_slack_kubeflow_URY59MKC7:matrix.org_slack_kubeflow_URY59MKC7 joined the room.07:13:41
@_slack_kubeflow_U01JLCXHSJY:matrix.orgJuergen Stary Thanks for letting me know. 07:16:06
@_slack_kubeflow_U03JHCGN0L8:matrix.org_slack_kubeflow_U03JHCGN0L8 joined the room.10:51:51
@_slack_kubeflow_U03HM661ZPY:matrix.org_slack_kubeflow_U03HM661ZPY joined the room.11:33:42
@_slack_kubeflow_U9UFLSBM4:matrix.orgcroberts I'm working with modelmesh-serving and I'm looking to add some labels/annotations to the runtimes. Per the docs, it looks like I can set runtimePodLabels and runtimePodAnnotations in the model-serving-config-defaults configmap, which I have done, but I'm not seeing those labels/annotations show-up on the runtime pods yet. Any ideas of where I've gone wrong? 14:06:08
@_slack_kubeflow_U9UFLSBM4:matrix.orgcroberts Oh wait....that feature might be too new for me, I'm still using the 0.8 release. 14:16:27
@_slack_kubeflow_U03CN7QAHN3:matrix.orgzorba(손주형) when 0.9.0 version may be release? is there any timeline? 15:23:48
@_slack_kubeflow_U03CM40MZQB:matrix.orgRaphael Chazelle joined the room.17:10:50

Show newer messages


Back to Room ListRoom Version: 6