| 3 May 2022 |
|  Chris Chase changed their display name from _slack_kubeflow_U013VE77D62 to Chris Chase. | 20:14:21 |
| Chris Chase set a profile picture. | 20:14:22 |
| Chris Chase | Hey Kserve folks. I deployed an example modelmesh-serving deployment using the quickstart install for someone to demo model mesh. Unfortunately, the cluster I deployed to upgraded over the weekend, meaning pods were evicted. As a result my sample installation was dead today before the demo because both the minio installation and the etcd installation had both disappeared. Is there a non-ephemeral deployment example I can use? I would think etcd disappearing would be very unfortunate since all the services died at that point. | 20:14:23 |
 John Daciuk | Does KServe give a FIFO guarantee for some subsection of the request path? | 23:43:42 |
| 4 May 2022 |
 Rachit Chauhan | kimwnasptd: in case of standalone installation, i had to mention namespace: kserve in kustomization.yaml otherwise the config map , kserve-models-web-app-config, was getting created in default namespace and was not available to the deployment | 01:06:34 |
 Surya Iyer | Was able to use serving.kubeflow.org/s3-usevirtualbucket to resolve the issue | 17:29:51 |
| Surya Iyer | Redacted or Malformed Event | 17:37:59 |
 Vivian Pan | Hi kserve team, my team is doing a POC on kServe v0.8.0. We have most of the components configured with knative. However it is not clear to how the Storage Initializer is configured for the namespace?
The knative deployments are in knative and knative-serving , whereas the kserve deployments are in kserve namespace.
The only way we have gotten the InferenceService to work is from within knative-serving namespace due to the storage-initializer not being available in other namespaces. Could someone help clarify how the storage-initializer is configured? | 18:47:52 |
|  _slack_kubeflow_U03DSUVTQM8 joined the room. | 18:48:48 |
 Dan Sun | There is no specific configuration it should work in all namespace except kserve control plane namespace | 23:22:09 |
| Dan Sun | Surya Iyer is there any doc we should update for this ? | 23:24:14 |
| Dan Sun | It does not work in kserve namespace because that’s for control plane, and the best practice is to deploy in user namespaces | 23:25:34 |
| Dan Sun | Vivian Pan | 23:25:54 |
| 5 May 2022 |
| Vivian Pan | ok thank you clarifying | 00:24:12 |
 wenyang zhou | I have deploy a model named realtime-losingorder-predict-xgb-neg-clean which name length is 43.
When I deployed the first revision, it works. And then I deployed the second revision, it showed Invalid value: "prev-realtime-losingorder-predict-xgb-neg-clean-predictor-default": must be no more than 63 characters | 06:15:16 |
| wenyang zhou | Is it a normal phenomenon? | 06:16:17 |
| wenyang zhou | I mean is it a normal phenomenon that the first revision can work while the second revision can not deploy. The two revisions do not meet the consistency. | 06:18:04 |
 Pierre Prange | Hey Folks, great work with kserve. Currently looking into leveraging KServe with KNative Eventing and OTEL Tracing. I Built and deployed some custom Python Models from kserve.model, configured Broker/Trigger aswell as Tracing in KNative. I use the InferenceService response Logger of Model A to send events to broker ingress and a KNative Trigger to forward those to Model B. I can see 2 Traces created but they don't correlate. Inspecting the Message Headers (Kafka Broker Implementation) i can see a traceparent field. What's in your Opinion the most convinient way to correlate those Traces? is it advisable to use otel SDK in custom model preprocessing to extract the trace-id and set context? | 08:58:58 |
 Paul Van Eck | Hey, sorry to hear that. The quickstart minio and etcd instances should probably be deployments, so I will convert them to provide some resiliency in these cases. These were meant for dev/experimentation, but I believe if you were to apply the quickstart dependencies again to bring the pods back up, the modelmesh controller would repopulate etcd based on the currently deployed predictors/isvcs. | 20:04:00 |
| Chris Chase | yup, I plan on recreating it as needed for dev. What do you guys use/expect on production deployments of modelmesh? | 20:51:18 |
| Chris Chase | (for etcd) | 20:52:26 |
| Paul Van Eck | Nick can answer better than I can, but definitely a multi-node cluster of etcd. At IBM, i believe we use some OpenShift based etcd operator for deploying and managing HA etcd | 21:08:11 |
| 6 May 2022 |
| Dan Sun | when you deploy the second revision it generates the prev tag for the previous version, so additional characters are added to the dns name. | 00:40:22 |
| Dan Sun | Pierre Prange sounds a great idea, would you be interested in to open an issue for this? | 00:41:10 |
| Dan Sun | We probably want to document a production etcd setup this is critical for production modelmesh deployment | 00:44:03 |
| wenyang zhou | Yea，but the first revision and the second revision can not deploy consistent , is it a right design ？ | 00:44:10 |
| Dan Sun | Well nothing wrong with the design rather a side effect adding the tag to previous deployment | 00:47:21 |
| Dan Sun | Probably need a validation for dns name length | 00:48:25 |
| wenyang zhou | Oh , get it | 00:48:43 |
|  _slack_kubeflow_U03E5S89214 joined the room. | 09:41:34 |
