26 May 2022 |
Dan Sun | you might need to create additional virtual service for the path rewrite | 19:47:29 |
Dan Sun | Kunming checkout this example https://github.com/kserve/kserve/blob/master/docs/samples/gcp-iap/virtual-service.yaml | 19:48:10 |
Dan Sun | should be fine, probably need a few configuration changes | 19:48:48 |
Rachit Chauhan | ok. I see there are references to knative-serving namespace (for ingresses) in kserve.yaml :
1. https://github.com/kserve/kserve/blob/master/install/v0.8.0/kserve.yaml#L14995
2. https://github.com/kserve/kserve/blob/master/install/v0.8.0/kserve.yaml#L14850
3. https://github.com/kserve/kserve/blob/master/install/v0.8.0/kserve.yaml#L14852 | 19:52:55 |
Rachit Chauhan | I am going to give it a try and will let here know what problems i see. | 19:53:42 |
27 May 2022 |
Rachit Chauhan | so, the problem that i am seeing when using a different namespace for knative-serving is that for webhooks domainmapping-webhook and webhook , the tls certs are missing.
{"severity":"WARNING","timestamp":"2022-05-27T00:12:49.070225477Z","logger":"domainmapping-webhook","caller":"webhook/webhook.go:154","message":"server key missing","knative.dev/pod":"domainmapping-webhook-775c44fdd7-nrfx4"}
2022/05/27 00:12:49 http: TLS handshake error from 10.1.0.1:62164: tls: no certificates configured
| 00:14:23 |
Rachit Chauhan | this is happening in a strange way. I deleted the NS and created it fresh and then tried installation and everything worked fine. | 00:46:59 |
Mark Winter | Triton's fil_backend has support for categorical features recently https://github.com/triton-inference-server/fil_backend#categorical-feature-support | 03:44:08 |
Mark Winter | You can using triton via kserve | 03:44:39 |
Mark Winter | Note that it only supports FP32 input/output at the moment | 03:44:47 |
| รัชพล เเขมภูเขียว joined the room. | 05:46:26 |
รัชพล เเขมภูเขียว | how to use pvc mount ? fails to reconcile predictor: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: Persistent volume claim support is disabled, but found persistent volume claim task-pv-claim: Persistent volume write support is disabled, but found persistent volume claim task-pv-claim that is not read-only: must not set the field(s): spec.template.spec.volumes[0].persistentVolumeClaim
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: nb-logistic-regression
spec:
predictor:
volumes:
- name: volume
persistentVolumeClaim:
claimName: task-pv-claim
containers:
- name: classifier
image: visairegistry.azurecr.io/xxxxxxxx
imagePullPolicy: Always
env:
- name: PROTOCOL
value: v2
volumeMounts:
- name: volume
mountPath: /opt/mlserver/outputs
subPath: inputs-bucket/inputs-sentiment
ports:
- containerPort: 8080
protocol: TCP | 05:48:30 |
Mark Winter | You should load pvc via STORAGE_URI instead of using volumeMounts directly
env:
- name: STORAGE_URI
value: pvc://pvc-name | 09:11:06 |
| Nithin R joined the room. | 10:06:23 |
Nithin R | Hi, i have grafana & prometheus running in different cluster and would like to intergrate to kubeflow model metrics. Can anyone please suggest | 10:08:00 |
zorba(손주형) | I`m using thanos | 12:27:01 |
zorba(손주형) | it`s good to manage multicluster prometheus | 12:27:21 |
Nithin R | Ok, how did you integrate it with kserve metrics? | 12:35:48 |
John Paulett | Thanks Mark Winter for the suggestion.
I'm also looking at submitting a PR to lightgbm to re-use the training's categories for the prediction | 14:47:04 |
zorba(손주형) | add prometheus operator annotation on isvc | 15:01:41 |
| _slack_kubeflow_U034329LRB2 joined the room. | 15:41:34 |
Rachit Chauhan | Dan Sun: what about kserve itself ? Does it need all of it’s control plane resources to be in kserve namespace or can we change that too ? Or any pointers I should be aware of ? | 22:37:21 |
28 May 2022 |
รัชพล เเขมภูเขียว | I have a very long Terminating problem, is there any solution? | 08:48:37 |
Dan Sun | you can tune the timeout to control the grace period for termination | 14:22:50 |
30 May 2022 |
| 레몬버터구이 changed their display name from _slack_kubeflow_U03EE7VFCDN to 레몬버터구이. | 02:19:23 |
| 레몬버터구이 set a profile picture. | 02:19:26 |
| _slack_kubeflow_U03ADPJCZBJ joined the room. | 07:36:48 |
Kuba Dawczynski | Dan Sun thx, it helped a lot, it's working 🙂 | 09:38:24 |
| @californiatl:matrix.org joined the room. | 15:45:21 |
Dan Sun | I think there is an issue to support out of the box | 16:07:41 |