26 May 2022 |
Alexandre Brown | Yes I believe what is under _example is not applied and only there for documentation purposes | 13:05:47 |
Alexandre Brown | Maybe try this :
config-network
apiVersion: v1
data:
domainTemplate: "{{.Domain}}/{{.Namespace}}/{{.Name}}"
_example: ,
config-domain
apiVersion: v1
data:
mydomain.com: ""
_example: , | 13:07:41 |
Kuba Dawczynski | ok give me the second | 13:08:14 |
Kuba Dawczynski | ```error: configmaps "config-network" could not be patched: admission webhook "config.webhook.serving.knative.dev" denied the request: validation failed: domain template has url path: /bar/foo
You can run `kubectl replace -f /var/folders/m1/c6dp897x6jb4p9k0j32_mqbh0000gq/T/kubectl-edit-2589593085.yaml` to try this update```
validation failed | 13:33:05 |
Alexandre Brown | Now this makes more sense, it clearly states that having url path /bar/foo is not allowed and so I'd be inclined to think it's simply not feasible | 13:34:15 |
Kuba Dawczynski | ok, thx for information and your time | 13:41:18 |
Rachit Chauhan | Due to some limitations at our org, we have to use namespace other than knative-serving for knative’s control plane. Will it have any problems with kserve ? | 19:45:23 |
Dan Sun | you might need to create additional virtual service for the path rewrite | 19:47:29 |
Dan Sun | Kunming checkout this example https://github.com/kserve/kserve/blob/master/docs/samples/gcp-iap/virtual-service.yaml | 19:48:10 |
Dan Sun | should be fine, probably need a few configuration changes | 19:48:48 |
Rachit Chauhan | ok. I see there are references to knative-serving namespace (for ingresses) in kserve.yaml :
1. https://github.com/kserve/kserve/blob/master/install/v0.8.0/kserve.yaml#L14995
2. https://github.com/kserve/kserve/blob/master/install/v0.8.0/kserve.yaml#L14850
3. https://github.com/kserve/kserve/blob/master/install/v0.8.0/kserve.yaml#L14852 | 19:52:55 |
Rachit Chauhan | I am going to give it a try and will let here know what problems i see. | 19:53:42 |
27 May 2022 |
Rachit Chauhan | so, the problem that i am seeing when using a different namespace for knative-serving is that for webhooks domainmapping-webhook and webhook , the tls certs are missing.
{"severity":"WARNING","timestamp":"2022-05-27T00:12:49.070225477Z","logger":"domainmapping-webhook","caller":"webhook/webhook.go:154","message":"server key missing","knative.dev/pod":"domainmapping-webhook-775c44fdd7-nrfx4"}
2022/05/27 00:12:49 http: TLS handshake error from 10.1.0.1:62164: tls: no certificates configured
| 00:14:23 |
Rachit Chauhan | this is happening in a strange way. I deleted the NS and created it fresh and then tried installation and everything worked fine. | 00:46:59 |
Mark Winter | Triton's fil_backend has support for categorical features recently https://github.com/triton-inference-server/fil_backend#categorical-feature-support | 03:44:08 |
Mark Winter | You can using triton via kserve | 03:44:39 |
Mark Winter | Note that it only supports FP32 input/output at the moment | 03:44:47 |
| รัชพล เเขมภูเขียว joined the room. | 05:46:26 |
รัชพล เเขมภูเขียว | how to use pvc mount ? fails to reconcile predictor: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: Persistent volume claim support is disabled, but found persistent volume claim task-pv-claim: Persistent volume write support is disabled, but found persistent volume claim task-pv-claim that is not read-only: must not set the field(s): spec.template.spec.volumes[0].persistentVolumeClaim
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: nb-logistic-regression
spec:
predictor:
volumes:
- name: volume
persistentVolumeClaim:
claimName: task-pv-claim
containers:
- name: classifier
image: visairegistry.azurecr.io/xxxxxxxx
imagePullPolicy: Always
env:
- name: PROTOCOL
value: v2
volumeMounts:
- name: volume
mountPath: /opt/mlserver/outputs
subPath: inputs-bucket/inputs-sentiment
ports:
- containerPort: 8080
protocol: TCP | 05:48:30 |
Mark Winter | You should load pvc via STORAGE_URI instead of using volumeMounts directly
env:
- name: STORAGE_URI
value: pvc://pvc-name | 09:11:06 |
| Nithin R joined the room. | 10:06:23 |
Nithin R | Hi, i have grafana & prometheus running in different cluster and would like to intergrate to kubeflow model metrics. Can anyone please suggest | 10:08:00 |
zorba(손주형) | I`m using thanos | 12:27:01 |
zorba(손주형) | it`s good to manage multicluster prometheus | 12:27:21 |
Nithin R | Ok, how did you integrate it with kserve metrics? | 12:35:48 |
John Paulett | Thanks Mark Winter for the suggestion.
I'm also looking at submitting a PR to lightgbm to re-use the training's categories for the prediction | 14:47:04 |
zorba(손주형) | add prometheus operator annotation on isvc | 15:01:41 |
| _slack_kubeflow_U034329LRB2 joined the room. | 15:41:34 |
Rachit Chauhan | Dan Sun: what about kserve itself ? Does it need all of it’s control plane resources to be in kserve namespace or can we change that too ? Or any pointers I should be aware of ? | 22:37:21 |
28 May 2022 |
รัชพล เเขมภูเขียว | I have a very long Terminating problem, is there any solution? | 08:48:37 |