| 18 May 2022 |
 Diego Kiner | I see the router.Dockerfile - is this what you mean? But even if built, how is it meant to be deployed? | 00:35:01 |
 Dan Sun | just run make deploy-dev that will deploy the graph orchestrator image and when you create the inference graph it will expose the endpoint for you | 00:42:46 |
| Diego Kiner | Ok thanks! will give it a try | 00:43:05 |
| Diego Kiner | ``kustomize build config/overlays/development , kubectl apply -f -
Error: trouble configuring builtin PatchStrategicMergeTransformer with config:
paths:
- configmap/inferenceservice_patch.yaml
- manager_image_patch.yaml
: evalsymlink failure on '.../kserve/config/overlays/development/configmap/inferenceservice_patch.yaml' : lstat .../kserve/config/overlays/development/configmap/inferenceservice_patch.yaml: no such file or directory`` | 01:28:59 |
 zorba(손주형) | Dan Sun I don’t know.. but it’s possible since containers yaml input list List of container | 02:40:04 |
 Shri Javadekar | I can imagine one container for the predictions and a sidecar container for logs/metrics, etc. | 04:18:29 |
| zorba(손주형) | Shri Javadekar I thought that too. However how kserve know which container is transformer and which is sidecar? | 05:24:27 |
| Dan Sun | well kserve already can inject a model agent sidecar, if any features need to be added I recommend contributing there | 07:42:13 |
| Dan Sun | KServe introduction talk on Kubecon AI Days is now on youtube !
https://www.youtube.com/watch?v=FX6naJLaq2Y&list=PLj6h78yzYM2PJdsIBxtDOyiFqP3wIbOcc&index=9 | 08:25:56 |
 Benjamin Tan | Hurray! Was looking forward to this ❤️ | 09:38:53 |
| zorba(손주형) | oh it will be in 0.9.0 | 11:20:03 |
|  Alexa Griffith changed their display name from _slack_kubeflow_U02UYNBU951 to Alexa Griffith. | 13:40:58 |
| Alexa Griffith set a profile picture. | 13:41:01 |
| Alexa Griffith | Yay!! Had so much fun | 13:41:12 |
 Vivian Pan | We tried the ensemble model, but the inference graph doesn’t seem to spin up the pod and can’t see any errors with the inference graph https://github.com/yuzisun/kserve/tree/graph/docs/samples/graph#34-ensemble
❯ kubectl -n kubeflow-demo get ig
NAME URL READY AGE
ensemble-model 23m
❯ k -n kubeflow-demo get isvc
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE
xgboost-iris http://xgboost-iris.kubeflow-demo. host-name True 100 xgboost-iris-predictor-default-00001 22h
sklearn-iris http://sklearn-iris.kubeflow-demo. host-name True 100 sklearn-iris-predictor-default-00001 22h
❯ k -n kubeflow-demo get pod
NAME READY STATUS RESTARTS AGE
sklearn-iris-predictor-default-00001-deployment-67df87c8ffddn8t 3/3 Running 0 22h
xgboost-iris-predictor-default-00001-deployment-7f88bf596crkdtt 3/3 Running 0 22h | 17:41:11 |
| Dan Sun | what’s the output for kubectl get ig ? | 19:38:44 |
| Vivian Pan | ❯ k -n kubeflow-demo get ig -o yaml
apiVersion: v1
items:
- apiVersion: serving.kserve.io/v1alpha1
kind: InferenceGraph
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: ,
{"apiVersion":"serving.kserve.io/v1alpha1","kind":"InferenceGraph","metadata":{"annotations":{},"name":"ensemble-model","namespace":"kubeflow-demo"},"spec":{"nodes":{"ensembleModel":{"routerType":"Ensemble","routes":[{"service":"sklearn-iris"},{"service":"xgboost-iris"}]}}}}
creationTimestamp: "2022-05-18T16:47:14Z"
generation: 1
managedFields:
- apiVersion: serving.kserve.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
.: {}
f:nodes:
.: {}
f:ensembleModel:
.: {}
f:routerType: {}
f:routes: {}
manager: kubectl-client-side-apply
operation: Update
time: "2022-05-18T16:47:14Z"
name: ensemble-model
namespace: kubeflow-demo
resourceVersion: "347617836"
uid: 6ae8f825-5624-47c7-b70d-f20d76eaf2be
spec:
nodes:
ensembleModel:
routerType: Ensemble
routes:
- service: sklearn-iris
- service: xgboost-iris
kind: List
metadata:
resourceVersion: ""
selfLink: ""
| 19:39:46 |
| Vivian Pan | no events listed
❯ k -n kubeflow-demo describe inferencegraph ensemble-model
Name: ensemble-model
Namespace: kubeflow-demo
Labels: none
Annotations: none
API Version: serving.kserve.io/v1alpha1
Kind: InferenceGraph
Metadata:
Creation Timestamp: 2022-05-18T16:47:14Z
Generation: 1
Managed Fields:
API Version: serving.kserve.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:nodes:
.:
f:ensembleModel:
.:
f:routerType:
f:routes:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-05-18T16:47:14Z
Resource Version: 347617836
UID: 6ae8f825-5624-47c7-b70d-f20d76eaf2be
Spec:
Nodes:
Ensemble Model:
Router Type: Ensemble
Routes:
Service: sklearn-iris
Service: xgboost-iris
Events: none | 19:40:18 |
|  shotarok joined the room. | 20:51:34 |
| Vivian Pan | i managed to resolve the issue, the validating webhook was not configured properly. But the router image seems unable right now. I’m guessing this is not pushed to the public container registry yet?
Revision "ensemble-model-00001" failed with message: Unable to fetch image "kserve/router:latest": | 22:53:36 |
| Dan Sun | if you run make deploy-dev that will deploy the development image, the latest image will be published once the PR is merged | 23:11:19 |
| Vivian Pan | I’m running your setup in our test cluster with a slightly different setup than what the make deploy-dev would generate. I build your image and pushed it to our container registry, I can see the inference graph is ready and pod is healthy
❯ k -n kubeflow-demo get ig
NAME URL READY AGE
ensemble-model http://ensemble-model.kubeflow-demo.iap.test. unity-host True 26m
❯ k -n kubeflow-demo get pods
NAME READY STATUS RESTARTS AGE
ensemble-model-00001-deployment-86b996545f-c9n48 3/3 Running 0 16m
Checking the virtualservice that gets created from this. Does the alpha version allow for usage of a custom gateway, other than the knative one?
❯ k -n kubeflow-demo get vs
NAME GATEWAYS HOSTS
ensemble-model-ingress ["knative-serving/knative-ingress-gateway","knative-serving/knative-local-gateway"] ["ensemble-model.kubeflow-demo","ensemble-model.kubeflow-demo.iap.test. unity-host ","ensemble-model.kubeflow-demo.svc","ensemble-model.kubeflow-demo.svc.cluster.local"] 17m
ensemble-model-mesh ["mesh"] ["ensemble-model.kubeflow-demo","ensemble-model.kubeflow-demo.svc","ensemble-model.kubeflow-demo.svc.cluster.local"] | 23:41:20 |
| Vivian Pan | similar to what the inference virtualservice has?
❯ k -n kubeflow-demo get vs
NAME GATEWAYS HOSTS
sklearn-iris ["knative-serving/knative-local-gateway","istio-system/gateway-iap"] ["sklearn-iris.kubeflow-demo.svc.cluster.local","sklearn-iris.kubeflow-demo.iap.test. unity-host "] 28h
sklearn-iris-predictor-default-ingress ["knative-serving/knative-ingress-gateway","knative-serving/knative-local-gateway"] ["sklearn-iris-predictor-default.kubeflow-demo","sklearn-iris-predictor-default.kubeflow-demo.iap.test. unity-host ","sklearn-iris-predictor-default.kubeflow-demo.svc","sklearn-iris-predictor-default.kubeflow-demo.svc.cluster.local"] 28h
sklearn-iris-predictor-default-mesh ["mesh"] ["sklearn-iris-predictor-default.kubeflow-demo","sklearn-iris-predictor-default.kubeflow-demo.svc","sklearn-iris-predictor-default.kubeflow-demo.svc.cluster.local"]
| 23:44:07 |
| 19 May 2022 |
 Ian Miller | Hi all, question on KServe and KFServing. We deployed KServe/KFServing (depending on the cluster + KF version) using the installation yaml for deploying with kubeflow. When we deploy inference services, the inference service gets a URL that doesn't work through the istio gateway. I see reference in the configmap (and therefore virtual services) referencing a kfserving (or kserve respectively) gateway which doesn't seem to exist. Additionally, when I switch that config to use cluster-local-gateway instead which does seem to exist, it still doesn't route the calls through to my deployed inference service. I am able to invoke my inference service via the route listed in the corresponding ksvc however. Wondering if anyone knows why I might not be able to invoke through the istio gateway. My kserve deployment is not altered other than as described above. The main issue here is the Model UI displays the route listed on the inference service so having trouble directing users how to interact with their deployed model. | 01:51:32 |
| Dan Sun | Yes you should be able to change via knative networking configuration | 08:14:38 |
| Dan Sun | Ian Miller You are probably on an old version of Kubeflow? | 08:49:37 |
| Ian Miller | Done the bulk of this testing on KF 1.4 with KFServing 0.6.1. Saw the same configuration on KF 1.5 + KServe 0.7 though, but admittedly have tried less to get it to work there so far as we're still upgrading our primary clusters. | 13:58:34 |
 _slack_kubeflow_U9UFLSBM4 | I'm working on a slightly strange deployment. KServe-raw on openshift without istio. I am able to serve models and get predictions no problem. I can hit /metrics on the kserve-controller-manager-metrics-service and I see metrics, but they all seem to be only for the controller itself rather than anything about the models being served (ie: number of predictions, time per prediction, etc). Is there something else I need to enable or somewhere else I need to be looking to get the metrics about the InferenceService itself? | 15:29:53 |
|  @californiatok:matrix.org joined the room. | 22:02:59 |
| 20 May 2022 |
|  레몬버터구이 joined the room. | 07:14:28 |
