kubeflow-kfserving

Diego Kiner We create graph orchestrator as a service to chain the requests, so the entry point is the graph orchestrator service

00:27:36

Ok, we're trying to work off of the changes in the PR to get it working - how is the orchestrator created?

00:28:32

what’s the use case of having two containers on transformer?

00:29:57

there is a docker file on the PR to build the graph orchestrator image

00:32:10

I see the router.Dockerfile - is this what you mean? But even if built, how is it meant to be deployed?

00:35:01

just run make deploy-dev that will deploy the graph orchestrator image and when you create the inference graph it will expose the endpoint for you

00:42:46

Ok thanks! will give it a try

00:43:05

``

kustomize build config/overlays/development , kubectl apply -f -
Error: trouble configuring builtin PatchStrategicMergeTransformer with config:

paths: - configmap/inferenceservice_patch.yaml - manager_image_patch.yaml

: evalsymlink failure on '.../kserve/config/overlays/development/configmap/inferenceservice_patch.yaml' : lstat .../kserve/config/overlays/development/configmap/inferenceservice_patch.yaml: no such file or directory

``

01:28:59

zorba(손주형)

Dan Sun I don’t know.. but it’s possible since containers yaml input list List of container

02:40:04

Shri Javadekar

I can imagine one container for the predictions and a sidecar container for logs/metrics, etc.

04:18:29

zorba(손주형)

Shri Javadekar I thought that too. However how kserve know which container is transformer and which is sidecar?

05:24:27

well kserve already can inject a model agent sidecar, if any features need to be added I recommend contributing there

07:42:13

KServe introduction talk on Kubecon AI Days is now on youtube ! https://www.youtube.com/watch?v=FX6naJLaq2Y&list=PLj6h78yzYM2PJdsIBxtDOyiFqP3wIbOcc&index=9

08:25:56

Benjamin Tan

Hurray! Was looking forward to this ❤️

09:38:53

zorba(손주형)

oh it will be in 0.9.0

11:20:03

Alexa Griffith changed their display name from _slack_kubeflow_U02UYNBU951 to Alexa Griffith.

13:40:58

Alexa Griffith set a profile picture.

13:41:01

Alexa Griffith

Yay!! Had so much fun

13:41:12

We tried the ensemble model, but the inference graph doesn’t seem to spin up the pod and can’t see any errors with the inference graph https://github.com/yuzisun/kserve/tree/graph/docs/samples/graph#34-ensemble

❯ kubectl -n kubeflow-demo get ig
NAME             URL   READY   AGE
ensemble-model                 23m
❯ k -n kubeflow-demo get isvc
NAME                       URL                                                                      READY   PREV   LATEST   PREVROLLEDOUTREVISION                              LATESTREADYREVISION                                AGE
xgboost-iris               http://xgboost-iris.kubeflow-demo. host-name                True           100                                                         xgboost-iris-predictor-default-00001               22h
sklearn-iris               http://sklearn-iris.kubeflow-demo. host-name                True           100                                                         sklearn-iris-predictor-default-00001               22h
❯ k -n kubeflow-demo get pod
NAME                                                              READY   STATUS    RESTARTS   AGE
sklearn-iris-predictor-default-00001-deployment-67df87c8ffddn8t   3/3     Running   0          22h
xgboost-iris-predictor-default-00001-deployment-7f88bf596crkdtt   3/3     Running   0          22h

17:41:11

what’s the output for kubectl get ig ?

19:38:44

❯ k -n kubeflow-demo get ig -o yaml
apiVersion: v1
items:
- apiVersion: serving.kserve.io/v1alpha1
  kind: InferenceGraph
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: ,
        {"apiVersion":"serving.kserve.io/v1alpha1","kind":"InferenceGraph","metadata":{"annotations":{},"name":"ensemble-model","namespace":"kubeflow-demo"},"spec":{"nodes":{"ensembleModel":{"routerType":"Ensemble","routes":[{"service":"sklearn-iris"},{"service":"xgboost-iris"}]}}}}
    creationTimestamp: "2022-05-18T16:47:14Z"
    generation: 1
    managedFields:
    - apiVersion: serving.kserve.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:spec:
          .: {}
          f:nodes:
            .: {}
            f:ensembleModel:
              .: {}
              f:routerType: {}
              f:routes: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: "2022-05-18T16:47:14Z"
    name: ensemble-model
    namespace: kubeflow-demo
    resourceVersion: "347617836"
    uid: 6ae8f825-5624-47c7-b70d-f20d76eaf2be
  spec:
    nodes:
      ensembleModel:
        routerType: Ensemble
        routes:
        - service: sklearn-iris
        - service: xgboost-iris
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

19:39:46

no events listed

❯ k -n kubeflow-demo describe inferencegraph ensemble-model
Name:         ensemble-model
Namespace:    kubeflow-demo
Labels:        none 
Annotations:   none 
API Version:  serving.kserve.io/v1alpha1
Kind:         InferenceGraph
Metadata:
  Creation Timestamp:  2022-05-18T16:47:14Z
  Generation:          1
  Managed Fields:
    API Version:  serving.kserve.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:nodes:
          .:
          f:ensembleModel:
            .:
            f:routerType:
            f:routes:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2022-05-18T16:47:14Z
  Resource Version:  347617836
  UID:               6ae8f825-5624-47c7-b70d-f20d76eaf2be
Spec:
  Nodes:
    Ensemble Model:
      Router Type:  Ensemble
      Routes:
        Service:  sklearn-iris
        Service:  xgboost-iris
Events:            none

19:40:18

shotarok joined the room.

20:51:34

i managed to resolve the issue, the validating webhook was not configured properly. But the router image seems unable right now. I’m guessing this is not pushed to the public container registry yet?

Revision "ensemble-model-00001" failed with message: Unable to fetch image "kserve/router:latest":

22:53:36

if you run make deploy-dev that will deploy the development image, the latest image will be published once the PR is merged

23:11:19

I’m running your setup in our test cluster with a slightly different setup than what the make deploy-dev would generate. I build your image and pushed it to our container registry, I can see the inference graph is ready and pod is healthy

❯ k -n kubeflow-demo get ig
NAME             URL                                                            READY   AGE
ensemble-model   http://ensemble-model.kubeflow-demo.iap.test. unity-host    True    26m

❯ k -n kubeflow-demo get pods
NAME                                                              READY   STATUS    RESTARTS   AGE
ensemble-model-00001-deployment-86b996545f-c9n48                  3/3     Running   0          16m

Checking the virtualservice that gets created from this. Does the alpha version allow for usage of a custom gateway, other than the knative one?

❯ k -n kubeflow-demo get vs
NAME                                                 GATEWAYS                                                                              HOSTS
ensemble-model-ingress                               ["knative-serving/knative-ingress-gateway","knative-serving/knative-local-gateway"]   ["ensemble-model.kubeflow-demo","ensemble-model.kubeflow-demo.iap.test. unity-host ","ensemble-model.kubeflow-demo.svc","ensemble-model.kubeflow-demo.svc.cluster.local"]                                                                                                                                                                                                                                                                                                                                                                                                                                  17m
ensemble-model-mesh                                  ["mesh"]                                                                              ["ensemble-model.kubeflow-demo","ensemble-model.kubeflow-demo.svc","ensemble-model.kubeflow-demo.svc.cluster.local"]

23:41:20