17 May 2022 |
Benjamin Tan | https://github.com/kserve/kserve/tree/master/docs/samples/istio-dex | 04:00:45 |
zorba(손주형) | What happens if I put more than 1 container in Transformer yaml??
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: torch-transformer
spec:
predictor:
model:
modelFormat:
name: pytorch
storageUri: gs://kfserving-examples/models/torchserve/image_classifier
transformer:
containers:
- image: kserve/image-transformer:latest
name: kserve-container
command:
- "python"
- "-m"
- "model"
args:
- --model_name
- mnist | 08:40:29 |
Dan Sun | You can horizontally scale up ingress gateway as you want, it is a reverse proxy so it can process ~1000/s for each replica | 08:58:10 |
Saurabh Agarwal | Download image.png | 12:11:44 |
Saurabh Agarwal | facing the issue where the inferenceservice is created but never gets into running(respective pod is not initializing) | 12:11:44 |
theofpa | https://twitter.com/techatbloomberg/status/1526554734680850432?s=21&t=_gw_HEdPK0TMgWhEnvtYxg | 14:23:10 |
| _slack_kubeflow_U02Q0DARM8B joined the room. | 15:46:18 |
Shri Javadekar | I've see the transformer (or even the predictor) components get translated into Kubernetes deployments. So, it could become a deployment with two containers (unless there are checks in place to prevent it).
Care to try it out and let us know? 🙂 | 16:01:07 |
Diego Kiner | It's also not clear how to actually make a request to a graph - there doesn't appear to be any extra virtual service stood up to provide an entrypoint? | 19:29:15 |
| Jonny Browning changed their display name from _slack_kubeflow_U036ZCFAFLP to Jonny Browning. | 22:20:28 |
| Jonny Browning set a profile picture. | 22:20:33 |
Rachit Chauhan | (commenting to follow the thread) | 22:58:55 |
18 May 2022 |
Dan Sun | Diego Kiner We create graph orchestrator as a service to chain the requests, so the entry point is the graph orchestrator service | 00:27:36 |
Diego Kiner | Ok, we're trying to work off of the changes in the PR to get it working - how is the orchestrator created? | 00:28:32 |
Dan Sun | what’s the use case of having two containers on transformer? | 00:29:57 |
Dan Sun | there is a docker file on the PR to build the graph orchestrator image | 00:32:10 |
Diego Kiner | I see the router.Dockerfile - is this what you mean? But even if built, how is it meant to be deployed? | 00:35:01 |
Dan Sun | just run make deploy-dev that will deploy the graph orchestrator image and when you create the inference graph it will expose the endpoint for you | 00:42:46 |
Diego Kiner | Ok thanks! will give it a try | 00:43:05 |
Diego Kiner | ``kustomize build config/overlays/development , kubectl apply -f -
Error: trouble configuring builtin PatchStrategicMergeTransformer with config:
paths:
- configmap/inferenceservice_patch.yaml
- manager_image_patch.yaml
: evalsymlink failure on '.../kserve/config/overlays/development/configmap/inferenceservice_patch.yaml' : lstat .../kserve/config/overlays/development/configmap/inferenceservice_patch.yaml: no such file or directory `` | 01:28:59 |
zorba(손주형) | Dan Sun I don’t know.. but it’s possible since containers yaml input list List of container | 02:40:04 |
Shri Javadekar | I can imagine one container for the predictions and a sidecar container for logs/metrics, etc. | 04:18:29 |
zorba(손주형) | Shri Javadekar I thought that too. However how kserve know which container is transformer and which is sidecar? | 05:24:27 |
Dan Sun | well kserve already can inject a model agent sidecar, if any features need to be added I recommend contributing there | 07:42:13 |
Dan Sun | KServe introduction talk on Kubecon AI Days is now on youtube !
https://www.youtube.com/watch?v=FX6naJLaq2Y&list=PLj6h78yzYM2PJdsIBxtDOyiFqP3wIbOcc&index=9 | 08:25:56 |
Benjamin Tan | Hurray! Was looking forward to this ❤️ | 09:38:53 |
zorba(손주형) | oh it will be in 0.9.0 | 11:20:03 |
| Alexa Griffith changed their display name from _slack_kubeflow_U02UYNBU951 to Alexa Griffith. | 13:40:58 |
| Alexa Griffith set a profile picture. | 13:41:01 |
Alexa Griffith | Yay!! Had so much fun | 13:41:12 |