!LuUSGaeArTeoOgUpwk:matrix.org

kubeflow-kfserving

433 Members
2 Servers

Load older messages


SenderMessageTime
2 Jun 2022
@_slack_kubeflow_U022U7KG24W:matrix.orgRachit Chauhan yeah, so the problem was a missing Destination Rule(DR). Everything was working fine at my local but the way my org manages certificates is diff from how istio’s default auth. istiod will configure each proxy with a TLS context it expects will work in istio’s default installation. Had to add a DR resource for my setup to work. 17:33:34
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun Not this field directly you need to use the timeout field on inference service yaml 17:53:52
@_slack_kubeflow_U02LE3KB53M:matrix.orgVivian Pan Hi KServe team, our team tested out the new inference graph alpha features and it works great. We are wondering if there’s a roadmap for implementing an SDK for this, if there’s any help needed for this. We’d love to contribute 17:59:03
@_slack_kubeflow_U03J0LZFTL5:matrix.org_slack_kubeflow_U03J0LZFTL5 joined the room.22:12:38
@_slack_kubeflow_UFVUV2UFP:matrix.orgDan Sun It should be pretty straight forward to do, will write up an issue and instruction for you to contribute 23:08:15
3 Jun 2022
@_slack_kubeflow_U0315UY2WRM:matrix.orgShri Javadekar Vivian Pan: How did you test the functionality? Did you download the patch and build the components yourself and deploy them before testing it? What components had to be built and deployed? 03:36:58
@_slack_kubeflow_U03FD3UU43V:matrix.orgHEM VATS joined the room.06:39:10
@_slack_kubeflow_U03CN7QAHN3:matrix.orgzorba(손주형) is it possible in go client?? there is only terminationGracePeriodSeconds in InferenceServiceSpec.PredictorSpec.PodSpec 07:19:05
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian Lehre joined the room.07:20:40
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian Lehre changed their display name from _slack_kubeflow_U03HW39JNLV to Christian Lehre.07:27:07
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian Lehre set a profile picture.07:27:08
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian LehreSkjermbilde 2022-06-03 kl. 09.26.09.png
Download Skjermbilde 2022-06-03 kl. 09.26.09.png
07:27:10
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian Lehre Hello! I have deployed Kubeflow v1.5 to GCP, and trying to apply a simple manifest for deploying an xgboost model that I have uploaded to the kubeflow-managed Cloud Storage. When i deploy to the kubeflow namespace that comes with the kubeflow deployment, the init container of the InferenceService that mounts the model to the volume of the pod is not running. However, when i deploy to another namespace the init container runs, but now the caller does not have access to mount the model. Any ideas what can be the problem? Another problem I have is that the models page in the Kubeflow UI simply renders a blank page. I inspect the page and see the following error messages in the console 07:27:10
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Don't deploy in the kubeflow namespace 08:01:44
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan You can create some other namespace and deploy it there 08:02:01
@californiatl:matrix.org@californiatl:matrix.orgI'll help 10 individuals how to earn $20,000 in just 72 hours from the crypto market. But you will pay me 10% commission when you receive your profit. if interested send me a direct message via WhatsApp by asking me HOW for more details on how to get started +1 (229778188108:03:28
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian Lehre Benjamin Tan Thanks for the reply! How would i then make sure that i have access to the storage? Im very new in the k8s world, so it might be a stupid question. 08:03:48
@_slack_kubeflow_U01N2J7JDAL:matrix.orgSebastian Lehrig joined the room.08:04:27
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan lol not stupid. sooo if u get logs from storage-initializer , you usually will get some hints. 08:11:24
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Where is your Kubeflow installation on? 08:11:54
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan GCP? 08:11:56
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan https://kserve.github.io/website/get_started/first_isvc/ 08:14:00
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian Lehre GCP, correct 🙂 I get the following Traceback in the storage-intializer container:
Traceback (most recent call last):
  File "/storage-initializer/scripts/initializer-entrypoint", line 14, in  module 
    kserve.Storage.download(src_uri, dest_path)
  File "/usr/local/lib/python3.7/site-packages/kserve/storage.py", line 67, in download
    Storage._download_gcs(uri, out_dir)
  File "/usr/local/lib/python3.7/site-packages/kserve/storage.py", line 152, in _download_gcs
    for blob in blobs:
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 212, in _items_iter
    for page in self._page_iter(increment=False):
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 243, in _page_iter
    page = self._next_page()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 372, in _next_page
    response = self._get_next_page_response()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 432, in _get_next_page_response
    method=self._HTTP_METHOD, path=self.path, query_params=params
  File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/_http.py", line 78, in api_request
    return call()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 290, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 188, in retry_target
    return target()
  File "/usr/local/lib/python3.7/site-packages/google/cloud/_http.py", line 479, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/akerbp-kubeflow-core-kfp/o?projection=noAcl&prefix=models%2Fxgboost_lithology%2Fmodel.bst%2F&prettyPrint=false: Caller does not have storage.objects.list access to the Google Cloud Storage bucket.
08:15:02
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Beautiful 08:15:30
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan So you need a few things. 08:16:40
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan 1. Set up another namespace to deploy your model (kubctl create ns kserve-test) 08:17:09
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan 2. Give permissions for the service account in the kfserve-test namespace to access the GCS bucket 08:18:06
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan You might need your K8s admin to help u do this 08:18:56
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan For the namespace:
kind: Namespace
apiVersion: v1
metadata:
  name: kfserving-test
  labels:
    serving.kubeflow.org/inferenceservice: "enabled"
08:21:16
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan
gcloud iam service-accounts add-iam-policy-binding  google-service-account-that-can-access-gcs-buckets  \ 
 --role roles/iam.workloadIdentityUser \ (or storage list role)
--member "serviceAccount: GOOGLE PROJECT NAME .svc.id.goog[kfserving-test/default]"
08:23:28

Show newer messages


Back to Room ListRoom Version: 6