!LuUSGaeArTeoOgUpwk:matrix.org

kubeflow-kfserving

433 Members
2 Servers

Load older messages


SenderMessageTime
3 Jun 2022
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan You can create some other namespace and deploy it there 08:02:01
@californiatl:matrix.org@californiatl:matrix.orgI'll help 10 individuals how to earn $20,000 in just 72 hours from the crypto market. But you will pay me 10% commission when you receive your profit. if interested send me a direct message via WhatsApp by asking me HOW for more details on how to get started +1 (229778188108:03:28
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian Lehre Benjamin Tan Thanks for the reply! How would i then make sure that i have access to the storage? Im very new in the k8s world, so it might be a stupid question. 08:03:48
@_slack_kubeflow_U01N2J7JDAL:matrix.orgSebastian Lehrig joined the room.08:04:27
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan lol not stupid. sooo if u get logs from storage-initializer , you usually will get some hints. 08:11:24
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Where is your Kubeflow installation on? 08:11:54
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan GCP? 08:11:56
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan https://kserve.github.io/website/get_started/first_isvc/ 08:14:00
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian Lehre GCP, correct 🙂 I get the following Traceback in the storage-intializer container:
Traceback (most recent call last):
  File "/storage-initializer/scripts/initializer-entrypoint", line 14, in  module 
    kserve.Storage.download(src_uri, dest_path)
  File "/usr/local/lib/python3.7/site-packages/kserve/storage.py", line 67, in download
    Storage._download_gcs(uri, out_dir)
  File "/usr/local/lib/python3.7/site-packages/kserve/storage.py", line 152, in _download_gcs
    for blob in blobs:
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 212, in _items_iter
    for page in self._page_iter(increment=False):
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 243, in _page_iter
    page = self._next_page()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 372, in _next_page
    response = self._get_next_page_response()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 432, in _get_next_page_response
    method=self._HTTP_METHOD, path=self.path, query_params=params
  File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/_http.py", line 78, in api_request
    return call()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 290, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 188, in retry_target
    return target()
  File "/usr/local/lib/python3.7/site-packages/google/cloud/_http.py", line 479, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/akerbp-kubeflow-core-kfp/o?projection=noAcl&prefix=models%2Fxgboost_lithology%2Fmodel.bst%2F&prettyPrint=false: Caller does not have storage.objects.list access to the Google Cloud Storage bucket.
08:15:02
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Beautiful 08:15:30
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan So you need a few things. 08:16:40
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan 1. Set up another namespace to deploy your model (kubctl create ns kserve-test) 08:17:09
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan 2. Give permissions for the service account in the kfserve-test namespace to access the GCS bucket 08:18:06
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan You might need your K8s admin to help u do this 08:18:56
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan For the namespace:
kind: Namespace
apiVersion: v1
metadata:
  name: kfserving-test
  labels:
    serving.kubeflow.org/inferenceservice: "enabled"
08:21:16
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan
gcloud iam service-accounts add-iam-policy-binding  google-service-account-that-can-access-gcs-buckets  \ 
 --role roles/iam.workloadIdentityUser \ (or storage list role)
--member "serviceAccount: GOOGLE PROJECT NAME .svc.id.goog[kfserving-test/default]"
08:23:28
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Something like that 08:23:34
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Relevant docs here: https://cloud.google.com/iam/docs/creating-managing-service-accounts#iam-service-accounts-create-gcloud 08:23:46
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian Lehre Very good input, thank you Benjamin Tan! I will let you know if it works 🙂 08:24:12
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan 👍 08:25:35
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian LehreRedacted or Malformed Event09:04:21
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian Lehre Benjamin Tan Seems like the role roles/storage.objectViewer (thats the least privileged role to have list access to GCS) is not supported for the service account...
ERROR: (gcloud.iam.service-accounts.add-iam-policy-binding) INVALID_ARGUMENT: Role roles/storage.objectViewer is not supported for this resource.
09:52:38
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Do u have workload identity set up? 09:53:29
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian Lehre I granted the service account the iam.workloadIdentityUser role. Apart from the kubeflow deployment I have not setup workload identity myself (not sure if its set up by the kubeflow deployment) 09:55:00
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Ooo try and see if workload identity works for u 09:55:36
@_slack_kubeflow_U03HW39JNLV:matrix.orgChristian Lehre Worload identity is enabled in the GKE cluster. Anything else I need to setup for it to work? 09:56:20
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan I'm heading out so responses would be slow 09:56:25
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Not really. That should be sufficient 09:56:36
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Double check that u have a default service account in the namespace too 09:56:56
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan kubectl get sa default -n namespaceupicked 09:57:34

Show newer messages


Back to Room ListRoom Version: 6