| 3 Jun 2022 |
|  Christian Lehre set a profile picture. | 07:27:08 |
| Christian Lehre | 
Download Skjermbilde 2022-06-03 kl. 09.26.09.png | 07:27:10 |
| Christian Lehre | Hello! I have deployed Kubeflow v1.5 to GCP, and trying to apply a simple manifest for deploying an xgboost model that I have uploaded to the kubeflow-managed Cloud Storage. When i deploy to the kubeflow namespace that comes with the kubeflow deployment, the init container of the InferenceService that mounts the model to the volume of the pod is not running. However, when i deploy to another namespace the init container runs, but now the caller does not have access to mount the model. Any ideas what can be the problem? Another problem I have is that the models page in the Kubeflow UI simply renders a blank page. I inspect the page and see the following error messages in the console | 07:27:10 |
 Benjamin Tan | Don't deploy in the kubeflow namespace | 08:01:44 |
| Benjamin Tan | You can create some other namespace and deploy it there | 08:02:01 |
 @californiatl:matrix.org | I'll help 10 individuals how to earn $20,000 in just 72 hours from the crypto market. But you will pay me 10% commission when you receive your profit. if interested send me a direct message via WhatsApp by asking me HOW for more details on how to get started
+1 (2297781881 | 08:03:28 |
| Christian Lehre | Benjamin Tan Thanks for the reply! How would i then make sure that i have access to the storage? Im very new in the k8s world, so it might be a stupid question. | 08:03:48 |
|  Sebastian Lehrig joined the room. | 08:04:27 |
| Benjamin Tan | lol not stupid. sooo if u get logs from storage-initializer , you usually will get some hints. | 08:11:24 |
| Benjamin Tan | Where is your Kubeflow installation on? | 08:11:54 |
| Benjamin Tan | GCP? | 08:11:56 |
| Benjamin Tan | https://kserve.github.io/website/get_started/first_isvc/ | 08:14:00 |
| Christian Lehre | GCP, correct 🙂 I get the following Traceback in the storage-intializer container:
Traceback (most recent call last):
File "/storage-initializer/scripts/initializer-entrypoint", line 14, in module
kserve.Storage.download(src_uri, dest_path)
File "/usr/local/lib/python3.7/site-packages/kserve/storage.py", line 67, in download
Storage._download_gcs(uri, out_dir)
File "/usr/local/lib/python3.7/site-packages/kserve/storage.py", line 152, in _download_gcs
for blob in blobs:
File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 212, in _items_iter
for page in self._page_iter(increment=False):
File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 243, in _page_iter
page = self._next_page()
File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 372, in _next_page
response = self._get_next_page_response()
File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 432, in _get_next_page_response
method=self._HTTP_METHOD, path=self.path, query_params=params
File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/_http.py", line 78, in api_request
return call()
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 290, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 188, in retry_target
return target()
File "/usr/local/lib/python3.7/site-packages/google/cloud/_http.py", line 479, in api_request
raise exceptions.from_http_response(response)
google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/akerbp-kubeflow-core-kfp/o?projection=noAcl&prefix=models%2Fxgboost_lithology%2Fmodel.bst%2F&prettyPrint=false: Caller does not have storage.objects.list access to the Google Cloud Storage bucket. | 08:15:02 |
| Benjamin Tan | Beautiful | 08:15:30 |
| Benjamin Tan | So you need a few things. | 08:16:40 |
| Benjamin Tan | 1. Set up another namespace to deploy your model (kubctl create ns kserve-test) | 08:17:09 |
| Benjamin Tan | 2. Give permissions for the service account in the kfserve-test namespace to access the GCS bucket | 08:18:06 |
| Benjamin Tan | You might need your K8s admin to help u do this | 08:18:56 |
| Benjamin Tan | For the namespace:
kind: Namespace
apiVersion: v1
metadata:
name: kfserving-test
labels:
serving.kubeflow.org/inferenceservice: "enabled" | 08:21:16 |
| Benjamin Tan | gcloud iam service-accounts add-iam-policy-binding google-service-account-that-can-access-gcs-buckets \
--role roles/iam.workloadIdentityUser \ (or storage list role)
--member "serviceAccount: GOOGLE PROJECT NAME .svc.id.goog[kfserving-test/default]" | 08:23:28 |
| Benjamin Tan | Something like that | 08:23:34 |
| Benjamin Tan | Relevant docs here: https://cloud.google.com/iam/docs/creating-managing-service-accounts#iam-service-accounts-create-gcloud | 08:23:46 |
| Christian Lehre | Very good input, thank you Benjamin Tan! I will let you know if it works 🙂 | 08:24:12 |
| Benjamin Tan | 👍 | 08:25:35 |
| Christian Lehre | Redacted or Malformed Event | 09:04:21 |
| Christian Lehre | Benjamin Tan Seems like the role roles/storage.objectViewer (thats the least privileged role to have list access to GCS) is not supported for the service account...
ERROR: (gcloud.iam.service-accounts.add-iam-policy-binding) INVALID_ARGUMENT: Role roles/storage.objectViewer is not supported for this resource. | 09:52:38 |
| Benjamin Tan | Do u have workload identity set up? | 09:53:29 |
| Christian Lehre | I granted the service account the iam.workloadIdentityUser role. Apart from the kubeflow deployment I have not setup workload identity myself (not sure if its set up by the kubeflow deployment) | 09:55:00 |
| Benjamin Tan | Ooo try and see if workload identity works for u | 09:55:36 |
| Christian Lehre | Worload identity is enabled in the GKE cluster. Anything else I need to setup for it to work? | 09:56:20 |
