21 May 2022 |
| _slack_kubeflow_U03GF0NC9QS joined the room. | 14:12:17 |
| Amalesh joined the room. | 16:45:25 |
22 May 2022 |
| mobin nikkhesal joined the room. | 11:47:09 |
| mobin nikkhesal changed their display name from _slack_kubeflow_U03GUM55T0R to mobin nikkhesal. | 11:58:34 |
| mobin nikkhesal set a profile picture. | 11:58:36 |
mobin nikkhesal | Hello,
I set up kubeflow 1.5 and added PodDefault namespace to access the kubeflow pipeline in a notebook based on this
apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
name: access-ml-pipeline
namespace: " YOUR_USER_PROFILE_NAMESPACE "
spec:
desc: Allow access to Kubeflow Pipelines
selector:
matchLabels:
access-ml-pipeline: "true"
volumes:
- name: volume-kf-pipeline-token
projected:
sources:
- serviceAccountToken:
path: token
expirationSeconds: 7200
audience: pipelines.kubeflow.org
volumeMounts:
- mountPath: /var/run/secrets/kubeflow/pipelines
name: volume-kf-pipeline-token
readOnly: true
env:
- name: KF_PIPELINES_SA_TOKEN_PATH
value: /var/run/secrets/kubeflow/pipelines/token
and I can see the token.
client.list_pipeline() work correctly
but when I want to list_experiment . I faced with this error:
ApiException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'date': 'Sun, 22 May 2022 11:54:05 GMT', 'x-envoy-upstream-service-time': '14', 'server': 'envoy', 'transfer-encoding': 'chunked'})
HTTP response body: {"error":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:279\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer)
could you please help me to solve this problem? | 13:02:16 |
Thomas Ounnas | pip install -U kfp 😉 | 14:08:37 |
mobin nikkhesal | kfp 1.6.3 | 16:27:30 |
| mobin nikkhesal changed their profile picture. | 17:00:13 |
mobin nikkhesal | The problem appears to be resolved since I updated the KFP client to '1.8.12'. Thomas Ounnas | 17:00:15 |
23 May 2022 |
| _slack_kubeflow_U03GBRE6GKG joined the room. | 09:46:29 |
24 May 2022 |
| Benjamin Tan changed their profile picture. | 15:25:15 |
25 May 2022 |
| Vu Dat joined the room. | 19:07:41 |
26 May 2022 |
| _slack_kubeflow_U03H27PMSMR joined the room. | 06:03:11 |
Vu Dat | Hi everyone, I want to build own my MLOps platform, so I want to integrate Kubeflow Notebooks to my platform not all of kubeflow. How can I do that? Thank you very much! | 06:10:34 |
Benjamin Tan | You can exclude stuff from the manifest if u dont want to install certain things | 09:09:27 |
Vu Dat | Sorry I am a newbie, can i chat private for you? | 09:10:24 |
Benjamin Tan | Sure | 09:10:42 |
| _slack_kubeflow_U03H4QLQ202 joined the room. | 15:36:26 |
27 May 2022 |
| Changshin Yoo changed their display name from _slack_kubeflow_U03C4M32V1U to Changshin Yoo. | 02:10:34 |
| Changshin Yoo set a profile picture. | 02:10:37 |
| _slack_kubeflow_U034329LRB2 joined the room. | 15:41:35 |
30 May 2022 |
| 레몬버터구이 changed their display name from _slack_kubeflow_U03EE7VFCDN to 레몬버터구이. | 02:19:24 |
| 레몬버터구이 set a profile picture. | 02:19:26 |
31 May 2022 |
| Harcharan Singh joined the room. | 14:20:54 |
Manjari singh | Yingding Wang Benjamin Tan - I tried restarting the pod and also did logging again but still getting same error. | 15:37:21 |
Manjari singh | Also webhook pod is not giving any logs. | 15:38:19 |
Yingding Wang | Which KF version do you use? 1.4 or 1.5, is it multi-user setting? Which KFP SDK version do you have? can you upgrade to the latest kfp 1.8.12, can you run %env in your notebook cell to see whether most kf service env variables are availabe? Can you post detailed error message from your notebook while instantiate kfp.Client() in this thread? | 16:04:48 |
1 Jun 2022 |
| _slack_kubeflow_U03HM661ZPY joined the room. | 11:33:39 |
Maciek Stopa | Hey Benjamin Tan, once again thank you for your help back a month ago 🙂. I got it working, the problem was way before Kubeflow, since I didn't configure nvidia-runtime in k0s 's containerd correctly. For anyone looking for hints on how to debug similar issues please see this: https://github.com/k0sproject/k0s/issues/1769 | 13:49:09 |