270 Members
1 Servers

Load older messages

10 May 2022
@_slack_kubeflow_UGZAH4DBQ:matrix.orgJoshBottum n unexpected exception has occurred Line: /usr/local/lib/python3.6/dist-packages/git/cmd.py:928 Type: GitCommandError Reason: Cmd('git') failed due to: exit code(129) cmdline: git diff --cached --abbrev=40 --full-index --raw stderr: 'Not a git repository To compare two paths outside a working tree: usage: git diff [--no-index] path path ' The logfile `/home/minikf/provision.log' may contain more information on this error, including a backtrace. *** MiniKF provisioning tool terminated. Press Enter to restart, Ctrl-C to exit... *** 02:36:24
@_slack_kubeflow_UGZAH4DBQ:matrix.orgJoshBottum As an fyi, I tried to deploy on AWS, it failed, then tried GCP and MiniKF deployed after 2nd try. I set-up the demo for presentation and then stopped it. When I went back in to restart, I am getting this error. 02:37:38
@_slack_kubeflow_UFCE0TABE:matrix.orgrawkintrevo Hiya JoshBottum while I do recommend minikf as a quick on ramp to kubeflow for trial purposes, i don't really play with it much myself anymore. Alex Aidun still monitors this channel- he may be able to help. best of luck. ... that said- what info was in that log file? the error is saying whatever git you're trying to pull from does not exist so if that's a configurable field I would check there- o/w someone bipped a repo they shouldn't have. 10:41:31
@_slack_kubeflow_U02DWFSKS1F:matrix.orgAlex Aidun JoshBottum rawkintrevo hey gents, asked internally to see if we recognize the error, that being said I would head over to kubeflow.arrikto.com and check out the new Kubeflow as a Service! https://www.arrikto.com/blog/introducing-kubeflow-as-a-service-giving-data-scientists-instant-access-to-a-complete-mlops-platform/ 12:59:23
@_slack_kubeflow_U02DWFSKS1F:matrix.orgAlex Aidun JoshBottum does this address your issue? https://support.arrikto.com/hc/en-us/articles/5449137426449 13:26:09
@_slack_kubeflow_UGZAH4DBQ:matrix.orgJoshBottum Hi Alex Aidun thanks for your prompt response. I have deleted my MiniKF and I am trying KaaS. I have a presentation in Austin tomorrow and wanted to provide a KF demo, and my plan is to use KaaS and the distributed pytorch tutorial. Josh 14:28:28
@_slack_kubeflow_U03BZ3KTBPX:matrix.orgBilal Obeidat Hi Maciek Stopa I am trying to use a custom image where I need to add only add an inter whl library to gcr.io/arrikto/jupyter-kale-py38:release-1.3-l0-release-1.3-rc6 14:29:55
@_slack_kubeflow_U03BZ3KTBPX:matrix.orgBilal Obeidat I followed your approach here but still Minikf notebook server complain about the custom image 14:31:04
@_slack_kubeflow_U03BZ3KTBPX:matrix.orgBilal Obeidat here is my dockerfile : 14:31:34
@_slack_kubeflow_U03BZ3KTBPX:matrix.orgBilal Obeidat FROM gcr.io/arrikto/jupyter-kale-py38:release-1.3-l0-release-1.3-rc6 COPY snowflake_snowpark_python-0.5.0-py3-none-any.whl /tmp/ WORKDIR /usr/local/ USER root RUN pip install '/tmp/snowflake_snowpark_python-0.5.0-py3-none-any.whl[pandas]' -t /usr/local/lib/python3.8/dist-packages 14:31:46
@_slack_kubeflow_U03BZ3KTBPX:matrix.orgBilal Obeidat any idea? 14:31:52
@_slack_kubeflow_U02DWFSKS1F:matrix.orgAlex Aidun Good to know thanks JoshBottum for the context 14:52:37
11 May 2022
@_slack_kubeflow_U03EX7S4N1H:matrix.org_slack_kubeflow_U03EX7S4N1H joined the room.00:24:28
@_slack_kubeflow_U035009NYQ2:matrix.orgEvans Kiplagat how do you allow locally deployed MiniKF to access the GPU resources? 06:45:31
@_slack_kubeflow_U02JYD39G57:matrix.orgZoltán R. Jánki changed their display name from _slack_kubeflow_U02JYD39G57 to Zoltán R. Jánki.11:24:06
@_slack_kubeflow_U02JYD39G57:matrix.orgZoltán R. Jánki set a profile picture.11:24:09
@_slack_kubeflow_U02DWFSKS1F:matrix.orgAlex Aidun Evans Kiplagat I think what you are looking for is in this tips and tricks post: https://www.arrikto.com/blog/kubeflow-tips-tricks-february-23rd-2022/. How do I enable GPUs on MiniKF local? 14:25:11
@_slack_kubeflow_U035009NYQ2:matrix.orgEvans Kiplagat Thanks Alex Aidun will be checking it out. 14:29:29
@_slack_kubeflow_U02DWFSKS1F:matrix.orgAlex Aidun Evans Kiplagat sounds good, please share any feedback good luck! 14:31:41
@_slack_kubeflow_U031LC9DKKJ:matrix.orgMaciek Stopa hey Bilal Obeidat, can you show the error message? 15:55:23
12 May 2022
@_slack_kubeflow_U03BZ3KTBPX:matrix.orgBilal Obeidat Hi Maciek Stopa thank you for the response. simply the notebook server creation never complete. I believe the whl library was causing some weird gcc issue so I fixed it. for other reasons i had to use the latest arkito image gcr.io/arrikto/jupyter-kale-py38:release-1.5-l0-release-1.5-rc1-17-gef3a04d6a 07:34:37
@_slack_kubeflow_U03BZ3KTBPX:matrix.orgBilal ObeidatRedacted or Malformed Event07:37:40
@_slack_kubeflow_U03BZ3KTBPX:matrix.orgBilal ObeidatScreen Shot 2022-05-12 at 12.28.04 AM.png
Download Screen Shot 2022-05-12 at 12.28.04 AM.png
@_slack_kubeflow_U03BZ3KTBPX:matrix.orgBilal Obeidat I am getting the attached rpc error in the screen shot every time I click compile and run or save a pipeline. I am using a custom image based on the latest arkito image gcr.io/arrikto/jupyter-kale-py38:release-1.5-l0-release-1.5-rc1-17-gef3a04d6aI know it is not my code and i think it is the image because I used the titanic notebook example and I still get the 400 error. any idea? 17:42:25
@_slack_kubeflow_U02DWFSKS1F:matrix.orgAlex Aidun Hey Bilal Obeidat, I'm Dr. Education here at Arrikto - I've raised the question internally, can I ask are you using MiniKF on GCP or AWS? Are you aware that we have a free Kubeflow as a Service offering now so you don't have to worry about MiniKF installs or managing your resources? That can be found here if you are curious: kubeflow.arrikto.com 17:53:26
@_slack_kubeflow_U03BZ3KTBPX:matrix.orgBilal Obeidat Thanks Alex Aidun I am using it on AWS 17:56:50
@_slack_kubeflow_U03BZ3KTBPX:matrix.orgBilal Obeidat I will take a look. I am using a custom image because I need an internal library 18:00:07
13 May 2022
@_slack_kubeflow_U02DWFSKS1F:matrix.orgAlex Aidun Hey Bilal Obeidat sorry for the delay here, the response I got back from engineering is below: "MiniKF though doesn't have the latest Kubeflow version with all the required updates in the frontend/backend in order to support private(namespaced) pipelines. Kale by default in the latest images uploads private pipeline definitions (pipeline definition in specific namespace that cannot be accessed by others living in different ones). This procedure requires latest Kubeflow version. From the UI I'm not sure if they can avoid this actually. However, with Kale's CLI they can use the --shared-pipeline flag to overcome this issue." Does this advice help? Again sorry for delay! 12:24:05
@_slack_kubeflow_U03BZ3KTBPX:matrix.orgBilal Obeidat Thank you Alex Aidun this is helpful in explaining the issue. I still workaround it by generating the pipeline.yaml file and continue running it that way. I will see if using the Kale's cli is helpful here 16:06:54
17 May 2022
@_slack_kubeflow_U03FLV8QGG6:matrix.org_slack_kubeflow_U03FLV8QGG6 joined the room.15:37:44

There are no newer messages yet.

Back to Room List