!caaCTdhYlLGpbJVvly:matrix.org

kubeflow-pipelines

741 Members
2 Servers

Load older messages


SenderMessageTime
21 May 2022
@_slack_kubeflow_U03FGUC1VGA:matrix.orgVu DatScreen Shot 2022-05-21 at 12.53.01 PM.png
Download Screen Shot 2022-05-21 at 12.53.01 PM.png
05:53:52
@_slack_kubeflow_U03FGUC1VGA:matrix.orgVu Dat Hello guys, I am following this tutorial about Pipeline REST API but having error: https://www.kubeflow.org/docs/components/pipelines/tutorials/api-pipelines/ 05:53:52
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Pipelines need to be associated with an experiment. So u need to create one first :) 12:23:11
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan Then pass in the experiment_id if I recall 12:23:27
@_slack_kubeflow_U03FGUC1VGA:matrix.orgVu Dat Actually, I created one experiment. Do you give some examples of code? 12:46:08
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan There's another way u can observe the API. Try running a pipeline in the UI and observe the network request and response in the browser console. 12:52:12
@_slack_kubeflow_U03GF0NC9QS:matrix.org_slack_kubeflow_U03GF0NC9QS joined the room.14:12:20
22 May 2022
@_slack_kubeflow_UQZ58R1KJ:matrix.org_slack_kubeflow_UQZ58R1KJ joined the room.07:01:54
@_slack_kubeflow_U03GUM55T0R:matrix.orgmobin nikkhesal joined the room.11:59:12
@_slack_kubeflow_U03GUM55T0R:matrix.orgmobin nikkhesal changed their profile picture.17:00:12
@_slack_kubeflow_U01QLMVBSMU:matrix.orgYuki Iwai joined the room.17:17:03
@_slack_kubeflow_U03FWQYAPR7:matrix.orgSteven Tobias Newbie question here. using pipelines components v2, and I can't save artifacts for the life of me. Just want to upload a video of my env at the end of my pipeline. I've tried multiple permutations of OutputPath[Artifact] Output[Artifact] etc, but getting various FileNotFound type errors. Any tips as to how to format this? Here's my component code.
@component(base_image='gcr.io/foo/robocode-mlflow-run')
def record_video(record_steps: int, output_path: OutputPath(Artifact)):
    import pathlib
    import os
    import gym
    import gym_robocode
    import string
    from stable_baselines3.common.vec_env import VecVideoRecorder, DummyVecEnv
    import random
    import boto3
    from stable_baselines3 import A2C, PPO
    import logging
    import shutil

    video_length = record_steps

    video_folder = './videos'
    env_id = 'RobocodeDownSample-v2'
    env = DummyVecEnv([lambda: gym.make(env_id)])

    model = PPO('CnnPolicy', env, verbose=1)

    obs = env.reset()
    # Record the video starting at the first step
    os.mkdir(output_path)
    env = VecVideoRecorder(env, video_folder,
                           record_video_trigger=lambda x: x == 0, video_length=video_length,
                           name_prefix=f"{output_path}/videos" + "-{}".format(env_id))
    env.reset()
    for _ in range(video_length + 1):
        action, states = model.predict(obs)
        obs, , ,  = env.step(action)
    # Save the video
    env.close()
    for file in os.listdir(f"{output_path}/videos"):
        print(f"Trying to list file {file}")
20:40:19
@_slack_kubeflow_U03FWQYAPR7:matrix.orgSteven Tobias Oof, just a dumb mistake. This iteration worked. The key for me was to use os.mkdir(output_path) It was still failing because I was trying to list a non existent dir at the end 😓 20:43:08
23 May 2022
@_slack_kubeflow_U02CKMPT47R:matrix.org_slack_kubeflow_U02CKMPT47R changed their display name from _slack_kubeflow_U02CKMPT47R to Lize Cai.08:25:52
@_slack_kubeflow_U02CKMPT47R:matrix.org_slack_kubeflow_U02CKMPT47R set a profile picture.08:25:53
@_slack_kubeflow_U03974MNYLB:matrix.orgNicolai Skogheim
In reply to@_slack_kubeflow_U03974MNYLB:matrix.org
Hello guys! New to kubeflow-pipeline, and struggling to set up a pipeline_root. I have a full kubeflow installation on minikube on local machine. I have followed a tutorial [1] that creates a simple component that needs to write to pipeline_root but I cannot get it to work. I have tried: • Using a Google Cloud Storage bucket, but this fails because I don't know how to give kubeflow access to it. I can specify a kubernetes service account when creating the run, but that doesn't help. • Using an existing Volume, but I get errors Not being able to write to disk at all prevents me from using kubeflow locally. How do you guys do it? (Pipelines not depending on pipieline_root work fine btw) [1] https://www.kubeflow.org/docs/components/pipelines/sdk-v2/build-pipeline/
(edited) ```minikube addons enable gcp-auth export GOOGLE_APPLICATION_CREDENTIALS=<creds-path>.json ... => ```export GOOGLE_APPLICATION_CREDENTIALS=<creds-path>.json ...
18:01:29
24 May 2022
@_slack_kubeflow_U015G7WJBUJ:matrix.orgFerdinand von den Eichen [Kineo.ai] Very nice, thanks for sharing! 09:22:00
@_slack_kubeflow_U015G7WJBUJ:matrix.orgFerdinand von den Eichen [Kineo.ai] Does anyone know if its possible to run the KF pipelines mysql (and minio) on storage other than EBS (on AWS)? Some websites we checked seemed to suggest that the Mysql database application may require block storage (rather than EFS, which basically file system NFS) 09:24:49
@_slack_kubeflow_UM56LA7N3:matrix.orgBenjamin Tan changed their profile picture.15:25:16
@_slack_kubeflow_UKM89RPD0:matrix.orgdroctothorpe Why would you want to use EFS? It’s more expensive than EBS and you’ll probably run into performance issues. If anything, use RDS. 18:25:39
@_slack_kubeflow_UKM89RPD0:matrix.orgdroctothorpe We just migrated to RDS for the metadata store. It’s pretty straightforward and provides a ton of advantages by virtue of it being externally managed. 18:27:34
@_slack_kubeflow_U03GTL2Q3HR:matrix.orgRicardo Martinelli de Oliveira joined the room.18:50:19
@_slack_kubeflow_U02AX5UF7L0:matrix.orgClark Updike Is there anyway to update a pipeline referenced in an experiment as a recurring run? If I bump the version of the pipeline to a later version, it doesn't seem to have any effect on the experiment which keeps running an older version. Hoping to not have to delete the recurring run experiment every time... 20:53:11
25 May 2022
@_slack_kubeflow_U02LW7FHWLS:matrix.org_slack_kubeflow_U02LW7FHWLS I also looked into this, and unfortunately couldn't find anything/wound up needing to delete and recreate the recurring run with the kfp client. This is especially unfortunate since a network flake when running this could result in inadvertently deleting a recurring run without recreating it 00:00:07
@_slack_kubeflow_U015G7WJBUJ:matrix.orgFerdinand von den Eichen [Kineo.ai] Our use case makes the kubeflow cluster(s) really fluent. On a given day we may be running 20 clusters and only 1 the next day, or none at all. In terms of cost efficiency it would be problematic to run 20 rds at all times, to support those clusters when they are up. We used EBS in the past, and it worked really well, however EBS has such huge limitations. Just the fact that it can only be attached to a single node, makes the whole setup much trickier (limitation to availability zone, scheduling tasks between nodes becomes harder). Regarding cost: we have found that EFS can be cheaper even, due to the nature of the pay per GB structure of EFS vs. EBS where you have to reserve 20, 30 GB and pay for it at all times. 07:33:39
@_slack_kubeflow_U015G7WJBUJ:matrix.orgFerdinand von den Eichen [Kineo.ai] Follow up question on RDS though droctothorpe : did you try the serverless aurora variant? That can shutdown and spin up again by any chance? 07:34:57
@_slack_kubeflow_U015G7WJBUJ:matrix.orgFerdinand von den Eichen [Kineo.ai] That actually looks very promising for our use case 😍 07:46:39
@_slack_kubeflow_U030D1EJY69:matrix.orgCornelis Boon Hi, I would like to keep track of the status of different branches in my pipeline. Specifically, I would like to be able to gracefully die and send something to my db that a step in the branch has failed. Are there tools in kubeflow (or kubernetes) for this or should I just write in the step’s code some lines that gracefully exit on failure and updates the state in my db before throwing an error? 11:33:50
@_slack_kubeflow_U02AX5UF7L0:matrix.orgClark Updike Hmmm, that's too bad. Ok, thanks for sharing your experience. 11:39:20
@_slack_kubeflow_UKM89RPD0:matrix.orgdroctothorpe I did not mostly because I couldn't find any precedent online and didn't want to spend my "innovation tokens" there, but I don't see any reason why it wouldn't work. It's all MySQL, regardless of the underlying infra, after all. I recommend starting with the AWS RDS overlay manifests and documentation, it's the most comprehensive. 13:20:23

Show newer messages


Back to Room ListRoom Version: 6