Sender | Message | Time |
---|---|---|
21 May 2022 | ||
Vu Dat | Download Screen Shot 2022-05-21 at 12.53.01 PM.png | 05:53:52 |
Vu Dat | Hello guys, I am following this tutorial about Pipeline REST API but having error: https://www.kubeflow.org/docs/components/pipelines/tutorials/api-pipelines/ | 05:53:52 |
Benjamin Tan | Pipelines need to be associated with an experiment. So u need to create one first :) | 12:23:11 |
Benjamin Tan | Then pass in the experiment_id if I recall | 12:23:27 |
Vu Dat | Actually, I created one experiment. Do you give some examples of code? | 12:46:08 |
Benjamin Tan | There's another way u can observe the API. Try running a pipeline in the UI and observe the network request and response in the browser console. | 12:52:12 |
_slack_kubeflow_U03GF0NC9QS joined the room. | 14:12:20 | |
22 May 2022 | ||
_slack_kubeflow_UQZ58R1KJ joined the room. | 07:01:54 | |
mobin nikkhesal joined the room. | 11:59:12 | |
mobin nikkhesal changed their profile picture. | 17:00:12 | |
Yuki Iwai joined the room. | 17:17:03 | |
Steven Tobias | Newbie question here. using pipelines components v2, and I can't save artifacts for the life of me. Just want to upload a video of my env at the end of my pipeline.
I've tried multiple permutations of OutputPath[Artifact] Output[Artifact] etc, but getting various FileNotFound type errors.
Any tips as to how to format this? Here's my component code.
@component(base_image='gcr.io/foo/robocode-mlflow-run') def record_video(record_steps: int, output_path: OutputPath(Artifact)): import pathlib import os import gym import gym_robocode import string from stable_baselines3.common.vec_env import VecVideoRecorder, DummyVecEnv import random import boto3 from stable_baselines3 import A2C, PPO import logging import shutil video_length = record_steps video_folder = './videos' env_id = 'RobocodeDownSample-v2' env = DummyVecEnv([lambda: gym.make(env_id)]) model = PPO('CnnPolicy', env, verbose=1) obs = env.reset() # Record the video starting at the first step os.mkdir(output_path) env = VecVideoRecorder(env, video_folder, record_video_trigger=lambda x: x == 0, video_length=video_length, name_prefix=f"{output_path}/videos" + "-{}".format(env_id)) env.reset() for _ in range(video_length + 1): action, states = model.predict(obs) obs, , , = env.step(action) # Save the video env.close() for file in os.listdir(f"{output_path}/videos"): print(f"Trying to list file {file}") | 20:40:19 |
Steven Tobias | Oof, just a dumb mistake.
This iteration worked. The key for me was to use os.mkdir(output_path)
It was still failing because I was trying to list a non existent dir at the end 😓 | 20:43:08 |
23 May 2022 | ||
_slack_kubeflow_U02CKMPT47R changed their display name from _slack_kubeflow_U02CKMPT47R to Lize Cai. | 08:25:52 | |
_slack_kubeflow_U02CKMPT47R set a profile picture. | 08:25:53 | |
Nicolai Skogheim | In reply to@_slack_kubeflow_U03974MNYLB:matrix.org(edited) ```minikube addons enable gcp-auth export GOOGLE_APPLICATION_CREDENTIALS=<creds-path>.json ... => ```export GOOGLE_APPLICATION_CREDENTIALS=<creds-path>.json ... | 18:01:29 |
24 May 2022 | ||
Ferdinand von den Eichen [Kineo.ai] | Very nice, thanks for sharing! | 09:22:00 |
Ferdinand von den Eichen [Kineo.ai] | Does anyone know if its possible to run the KF pipelines mysql (and minio) on storage other than EBS (on AWS)? Some websites we checked seemed to suggest that the Mysql database application may require block storage (rather than EFS, which basically file system NFS) | 09:24:49 |
Benjamin Tan changed their profile picture. | 15:25:16 | |
droctothorpe | Why would you want to use EFS? It’s more expensive than EBS and you’ll probably run into performance issues. If anything, use RDS. | 18:25:39 |
droctothorpe | We just migrated to RDS for the metadata store. It’s pretty straightforward and provides a ton of advantages by virtue of it being externally managed. | 18:27:34 |
Ricardo Martinelli de Oliveira joined the room. | 18:50:19 | |
Clark Updike | Is there anyway to update a pipeline referenced in an experiment as a recurring run? If I bump the version of the pipeline to a later version, it doesn't seem to have any effect on the experiment which keeps running an older version. Hoping to not have to delete the recurring run experiment every time... | 20:53:11 |
25 May 2022 | ||
_slack_kubeflow_U02LW7FHWLS | I also looked into this, and unfortunately couldn't find anything/wound up needing to delete and recreate the recurring run with the kfp client. This is especially unfortunate since a network flake when running this could result in inadvertently deleting a recurring run without recreating it | 00:00:07 |
Ferdinand von den Eichen [Kineo.ai] | Our use case makes the kubeflow cluster(s) really fluent. On a given day we may be running 20 clusters and only 1 the next day, or none at all. In terms of cost efficiency it would be problematic to run 20 rds at all times, to support those clusters when they are up. We used EBS in the past, and it worked really well, however EBS has such huge limitations. Just the fact that it can only be attached to a single node, makes the whole setup much trickier (limitation to availability zone, scheduling tasks between nodes becomes harder). Regarding cost: we have found that EFS can be cheaper even, due to the nature of the pay per GB structure of EFS vs. EBS where you have to reserve 20, 30 GB and pay for it at all times. | 07:33:39 |
Ferdinand von den Eichen [Kineo.ai] | Follow up question on RDS though droctothorpe : did you try the serverless aurora variant? That can shutdown and spin up again by any chance? | 07:34:57 |
Ferdinand von den Eichen [Kineo.ai] | That actually looks very promising for our use case 😍 | 07:46:39 |
Cornelis Boon | Hi, I would like to keep track of the status of different branches in my pipeline. Specifically, I would like to be able to gracefully die and send something to my db that a step in the branch has failed. Are there tools in kubeflow (or kubernetes) for this or should I just write in the step’s code some lines that gracefully exit on failure and updates the state in my db before throwing an error? | 11:33:50 |
Clark Updike | Hmmm, that's too bad. Ok, thanks for sharing your experience. | 11:39:20 |
droctothorpe | I did not mostly because I couldn't find any precedent online and didn't want to spend my "innovation tokens" there, but I don't see any reason why it wouldn't work. It's all MySQL, regardless of the underlying infra, after all. I recommend starting with the AWS RDS overlay manifests and documentation, it's the most comprehensive. | 13:20:23 |