!caaCTdhYlLGpbJVvly:matrix.org

kubeflow-pipelines

741 Members
2 Servers

Load older messages


SenderMessageTime
25 May 2022
@_slack_kubeflow_UKM89RPD0:matrix.orgdroctothorpe Everything past provisioning is the same. All KF needs is an endpoint and credentials. 13:20:54
@_slack_kubeflow_U015G7WJBUJ:matrix.orgFerdinand von den Eichen [Kineo.ai] Yeah I looked into it earlier today. Seems doable. I might reach out to you if we run into major issues 👼 13:21:44
@_slack_kubeflow_UKM89RPD0:matrix.orgdroctothorpe Happy to help (assuming I can, heh). 13:22:08
@_slack_kubeflow_U03GW0Y4L3X:matrix.org_slack_kubeflow_U03GW0Y4L3X joined the room.15:41:16
@_slack_kubeflow_U03GTL2Q3HR:matrix.orgRicardo Martinelli de Oliveira changed their display name from _slack_kubeflow_U03GTL2Q3HR to Ricardo Martinelli de Oliveira.16:22:02
@_slack_kubeflow_U03GTL2Q3HR:matrix.orgRicardo Martinelli de Oliveira set a profile picture.16:22:04
@_slack_kubeflow_U03H39N6V28:matrix.org_slack_kubeflow_U03H39N6V28 joined the room.21:39:15
@_slack_kubeflow_U02LW7FHWLS:matrix.orgRahul Mehta If a recurring run is deleted from Kubeflow during scheduled execution, will Kubeflow lose track of the job? We've observed this behavior with a particular recurring run that's scheduled around the same time that we re-deploy recurring runs (we remove and re-schedule them since they involve different dependencies etc). 21:56:44
26 May 2022
@_slack_kubeflow_U01NF87E6NS:matrix.orgGerard Casas Saez Hi folks, qq, is there any estimate on dates for kfp==2.0.0rc0 release and 2.0.0 full support from backend (trying to undestand better the current story around this) 00:50:57
@_slack_kubeflow_U034BN4GGDB:matrix.orgPayman Touliat joined the room.01:46:11
@_slack_kubeflow_U03H27PMSMR:matrix.org_slack_kubeflow_U03H27PMSMR joined the room.06:03:21
@_slack_kubeflow_U0385A64B1B:matrix.orgAtra Akandeh Hello everyone. Could someone please direct me to a link that provides an example of pipeline that consume large amounts of data stored in Google Cloud Storage? or any suggestion on that. 14:07:55
@_slack_kubeflow_U03H4QLQ202:matrix.org_slack_kubeflow_U03H4QLQ202 joined the room.15:36:23
@_slack_kubeflow_U037P7VJ6MN:matrix.orgKishan Savant I think you can download the data using the URL from GCS and then pass on the path of the data using InputPath() and OutputPath(). Check out the following link. There are some examples with implementation that you can check out in the kubeflow/examples repo 17:19:31
@_slack_kubeflow_U0385A64B1B:matrix.orgAtra Akandeh Thanks, Kishan. Why would you not use pvc over InputPath()/OutputPath()? 23:19:19
27 May 2022
@_slack_kubeflow_U03C4M32V1U:matrix.orgChangshin Yoo changed their display name from _slack_kubeflow_U03C4M32V1U to Changshin Yoo.02:10:35
@_slack_kubeflow_U03C4M32V1U:matrix.orgChangshin Yoo set a profile picture.02:10:37
@_slack_kubeflow_U037P7VJ6MN:matrix.orgKishan Savant https://kubeflow.slack.com/archives/CE10KS9M4/p1645603722299249?thread_ts=1645458077.332789&cid=CE10KS9M4 05:17:18
@_slack_kubeflow_U023Z696N9E:matrix.orglaserK3000 joined the room.12:11:18
@_slack_kubeflow_U023Z696N9E:matrix.orglaserK3000 Hi, what is the intended way in v2 to output multiple files of a priori unknown number as an Artifact ? Do I have to compress everything into a single file or can I treat Output.path as a directory? I did the latter. I works (files end up in minio) but I wanted to know if this is intended behavior or has some caveats. One thing that did not work with this approach though was to download the artifact via the UI. 12:31:00
@_slack_kubeflow_U034KDRM3U7:matrix.orgChase Christensen laserK3000 I believe output path is supposed to be a directory. Everything is pushed to S3 (minIO) for outputs. Now you can choose the URL or S3 endpoint in V2 is my understanding. You should be able to look at the DAG and see where the output artifacts are stored. I think its a good approach. You can also get real spicy and do VolumeOps and just write to local volumes for marshaling and build pipelines that write to expected local directories. Then you aren't as reliant on Minio and can just write plain ol Python functions and drop stuff wherever without worrying about how KFP handles marshaling. 14:01:17
@_slack_kubeflow_U035NJ80VFS:matrix.org_slack_kubeflow_U035NJ80VFS joined the room.15:07:50
@_slack_kubeflow_U034329LRB2:matrix.org_slack_kubeflow_U034329LRB2 joined the room.15:41:31
28 May 2022
@_slack_kubeflow_U023Z696N9E:matrix.orglaserK3000 Thank you! 15:42:45
@_slack_kubeflow_U023Z696N9E:matrix.orglaserK3000 I used mount_pvc from kfp.onprem before but apparently this has not yet been ported to the final v2 implementation. That's why I was looking into using minio for passing large amounts of data. 15:45:48
@_slack_kubeflow_U034KDRM3U7:matrix.orgChase Christensen You can just create the volumOp and pass the object to steps 16:07:29
@_slack_kubeflow_U034KDRM3U7:matrix.orgChase Christensen Especially if you are using light weight python components. I'm away from my desk..i can get you an example when im back 16:08:02
29 May 2022
@_slack_kubeflow_U03GUM55T0R:matrix.orgmobin nikkhesal Hello everyone, When using kfp, I would like to pull images from a private container registry. Could you please explain how to set this up? --- kubeflow: 1.5 (wg distribution) Kubernetes: 1.21 (on perimise) kfp: 1.8 06:17:34
@_slack_kubeflow_U02JN8ZRU2E:matrix.orgYingding Wang Please take a look at this kubeflow notebook from which I start a pipeline with a component using a private container registry. https://github.com/careforrare/kf-pipelines/blob/main/demo-examples/pipeline_image_registry_builder_sdk_v2.ipynb I am still on kfp 1.7.0 kubeflow 1.4 und K8s 1.21 (on prem), but I think the prinzip for kf 1.5 should be the same. 09:58:33
@_slack_kubeflow_U03GUM55T0R:matrix.orgmobin nikkhesal thanks Yingding Wang 11:31:14

Show newer messages


Back to Room ListRoom Version: 6