!caaCTdhYlLGpbJVvly:matrix.org

kubeflow-pipelines

739 Members
2 Servers

Load older messages


SenderMessageTime
3 Jun 2022
@_slack_kubeflow_U03HVCQDWVB:matrix.orgBalu set a profile picture.08:53:26
@_slack_kubeflow_U03HVCQDWVB:matrix.orgBalu Can someone help me on How to add label to dsl.ResourceOp. That label value must be dynamic one. 08:53:27
@_slack_kubeflow_U032CK41Z8V:matrix.orgNSLog0 any experience with GCP Transfer jobs service? I have a requirement to build pipeline which related with Transfer jobs. the job will be sync data from s3 to gcs daily. after data sync done then next pipeline will be run after. 1. sync data -> wait until sync done 2. (train data after sync done) Does the pipeline can work with Transfer jobs like above scenario ? 11:02:01
@_slack_kubeflow_U01A4GE3FFB:matrix.orgBrett Koonce i have done that 13:24:43
@_slack_kubeflow_U01A4GE3FFB:matrix.orgBrett Koonce i believe you can have gcp emit an event after the transfer is done 13:24:54
@_slack_kubeflow_U01A4GE3FFB:matrix.orgBrett Koonce you could use that to trigger your job 13:25:03
@_slack_kubeflow_U01A4GE3FFB:matrix.orgBrett Koonce if you don't have a ton of data changes 13:25:17
@_slack_kubeflow_U01A4GE3FFB:matrix.orgBrett Koonce then you might do a simple cron 13:25:25
@_slack_kubeflow_U01A4GE3FFB:matrix.orgBrett Koonce eg sync at 12am 13:25:30
@_slack_kubeflow_U01A4GE3FFB:matrix.orgBrett Koonce run job at 12 pm 13:25:34
@_slack_kubeflow_UKM89RPD0:matrix.orgdroctothorpe Very cool. Any plans to open source that logic? If not, no worries. We can always reinvent the wheel, heh. That script sounds like a good candidate for an upstream contribution IMO. 13:27:28
@_slack_kubeflow_U01K24XKKK9:matrix.orgJoseph Olaide Hi Nicholas Kosteski, I noticed it shows the output artifact as soon as the pipeline run ends however after some time, it disappears. I don't know why this is happening though. Trying to figure out the reason 20:35:11
@_slack_kubeflow_U0389FWMZ8X:matrix.orgJames Harr Hi, I'm writing components and pipelines using the Python v2 SDK (kfp==1.8.12) so I have several functions that are decorated with @component and then a @pipeline that assembles the component functions. I'm attempting to use set_env_variable to assign a runtime environment variable with a value obtained from a different environment variable that is available when the pipeline is compiled.
import os

import kfp
from kfp.v2 import dsl
from kfp.v2.dsl import component


@component
def do_a_thing(input: str):
    import os

    print(f"--->>> {input} {os.getenv('MY_ENV_VAR')}")


@dsl.pipeline(name="A name", description="A description")
def my_pipeline(start_with_this: str):
    the_thing = do_a_thing(start_with_this).set_env_variable(
        "MY_ENV_VAR", os.getenv("ENV_VAR_FROM_BUILD")
    )


def compile_pipeline():
    print(f"--->>> {os.getenv('ENV_VAR_FROM_BUILD')}")
    kfp.compiler.Compiler(mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE).compile(
        pipeline_func=my_pipeline, package_path="my_pipeline.yaml"
    )


if __name__ == "__main__":
    compile_pipeline()
If I save this file as experiment.py and then compile this pipeline to YAML with the following command, ENV_VAR_FROM_BUILD=Hello python experiment.py, I get a YAML file that looks like a pipeline I can run. However, I don't see my compile-time environment variable MY_ENV_VAR or the value (Hello) in the YAML file (but I do see the print statement that show the compile-time value of ENV_VAR_FROM_BUILD). Shouldn't it be there in the env: section of the do-a-thing component? Is there a better way to achieve what I'm trying to do?
21:29:04
4 Jun 2022
@_slack_kubeflow_U02QG484SPM:matrix.orgAndrej Albrecht changed their display name from _slack_kubeflow_U02QG484SPM to Andrej Albrecht.18:35:43
@_slack_kubeflow_U02QG484SPM:matrix.orgAndrej Albrecht set a profile picture.18:35:48
5 Jun 2022
@_slack_kubeflow_U03J0E58XJN:matrix.orgwafaa abdelhafez joined the room.13:20:17
6 Jun 2022
@clifontaive:matrix.orgclifontaive joined the room.06:00:13
@_slack_kubeflow_U03J0E58XJN:matrix.orgwafaa abdelhafez changed their display name from _slack_kubeflow_U03J0E58XJN to wafaa abdelhafez.13:06:59
@_slack_kubeflow_U03J0E58XJN:matrix.orgwafaa abdelhafez set a profile picture.13:07:02
@_slack_kubeflow_U03J0E58XJN:matrix.orgwafaa abdelhafez Hello everyone 🙂 , I hope every one is doing great. I just started to use kubeflow weeks ago, and I am facing some issues that I hope you can help me in it. I am using • kubeflow version 1.4 • created pipeline (load data, preprocess, create model , evaluate) my question is I want to make my model to show up in artifacts screen, I know how in kubeflow V2 but cannot get it in version 1.x. I can see model artifact in pipeline stage and cad download it but cannot see it in artifacts screen, dont' know if i have to save it in a specific way to the volume itself or something else. I hope my question is clear to get. Thanks in advance. 13:07:02
9 Jun 2022
@wyb:matrix.orgA-wandering-bot joined the room.16:44:35
@wyb:matrix.orgA-wandering-bot left the room.16:44:37
13 Jun 2022
@better_sleeping:converser.eubetter_sleeping joined the room.01:28:47
@better_sleeping:converser.eubetter_sleeping 01:30:02
@better_sleeping:converser.eubetter_sleeping left the room.01:30:05
24 Jun 2022
@frankcryptic:matrix.orgfrankcryptic joined the room.06:02:41
@clifontaive:matrix.orgclifontaive left the room.14:21:57
29 Jun 2022
@frankcryptic:matrix.orgfrankcrypticI'll help 10 individuals how to earn $20,000 in just 72 hours from the crypto market. But you will pay me 10% commission when you receive your profit. if interested send me a direct message via Whatapp by asking me HOW for more details on how to get started +1 (570) 801-086201:40:20
1 Jul 2022
@frankcryptic:matrix.orgfrankcryptic left the room.14:38:11
16 Jul 2022
@tokenshub:matrix.orgtokenshub joined the room.23:12:20

Show newer messages


Back to Room ListRoom Version: 6