3 Jun 2022 |
| Balu set a profile picture. | 08:53:26 |
Balu | Can someone help me on How to add label to dsl.ResourceOp . That label value must be dynamic one. | 08:53:27 |
NSLog0 | any experience with GCP Transfer jobs service? I have a requirement to build pipeline which related with Transfer jobs. the job will be sync data from s3 to gcs daily. after data sync done then next pipeline will be run after.
1. sync data -> wait until sync done
2. (train data after sync done)
Does the pipeline can work with Transfer jobs like above scenario ? | 11:02:01 |
Brett Koonce | i have done that | 13:24:43 |
Brett Koonce | i believe you can have gcp emit an event after the transfer is done | 13:24:54 |
Brett Koonce | you could use that to trigger your job | 13:25:03 |
Brett Koonce | if you don't have a ton of data changes | 13:25:17 |
Brett Koonce | then you might do a simple cron | 13:25:25 |
Brett Koonce | eg sync at 12am | 13:25:30 |
Brett Koonce | run job at 12 pm | 13:25:34 |
droctothorpe | Very cool. Any plans to open source that logic? If not, no worries. We can always reinvent the wheel, heh. That script sounds like a good candidate for an upstream contribution IMO. | 13:27:28 |
Joseph Olaide | Hi Nicholas Kosteski, I noticed it shows the output artifact as soon as the pipeline run ends however after some time, it disappears. I don't know why this is happening though. Trying to figure out the reason | 20:35:11 |
James Harr | Hi, I'm writing components and pipelines using the Python v2 SDK (kfp==1.8.12) so I have several functions that are decorated with @component and then a @pipeline that assembles the component functions. I'm attempting to use set_env_variable to assign a runtime environment variable with a value obtained from a different environment variable that is available when the pipeline is compiled.
import os
import kfp
from kfp.v2 import dsl
from kfp.v2.dsl import component
@component
def do_a_thing(input: str):
import os
print(f"--->>> {input} {os.getenv('MY_ENV_VAR')}")
@dsl.pipeline(name="A name", description="A description")
def my_pipeline(start_with_this: str):
the_thing = do_a_thing(start_with_this).set_env_variable(
"MY_ENV_VAR", os.getenv("ENV_VAR_FROM_BUILD")
)
def compile_pipeline():
print(f"--->>> {os.getenv('ENV_VAR_FROM_BUILD')}")
kfp.compiler.Compiler(mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE).compile(
pipeline_func=my_pipeline, package_path="my_pipeline.yaml"
)
if __name__ == "__main__":
compile_pipeline()
If I save this file as experiment.py and then compile this pipeline to YAML with the following command, ENV_VAR_FROM_BUILD=Hello python experiment.py , I get a YAML file that looks like a pipeline I can run. However, I don't see my compile-time environment variable MY_ENV_VAR or the value (Hello) in the YAML file (but I do see the print statement that show the compile-time value of ENV_VAR_FROM_BUILD ). Shouldn't it be there in the env: section of the do-a-thing component? Is there a better way to achieve what I'm trying to do? | 21:29:04 |
4 Jun 2022 |
| Andrej Albrecht changed their display name from _slack_kubeflow_U02QG484SPM to Andrej Albrecht. | 18:35:43 |
| Andrej Albrecht set a profile picture. | 18:35:48 |
5 Jun 2022 |
| wafaa abdelhafez joined the room. | 13:20:17 |
6 Jun 2022 |
| @clifontaive:matrix.org joined the room. | 06:00:13 |
| wafaa abdelhafez changed their display name from _slack_kubeflow_U03J0E58XJN to wafaa abdelhafez. | 13:06:59 |
| wafaa abdelhafez set a profile picture. | 13:07:02 |
wafaa abdelhafez | Hello everyone 🙂 , I hope every one is doing great.
I just started to use kubeflow weeks ago, and I am facing some issues that I hope you can help me in it.
I am using
• kubeflow version 1.4
• created pipeline (load data, preprocess, create model , evaluate)
my question is I want to make my model to show up in artifacts screen, I know how in kubeflow V2 but cannot get it in version 1.x.
I can see model artifact in pipeline stage and cad download it but cannot see it in artifacts screen, dont' know if i have to save it in a specific way to the volume itself or something else.
I hope my question is clear to get. Thanks in advance. | 13:07:02 |
9 Jun 2022 |
| @wyb:matrix.org joined the room. | 16:44:35 |
| @wyb:matrix.org left the room. | 16:44:37 |
13 Jun 2022 |
| @better_sleeping:converser.eu joined the room. | 01:28:47 |
| @better_sleeping:converser.eu | 01:30:02 |
| @better_sleeping:converser.eu left the room. | 01:30:05 |
24 Jun 2022 |
| @frankcryptic:matrix.org joined the room. | 06:02:41 |
| @clifontaive:matrix.org left the room. | 14:21:57 |
29 Jun 2022 |
@frankcryptic:matrix.org | I'll help 10 individuals how to earn $20,000 in just 72 hours from the crypto market. But you will pay me 10% commission when you receive your profit. if interested send me a direct message via Whatapp by asking me HOW for more details on how to get started
+1 (570) 801-0862 | 01:40:20 |
1 Jul 2022 |
| @frankcryptic:matrix.org left the room. | 14:38:11 |
16 Jul 2022 |
| tokenshub joined the room. | 23:12:20 |