!GVzqqhOOirSFrkrnzX:matrix.org

platform-aws

229 Members
3 Servers

Load older messages


SenderMessageTime
28 Apr 2022
@_slack_kubeflow_U014PVBKWEQ:matrix.orgSuraj Kota Is your teraform stack public? 07:48:17
@_slack_kubeflow_UL38EU0BG:matrix.orgThomas Ounnas Not yet, but I plan to create a public version later. I’m available for a call if you want to exchange about it 😉 07:50:46
@_slack_kubeflow_UL38EU0BG:matrix.orgThomas Ounnas I’ve also created a small python package to be able to run kfp stuff from outside cluster without cookies, using only kube jobs 🙂 07:59:30
@_slack_kubeflow_U014PVBKWEQ:matrix.orgSuraj Kota Interesting stuff. Will find a time to chat 08:27:20
@_slack_kubeflow_UL38EU0BG:matrix.orgThomas Ounnas And here the yq cmd for those interested 🙂 yq e ‘select(.kind == “ServiceAccount” and .metadata.name == “aws-load-balancer-controller”).metadata.annotations.“eks.amazonaws.com/role-arn ,= env(ALB_SA_ROLE_ARN) ’ -i awsconfigs/common/aws-alb-ingress-controller/base/load_balancer_controller.yaml 09:02:01
@_slack_kubeflow_UL38EU0BG:matrix.orgThomas Ounnas And Suraj Kota, quick question, do I really need to apply the istio-ingress https overlay as mentioned in the LB conf doc (aka kustomize) since I’ll apply the cognito overlay later ? 🤔 09:29:35
@_slack_kubeflow_U014PVBKWEQ:matrix.orgSuraj Kota No, only need to follow the configure load balancer controller section 10:56:05
@_slack_kubeflow_UL38EU0BG:matrix.orgThomas Ounnas Yes I saw, but this doc use a kustomize which install the https overlay, and since in the kustomization of the cognito we install the cognito one, I was just wondering if needed ? 🙂 11:41:49
@_slack_kubeflow_UL38EU0BG:matrix.orgThomas Ounnas By the way there is a dead link here: 11:44:29
@_slack_kubeflow_UL38EU0BG:matrix.orgThomas Ounnas The url should point to “https://awslabs.github.io/kubeflow-manifests/docs/troubleshooting-aws/” I guess 11:44:42
@_slack_kubeflow_UL38EU0BG:matrix.orgThomas Ounnas NB: I did succeed to deploy the release with my stack (no eksctl) adding the annotations as mentioned previously and I also add to increase CreatTags access statement in the provided policy, otherwise I had issues with my ingress 12:58:16
29 Apr 2022
@_slack_kubeflow_U03E7U44FC0:matrix.orgBogdan Kowalczyk joined the room.15:13:49
@_slack_kubeflow_U03E7U44FC0:matrix.orgBogdan Kowalczyk changed their display name from _slack_kubeflow_U03E7U44FC0 to Bogdan Kowalczyk.15:14:53
@_slack_kubeflow_U03E7U44FC0:matrix.orgBogdan Kowalczyk set a profile picture.15:15:01
@_slack_kubeflow_U0139T1KGLR:matrix.orgjcostanzo for people using kubeflow in a multi-tenanted fashion AND are attempting to calculate the effective costs per team, how are you handling it? Completely separate AWS accounts / EKS clusters for Kubeflow per team and relying on the natural separation there? Kubecost or some other tool to approximate cost per team? Something else? Or maybe nobody does this 🙂 20:09:01
30 Apr 2022
@_slack_kubeflow_U034NP55CPQ:matrix.orgVinay Chandrasekharan We spin up separate node groups for each kubeflow profile/ tenant so that each tenant specific workloads are isolated - notebooks , pipelines or kserve pods 12:52:14
@_slack_kubeflow_U0139T1KGLR:matrix.orgjcostanzo I'm assuming it's a manual process for you for adding new node groups and teams. How are you handling assigning users to whichever node groups they should be allowed to schedule jobs on? Something automated to setup pod defaults or is it still manual? 12:58:30
@_slack_kubeflow_U034NP55CPQ:matrix.orgVinay Chandrasekharan Automated thru pod defaults 18:50:45
@_slack_kubeflow_U033RA6APQQ:matrix.orgHaris FarooquiScreen Shot 2022-04-30 at 1.55.36 PM.png
Download Screen Shot 2022-04-30 at 1.55.36 PM.png
21:04:57
@_slack_kubeflow_U033RA6APQQ:matrix.orgHaris Farooqui Kubeflow-pipeline version:1.8.1 Artifacts backed: S3 Issue: If the S3 bucket is not in default us-east-2 region following error is thrown:
Failed to get object in bucket pdx2-zpa-kubeflow-qa-intpol-qa at path artifacts/conditional-execution-pipeline-with-exit-handler-x8z92/conditional-execution-pipeline-with-exit-handler-x8z92-2049231942/main.log: S3Error: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
21:04:57
@_slack_kubeflow_U033RA6APQQ:matrix.orgHaris Farooqui Tried version options as setting region in minio config.json, environment variables AWS_DEFAULT_REGION , MINIO_SERVICE_REGION in mi-pipline-ui deployement but nothing worked 21:06:51
@_slack_kubeflow_U033RA6APQQ:matrix.orgHaris Farooqui Any suggestions? 21:06:56
1 May 2022
@_slack_kubeflow_U03DQAW3Z36:matrix.org_slack_kubeflow_U03DQAW3Z36 joined the room.03:29:12
@wybpip:matrix.org@wybpip:matrix.org joined the room.16:48:38
@wybpip:matrix.org@wybpip:matrix.org left the room.16:48:39
2 May 2022
@_slack_kubeflow_U02L9TZTN64:matrix.org_slack_kubeflow_U02L9TZTN64 joined the room.15:17:31
@_slack_kubeflow_U02H4ARNFEK:matrix.orgRahul Kharse Hi Haris, when setting up the s3 integration did you set the minioServiceRegion to the desired region? https://github.com/awslabs/kubeflow-manifests/blob/main/awsconfigs/apps/pipeline/s3/params.env#L3 17:34:01
@_slack_kubeflow_UK2BNJCJW:matrix.orgGautam Kumar Haris Farooqui In addition to what Rahul, said you can also check all steps described in the guide https://github.com/awslabs/kubeflow-manifests/blob/v1.4.1-aws-b1.0.0/docs/deployment/rds-s3/README.md 18:52:50
@_slack_kubeflow_U03E5L0V0SV:matrix.org_slack_kubeflow_U03E5L0V0SV joined the room.19:29:25
@_slack_kubeflow_U033RA6APQQ:matrix.orgHaris Farooqui This should be sorted. Had to patch pipeline service:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-pipeline-ui
spec:
  template:
    metadata:
      labels:
        app: ml-pipeline-ui
    spec:
      containers:
        - name: ml-pipeline-ui
          env:
            - name: AWS_REGION
              value: ${region}
19:41:15

Show newer messages


Back to Room ListRoom Version: 6