28 Apr 2022 |
Suraj Kota | Is your teraform stack public? | 07:48:17 |
Thomas Ounnas | Not yet, but I plan to create a public version later.
I’m available for a call if you want to exchange about it 😉 | 07:50:46 |
Thomas Ounnas | I’ve also created a small python package to be able to run kfp stuff from outside cluster without cookies, using only kube jobs 🙂 | 07:59:30 |
Suraj Kota | Interesting stuff. Will find a time to chat | 08:27:20 |
Thomas Ounnas | And here the yq cmd for those interested 🙂
yq e ‘select(.kind == “ServiceAccount” and .metadata.name == “aws-load-balancer-controller”).metadata.annotations.“eks.amazonaws.com/role-arn”
,= env(ALB_SA_ROLE_ARN)
’ -i awsconfigs/common/aws-alb-ingress-controller/base/load_balancer_controller.yaml | 09:02:01 |
Thomas Ounnas | And Suraj Kota, quick question, do I really need to apply the istio-ingress https overlay as mentioned in the LB conf doc (aka kustomize) since I’ll apply the cognito overlay later ? 🤔 | 09:29:35 |
Suraj Kota | No, only need to follow the configure load balancer controller section | 10:56:05 |
Thomas Ounnas | Yes I saw, but this doc use a kustomize which install the https overlay, and since in the kustomization of the cognito we install the cognito one, I was just wondering if needed ? 🙂 | 11:41:49 |
Thomas Ounnas | By the way there is a dead link here: | 11:44:29 |
Thomas Ounnas | The url should point to “https://awslabs.github.io/kubeflow-manifests/docs/troubleshooting-aws/” I guess | 11:44:42 |
Thomas Ounnas | NB: I did succeed to deploy the release with my stack (no eksctl) adding the annotations as mentioned previously and I also add to increase CreatTags access statement in the provided policy, otherwise I had issues with my ingress | 12:58:16 |
29 Apr 2022 |
| Bogdan Kowalczyk joined the room. | 15:13:49 |
| Bogdan Kowalczyk changed their display name from _slack_kubeflow_U03E7U44FC0 to Bogdan Kowalczyk. | 15:14:53 |
| Bogdan Kowalczyk set a profile picture. | 15:15:01 |
jcostanzo | for people using kubeflow in a multi-tenanted fashion AND are attempting to calculate the effective costs per team, how are you handling it? Completely separate AWS accounts / EKS clusters for Kubeflow per team and relying on the natural separation there? Kubecost or some other tool to approximate cost per team? Something else? Or maybe nobody does this 🙂 | 20:09:01 |
30 Apr 2022 |
Vinay Chandrasekharan | We spin up separate node groups for each kubeflow profile/ tenant so that each tenant specific workloads are isolated - notebooks , pipelines or kserve pods | 12:52:14 |
jcostanzo | I'm assuming it's a manual process for you for adding new node groups and teams. How are you handling assigning users to whichever node groups they should be allowed to schedule jobs on? Something automated to setup pod defaults or is it still manual? | 12:58:30 |
Vinay Chandrasekharan | Automated thru pod defaults | 18:50:45 |
Haris Farooqui | Download Screen Shot 2022-04-30 at 1.55.36 PM.png | 21:04:57 |
Haris Farooqui | Kubeflow-pipeline version:1.8.1
Artifacts backed: S3
Issue: If the S3 bucket is not in default us-east-2 region following error is thrown:
Failed to get object in bucket pdx2-zpa-kubeflow-qa-intpol-qa at path artifacts/conditional-execution-pipeline-with-exit-handler-x8z92/conditional-execution-pipeline-with-exit-handler-x8z92-2049231942/main.log: S3Error: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2' | 21:04:57 |
Haris Farooqui | Tried version options as setting region in minio config.json, environment variables AWS_DEFAULT_REGION , MINIO_SERVICE_REGION in mi-pipline-ui deployement but nothing worked | 21:06:51 |
Haris Farooqui | Any suggestions? | 21:06:56 |
1 May 2022 |
| _slack_kubeflow_U03DQAW3Z36 joined the room. | 03:29:12 |
| @wybpip:matrix.org joined the room. | 16:48:38 |
| @wybpip:matrix.org left the room. | 16:48:39 |
2 May 2022 |
| _slack_kubeflow_U02L9TZTN64 joined the room. | 15:17:31 |
Rahul Kharse | Hi Haris, when setting up the s3 integration did you set the minioServiceRegion to the desired region?
https://github.com/awslabs/kubeflow-manifests/blob/main/awsconfigs/apps/pipeline/s3/params.env#L3 | 17:34:01 |
Gautam Kumar | Haris Farooqui In addition to what Rahul, said you can also check all steps described in the guide https://github.com/awslabs/kubeflow-manifests/blob/v1.4.1-aws-b1.0.0/docs/deployment/rds-s3/README.md | 18:52:50 |
| _slack_kubeflow_U03E5L0V0SV joined the room. | 19:29:25 |
Haris Farooqui | This should be sorted. Had to patch pipeline service:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ml-pipeline-ui
spec:
template:
metadata:
labels:
app: ml-pipeline-ui
spec:
containers:
- name: ml-pipeline-ui
env:
- name: AWS_REGION
value: ${region} | 19:41:15 |