Calico with EMR on EKS

We are running Calico in our EKS clusters in AWS and are now trying to migrate our management of Spark jobs to EMR on EKS. It turns out that EMR on EKS generates service accounts with very long names and we are experiencing errors like this:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “6989d862f6a83506d31d36f39a9341979d8b1e3d689440003654ab52f37c03ea” network for pod “00000002u4k06332q3b-rs9bp”: networkPlugin cni failed to set up pod “00000002u4k06332q3b-rs9bp_ipedia-01” network: error with field Metadata.Labels (value) = ‘emr-containers-sa-spark-client-029354679212-2nuauonu2i0xe6wam1s74smvgqb0fyib5’ (must be no more than 63 characters)kubelet ip-10-135-235-56.ec2.internal

The service account name is: emr-containers-sa-spark-client-029354679212-2nuauonu2i0xe6wam1s74smvgqb0fyib5

I understand that service account is a selection criteria for network policy. I was wondering if it is possible to configure Calico to not provide service account as an option for network policies? If that is not possible does Calico have any recommendations regarding running EMR on EKS?

This isn’t really a Calico thing, its a Kubernetes thing. Resource names (including labels from the error above) must be less than 64 characters.

Removing text. It is a Kubernetes thing, but still wondering if Calico can not use service account in the label.

Some more info to see if I can get some traction. We are deploying Calico into AWS using the aws-calico helm chart. In the Calico docs regarding service accounts it mentions using service account labels as opposed to service account names. Is it possible to run the helm chart so that Calico uses labels instead of names to create network policies?

Apparently using the service account name is not optional:

Can a feature be added to make this optional?

Can you raise an enhancement issue for this at Issues · projectcalico/calico · GitHub please? Please explain the use case clearly so that it can be prioritised appropriately.

If I’m honest, at this point, I can’t see that this would be a high priority for the core calico team - but raising the issue provides somewhere for other users to say “Yes, I want this too” which will help bump it up the list.

And of course the issue would be a good place to ask the core team whether it would be accepted if you raised a PR adding this feature.

Posted the enhancement request:

Thanks.

I think the issue here is that kubernetes validates label values as max 63 chars but the rule for service account names is different; it allows a service account name to be up to 253 characters. For some reason label values must be “DNS labels” whereas service account names area allowed to be “DNS subdomains” ; i.e. a string of DNS labels separated by .. It appears that there is a bug in the validation in that it doesn’t check the length of the DNS labels that form the subdomain so the above name passes validation where it probabaly shouldn’t (because it has >63 chars without a .).