Calico is not exposing services on self managed k8s cluster on aws using elb as VIP

Unable to curl or access any exposed service (NodePort Service , ClusterIP )

Below providing “kubeadm-config.yaml” used for bootstrapping cluster

apiVersion: kubeadm_dot_k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: “10.10.0.17”

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
podSubnet: 10.233.64.0/18
kubernetesVersion: v1.19.1
controlPlaneEndpoint: *******-k8s-lb-175805704.ap-south-1.elb.amazonaws.com:6443
useHyperKubeImage: False
apiServer:
extraArgs:
bind-address: 0.0.0.0
apiserver-count: “1”
endpoint-reconciler-type: lease
controllerManager:
extraArgs:
node-monitor-grace-period: 40s
node-monitor-period: 5s
pod-eviction-timeout: 5m0s
bind-address: 0.0.0.0
scheduler:
extraArgs:
bind-address: 0.0.0.0

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
bindAddress: 0.0.0.0

Below providing part of “calico.yaml” used for cluster


Source: calico/templates/calico-config.yaml

This ConfigMap is used to configure a self-hosted Calico installation.

kind: ConfigMap
apiVersion: v1
metadata:
name: calico-config
namespace: kube-system
data:

Typha is disabled.

typha_service_name: “none”

Configure the backend to use.

calico_backend: “bird”

Configure the MTU to use for workload interfaces and tunnels.

By default, MTU is auto-detected, and explicitly setting this field should not be required.

You can override auto-detection by providing a non-zero value.

veth_mtu: “0”

The CNI network configuration to install on each node. The special

values in this config will be automatically populated.

cni_network_config: |-
{
“name”: “k8s-pod-network”,
“cniVersion”: “0.3.1”,
“plugins”: [
{
“type”: “calico”,
“log_level”: “info”,
“log_file_path”: “/var/log/calico/cni/cni.log”,
“datastore_type”: “kubernetes”,
“nodename”: “KUBERNETES_NODE_NAME”,
“mtu”: CNI_MTU,
“ipam”: {
“type”: “calico-ipam”
},
“policy”: {
“type”: “k8s”
},
“kubernetes”: {
“kubeconfig”: “KUBECONFIG_FILEPATH
}
},
{
“type”: “portmap”,
“snat”: true,
“capabilities”: {“portMappings”: true}
},
{
“type”: “bandwidth”,
“capabilities”: {“bandwidth”: true}
}
]
}


output of “calicoctl get ippool -o yaml”

apiVersion: projectcalico.org/v3
items:

  • apiVersion: projectcalico.org/v3
    kind: IPPool
    metadata:
    creationTimestamp: “2021-03-23T12:16:44Z”
    name: default-ipv4-ippool
    resourceVersion: “717”
    uid: 0d6cb830-8cb9-4fee-8742-4e371890562d
    spec:
    blockSize: 26
    cidr: 10.233.64.0/18
    ipipMode: Always
    natOutgoing: true
    nodeSelector: all()
    vxlanMode: Never
    kind: IPPoolList
    metadata:
    resourceVersion: “2809”

Are you talking about kubernetes services with type: LoadBalancer ? If so, you’ll need to enable the AWS cloud integration.

See Cloud Providers | Kubernetes

Note that Calico provides pod networking and security, this sort of general kubernetes question is probably best answered on the kubernetes slack.

No , Unable to curl or access any exposed service (NodePort Service , ClusterIP ) even from inside of cluster

Can you curl direct between pod IPs?

Services are handled by kube-proxy, while Calico deals with networking pods. If you can curl between pods, that suggests that Calico is working correctly and that you should look at kube-proxy.

If pod-pod traffic isn’t working, check AWS security groups are allowing IPIP traffic (protocol 4) within the cluster.