Trying to understand network policies for host network <-> pod network

Help
#1

Hi!

While trying to configure global and namespaced network policies for my first time, I struggle to completely understand how to specify my selectors.

The specific issues currently is with pods running hostNetwork: true.
I’m trying to allow access to the cluster dns service in kube-system namespace.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: cluster-dns
spec:
  # TODO: doesn't work?
  # selector: k8s-app == 'kube-dns'
  types:
    - Ingress
  ingress:
    # Allow from host & pod networks
    - action: Allow
      protocol: UDP
      destination:
        ports:
          - 53
      source:
        nets:
          - 127.0.0.1/32
          - 10.96.0.0/16
          - 192.168.0.0/16

This works fine. But trying to use a selector to specify that only the destination of kube-dns pods should be allowed fails miserably.

At first I thought it would even work with a namespaced network policy targeting the pod specifically. But this seem to only work for pods not running with the host network.
I’ve also tried enabling the applyOnForward and preDNAT without luck. I didn’t however try to target the service cluster IP itself

This must be a common issue, for allowing the cluster dns to function?

#2

Another thing I discovered is that even without the above config, I’m allowed to query the cluster IP of the dns pods from the host itself as well as a different subnet than listed above (via wireguard).

But pods with hostNetwork: true can’t resolve dns for pulling their image.
I’m very confused

#3

hostNetwork: true tells Kubernetes “do not use Calico for this pod, treat it as part of the host itself” so Calico is not aware of host-networked pods and, in the networking sense, they’re not pods at all, they run as part of the host. You can make policy that matches traffic from them but you have to match traffic from that host.

#4

Hey, thanks for replying.

Yeah that’s what I figured. But I do have “auto endpoints” to create HEPs for interfaceName: *.
I would have thought that I could then let the pod IPs of kube-dns accept ingress from HEPs as well, by using “GlobalNetworkPolicy” and specifically select the k8s-app == kube-dns.

And I’m still confused why I’m able to talk to the kube-dns directly from the host as well as from a subnet on a wireguard interface (not a felix managed wireguard overlay) even without the above configuration, but the hostNetwork pod can’t (while resolving the docker image hostname).

I seem unable to block access to the kube-dns clusterIP to the subnet on the wireguard interface no matter what I do.

This type of config does however work as expected for a different service that is in the default namespace. Perhaps the cluster dns is handled in a special way.

The other similar config that does work looks like this:

otherdns.yaml 
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: otherdns
spec:
  selector: "!has(node-role.kubernetes.io/master)"
  types:
    - Ingress
  ingress:
    - action: Allow
      source:
        nets:
          - 10.1.2.3/16 # the wireguard subnet
      protocol: UDP
      destination:
        ports:
          - 53
        selector: app == 'otherdns-resolver'
#5

Ah, I think the problem is that you’re using wireguard and we have a known issue with host to remote pod connections.

Auto-heps do solve most of the issues here since they automatically contain the right IPs including all the tunnel IPs.

To explain the other issue you saw: host to local pod traffic is always allowed; this is for kubelet to access pods.

#6

oh ok. But it does work for accessing my own dns service on default namespace. While denying access to other services not listed in the policy as expected, except kube-dns.

I did make it block the kube dns for the wireguard subnet by specifically adding a deny like this:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: otherdns-default-deny
spec:
  applyOnForward: true
  preDNAT: true
  types:
    - Ingress
  ingress:
    - action: Deny
      source:
        nets:
          - 10.1.2.3/16 # the wireguard subnet
      destination:
        nets:
          - 10.96.0.10/32 # the kube-dns cluster IP

but if I try to delete the destination to become default deny while having applyOnForward. It will also deny the first Allow and basically make everything inaccessible for the wireguard subnet.

#7

And finally an off topic question,
is there a recommended way to calicoctl apply these policies that also removes ones that no longer exist in the provided yaml? kind of like kubectl apply --prune