Trying to make sense of Kubernetes networking in my own personal network at 1 AM on a Wednesday and somehow I feel like I may have made some poor decisions on how to spend my Tuesday night.

Follow

"run BGP at home", nobody said, "it'll be fun"

i haven't figured out what my problem is yet, but i think this particular problem _isn't_ BGP-related, but i won't say that too confidently

i just want one of my kubernetes pods to talk to my unifi cloud key so it can scrape metrics, but also keep my homelab stuff on a separate network, how hard can it be

the kubernetes nodes can all talk to the cloud key api, but for some reason pods cannot

Good news, I solved my problem and I still have internet access (my Kubernetes pod CIDR conflicted with the subnet the Cloud Key was running on).

Moving all my pods to a new subnet was surprisingly easy and pleasant. Just tell Calico to use a new subnet, disable the old one, roll all the pods.

I also switched things around so now Calico is doing the BGP announcing to my router instead of MetalLB, and then I also have the subnets added to my Tailscale network, so now in theory as long as I'm on my Tailscale network I can just talk "directly" to Kubernetes pods.

I am mildly amazed that I still have internet connectivity and didn't seem to break anything, except a short interruption in connectivity as routes were withdrawn and re-announced…

Tempted to switch back to my EdgeRouter this weekend to see if I can get ECMP working (the USG doesn't technically support BGP.

I guess there's not really much stopping me from having my EdgeRouter do all the internal routing for all my local subnets and then just have the USG be my gateway with all that entails…

Sign in to participate in the conversation
The Vulpine Club

The Vulpine Club is a friendly and welcoming community of foxes and their associates, friends, and fans! =^^=