Am I correct in assuming that there is the Kube Service IP routing happening via iptables DNAT to get the request into the Kube running the Ingress Controller, and then the Ingress Controller is on top of that routing traffic to another Service IP which also has to go through the iptables DNAT?
Thank you. So there is a DNAT to get to the Ingress Controller but from there at least it's direct routing to the service endpoint(s)? Does that mean the Virtual IP given to the Service is basically bypassed when using Ingress Controller?
TLS termination at the Ingress Controller and by default unencrypted from there to the service endpoint?
Regarding ways of updating of the NGINX upstreams without requiring a reload, I was just made aware of modules like ngx_dynamic_upstream[1]. I'm sure there are other ways to address this in a less disruptive way than reloading everything, so this is probably something that could be improved in the future.
May I ask how you are automating the ELB/TLS configuration and how that ties into the Ingress controller? Do you somehow specify which ELB it should use? We're in a similar situation.
You can annotate any Service of type LoadBalancer in order to configure various aspects[1] of the associated ELB, including which ACM-managed certificate you want to attach to each listener port.
