Our bottleneck was serialization of objects to bytes to send to etcd. Etcd cpu should be about 0.01-0.001 control plane CPU, and control plane apiserver CPU has been dominated by serialization since day 1 (except for brief periods around when we introduced inefficient custom resource code, because of the generic nature of that code).
Huh, I'd have guessed that distributed synchronization of etcd (which goes over the network and has multiple events, and uses raft, which has unbounded worst case latency) would be the limiting step.
