They pay Amazon for that middle-man position already. And then Amazon gets to determine that it's more profitable for them to also compete with you in addition to taking the cut from your sales.
I find it interesting the FedEx hasn’t entered this market by purchase someone like Square and competing with Amazon. At this pace, left unchecked, Amazon is going to destroy their business as well.
I don't know about the US, in Europe FBA (Fulfillment by Amazon) is getting competition from affiliates from Hermes (the logistics arm of Otto Group, one Amazon's biggest competition in Germany) and others. Multi marketplace capability and cost being among the main reasons.