ARP is also nice and abstract and well-defined; it can bridge from any multi-endpoint subnet's layer-2 address to an IP address. Not sure if anyone actually uses it for non-802, but the generality has forced a clean design.
To add to your praises of DHCP - it can also configure routers, and is in fact the standard solution for that in IPv6. Instead of giving you one or several addresses for NAT through DHCP, it gives the router an address for itself, and also a prefix to assign to clients on its internal network. Super neat stuff, and a boon to administrators.
Also to nitpick your summary, because nitpicking is what I do - layering violations are a thing, but only in the same way that violating software abstraction barriers are a thing. Not a hard-and-fast rule, and sometimes if you're doing weird enough stuff you just have to do it.
Got to disagree. The level of abstraction is very useful as a means of swapping out one layer without changing the other technologies - e.g. running IP over point-to-point fiber, or AlohaNet, or 802, or carrier pigeon. Or running running ethernet over a phy with whatever ridiculous number of Mbps is the latest thing. (802.11, of course, has effectively zero phy/link distinction, but anything that has to deal with such high packet drop rates and negotiation of physical layer between endpoints is going to be a mess.)
There's an issue with the specific OSI layering, but that's higher in the stack: it has waaay too many layers at the top. Everything up to maybe the transport layer (TCP/UDP/SCTP) is very well delinked in most implementations, but the session/presentation/application layer distinctions are total BS.
They're a useful tool for understanding the mindset of the original developers, but as you go "up" in the layers, the division of responsibilities becomes more and more arbitrary, with a very sharp uptick after "layer 3".
But more importantly, the notion that routing and forwarding "belongs" in IP, because that's the layer 3 protocol --- that's just false. There's no validity to it, and lots of systems have built overlays with layer 3 function on top of UDP (which in the "layering" model is a "layer 4" protocol, but is really best thought of as an escape hatch with which to build any new system you want on top of IP).
1. layers are a thing (and while any given piece of hardware or software can be serving as an amalgam of any contiguous sequence of layers, you can still analyze the behavior of such a component as if it were N separate abstract components, one for each layer it embodies);
2. layering and layering violations are a thing, in the particular sense of code that intermingles and entangles the concerns of different network layers being automatically a design smell (e.g. OpenVPN smells because, rather than building a clean layer-1 circuit abstraction on top of a layer-4/5/7 stream, and then running a regular substrate-oblivious layer-2 on top, OpenVPN runs a "dirty" layer-2 implementation directly on top of a layer-7 protocol (HTTP), where the layer-2 implementation knows things about HTTP and uses HTTP features to signal layer-2 data, such that it can no longer freely interoperate with other layer-2 implementations);
3. but just going down the layer stack, repeating layers, is not a layering violation. You can build all the way up to a circuit-switching abstraction like TCP, and then put PPP on that to go down to layer 2, and come back up again, and that's not even bad engineering.
"1. layers are a thing (and while any given piece of hardware or software can be serving as an amalgam of any contiguous sequence of layers, you can still analyze the behavior of such a component as if it were N separate abstract components, one for each layer it embodies);"
* Path MTU discovery: For proper operation, TCP needs to know a link-layer property for each of the links between a source and destination.
This bypasses the IP layer, because IP fragmentation does not play well with TCP. On the other hand, TCP does not even see the concept of a "path" between the source and destination; IP may route each segment uniquely.
* TCP over wireless links: TCP makes the assumption that segment loss implies congestion; wireless links have the propensity to drop packets for a plethora of reasons that have nothing to do with congestion. Hey, it's a bad assumption, and there's work on congestion controls that don't make that assumption, but maybe we ought to ask Van Jacobson if life mightn't be easier if the link could tell the transport protocol, "My bad! That was me, I did that?"
* Path MTU discovery: that's part of the IP contract. IP provides an unreliable datagram service with an MTU that varies based on destination endpoint but will never be below 1280b (in IPv6 - IPv4 was 576b). IPv6 also wisely doesn't do fragmentation; sizing your packets correctly is the job of layer 4.
* TCP over wireless links: TCP's congestion control mechanism is a heuristic based on ever-evolving understanding of the characteristics of links in the wild. There are things that layer 3 can do that unambiguously get in layer 4's way (bufferbloat makes low-latency response unfeasible), but it's layer 4's job to deal with reliability and congestion control. (By the way - unlike LFNs, WiFi is actually not a pathological case for TCP congestion control and buffering. A good mental model for those periodic WiFi drops is of an Ethernet cable being disconnected and reconnected with a different one picked at random from a supply closet. In a lot of very common cases, when traffic gets passed again it will not be at the same throughput as before and so the endpoints need to rediscover the available throughput.)
To your more general suggestions about alternative designs: generally, schemes that have the link layer communicate with the endpoints using them scale BADLY to large internetworks, and the global internet is the largest.
What does "on UDP" mean? UDP is just a means of running an arbitrary datagram protocol that rides on top of IP; it's how you'd build a system that treats IP the way IP treats Ethernet.
Sure, but you mentioned protocols that have "built overlays with layer 3 function on top of UDP". What are the examples you're referring to?
EDIT: My comment in reply to the sibling comment, which mentioned vxlan:
That's more of a recursive version of the lower layers; using layers 1-4 of one instance of the OSI model as layer 2 of another instance. If anything, this demonstrates just how useful the clear abstraction barrier between layer 2 and layer 3 is; you can have a very complicated software package (like a VPN) as a layer 2 instead of a physical network and all the code from layer 3 up doesn't even need to know.
That's more of a recursive version of the lower layers; using layers 1-4 of one instance of the OSI model as layer 2 of another instance.
If anything, this demonstrates just how useful the clear abstraction barrier between layer 2 and layer 3 is; you can have a very complicated software package (like a VPN) as a layer 2 instead of a physical network and all the code from layer 3 up doesn't even need to know.
There are other models of modularity that make it easy to separate transport, routing, link, and physical protocols without starting from the assumption that "layer X can only interact with the minimum common denominator interface for layers X-1 and X+1". That assumption leads to everything from the PMTU discovery silliness to the pain of getting TCP to work correctly over links like wireless where packet loss does not imply congestion.
I've heard some folks talk about TLS as a "session" layer, and it is fortunate that we no longer have to translate between ASCII and EBCDIC underneath the application, so the "presentation" layer now seems like it is mis-named. Ah how times change.
To add to your praises of DHCP - it can also configure routers, and is in fact the standard solution for that in IPv6. Instead of giving you one or several addresses for NAT through DHCP, it gives the router an address for itself, and also a prefix to assign to clients on its internal network. Super neat stuff, and a boon to administrators.
Also to nitpick your summary, because nitpicking is what I do - layering violations are a thing, but only in the same way that violating software abstraction barriers are a thing. Not a hard-and-fast rule, and sometimes if you're doing weird enough stuff you just have to do it.