> I'm having a hard time mentally come up with a way a larger L1$ could be faster.
It’s faster because you’re comparing a two completely different physical implementations. SRAM does not use capacitors. DRAM does. You trade speed for density.
Except that L2 cache is also SRAM. In fact, usually all RAM inside the CPU is SRAM, because the process for making DRAM is not very compatible with the process for making SRAM and the rest of the CPU. We are only just recently been seeing devices which merge DRAM and a CPU into one package, but internally it is still on different chips (for example, the Apple M1 processor).
Also a few IBM designs. An interesting thing is that IBM claimed that eDRAM was faster than SRAM for their particular design because at the scale they were using it the reduced size of the array increased speed more than eDRAM's access time decreased it.
It’s faster because you’re comparing a two completely different physical implementations. SRAM does not use capacitors. DRAM does. You trade speed for density.