You don't have to carry it, no. But if you don't carry your ID (or equivalent) and they have a reason to ask for it (e.g. you are a suspect of a crime), they can require you to take them wherever you keep your ID or to the station.

I figured it worked the same in the Netherlands, but from what I can tell it's true that you always need to carry it and you can get fined if you don't. So quite a bit more strict. That sucks.

https://de.m.wikibooks.org/wiki/Rechte_und_Pflichten_im_Umga... (German)