Didn't you still run into times when most people wanted the time off? For example, times when schools are off, so everybody who is a parent wants that time off, regardless of whether it lines up with their cultural or religious important dates?
Not everyone has kids. Even if everyone has kids, most of the time employees will be in different school districts and have different holiday calendars.
