Everything that I'd expect to 404 gives me 500s.

Probably using a framework (Django, for instance) that doesn't have a default 404 path.

If 404 template is absent, in the case of django it will show Not Found The requested URL /whypost was not found on this server.

Guessing by what curl -I says, looks like lighttpd + webpy.

BuiltWith shows it's using Rails.