There was never a time when people had to fear for their life for not wanting to work at a concentration camp. It was a volunteer assignment. Transferring to a different assignment would lead to a loss of status, but not serious punishment.