To clarify, from my understanding, in a libertarian society, individuals and groups would be able to hire armed mercenary groups a la private security forces, but with presumably less regulation. That's what I mean by "monopoly on violence", eg only the ruling government has the right to use violence to enforce contracts.

That sounds like anarchism.

Most libertarians want the government to retain a monopoly on violence so that the institution that uses violence is accountable to the public.