I had a very similar experience in NW New Mexico. I expect to go off pavement for this sort of thing, but eventually we faded into what was very uneven dirt. Luckily we could tell someone had come that way in the last day because of an inch or so of fresh snow, so we made it through successfully to the destination where everything was well maintained. We went out the more common route south, but it wasn't that much better.

https://goo.gl/maps/EFik9xuw9o8j9SaM7