One of the biggest hurdles in the manufacturing of dirty bombs is that they are made from highly active sources. You can more or less safely manipulate a plutonium ball to manufacture a conventional a-bomb. But you just can't do it with highly active sources like cesium-137 without a significant amount of shielding during assembly and transportation. With this kind of source, acute radiation poisoning is a matter of hours, even minutes.