Once you've agreed on who the eligible judges are, you can just have the plaintiff's lawyer and the defendant's lawyer meet up in a room and roll dice. This isn't a technical problem requiring advanced cryptography, it's a social or political problem.
Also, your method is biased towards lower-number judges when the number of judges is not a factor of the range of the output of your hash function. Use an existing library instead.
I mentioned as well it just was quick thought off the top my head. However, then you have the question can you trust the dice. It's a lot easier to trust math.
I could sit down try to figure out a construction that adjusts/accounts for any given N judges. Again though like I said this was just quick thought off the top of my head showing it seems possible to make a construction that can choose judges. Not something with every possible detail refined.
However, modular bias becomes smaller the larger your max number gets. We are not talking a small 32 bit random generator here. While it would be ideal to eliminate the bias it's not exactly easy. One could try the method of re-rolling/generating till you get an output in the target range. This has a massive problem because it is improbably at the scale of these numbers. Secondly, one can't control the modulo N in this case. There are things we can try to smear the bias, but that does not eliminate it.
Moreover, as mentioned before the scale of max output works in our favour here. The larger your max output/range is the smaller the bias is. Anyways there is a reason the CPRNGs will just truncate their output when you tell one to return X bytes. The range of possible bias is so small because how large cryptographic numbers are. Although some algorithms max output is a power 2 so the modulo N works out nicely.
The worst a bias like this can do is make partitioning cryptanalysis stronger, but unless you made some really bad choices generally should only be a theoretic break. At which point your designing primitives and not a protocol. However, that part we are using the modulus on is not secret so we are not gaining/losing anything there. For this instance it's just extremely tiny bias towards a certain judge.
I think it's very solvable: Given N numbered judges, you have both sides, perhaps plus an allegedly neutral third party, submit a number from 1 to N in a sealed envelope. Then, in full view of the public, someone opens the envelopes, and you add up the numbers and take the result mod N.
If even one of the parties submits a fairly chosen random number, and manages to conceal it from the others until they've submitted their numbers, then the result will be a fairly chosen random number. Then no one involved will have standing to complain that the result wasn't chosen fairly enough.
Also, your method is biased towards lower-number judges when the number of judges is not a factor of the range of the output of your hash function. Use an existing library instead.