This is basically the same way as the basic primitive of an FPGA, a LUT (look-up table) works. It's a small ROM or RAM of size 2^N x 1, that is looked up by the N-bit "address" of the inputs. Modern FPGAs tend to use N=4 to N=6, with some additional fanciness occasionally present to make the up-to-64-bits of ROM/RAM useful in other ways as well.
VPTERNLOGD basically works by constructing a truth table for 3 inputs.
You pick the values you want for R, then pass this 8-bit value as the operand to the instruction along with the 3 values.For example, A ∧ B ∧ C would be 0b10000000. A ∧ ¬B ∧ ¬C would be 0xb00010000
There are 256 such tables and many of them can be represented by multiple boolean expressions.