"Fuzzing" is the technique for finding bugs by varying the inputs of a program and looking at how this affects code paths. A simpler place to start would be http://lcamtuf.coredump.cx/afl/