I might say that there are an infinite number of copies of the image always there, all washing each other out. Each is a slightly different image (each has a slightly different centre of projection, in the jargon). If your pinhole is infinitely small, you get an infinitely-weak shot of a single one of those images. As you enlarge the pinhole, you get a larger collection of images, all added together, so you get more total light, but a bit of blurriness (because the images you're adding are all just a touch different).