Its cause "Hello world!" is multiple tokens and "A " would be one token. Its probably just hitting the repeat penalty for the same token making it unable to continue the sequence.

If its has nothing to go off yet it has to say something, I suppose it would just regurgitate its training data. Similar to the situation with the reddit usernames.