This idea that people objecting to you means you are on the right path is also known as "they laughed at da Vinci, they laughed at Edison, and they laughed at bozo the clown" - don't just assume you're not the third one.
That's the most important point. Don't drop everything just for a project. But if you have looked at your idea from all angles, and you are just so convinced that it will work, then give it your all and go all in.
Have a look though that you still take care of yourself a bit and don't let your endeavors affect your family or friends too much/someone other than you, then it's all good.