It could, and indeed there are people who hold this belief. They are called "Fruititarians" and only eat things which can be taken from the plant without killing or harming it, see: https://en.wikipedia.org/wiki/Fruitarianism

I don't personally subscribe to this.