Releasing a basic game with the same mechanics/concepts before expanding on it through updates?
Well, that seems to be how many multiplayer games are released nowadays anyway. Look at ARMS or Mario Tennis Aces on the Nintendo Switch, or Pokemon GO, or Sea of Thieves* on Xbox. Released in very basic states, then slowly expanded upon via regular updates. You could even possibly say Minecraft went the same way.
That said, this sort of 'Minimum Viable Game' idea may not work as well here as it does for business products or web services. People are practically spoilt for choice when it comes to what games to buy, and a game that leaves an initial 'meh' impression (due to a lack of content/replay value) can often die out before the updates ever come. And the critics will certainly not be kind to it either...
* Admittedly, that one took four years to develop, which may not have been the best setup given the lack of content.
Other ways this is sometimes done are:
1. By releasing demos on a regular basis to test the waters
2. Splitting the game up into episodes and selling them one at a time. Valve did this with Half Life, but it was arguably TellTale Games who ran with it.
3. Or by running a beta test for the game and gauging reactions from that.
Of course, all the above assumes you can build at least a somewhat sizable portion of the game in a reasonable timeframe. If you want to know whether a completely untested idea will be viable... well good luck with that in this industry. You'll always need at least a core gameplay loop setup to know whether the idea is fun, and you'll need much more if you want to know whether anyone will buy it.
Looking at the industry they also have the approach of announcing or teasing things which then never get build. So thats probably a way big companies test too
Don't start by trying to test your idea. That's already getting ahead of yourself. Instead, start by identifying and checking the assumptions behind your idea.
For example, let's say you want to make a retro platformer. The assumption is that many people would be interested in a new retro platformer. That's easy to check: browse through Steam's new releases, find the 10 newest retro platformers, and see how much interest they get. Heck, even imagining that exercise could be enough to make you change course.
And yet Shovel Knight is doing so so great. Cuphead too. Turns out retro platformers can do great. But way too many indie retro platformers are just trash with nothing to save them.
I don't think you can gauge success by looking at existing titles. It's like asking if there's a market for scifi novels. What do you find? People like scifi novels, and there are lots of scifi novels that suck and nobody reads them. Do people like metal music? There's tons of successful metal bands, and there's an ocean of trash.
"Papers, Please" indie game wikipedia: Pope publicly shared details of the game's development from its onset, leading to high interest in the title and encouraging him to put more effort into it; though he initially planned to only spend a few weeks, Pope ended up spending about nine months on the game.
7DRLs[1] are a good example of prototyping gameplay. Some of them ultimately go on to become full fledged games. Cogmind [2] is an excellent example of this.
Both are prepared from a completed game. Except for some very specific genres, it's impossible to make a demo without building the whole game behind it. That's not a format you use to test a game idea.
