All house structures depreciate in value over time. The land they sit on is what appreciates, usually dramatically more than the depreciation of the structure. The prefabs associated with poverty are prefab structures on rented land, so the only thing you own is the depreciating structure, not the appreciating land.
That is true of "manufactured homes", a.k.a. "trailers", but not of "prefab" or "modular" homes, which are built using the same components as conventional homes.
They're the same components but not. If you would frame a building and then build the wall assemblies traditionally your ability to integrate parts of the structure and seal it is different than when you put together assemblies and try to seal the thick boundaries between them. You also don't have any quality control - whatever shows up on site is installed.