I like wearing tailoring in wool, as it has a bounciness not present in cotton. That's because wool fibers have a natural crimp to them. You can feel this when you wear a wool suit or sport coat. Put your hands in the hip pockets and you'll notice how you can push down. When you remove you hands, the jacket springs back up. It's also something you can feel when you move around. When you go to hug someone, the back seam stretches comfortably.
Since cotton fibers don't have this natural crimp, they will feel much stiffer as a suit, sport coat, or pair of trousers. If you put your hands in a cotton jacket's hip pockets, they won't stretch down or bounce up. If you go to hug someone, the back seam can feel a bit tight. This is why good tailors typically cut cotton and linen suits a bit bigger than their wool counterparts.
But if you add just a touch of stretch, you get those same qualities. I don't think stretch cottons have to be low quality, a much depends on how the fibers were produced and how the fabric was made. For instance, Zegna offers stretch blends in their Winter Cottons book. I find they're quite good. Harrison's of Edinburgh is also about to introduce a cotton-stretch blend as a moleskin this winter. I commissioned a pair of moleskin trousers about 15 years ago and never wore them, always preferring the wool options in my wardrobe instead.
I wish there was a cotton-wool or cotton-stretch option for corduroy. Loro Piana does cotton-cashmere corduroy, but I've heard it's not very durable.
I've also heard from a bespoke tailor that stretch fabrics are better for industrial production, as the methods there are more controlled. For benchmade tailoring, a tailor's hot and heavy iron and break the elastane and make the cloth misshapen.
Stretch affects how a garment can be repaired. For instance, if you get stretch jeans, they will be harder to darn. But I don't darn my suits and I have so many clothes, each item only gets worn a few time a year, so wear is not an issue. To me, the most important part of a suit is whether it fits and flatters. To me, adding a touch of stretch to cotton doesn't really affect this, assuming your tailor doesn't run into the aforementioned issue.