A difference between conservative and libertarian conceptions of freedom, explained to me many years ago, is whether you think the fundamental unit of society is the family, or the individual.
In the conservative view, a family is a microcosm of a society and its shared future, and has interests across time in a way that an individual would not.
So, both the family and the individual would have rational selfish interests like "I want lower taxes to maximize our well being" or "I want my trade to be protected". And they might have ideological interests like "I want my kids to be raised in our faith".
But something like "I should have the freedom to use drugs" only makes sense for an individual - no family ever sits down at the kitchen table and chooses as their shared direction to allocate money in the budget for hard drugs.
When the locus of voting becomes individual expression, you get unaligned behavior, because pure individual freedom isn't a functional basis for a civilization across time.
The separation of voting by sex is somewhat destructive in the same way. Optimistically, men and women would each contribute their unique perspectives to a better shared future.
But what appeals to sex in voting are actually starting to look like with today's polarization are messages about how *your sex* has an identity, interests, and future of its own across time, like a permanent society, not half of a shared country.