Humans are sexually dimorphic, which does only mean there are sex-driven differences. Dimorphism is not limited to chromosomes. And there are actually a large number of traits on which you could draw bimodal graphs (=something with 2, you could almost say binary) affected by this.
The traits' distributions actually stack and bimodality in a set is a strong indicator, that you actually have two sets.
Height distribution looks like this, the distance of the peaks is almost 2 standarddeviations and even the standard deviations are different with men having a bigger one.