Our approach improves the performance of existing representation learning methods by up to 8%. We also observe higher performance than a fully-supervised model in some cases. So, HistoPerm provides a viable approach for jointly using unlabeled and labeled data.