Our new bilingual protein language model (pLM), ProstT5, translates between protein sequence and structure. Besides producing more structure-aware embeddings that are better at remote homology detection than sequence-pLMs, its translation capability enables inverse folding.