ProtTrans: Understanding the Language of Life Through Self-Supervised Learning(pubmed.ncbi.nlm.nih.gov)
pubmed.ncbi.nlm.nih.gov
ProtTrans: Understanding the Language of Life Through Self-Supervised Learning
https://pubmed.ncbi.nlm.nih.gov/34232869/
https://pubmed.ncbi.nlm.nih.gov/34232869/
Here are quotes from the abstract:
> we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids.
> The protein LMs (pLMs) were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores.
> Dimensionality reduction revealed that the raw pLM-embeddings from unlabeled data captured some biophysical features of protein sequences.
> We validated the advantage of using the embeddings as exclusive input for several subsequent tasks: (1) a per-residue (per-token) prediction of protein secondary structure (3-state accuracy Q3=81%-87%); (2) per-protein (pooling) predictions of protein sub-cellular location (ten-state accuracy: Q10=81%) and membrane vs. water-soluble (2-state accuracy Q2=91%).
> For secondary structure, the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without multiple sequence alignments (MSAs) or evolutionary information thereby bypassing expensive database searches.
> Taken together, the results implied that pLMs learned some of the grammar of the language of life.