A preprint titled PG-LLM: Benchmarking General-Purpose Language Models for Protein Variant Ranking, recently released by a research collaboration between Harvard University and Capable, proposes a novel protein variant ranking evaluation framework for general-purpose language models.

Preprint access link: 

https://www.proteingymllm.com/paper

Built on the ProteinGym v1.3 dataset, the study compares 13 general-purpose language models (LLMs) and 95 specialized protein prediction methods on their zero-shot ability to rank the functional performance of protein variants.

Results show that leading general-purpose LLMs including Claude Opus and GPT-5.6 Sol achieved Spearman correlation coefficients (ρ) of 0.406 and 0.402 on this domain-specific task, outperforming a majority of specialized models (49 of 95), but still fall well short of top-tier specialized models. The preprint ranks VenusREM (ρ=0.523), led by Matwings Technology Gen Z research scientist Tan Yang, as the top performer across all 108 evaluated models, and sets it as the authoritative reference benchmark for general-purpose LLM comparison (see Figure 1).

Figure 1: Performance rankings of 13 general-purpose language models under N=50 evaluation conditions

The findings confirm that while general-purpose LLMs demonstrate preliminary capability in protein variant screening, specialized domain models retain clear performance advantages.

 01 From "can it work?" to "how well does it work?"

In recent years, general-purpose LLMs including GPT, Claude, and Gemini have become increasingly integrated into life science research workflows, supporting researchers with experimental protocol interpretation, function explanation, and code generation. However, "understanding protein-related questions" is not equivalent to accurately predicting mutational effects: the core task of protein engineering is quantitative ranking of functional impacts across a set of candidate mutations, not general knowledge question answering, and the PG-LLM benchmark is purpose-built for this specific, high-stakes task.

The evaluation is built on ProteinGym v1.3 deep mutational scanning (DMS) data, covering 186 proteins and 217 sets of experimentally measured assay results. Under test conditions, general-purpose LLMs were provided only with protein sequence, experimental protocol, and candidate mutation information, with no access to multiple sequence alignments (MSA) or 3D structural data; specialized models were allowed full access to structural, MSA, and other domain-specific inputs. Performance was measured via Spearman correlation ρ between model rankings and experimental ground truth results.

Results show Claude Opus 5 topped the general-purpose LLM leaderboard with ρ=0.406, followed closely by GPT-5.6 Sol at ρ=0.402. Claude Opus 5 outperformed 49 of the 95 total tested methods, including 41 of 46 sequence-only methods and 8 of 49 composite methods using MSA or structural data. Its score exceeded the sequence-only method median of ρ=0.374, approaching the performance of the dedicated protein LLM ESM2-650M at 0.411, but still lagged far behind VenusREM's score of 0.523.

Further ablation testing of whether scaling inference-time compute could improve general LLM performance found that allowing models longer thinking chains and more verbose reasoning steps improved correlation across all general LLMs before plateauing at a maximum of ρ=0.411 — still unable to catch up with dedicated specialized protein language models (see Figure 2).

Figure 2: Impact of inference-time scaling on model performance

 02 Gen Z scientist-led VenusREM emerges as the specialized reference for protein mutation prediction

VenusREM is a multimodal pre-trained model for protein mutational effect prediction and protein engineering, led by Tan Yang — Gen Z research scientist at Matwings Technology AI Lab and joint PhD candidate at Shanghai Jiao Tong University and Shanghai Chuangzhi Institute. The core design goal of VenusREM is to jointly leverage amino acid sequence, 3D structure, and homologous evolutionary information to more accurately assess the potential impact of candidate mutations on functional properties including protein activity and stability.

The VenusREM paper is published in Bioinformatics, full text available at:

 https://academic.oup.com/bioinformatics/article/41/Supplement_1/i401/8199374 

Figure 3: Tan Yang, Research Scientist at Matwings Technology AI Lab, joint PhD candidate at Shanghai Jiao Tong University & Shanghai Chuangzhi Institute

Unlike traditional protein language models that rely solely on single sequence information, VenusREM encodes amino acid sequences into sequence tokens, discretizes the local 3D spatial environment of each residue into structure tokens, and uses decoupled multi-head cross-attention to jointly model sequence context, local structure, and inter-residue spatial relationships — enabling it to capture long-range sequence interactions between residues that are spatially adjacent in folded structure.

Building on this representation, VenusREM integrates a retrieval-augmented module that extracts per-residue evolutionary conservation and amino acid substitution preferences from multiple sequence alignments, and fuses sequence-structure representations with evolutionary information via weighted integration, delivering fitness scoring and ranking for both single-point and multi-point mutations.

Figure 4: VenusREM model architecture

On the ProteinGym mutational effect benchmark developed by the Harvard Medical School team, VenusREM achieved leading zero-shot prediction performance, outperforming representative models developed by teams including Meta and Microsoft. The model and associated code have been fully open-sourced, with over 4,000 downloads on Hugging Face in the past 30 days and total cumulative downloads exceeding 250,000.

The VenusREM project is open source, available at:

 https://github.com/ai4protein/VenusREM

These results confirm that VenusREM has become a widely adopted representative open-source model and research tool in the field of protein mutational effect prediction.

Closing

Can general-purpose LLMs directly take on protein variant screening tasks? The PG-LLM benchmark provides a clear answer: they can, but with clear boundaries.

General-purpose large models have demonstrated measurable variant ranking capability, with some top models outperforming more than half of dedicated specialized models, but domain-specific protein models still retain significant performance advantages.

VenusREM's recognition as an authoritative industry reference in this evaluation marks a shift for specialized protein models: from research tools to an important benchmark for measuring general-purpose LLM capability. The goal our field is collectively advancing is to let general models handle complex task interpretation, while specialized models deliver high-precision prediction, ultimately building a human-AI collaborative protein R&D system.