Choosing Protein Models: Lessons from CASP17

A predicted protein structure is useful when it helps us choose an experiment: which residues to mutate, where to test binding, or how to investigate an assembly. When thousands of models suggest different answers, the challenge is deciding which structural ideas deserve that investment.

In CASP17, UniBio Intelligence used UbiStruct-QA to estimate the quality of 70,697 supplied models across five targets: two antibody–antigen complexes, two peptide assemblies, and a 24-subunit protein cage. CASP tests predictions against experimental structures withheld during prediction. Its assessment results are still pending. [20]

Our contribution was a method for scoring those candidates. It builds on a long-standing challenge in CASP: judging whether the parts of a complex are assembled correctly. [1][2][3][4] The lesson from our analysis is that a useful ranking should make its biological evidence and its sensitivity to assumptions visible.

Where the models came from

We received structures from CASP participants and large prediction pools supplied by the MassiveFold team. MassiveFold makes extensive sampling practical: repeated runs explore alternative structures rather than returning just one prediction. [5] Each antibody and peptide target included 17,085 MassiveFold entries, alongside 403–606 participant models. The cage pool contained 342 participant models only.

The supplied MassiveFold files were labelled AlphaFold-Multimer, the AlphaFold2 extension for complexes; [6][7] AlphaFold 3, which uses diffusion to generate atomic structures; [8] ColabFold, which makes sequence searches and AlphaFold prediction more accessible; [9] and ESMFold2, a protein language-model-based predictor. [10] File names recorded variations in sampling seeds, sequence alignments, and template settings. We could identify the predictor family from these file names, but could not reconstruct every setting used to generate each model. Method information was more limited for the CASP participant models.

We thank the CASP participants and the MassiveFold team for generating the models, and the method developers cited above for the prediction tools. UbiStruct-QA scored the supplied structures using their confidence outputs and our structural checks. We did not train a new quality-estimation model or generate new sequence alignments for the submitted scores.

Count evidence, not repeated predictions

We found byte-for-byte identical peptide structure files under different group IDs. Related entries may have reused the same prediction, or shared a prediction pipeline; the files alone do not establish why. Counting each copy separately would make one prediction look like agreement between independent predictions.

For the antibody targets, we calculated how often each contact appeared within each CASP group’s models and within each MassiveFold predictor family. We averaged these contact-frequency summaries separately for the CASP participant pool and the MassiveFold pool, giving each distinct summary equal weight. If two groups or families within a pool had identical summaries, we counted that summary once. We then combined contact support from the CASP participant pool and the MassiveFold pool. For peptides, we applied the same principle to groups of structurally similar models. This reduced the influence of repeated predictions and unequal numbers of submitted models.

This does not make all remaining predictions independent: methods can share training data or assumptions, and each candidate still contributes to its own agreement score. Peptide scoring had another limitation. CASP participant models carried labels for the requested assembly states; MassiveFold models did not. A labelled participant model could receive support from participant models with the same label and from the MassiveFold pool. An unlabelled MassiveFold model received support only from the MassiveFold pool. The agreement scores therefore used different evidence for the two pools.

Check the biology of the assembly

Agreement alone is not enough. We combined it with predictor confidence, completeness, packing, and clashes. Confidence values were not calibrated across predictors; interface-specific scores such as ipTM and actifpTM supplied additional evidence where available. [11] The structural questions differed by target:

• Antibody complexes: do the antibody and antigen make plausible contacts? Published NDM-1 and MYDGF structures provided antigen references; [12][13] ImmuneBuilder supplied antibody references. [15] Distance comparisons adapted from lDDT checked component geometry. [14] Well-folded components can still be docked in the wrong orientation.
• Peptide assemblies: do the chains form a consistent repeating arrangement? One target was annotated as an amyloid, motivating cross-β checks; [16] public metadata for the other motivated helical packing checks. [17] That metadata guided an assumption; no unreleased coordinates were used.
• The protein cage: do 24 subunits form a complete, well-packed shell? We used structural checks and confidence, with a related experimental cage as a reference, rather than consensus across candidates. [18][19]

Test the shortlist

For peptide target T2464, we ranked all 17,683 candidates by our baseline score. Here, “top 100” means the 100 models with the highest scores for that target. We then kept the models fixed, changed the scoring weights, and counted how many original top-100 model IDs remained in the new top 100.

Giving confidence more weight retained 64 of the original 100: 36 left the shortlist. Yet the rank correlation across the entire pool was 0.993, close to the value of 1 for identical ordering. Removing other evidence produced larger changes; the strongest stress test, using structural checks alone, retained 19.

A model leaving the top 100 could be replaced by a nearly identical structure, so 36 replacements do not necessarily mean 36 different biological interpretations. Even so, the result shows that a ranking can look almost unchanged across thousands of models while its shortlist changes substantially. Comparison with experimental structures is needed to determine which scoring weights select more accurate models.

For computational teams, a useful report should show the top-ranked models and how that selection changes when scoring assumptions change. For wet-lab teams, the next step is to compare the shortlisted structures: which binding contacts or assembly features recur, and where do they disagree? Those differences can guide an experiment—for example, testing binding after mutating a residue predicted to contact the partner in only some models. CASP’s independent assessment will test how well our scores reflect structural accuracy.

Explore the UbiStruct-QA code and methodology.