从计算模拟到湿实验：评估AI蛋白质设计性能

From In-Silico to Wet-Lab: Evaluating AI Protein Design Performance

In this tutorial, we use Anthropic’s claude-protein-binder-design dataset, which contains 1,440 AI-designed miniprotein binders tested against 16 targets. Because the release includes both computational predictions and real wet-lab results from two independent labs, we can go beyond simply studying the designs. We evaluate how well structure predictors identify successful binders, whether combining predictions improves performance, how rankings translate into practical testing budgets, and how much disagreement comes from the assays themselves. Also, we train a target-aware classifier to test whether these signals can reliably predict experimental success.

Copy CodeCopiedUse a different Browser

We start by installing only what the runtime is actually missing, then enumerate the repository once and build a {subset to path} map instead of hard-coding file locations. This matters because the naming is not uniform; the subset wetlab_summary lives at data/tables/wetlab/summary.parquet, and a guessed path would fail silently. With the map in place we pull design_summary, one row per design, 1,440 rows wide enough to carry every join we need downstream.

We define the evaluable set by filtering on actual vendor calls rather than on binder_final, because that column is a bool and so records the 120 never-measured designs as False rather than missing. From there we compute hit rates by model, campaign, generator, and target, wrapping each in a Wilson interval since several subgroups sit in the small-n regime where the normal approximation misbehaves. The target plot is the one to read first: it shows antigen choice swamping every other factor we compare.

We score all ten predictors against the wet-lab label, on both ipSAE and self-consistency DockQ, with bootstrapped confidence intervals so we can see which differences are real. We then rank-normalize each column to percentiles and aggregate them, which keeps the comparison scale-free across metrics that live on different ranges and pile up differently at zero. The Spearman heatmap explains why the ensemble helps at all; if the predictors agreed perfectly there would be nothing left to harvest.

We convert ranking performance into precision@N, because no lab orders 1,300 constructs and AUC quietly hides how a score behaves at the top of the list. The enrichment table then tells us what triage actually buys at budgets of 25, 50, 100, and 200. We follow it with Cohens κ and a log-log KD comparison between vendors, which sets the ceiling: label noise bounds how high any AUC above can honestly climb.

We test whether expression titer alone discriminates binders, and if it does, we know part of every score from above is solubility riding along under another name. Re-running consensus on expressed-only designs isolates interface quality from biophysics. We then parse the epitope contact lists into residue sets and ask, per target and paired across targets, whether confirmed binders converge on a shared patch more than the failures do.

We featurize sequences by composition, charge, hydropathy, entropy, and hydrophobic run length, then fit gradient boosting under two schemes: random folds and target-grouped folds. The gap between them is the leakage, since designs nest inside targets with very different base rates and a random split lets the model memorize which antigens are easy. The one-hot control makes that explicit, and permutation importance on a held-out target block shows what survives when we remove the shortcut.

In conclusion, in-silico scoring was helpful, but it did not tell the whole story. The target strongly influenced the results, so comparing models without accounting for it could easily give us a misleading picture. The structure predictors showed useful signals, and combining them gave a modest improvement, but they were still not reliable enough to use on their own. We also found that differences between experiments and protein expression could make a design look like a binding failure even when the real issue was poor expression. Overall, we learned that careful evaluation mattered more than chasing impressive individual metrics. By grouping our cross-validation by target, we got a more realistic view of how well the models could generalize to new targets.

Check out the FULL CODES here. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.

Scientific Data Analysis with LabPlot in Python: Signal Processing, Spectral Peak Fitting, Visualization, and Batch Automation

Building an End-to-End Document Intelligence Pipeline with deepDoctection

The Developer’s Guide to NeMo Guardrails for Enterprise AI Safety

Building Agentic Document Intelligence Pipelines: Creating Scientific Figures with AutoFigure

Previous articleGoogle Research Introduces GlucoFM: A 0.72M-Parameter Dual-Stream Foundation Model for Continuous Glucose Monitoring

Next articleBest Agent Sandboxes in 2026: Cold Start, Per-Second Pricing, and Network Policy Across E2B, Daytona, Modal, Cloudflare, and Vercel