> ## Documentation Index
> Fetch the complete documentation index at: https://wb-21fd5541-style-guide-models-tables-20260604-114339.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluate models with W&B Weave and W&B Tables
> Learn how to evaluate machine learning models using W&B Weave and Tables.
This page shows you two complementary ways to evaluate models tracked in W\&B: use W\&B Weave for LLM and GenAI evaluations, and use W\&B Tables for prediction analysis across runs and epochs.
## Evaluate models with Weave
[W\&B Weave](/weave) is a purpose-built toolkit for evaluating LLMs and GenAI applications. It provides evaluation capabilities including scorers, judges, and detailed tracing to help you understand and improve model performance. Weave integrates with W\&B Models so you can evaluate models stored in W\&B Registry.
### Key features for model evaluation
Weave provides the following capabilities for model evaluation:
* **Scorers and judges**: Pre-built and custom evaluation metrics for accuracy, relevance, coherence, and more.
* **Evaluation datasets**: Structured test sets with ground truth for systematic evaluation.
* **Model versioning**: Track and compare different versions of your models.
* **Detailed tracing**: Debug model behavior with complete input/output traces.
* **Cost tracking**: Monitor API costs and token usage across evaluations.
### Evaluate a model from Registry
Download a model from W\&B Registry and evaluate it using Weave:
```python theme={null}
import weave
import wandb
from typing import Any
# Initialize Weave
weave.init("your-entity/your-project")
# Define a ChatModel that loads from W&B Registry
class ChatModel(weave.Model):
model_name: str
def model_post_init(self, __context):
# Download model from W&B Registry
with wandb.init(project="your-project", job_type="model_download") as run:
artifact = run.use_artifact(self.model_name)
self.model_path = artifact.download()
# Initialize your model here
@weave.op()
async def predict(self, query: str) -> str:
# Your model inference logic
return self.model.generate(query)
# Create evaluation dataset
dataset = weave.Dataset(name="eval_dataset", rows=[
{"input": "What is the capital of France?", "expected": "Paris"},
{"input": "What is 2+2?", "expected": "4"},
])
# Define scorers
@weave.op()
def exact_match_scorer(expected: str, output: str) -> dict:
return {"correct": expected.lower() == output.lower()}
# Run evaluation
model = ChatModel(model_name="wandb-entity/registry-name/model:version")
evaluation = weave.Evaluation(
dataset=dataset,
scorers=[exact_match_scorer]
)
results = await evaluation.evaluate(model)
```
### Integrate Weave evaluations with W\&B Models
To connect Weave evaluation results with the models and runs you track in W\&B, use the integration workflow described next.
The [Models and Weave Integration Demo](/weave/cookbooks/Models_and_Weave_Integration_Demo) shows the complete workflow for:
1. **Load models from Registry**: Download fine-tuned models stored in W\&B Registry.
2. **Create evaluation pipelines**: Build evaluations with custom scorers.
3. **Log results back to W\&B**: Connect evaluation metrics to your model runs.
4. **Version evaluated models**: Save improved models back to the Registry.
Log evaluation results to both Weave and Models:
```python theme={null}
# Run evaluation with W&B tracking
with weave.attributes({"wandb-run-id": wandb.run.id}):
summary, call = await evaluation.evaluate.call(evaluation, model)
# Log metrics to W&B Models
wandb.run.log(summary)
wandb.run.config.update({
"weave_eval_url": f"https://wandb.ai/{entity}/{project}/r/call/{call.id}"
})
```
### Advanced Weave features
#### Custom scorers and judges
Create evaluation metrics tailored to your use case:
```python theme={null}
@weave.op()
async def llm_judge_scorer(expected: str, output: str, judge_model) -> dict:
prompt = f"Is this answer correct? Expected: {expected}, Got: {output}"
judgment = await judge_model.predict(prompt)
return {"judge_score": judgment}
```
#### Batch evaluations
Evaluate multiple model versions or configurations:
```python theme={null}
models = [
ChatModel(model_name="model:v1"),
ChatModel(model_name="model:v2"),
]
for model in models:
results = await evaluation.evaluate(model)
print(f"{model.model_name}: {results}")
```
### Next steps
For more information, see the following:
* [Complete Weave evaluation tutorial](/weave/tutorial-eval/)
* [Models and Weave integration example](/weave/cookbooks/Models_and_Weave_Integration_Demo)
## Evaluate models with W\&B Tables
W\&B Tables let you log structured predictions and inspect them interactively in the W\&B UI. Use Tables to:
* **Compare model predictions**: View side-by-side comparisons of how different models perform on the same test set.
* **Track prediction changes**: Monitor how predictions evolve across training epochs or model versions.
* **Analyze errors**: Filter and query to find commonly misclassified examples and error patterns.
* **Visualize rich media**: Display images, audio, text, and other media types alongside predictions and metrics.
### Basic example: Log evaluation results
```python theme={null}
import wandb
# Initialize a run
run = wandb.init(project="model-evaluation")
# Create a table with evaluation results
columns = ["id", "input", "ground_truth", "prediction", "confidence", "correct"]
eval_table = wandb.Table(columns=columns)
# Add evaluation data
for idx, (input_data, label) in enumerate(test_dataset):
prediction = model(input_data)
confidence = prediction.max()
predicted_class = prediction.argmax()
eval_table.add_data(
idx,
wandb.Image(input_data), # Log images or other media
label,
predicted_class,
confidence,
label == predicted_class
)
# Log the table
run.log({"evaluation_results": eval_table})
```
### Advanced table workflows
#### Compare multiple models
Log eval tables from different models to the same key for direct comparison:
```python theme={null}
# Model A evaluation
with wandb.init(project="model-comparison", name="model_a") as run:
eval_table_a = create_eval_table(model_a, test_data)
run.log({"test_predictions": eval_table_a})
# Model B evaluation
with wandb.init(project="model-comparison", name="model_b") as run:
eval_table_b = create_eval_table(model_b, test_data)
run.log({"test_predictions": eval_table_b})
```
#### Track predictions over time
Log tables at different training epochs to visualize improvement:
```python theme={null}
for epoch in range(num_epochs):
train_model(model, train_data)
# Evaluate and log predictions for this epoch
eval_table = wandb.Table(columns=["image", "truth", "prediction"])
for image, label in test_subset:
pred = model(image)
eval_table.add_data(wandb.Image(image), label, pred.argmax())
wandb.log({f"predictions_epoch_{epoch}": eval_table})
```
### Interactive analysis in the W\&B UI
After you log a table, you can use the W\&B UI to explore the results. You can:
* **Filter results**: Click column headers to filter by prediction accuracy, confidence thresholds, or specific classes.
* **Compare tables**: Select multiple table versions to see side-by-side comparisons.
* **Query data**: Use the query bar to find specific patterns (for example, `"correct" = false AND "confidence" > 0.8`).
* **Group and aggregate**: Group by predicted class to see per-class accuracy metrics.
### Example: Error analysis with enriched tables
The following example creates a mutable table, logs initial predictions, then adds confidence and error type columns for deeper analysis:
```python theme={null}
# Create a mutable table to add analysis columns
eval_table = wandb.Table(
columns=["id", "image", "label", "prediction"],
log_mode="MUTABLE" # Allows adding columns later
)
# Initial predictions
for idx, (img, label) in enumerate(test_data):
pred = model(img)
eval_table.add_data(idx, wandb.Image(img), label, pred.argmax())
run.log({"eval_analysis": eval_table})
# Add confidence scores for error analysis
confidences = [model(img).max() for img, _ in test_data]
eval_table.add_column("confidence", confidences)
# Add error types
error_types = classify_errors(eval_table.get_column("label"),
eval_table.get_column("prediction"))
eval_table.add_column("error_type", error_types)
run.log({"eval_analysis": eval_table})
```