Static Embeddings Tutorial

Overview

Static embeddings capture semantic relationships between words using pre-trained models like Word2Vec or GloVe. These embeddings provide rich semantic representations that can be highly predictive of brain activity.

Key Components

• Assembly: Pre-packaged LeBel assembly containing brain data and stimuli
• Feature Extractor: StaticEmbeddingFeatureExtractor using pre-trained embeddings
• Embedding Models: Word2Vec, GloVe, or other static embedding models
• Downsampler: Aligns word-level features with brain data timing
• Model: Ridge regression with nested cross-validation
• Trainer: AbstractTrainer orchestrates the entire pipeline

Step-by-Step Tutorial

1. Load the Assembly

from encoding.assembly.assembly_loader import load_assembly

# Load the pre-packaged LeBel assembly
assembly = load_assembly("assembly_lebel_uts03.pkl")

2. Create Static Embedding Feature Extractor

from encoding.features.factory import FeatureExtractorFactory

# You need to provide the path to your embedding file
vector_path = "/path/to/your/embeddings.bin.gz"  # Replace with your path

extractor = FeatureExtractorFactory.create_extractor(
    modality="embeddings",
    model_name="word2vec",  # Can be "word2vec", "glove", or any identifier
    config={
        "vector_path": vector_path,
        "binary": True,  # Set to True for .bin files, False for .txt files
        "lowercase": False,  # Set to True if your embeddings expect lowercase tokens
        "oov_handling": "copy_prev",  # How to handle out-of-vocabulary words
        "use_tqdm": True,  # Show progress bar
    },
    cache_dir="cache",
)

3. Set Up Downsampler and Model

from encoding.downsample.downsampling import Downsampler
from encoding.models.nested_cv import NestedCVModel

downsampler = Downsampler()
model = NestedCVModel(model_name="ridge_regression")

4. Configure Training Parameters

# FIR delays for hemodynamic response modeling
fir_delays = [1, 2, 3, 4]

# Trimming configuration for LeBel dataset
trimming_config = {
    "train_features_start": 10,
    "train_features_end": -5,
    "train_targets_start": 0,
    "train_targets_end": None,
    "test_features_start": 50,
    "test_features_end": -5,
    "test_targets_start": 40,
    "test_targets_end": None,
}

# No additional downsampling configuration needed
downsample_config = {}

5. Create and Run Trainer

from encoding.trainer import AbstractTrainer

trainer = AbstractTrainer(
    assembly=assembly,
    feature_extractors=[extractor],
    downsampler=downsampler,
    model=model,
    fir_delays=fir_delays,
    trimming_config=trimming_config,
    use_train_test_split=True,
    logger_backend="wandb",
    wandb_project_name="lebel-embeddings",
    dataset_type="lebel",
    results_dir="results",
    downsample_config=downsample_config,
)

metrics = trainer.train()
print(f"Median correlation: {metrics.get('median_score', float('nan')):.4f}")

Understanding Static Embeddings

Key Parameters

• modality: "embeddings" - specifies the feature type
• model_name: "word2vec" - identifier for the extractor
• vector_path: Path to the embedding file
• binary: True for .bin files, False for .txt files
• lowercase: Whether to lowercase tokens before lookup
• oov_handling: How to handle out-of-vocabulary words
• use_tqdm: Whether to show progress bar
• cache_dir: "cache" - directory for caching

Embedding Models

• Word2Vec: Google News vectors, custom Word2Vec models
• GloVe: Stanford GloVe embeddings
• Custom embeddings: Any compatible embedding format

File Formats

• Binary files (.bin): Set binary=True
• Text files (.txt): Set binary=False
• Compressed files (.gz): Automatically handled

OOV Handling

• "copy_prev": Use the previous word's embedding
• "zero": Use zero vector
• "random": Use random vector
• "mean": Use mean of all embeddings

Choose based on your research question and data characteristics.

Training Configuration

• fir_delays: [1, 2, 3, 4] - temporal delays for hemodynamic response
• trimming_config: LeBel-specific trimming to avoid boundary effects
• downsample_config: {} - no additional downsampling configuration needed