TFIDF

class sdm.processing.text.TFIDF(*, ngram_range: tuple[int, int], max_features: int | None = None, lowercase: bool = True)

Bases: EnsembleProcessor

Encode text columns as character n-gram TF-IDF vectors.

Tokenization follows scikit-learn’s char_wb analyzer: whitespace- delimited words are space-padded before windowing, so a word shorter than n still yields one n-gram. Fit learns a vocabulary and smoothed idf weights per text column. Transform replaces text with concatenated numerical features (one per retained n-gram), applies those idf weights, L2-normalizes each row, and ignores n-grams unseen at fit time.

Parameters:

• ngram_range (tuple[int, int]) – Inclusive (min_n, max_n) character-window sizes.

• max_features (int | None) – If set, keep only this many most frequent n-grams per column. None keeps the full vocabulary.

• lowercase (bool) – If True, lowercase text before tokenizing.

Capabilities

Handled Semantic Types	text
Requires Fitting	✅