nvalchemi.training.peft.lora.LoRAConfig#
- class nvalchemi.training.peft.lora.LoRAConfig(*, peft_method='lora', rank=8, alpha=1.0, lora_target_patterns, lora_dropout=0.0, wrap_mlp=False, wrapper_registrations=None)[source]#
ALCHEMI LoRA configuration for
FineTuningStrategy.This configuration is passed to
FineTuningStrategyto enable LoRA PEFT. The strategy prepends aLoRAHookthat applies the adapters during workflow registration.- Parameters:
peft_method (Literal["lora"]) – Fixed PEFT method name as
"lora".lora_target_patterns (tuple[str, ...]) – Shell-style glob patterns matched against model-prefixed module names, using the same
*,?, and[...]syntax astrainable_patterns. Dots are literal path separators. For example,"main.model.projection"selects exactly"main.model.projection","student.model.*projection"selects projection-like modules understudent.model, and"main.model.readout*"selects modules whose final path component starts withreadout. Patterns without glob characters are exact matches.rank (int) – Rank of the low-rank adapter factors. Defaults to
8.alpha (float, optional) – Scaling numerator for adapter updates. Defaults to
1.0.lora_dropout (float, optional) – Dropout probability on the adapter input path. Defaults to
0.0.wrap_mlp (bool, optional) – Also target supported feed-forward sub-blocks discovered by the adapter implementation. This is only supported for single-model strategies. Defaults to
False.wrapper_registrations (LoRAWrapperRegistrations, optional) – Custom layer-to-wrapper registrations installed before adapter injection. Each pair maps a base layer class to the adapter wrapper class that should handle it. The registrations only apply during adapter injection, and the global wrapper registry is restored to its prior state as soon as injection finishes. Replacing an existing wrapper emits a
UserWarning; assigning two different wrappers to one layer class in the same configuration raisesValueError. Defaults to().
Examples
Exact module names can be written as patterns without glob characters:
>>> config = LoRAConfig( ... lora_target_patterns=("main.model.projection",), ... )
Glob patterns match module names from
model.named_modules(), prefixed with the model key such as"main":>>> config = LoRAConfig( ... lora_target_patterns=("main.model.*projection",), ... rank=4, ... alpha=1.0, ... )
In multi-model workflows, the prefix helps to identify which model receives adapters:
>>> config = LoRAConfig( ... lora_target_patterns=("student.model.projection",), ... )
- model_config = {'arbitrary_types_allowed': True, 'extra': 'forbid'}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].