nvalchemi.training.peft.lora.LoRAConfig#

class nvalchemi.training.peft.lora.LoRAConfig(*, peft_method='lora', rank=8, alpha=1.0, lora_target_patterns, lora_dropout=0.0, wrap_mlp=False, wrapper_registrations=None)[source]#

ALCHEMI LoRA configuration for FineTuningStrategy.

This configuration is passed to FineTuningStrategy to enable LoRA PEFT. The strategy prepends a LoRAHook that applies the adapters during workflow registration.

Parameters:
  • peft_method (Literal["lora"]) – Fixed PEFT method name as "lora".

  • lora_target_patterns (tuple[str, ...]) – Shell-style glob patterns matched against model-prefixed module names, using the same *, ?, and [...] syntax as trainable_patterns. Dots are literal path separators. For example, "main.model.projection" selects exactly "main.model.projection", "student.model.*projection" selects projection-like modules under student.model, and "main.model.readout*" selects modules whose final path component starts with readout. Patterns without glob characters are exact matches.

  • rank (int) – Rank of the low-rank adapter factors. Defaults to 8.

  • alpha (float, optional) – Scaling numerator for adapter updates. Defaults to 1.0.

  • lora_dropout (float, optional) – Dropout probability on the adapter input path. Defaults to 0.0.

  • wrap_mlp (bool, optional) – Also target supported feed-forward sub-blocks discovered by the adapter implementation. This is only supported for single-model strategies. Defaults to False.

  • wrapper_registrations (LoRAWrapperRegistrations, optional) – Custom layer-to-wrapper registrations installed before adapter injection. Each pair maps a base layer class to the adapter wrapper class that should handle it. The registrations only apply during adapter injection, and the global wrapper registry is restored to its prior state as soon as injection finishes. Replacing an existing wrapper emits a UserWarning; assigning two different wrappers to one layer class in the same configuration raises ValueError. Defaults to ().

Examples

Exact module names can be written as patterns without glob characters:

>>> config = LoRAConfig(
...     lora_target_patterns=("main.model.projection",),
... )

Glob patterns match module names from model.named_modules(), prefixed with the model key such as "main":

>>> config = LoRAConfig(
...     lora_target_patterns=("main.model.*projection",),
...     rank=4,
...     alpha=1.0,
... )

In multi-model workflows, the prefix helps to identify which model receives adapters:

>>> config = LoRAConfig(
...     lora_target_patterns=("student.model.projection",),
... )
model_config = {'arbitrary_types_allowed': True, 'extra': 'forbid'}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_spec_dict()[source]#

Return a JSON-safe representation of this LoRA config.

Returns:

JSON-safe LoRA configuration data suitable for serialization.

Return type:

dict[str, Any]