fastnas
Module implementing fasnas
pruning algorithm for search.
Classes
An iterative searcher that uses binary search to find the best configuration. |
|
Class to describe the |
|
A patch manager for FastNAS (same as AutoNAS except no sampling during training). |
Functions
Convert search space for FastNAS mode with correct patch manager. |
|
Restore search space for FastNAS mode with correct patch manager. |
- class BinarySearcher
Bases:
IterativeSearcher
An iterative searcher that uses binary search to find the best configuration.
- after_step()
Update boundaries of the interval after recursing.
- Return type:
None
- before_search()
Build sensitivity map before search that we use to approximate the cost function.
- Return type:
None
- before_step()
Check what the middle value is to determine where we recurse.
- Return type:
None
- property default_state_dict: dict[str, Any]
We also store the sensitivity map and related arguments.
- early_stop()
Early stop if the interval is small enough.
- Return type:
bool
- property hparam_names_for_search: set[str]
We can only optimize over certain types of hparams in binary search.
- property hparam_types_for_search: tuple[type]
We can only optimize over certain types of hparams in binary search.
- load_search_checkpoint()
We only want to load sensitivity map and original_score here and keep the rest.
- Return type:
bool
- max_degrade: float
- middle_value: float
- min_degrade: float
- original_score: float
- sample()
Check in which interval we should recurse and sets the corresponding subnet.
- Return type:
dict[str, Any]
- sensitivity_map: dict[str, dict[int, float]]
- ModeloptConfig FastNASConfig
Bases:
ModeloptBaseRuleConfig
Configuration for the
"fastnas"
mode.Show default config as JSON
- Default config (JSON):
{ "nn.Conv1d": { "*": { "channel_divisor": 32, "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "kernel_size": [] } }, "nn.Conv2d": { "*": { "channel_divisor": 32, "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "kernel_size": [] } }, "nn.Conv3d": { "*": { "channel_divisor": 32, "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "kernel_size": [] } }, "nn.ConvTranspose1d": { "*": { "channel_divisor": 32, "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "kernel_size": [] } }, "nn.ConvTranspose2d": { "*": { "channel_divisor": 32, "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "kernel_size": [] } }, "nn.ConvTranspose3d": { "*": { "channel_divisor": 32, "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "kernel_size": [] } }, "nn.Linear": { "*": { "feature_divisor": 32, "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ] } }, "nn.BatchNorm1d": { "*": { "feature_divisor": 32, "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ] } }, "nn.BatchNorm2d": { "*": { "feature_divisor": 32, "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ] } }, "nn.BatchNorm3d": { "*": { "feature_divisor": 32, "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ] } }, "nn.SyncBatchNorm": { "*": { "feature_divisor": 32, "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ] } }, "nn.InstanceNorm1d": { "*": { "feature_divisor": 32, "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ] } }, "nn.InstanceNorm2d": { "*": { "feature_divisor": 32, "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ] } }, "nn.InstanceNorm3d": { "*": { "feature_divisor": 32, "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ] } }, "nn.LayerNorm": { "*": { "feature_divisor": 32, "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ] } }, "nn.GroupNorm": { "*": { "channel_divisor": 32, "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ] } } }
- field nn.BatchNorm1d: DynamicBatchNorm1dConfig | None | dict[str, DynamicBatchNorm1dConfig | None]
Show details
Configuration for dynamic nn.BatchNorm1d module.
If the
"nn.BatchNorm1d"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "feature_divisor": 32 } }
To deactivate any dynamic nn.BatchNorm1d module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.BatchNorm1d
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.BatchNorm2d: DynamicBatchNorm2dConfig | None | dict[str, DynamicBatchNorm2dConfig | None]
Show details
Configuration for dynamic nn.BatchNorm2d module.
If the
"nn.BatchNorm2d"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "feature_divisor": 32 } }
To deactivate any dynamic nn.BatchNorm2d module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.BatchNorm2d
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.BatchNorm3d: DynamicBatchNorm3dConfig | None | dict[str, DynamicBatchNorm3dConfig | None]
Show details
Configuration for dynamic nn.BatchNorm3d module.
If the
"nn.BatchNorm3d"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "feature_divisor": 32 } }
To deactivate any dynamic nn.BatchNorm3d module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.BatchNorm3d
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.Conv1d: DynamicConv1dConfig | None | dict[str, DynamicConv1dConfig | None]
Show details
Configuration for dynamic nn.Conv1d module.
If the
"nn.Conv1d"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "kernel_size": [], "channel_divisor": 32 } }
To deactivate any dynamic nn.Conv1d module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.Conv1d
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.Conv2d: DynamicConv2dConfig | None | dict[str, DynamicConv2dConfig | None]
Show details
Configuration for dynamic nn.Conv2d module.
If the
"nn.Conv2d"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "kernel_size": [], "channel_divisor": 32 } }
To deactivate any dynamic nn.Conv2d module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.Conv2d
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.Conv3d: DynamicConv3dConfig | None | dict[str, DynamicConv3dConfig | None]
Show details
Configuration for dynamic nn.Conv3d module.
If the
"nn.Conv3d"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "kernel_size": [], "channel_divisor": 32 } }
To deactivate any dynamic nn.Conv3d module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.Conv3d
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.ConvTranspose1d: DynamicConvTranspose1dConfig | None | dict[str, DynamicConvTranspose1dConfig | None]
Show details
Configuration for dynamic nn.ConvTranspose1d module.
If the
"nn.ConvTranspose1d"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "kernel_size": [], "channel_divisor": 32 } }
To deactivate any dynamic nn.ConvTranspose1d module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.ConvTranspose1d
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.ConvTranspose2d: DynamicConvTranspose2dConfig | None | dict[str, DynamicConvTranspose2dConfig | None]
Show details
Configuration for dynamic nn.ConvTranspose2d module.
If the
"nn.ConvTranspose2d"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "kernel_size": [], "channel_divisor": 32 } }
To deactivate any dynamic nn.ConvTranspose2d module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.ConvTranspose2d
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.ConvTranspose3d: DynamicConvTranspose3dConfig | None | dict[str, DynamicConvTranspose3dConfig | None]
Show details
Configuration for dynamic nn.ConvTranspose3d module.
If the
"nn.ConvTranspose3d"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "kernel_size": [], "channel_divisor": 32 } }
To deactivate any dynamic nn.ConvTranspose3d module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.ConvTranspose3d
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.GroupNorm: DynamicGroupNormConfig | None | dict[str, DynamicGroupNormConfig | None]
Show details
Configuration for dynamic nn.GroupNorm module.
If the
"nn.GroupNorm"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "channels_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "channel_divisor": 32 } }
To deactivate any dynamic nn.GroupNorm module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.GroupNorm
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.InstanceNorm1d: DynamicInstanceNorm1dConfig | None | dict[str, DynamicInstanceNorm1dConfig | None]
Show details
Configuration for dynamic nn.InstanceNorm1d module.
If the
"nn.InstanceNorm1d"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "feature_divisor": 32 } }
To deactivate any dynamic nn.InstanceNorm1d module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.InstanceNorm1d
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.InstanceNorm2d: DynamicInstanceNorm2dConfig | None | dict[str, DynamicInstanceNorm2dConfig | None]
Show details
Configuration for dynamic nn.InstanceNorm2d module.
If the
"nn.InstanceNorm2d"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "feature_divisor": 32 } }
To deactivate any dynamic nn.InstanceNorm2d module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.InstanceNorm2d
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.InstanceNorm3d: DynamicInstanceNorm3dConfig | None | dict[str, DynamicInstanceNorm3dConfig | None]
Show details
Configuration for dynamic nn.InstanceNorm3d module.
If the
"nn.InstanceNorm3d"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "feature_divisor": 32 } }
To deactivate any dynamic nn.InstanceNorm3d module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.InstanceNorm3d
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.LayerNorm: DynamicLayerNormConfig | None | dict[str, DynamicLayerNormConfig | None]
Show details
Configuration for dynamic nn.LayerNorm module.
If the
"nn.LayerNorm"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "feature_divisor": 32 } }
To deactivate any dynamic nn.LayerNorm module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.LayerNorm
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.Linear: DynamicLinearConfig | None | dict[str, DynamicLinearConfig | None]
Show details
Configuration for dynamic nn.Linear module.
If the
"nn.Linear"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "feature_divisor": 32 } }
To deactivate any dynamic nn.Linear module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.Linear
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- field nn.SyncBatchNorm: DynamicSyncBatchNormConfig | None | dict[str, DynamicSyncBatchNormConfig | None]
Show details
Configuration for dynamic nn.SyncBatchNorm module.
If the
"nn.SyncBatchNorm"
key is not specified, the default configuration (shown in JSON) will be used:{ "*": { "features_ratio": [ 0.05, 0.1, 0.15000000000000002, 0.2, 0.25, 0.30000000000000004, 0.35000000000000003, 0.4, 0.45, 0.5, 0.55, 0.6000000000000001, 0.65, 0.7000000000000001, 0.75, 0.8, 0.8500000000000001, 0.9, 0.9500000000000001, 1.0 ], "feature_divisor": 32 } }
To deactivate any dynamic nn.SyncBatchNorm module, use
None
instead of providing a dictionary{}
.To specify layer-specific configurations, you can specify a config for each submodule with the key specifying a glob pattern that matches the submodule name. For example, to convert to a dynamic module for all
nn.SyncBatchNorm
layers except for those in the"lm_head"
submodule use:{ "*": {...}, "*lm_head*": None, }
Note that glob expressions are processed sequentially in the order they are specified. Later keys in the config will overwrite earlier keys if they match the same submodule name.
If you want to specify the same configuration for all submodules, you can provide an unnested dictionary as well:
{...}
which is short for
{ "*": {...}, }
- class FastNASModeDescriptor
Bases:
ModeDescriptor
Class to describe the
"fastnas"
mode.The properties of this mode can be inspected via the source code.
- property config_class: type[ModeloptBaseConfig]
Specifies the config class for the mode.
- property convert: Callable[[Module, ModeloptBaseConfig], tuple[Module, dict[str, Any]]] | Callable[[Module, ModeloptBaseConfig, Any], tuple[Module, dict[str, Any]]]
The mode’s entrypoint for converting a model.
- property export_mode: str | None
The mode that corresponds to the export mode of this mode.
- property name: str
Returns the value (str representation) of the mode.
- property next_modes: set[str] | None
Modes that must immediately follow this mode.
- property restore: Callable[[Module, ModeloptBaseConfig, dict[str, Any]], Module]
The mode’s entrypoint for restoring a model.
- property search_algorithm: type[BaseSearcher]
Specifies the search algorithm to use for this mode (if any).
- property update_for_new_mode: Callable[[Module, ModeloptBaseConfig, dict[str, Any]], None]
The mode’s entrypoint for updating the models state before new mode.
- property update_for_save: Callable[[Module, ModeloptBaseConfig, dict[str, Any]], None]
The mode’s entrypoint for updating the models state before saving.
- class FastNASPatchManager
Bases:
AutoNASPatchManager
A patch manager for FastNAS (same as AutoNAS except no sampling during training).
- property sample_during_training
Indicates whether we should sample a new subnet during training.
- convert_fastnas_searchspace(model, config)
Convert search space for FastNAS mode with correct patch manager.
- Parameters:
model (Module)
config (ModeloptBaseConfig)
- Return type:
tuple[Module, dict[str, Any]]
- restore_fastnas_searchspace(model, config, metadata)
Restore search space for FastNAS mode with correct patch manager.
- Parameters:
model (Module)
config (ModeloptBaseConfig)
metadata (dict[str, Any])
- Return type:
Module