Api

FineTuneSeqLenBioBertConfig dataclass

Bases: BioBertConfig[MegatronBioBertFineTuneSeqLengthModel, SequenceLengthRMSEPlusBERTMLMLossWithReduction], IOMixinWithGettersSetters

BioBert fine-tuning sequence length model configuration.

Source code in bionemo/geneformer/model/finetune_token_regressor.py

@dataclass
class FineTuneSeqLenBioBertConfig(
    BioBertConfig[MegatronBioBertFineTuneSeqLengthModel, SequenceLengthRMSEPlusBERTMLMLossWithReduction],
    iom.IOMixinWithGettersSetters,
):
    """BioBert fine-tuning sequence length model configuration."""

    # When overriding fields in a dataclass _always_ declare types: https://github.com/python/cpython/issues/123269
    model_cls: Type[MegatronBioBertFineTuneSeqLengthModel] = MegatronBioBertFineTuneSeqLengthModel
    # typical case is fine-tune the base biobert that doesn't have this head. If you are instead loading a checkpoint
    # that has this new head and want to keep using these weights, please drop this next line or set to []
    initial_ckpt_skip_keys_with_these_prefixes: List[str] = field(default_factory=lambda: ["regression_head"])

    def get_loss_reduction_class(self) -> Type[SequenceLengthRMSEPlusBERTMLMLossWithReduction]:
        """Loss function type."""
        return SequenceLengthRMSEPlusBERTMLMLossWithReduction

get_loss_reduction_class()

Loss function type.

Source code in bionemo/geneformer/model/finetune_token_regressor.py

def get_loss_reduction_class(self) -> Type[SequenceLengthRMSEPlusBERTMLMLossWithReduction]:
    """Loss function type."""
    return SequenceLengthRMSEPlusBERTMLMLossWithReduction

GeneformerConfig dataclass

Bases: BioBertConfig[GeneformerModel, MegatronLossType], IOMixinWithGettersSetters

A geneformer config.

The geneformer config overrides the parent config, and adds a leaf-level iomixin, please do not inherit from this directly, as your parameters will likely be reset to this method's parameters silently.

Source code in bionemo/geneformer/api.py

@dataclass
class GeneformerConfig(BioBertConfig[GeneformerModel, MegatronLossType], iom.IOMixinWithGettersSetters):
    """A geneformer config.

    The geneformer config overrides the parent config, and adds a leaf-level iomixin, please do not inherit from this
    directly, as your parameters will likely be reset to this method's parameters silently.
    """

    num_layers: int = 6
    hidden_size: int = 256
    ffn_hidden_size: int = 512
    num_attention_heads: int = 4
    seq_length: int = 2048
    fp32_residual_connection: bool = False
    # Dropout
    attention_dropout: float = 0.1  # NeMo1 hard-coded, differs from publication of ReLU
    hidden_dropout: float = 0.02
    init_method_std: float = 0.02
    apply_query_key_layer_scaling: bool = False
    make_vocab_size_divisible_by: int = 128
    fp16_lm_cross_entropy: bool = False
    layernorm_zero_centered_gamma: bool = False
    layernorm_epsilon: float = 1.0e-12
    activation_func: Callable = F.gelu  # NeMo1 hard-coded, differes from publication of ReLU
    qk_layernorm: bool = False
    apply_residual_connection_post_layernorm: bool = False  # False is new default, True was BERT pub.
    share_embeddings_and_output_weights: bool = True
    # FUSION SETTINGS
    parallel_output: bool = True
    bias_dropout_fusion: bool = True
    bias_activation_fusion: bool = True
    masked_softmax_fusion: bool = True
    persist_layer_norm: bool = True
    get_attention_mask_from_fusion: bool = True

    position_embedding_type: PositionEmbeddingKinds = "learned_absolute"
    biobert_spec_option: BiobertSpecOption = BiobertSpecOption.bert_layer_with_transformer_engine_spec
    qk_layernorm: bool = False

    enable_autocast: bool = False
    model_cls: Type[GeneformerModel] = GeneformerModel
    loss_reduction_class: Type[MegatronLossType] = BERTMLMLossWithReduction