ggml#

GGML-compatible block quantization formats.

Functions

dequantize_iq1_s

Decode GGML-compatible IQ1_S payload bytes.

iq1_s_fake_quant

IQ1_S weight backend for TensorQuantizer, with pass-through backward.

iq1_s_grid

Return the canonical IQ1_S ternary grid as float32.

quantize_iq1_s

Pack a floating-point weight into GGML-compatible IQ1_S blocks.

dequantize_iq2_xs

Decode GGML-compatible IQ2_XS payload bytes.

iq2_xs_fake_quant

IQ2_XS weight backend for TensorQuantizer, with pass-through backward.

iq2_xs_grid

Return the canonical IQ2_XS magnitude grid as float32.

quantize_iq2_xs

Pack a floating-point weight into GGML-compatible IQ2_XS blocks.
dequantize_iq1_s(packed_weights, weight_shape, *, dtype=torch.bfloat16, block_chunk_size=1024)#

Decode GGML-compatible IQ1_S payload bytes.

Parameters:

  • packed_weights (Tensor)

  • weight_shape (Tensor)

  • dtype (dtype)

  • block_chunk_size (int)

Return type:

Tensor

dequantize_iq2_xs(packed_weights, weight_shape, *, dtype=torch.bfloat16, block_chunk_size=256)#

Decode GGML-compatible IQ2_XS payload bytes.

Parameters:

  • packed_weights (Tensor)

  • weight_shape (Tensor)

  • dtype (dtype)

  • block_chunk_size (int)

Return type:

Tensor

iq1_s_fake_quant(inputs, quantizer, *, block_chunk_size=1024)#

IQ1_S weight backend for TensorQuantizer, with pass-through backward.

Parameters:

  • inputs (Tensor)

  • block_chunk_size (int)

Return type:

Tensor

iq1_s_grid(device=None)#

Return the canonical IQ1_S ternary grid as float32.

Parameters:

device (device | str | None)

Return type:

Tensor

iq2_xs_fake_quant(inputs, quantizer, *, block_chunk_size=256)#

IQ2_XS weight backend for TensorQuantizer, with pass-through backward.

Parameters:

  • inputs (Tensor)

  • block_chunk_size (int)

Return type:

Tensor

iq2_xs_grid(device=None)#

Return the canonical IQ2_XS magnitude grid as float32.

Parameters:

device (device | str | None)

Return type:

Tensor

quantize_iq1_s(weight, *, block_chunk_size=1024)#

Pack a floating-point weight into GGML-compatible IQ1_S blocks.

Returned shapes are [*weight.shape[:-1], weight.shape[-1] // 256, 50] and [weight.ndim]. The packed payload remains on the weight’s device; the logical-shape metadata is kept on CPU. Non-finite input elements are treated as zero during packing.

Parameters:

  • weight (Tensor)

  • block_chunk_size (int)

Return type:

tuple[Tensor, Tensor]

quantize_iq2_xs(weight, *, block_chunk_size=256)#

Pack a floating-point weight into GGML-compatible IQ2_XS blocks.

Returned shapes are [*weight.shape[:-1], weight.shape[-1] // 256, 74] and [weight.ndim]. The packed payload remains on the weight’s device; the logical-shape metadata is kept on CPU. Non-finite input elements are treated as zero during packing.

Parameters:

  • weight (Tensor)

  • block_chunk_size (int)

Return type:

tuple[Tensor, Tensor]