Run inference with the Python runtime#

Use the Edge-LLM C++ runtime through Python to generate text from local model files. This example uses the wheel without optional dependencies. For chat or HTTP serving, follow the trtedge CLI guide.

Prepare your environment and model#

Complete Install without optional dependencies and keep that environment activated for the commands below.

Provide a complete local Qwen3.5-0.8B Safetensors checkpoint, including tokenizer.json. The example does not download model files. Replace /path/to/Qwen3.5-0.8B with your checkpoint directory.

Build an engine and generate text#

  1. Download the example script and save it outside a source checkout.

  2. Choose an output directory for the generated engine. The script deletes and replaces this directory, so use a disposable directory.

  3. Run the script, replacing its path and the checkpoint path below:

python -I /path/to/installed_wheel_build_and_infer.py \
  /path/to/Qwen3.5-0.8B /tmp/edgellm-base-engines \
  --workflow base

The script builds a text engine, runs a prompt through tensorrt_edgellm.runtime.LLMRuntime, and checks for generated text and tokens. On success, it prints the result and INSTALLED_WHEEL_BUILD_AND_INFERENCE_PASSED.

For more engine-build options, see the direct builder guide.

Use an existing engine#

If you already have a compatible text engine and its local checkpoint, save the following code as run_engine.py. Replace the checkpoint and engine paths in the final line. This example runs inference without rebuilding the engine.

from pathlib import Path
from typing import Tuple

def _infer_base(model_dir: Path, engine_dir: Path, prompt: str,
                max_tokens: int) -> Tuple[str, int]:
    from tensorrt_edgellm import runtime

    native = runtime.load()
    llm = native.LLMRuntime(engine_dir=str(engine_dir),
                            checkpoint_dir=str(model_dir))
    request = native.create_generation_request(
        [[native.create_text_message("user", prompt)]],
        temperature=0.0,
        max_generate_length=max_tokens,
    )
    response = llm.handle_request(request)
    if (len(response.output_texts) != 1
            or not response.output_texts[0].strip()
            or len(response.output_ids) != 1 or not response.output_ids[0]):
        raise RuntimeError("EdgeLLM returned no generated output.")
    return response.output_texts[0], len(response.output_ids[0])
print(_infer_base(Path("/path/to/checkpoint"), Path("/path/to/engines"), "Hello", 32))

Run it in the same environment:

python /path/to/run_engine.py

Successful execution prints the generated text and token count.

C++ example executables require a source installation.