AI Dynamo

This workload (test_template_name is AIDynamo) runs AI inference benchmarks using the Dynamo framework with distributed prefill and decode workers.

Usage Example

See Running AI Dynamo with CloudAI for details.

API Documentation

Command Arguments

class cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoCmdArgs(

*,

docker_image_url: str,

huggingface_home_host_path: Path = PosixPath('/home/runner/.cache/huggingface'),

huggingface_home_container_path: Path = PosixPath('/root/.cache/huggingface'),

dynamo: AIDynamoArgs,

genai_perf: GenAIPerfArgs,

run_script: str = '',

**extra_data: Any,

)

Bases: CmdArgs

Arguments for AI Dynamo.

docker_image_url: str

huggingface_home_host_path: Path

huggingface_home_container_path: Path

dynamo: AIDynamoArgs

genai_perf: GenAIPerfArgs

run_script: str

Test Definition

class cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoTestDefinition(*, name: str, description: str, test_template_name: str, cmd_args: ~cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoCmdArgs, extra_env_vars: dict[str, str | ~typing.List[str]] = {}, extra_cmd_args: dict[str, str] = {}, extra_container_mounts: list[str] = [], git_repos: list[~cloudai._core.installables.GitRepo] = [], nsys: ~cloudai.models.workload.NsysConfiguration | None = None, predictor: ~cloudai.models.workload.PredictorConfig | None = None, agent: str = 'grid_search', agent_steps: int = 1, agent_metrics: list[str] = ['default'], agent_reward_function: str = 'inverse', script: ~cloudai._core.installables.File = File(src=PosixPath('/home/runner/work/cloudai/cloudai/src/cloudai/workloads/ai_dynamo/ai_dynamo.sh')), dynamo_repo: ~cloudai._core.installables.GitRepo = GitRepo(url=https://github.com/ai-dynamo/dynamo.git, commit=f7e468c7e8ff0d1426db987564e60572167e8464))

Bases: TestDefinition

Test definition for AI Dynamo.

cmd_args: AIDynamoCmdArgs

script: File

dynamo_repo: GitRepo

property docker_image: DockerImage

property installables: list[Installable]

property huggingface_home_host_path: Path