Timeout dumps ============= NVRx can capture Python, native CPU, and CUDA diagnostics when a selected workload section times out. Diagnostics are opt-in; the default capture section is ``step``. Once enabled, all completed captures are uploaded automatically. Heartbeat and out-of-section timeouts do not trigger captures by themselves. Enable capture -------------- Install the requested tools in the training container and enable diagnostics on the launcher: .. code-block:: bash ft_launcher \ --ft-timeout-dump-tools=pyspy,gdb,cuda-gdb \ --ft-timeout-dump-gdb-command-file=/lustre/config/gdb-timeout.txt \ --ft-timeout-dump-cuda-gdb-command-file=/lustre/config/cuda-gdb-timeout.txt \ --ft-nvrx-logfile /lustre/run/nvrx/12345/logs/launcher.log \ --ft-per-cycle-applog-prefix /lustre/run/logs/training.log \ --ft-timeout-dump-local-root /tmp \ ... train.py Select additional named sections with, for example:: --ft-timeout-dump-sections=step,checkpointing The value is a comma-separated list of exact, case-sensitive workload section names. Surrounding whitespace is ignored; empty or duplicate names are rejected. Megatron uses ``checkpointing``, while the NVRx sections example uses ``checkpoint``. Use the names passed to ``start_section()`` by your workload. Other names, such as ``setup`` or application-defined sections, are supported. There are no wildcard or timeout-type aliases. Selecting a section does not enable diagnostics or configure its timeout: it must also have a finite timeout in ``rank_section_timeouts`` and actually expire. Heartbeat and out-of-section timeouts are separate detection paths, not section names in this option. Selecting tools with ``--ft-timeout-dump-tools`` enables capture; ``--ft-timeout-dump-enabled=true`` enables it with the default tools, ``pyspy,gdb,cuda-gdb``, in that order. ``--ft-timeout-dump-enabled=false`` wins over ``--ft-timeout-dump-tools``. Options for a tool that is not listed, such as ``--ft-timeout-dump-cuda-gdb-mode`` without ``cuda-gdb``, are unused; they never select the tool. Option values are checked when the command line is parsed: an invalid value, such as an unknown tool name or a relative local root, fails the launch like any other bad flag. Problems with the node itself never stop training. When dumps are enabled, the launcher checks every requested executable, each selected debugger's command file, the log layout below, and the local staging root before starting workers. If any check fails, the launcher logs an error on node0 and runs without dumps. There is no automatic detection or partial fallback: a missing tool disables dumps on that node rather than being skipped. Supply a smaller tool list or install/configure the missing tools. Dumps require an absolute ``--ft-nvrx-logfile=/logs/`` and always use ``/diags/``. The existing requirement for ``--ft-per-cycle-applog-prefix`` with ``--ft-nvrx-logfile`` still applies. Deployment scripts choose ``/nvrx/`` for ordinary jobs and ``/nvrx/`` for job arrays. Diagnostics reuse that namespace without reconstructing Slurm IDs. A root must distinguish launch attempts whose cycle numbers restart at zero. Successful startup validation logs one summary of the resolved tools (executable, CUDA-GDB mode, and command files) only from node0 at DEBUG (``NVRX_LOG_DEBUG=1``). Node0 uses the launcher's infrastructure rank with hostname/topology-derived IDs disabled; Slurm job-array offsets still apply. If the startup rank is unknown, the summary is omitted. Every node still checks its own dependencies, but only node0 logs startup errors, since every node normally runs the same image. INFO is reserved for actual capture results and upload completion. Runtime warnings and capture/upload results remain visible from every node at their existing levels. The worker obtains its monitor PID from the existing Unix socket and calls ``PR_SET_PTRACER`` to permit the monitor and its descendants to attach under Linux Yama mode 1. This does not override container restrictions, other security checks, or stricter Yama modes. It replaces any previously nominated ptracer; qualify applications that already configure a debugger or crash handler. Permission failures are logged and do not disable fault handling. Validate attachment inside the actual training container before production rollout. Capture targets only the training worker PID. The ptrace authorization does not extend to separate processes created by that worker, such as DataLoader workers or an async checkpoint-save worker. Capturing them would require their own authorization and explicit targeting. Selecting a checkpoint section captures the training process, which may show where checkpoint finalization is waiting; it does not collect stacks from the async checkpoint worker itself. CUDA-GDB launch mode -------------------- CUDA-GDB defaults to ``batch``, preserving the existing ``--batch``, ``-ex``, and ``-x`` invocation. To use its normal stdin command reader instead:: --ft-timeout-dump-cuda-gdb-mode=stdin The mode option applies only when CUDA-GDB is a selected tool; it does not add it. Native GDB and py-spy are unaffected. Switch back with ``--ft-timeout-dump-cuda-gdb-mode=batch``; there is no automatic fallback. ``stdin`` is fully automated: no terminal, prompts, or monitor command exchange is required. The startup thread writes setup, ``attach``, ``source`` of the configured command file, ``detach``, and ``quit`` into a Linux anonymous in-memory file (``memfd``), rewinds it, and passes it as the debugger's stdin without ``--batch``. The parent closes its descriptor after subprocess startup, including failed or abandoned launches. The shared command file is still read only by the debugger; its path must not contain a newline or carriage return in stdin mode. Both modes use the same capture deadline, output cap, cancellation, and cleanup. Use stdin mode for environments where batch attach does not expose usable CUDA state, after qualifying it in the deployed debugger/driver/container combination. A zero exit code in stdin mode is recorded as ``completed``, not ``success``: command errors can be printed while the debugger continues to ``detach`` and ``quit`` and exits zero. NVRx retains merged stdout/stderr and partial output, and records the selected CUDA-GDB mode in the capture header; every capture's footer records the tool's process exit code. It does not parse command errors or certify GPU-state availability. Nonzero exits, timeouts, and output limits retain their existing failure statuses. Diagnostic failure does not add retries or extend recovery beyond the capture budget. Debugger command files ---------------------- Users must supply a command file for each selected debugger. NVRx does not bundle default GDB or CUDA-GDB commands. Set the paths on the command line:: --ft-timeout-dump-gdb-command-file=/lustre/config/gdb-timeout.txt --ft-timeout-dump-cuda-gdb-command-file=/lustre/config/cuda-gdb-timeout.txt GDB timeout reference ~~~~~~~~~~~~~~~~~~~~~ Use this as the default deployment reference: save the following text as ``/lustre/config/gdb-timeout.txt`` and pass its path explicitly. It is a reference for users to copy, not an automatic runtime fallback:: # Native CPU call stacks for all host threads, up to 32 frames each. thread apply all bt 32 # Loaded shared libraries and their symbol-loading status. info sharedlibrary # Process virtual address ranges and mapped files (Linux). info proc mappings These commands produce text. They do not save process memory or create a core file. Native backtraces do not automatically reconstruct Python call stacks; py-spy provides that view. The 32-frame limit is a starting point that users can adjust in their deployed file. CUDA-GDB timeout reference ~~~~~~~~~~~~~~~~~~~~~~~~~~ CUDA-GDB is NVIDIA's extension of GDB for inspecting both CPU code and CUDA GPU execution. It is a separate executable, ``cuda-gdb``. A command file is plain text containing the commands one would otherwise type at its debugger prompt; the ``.txt`` extension is only a naming convention. It is not a CUDA kernel, Python program, or binary core dump. The following reference was validated with CUDA-GDB 13.2 on four-GPU GB300 nodes. Save it as ``/lustre/config/cuda-gdb-timeout.txt``:: # Explicit device filters work around the unfiltered context-query issue # observed with CUDA-GDB 13.2 on four-GPU GB300 nodes. echo NVRX_CUDA_GDB_STACKS_BEGIN\n thread apply all bt 32 info cuda devices info cuda contexts device 0 info cuda contexts device 1 info cuda contexts device 2 info cuda contexts device 3 info cuda kernels info cuda threads echo NVRX_CUDA_GDB_COMMANDS_DONE\n Adjust the device queries to the device IDs reported by ``info cuda devices`` in your deployment; the four-device list above is specific to the tested setup. Explicit queries worked around an unfiltered ``info cuda contexts`` issue observed in that environment. A worker using only one GPU can legitimately report ``No CUDA contexts.`` for the other devices. Check the query for its active device before concluding that CUDA context capture failed. NVRx launches CUDA-GDB against the timed-out worker PID and passes this file with ``-x`` in batch mode or ``source`` in stdin mode. Its output goes into the CUDA-GDB text capture. Both modes use the same command file; leave ``attach``, ``detach``, and ``quit`` out of that file because NVRx supplies them. To use this reference through stdin, add ``--ft-timeout-dump-cuda-gdb-mode=stdin`` to the launcher options. The equivalent manual invocation below shows the command stream NVRx supplies. Replace ```` with the target PID and use a command-file path available inside the training container: .. code-block:: bash cuda-gdb --nx -q -iex 'set debuginfod enabled off' <<'EOF' set confirm off set pagination off set width 0 set height 0 attach source /lustre/config/cuda-gdb-timeout.txt detach quit EOF This invocation omits ``--batch`` and supplies ``attach`` as a top-level stdin command. NVRx uses an anonymous in-memory file instead of a shell here-document and also enforces the capture deadline and output limit. This manual example does not impose those limits. The ``info cuda`` commands describe GPU execution state; they do not dump a backtrace for every GPU thread. Inspecting a particular GPU thread's stack requires selecting a CUDA execution focus and then issuing ``bt``. Available detail depends on the kernel's symbols/debug information and the debugger/driver state. See the `NVIDIA CUDA-GDB manual `_ for CUDA focus selection and the ``info cuda`` commands. CUDA-GDB capture remains best effort on a hung GPU and needs qualification in the deployed container. The files are independent; only selected debuggers require a file. A py-spy-only configuration does not need either file. NVRx disables initialization scripts and debuginfod downloads, attaches to the worker, disables pagination, executes the file, and detaches. NVRx still owns the subprocess deadline, output cap, and text upload. GDB stops processing a sourced command file on an error. Batch mode reports command failures through its exit code; stdin mode continues with subsequent top-level commands and can exit zero despite a command error, as described above. Each launcher checks that the selected files exist, are regular files, and can be read before starting workers. Relative paths are resolved once at startup from the launcher's working directory. The monitor passes the validated path to the debugger without resolving or reading it during capture. Files may live on shared Lustre storage; mount the path inside the training container on every node, including replacement nodes. Debuggers read the files at capture time, so startup validation does not guarantee their later availability. Command files are trusted deployment code, not a sandboxed query language. They should emit text to the captured output and leave attach/detach and process lifecycle to NVRx. The capture byte limit applies to captured stdout/stderr; it cannot bound files or other side effects produced by arbitrary commands. Changes to a file take effect on the next capture; keep it stable during a capture or deploy updates with an atomic replacement. Binary core generation and binary artifact collection are **future extensions, not part of the current feature**. Do not add ``gcore`` or ``generate-core-file`` to these scripts expecting NVRx to collect the result. The collector publishes only text captures. Future core support requires its own artifact lifecycle and capture-budget design; no core-specific options are provided now. Capture and upload ------------------ Each launcher forks one diagnostic collector before threaded logging and attribution services start. The collector closes unnecessary inherited descriptors, keeps the existing launcher log route, and remains outside training-worker cleanup. On a selected section timeout, the rank monitor attempts one diagnostic sequence per worker incarnation. Each tool runs in its own subprocess, sequentially in the configured order, with bounded output in memory. The capture grace defaults to **30 seconds total**, including startup and cleanup. Each tool receives at most the remaining time divided by the number of remaining tools; unused time becomes available to later tools. A small part of each share is reserved for killing and reaping the diagnostic child. If that child remains alive, later tools are skipped to prevent overlapping debugger attachments. Worker termination is requested by the overall deadline, independently of diagnostic subprocess startup and cleanup. Startup runs on a daemon thread so a blocked executable lookup cannot hold the monitor event loop or its shutdown. A pending launch prevents further captures; if it returns after cancellation or timeout, its child is killed and reaped without delaying recovery. Other launchers may terminate the target sooner, so capture is best effort. The timeout retains the original process identity (PID and creation time): disconnecting or reconnecting its monitor socket does not cancel termination, and a replacement process is not targeted by the old timeout. ``pyspy`` captures Python thread stacks. ``gdb`` and ``cuda-gdb`` attach to the live worker and execute the user-supplied command files. The files determine which native CPU stacks or CUDA state are captured. Debuggers disable initialization scripts and debuginfod downloads, then explicitly detach. Attachment temporarily stops the worker; detaching resumes it, so successive captures may observe different states. Runtime capture errors are retained and recovery continues. GPU inspection needs GPU device access and a compatible debugger/driver stack. CUDA-GDB receives the worker's ``TMPDIR``, ``CUDA_VISIBLE_DEVICES``, and ``CUDA_DEVICE_ORDER`` from a bounded read of its process environment. The worker and debugger must share a writable temporary directory. Symbols and device line/debug information improve results; production builds need not enable ``-G``. Executable validation does not prove GPU attachment will work. Qualify GPU capture and timeout cleanup in the deployed training container. See the `CUDA-GDB setup guide `_. After requesting worker termination, the monitor stages the output locally:: /tmp/nvrx-diags/cycle/ rank_pid_.txt.tmp rank_pid_.txt.ready Each file contains the host, cycle, global/local rank, PID, timeout reason, timestamp, and capture outcome. The monitor closes the file before renaming it to ``.ready``. Failed or truncated output is retained with its status and, once the tool has started, its exit code. A still-running diagnostic child only has a pipe, never a writable staging file. After staging, the monitor emits one INFO summary per rank and capture sequence, combining the attempted tools' capture statuses and the local cycle directory:: Timeout captures: cycle=0 global_rank=246 local_rank=2 pid=1536944 pyspy=success gdb=success cuda-gdb=completed dir=/tmp/nvrx-diags/cycle0 There are no per-tool staging messages at INFO or DEBUG. Individual files retain their full metadata and status; staging failures still produce warnings. Upload completion messages remain separate and report the shared output path and number of captures in each node/tool upload batch. The collector polls this local directory every 30 seconds. The launcher supplies the local worker count; it is not hard-coded to four. A newly discovered batch with captures from all local ranks is published immediately. An incomplete batch waits one additional poll to collect late arrivals, then publishes whatever is available even if ranks are still missing. New arrivals do not extend that wait. Distinct local ranks are counted, so two PIDs for one rank do not make a batch complete. Tools and cycles are handled independently. Waiting does not consume an upload attempt; inputs become fixed when the first upload is attempted. The initial polling offset is randomized within the interval to spread writes across nodes. There are no separate quiet-interval, maximum batch-delay, upload-jitter, or retry-delay settings, and no new timer. Training recovery never waits for uploading. Shutdown bypasses incomplete-batch deferral for one final collection pass within the existing shutdown grace period. There are no new sockets, collection requests, metadata sidecars, archives, or compression. Each node publishes one text file per cycle and tool:: /diags/cycle/_.txt The collector selects a fixed batch, writes a complete ``.txt.partial`` snapshot, syncs it, and atomically renames it over the final file in the same directory. It syncs the directory before deleting the included local files. A late capture requires reading the previously published contents to build a new snapshot. Successful uploads delete their local inputs. Late captures produce updated snapshots. Failures and retention ---------------------- * Upload failure retains local files. Each subsequent collection poll makes one retry per pending batch, up to the attempt limit. Every copy attempt rebuilds ``.partial`` from scratch. If publication already succeeded, an ordinary retry only finishes syncing and local cleanup. New arrivals are not deleted with an earlier batch. Empty cycle directories are retained to avoid racing with producers opening capture files; successful publication removes the capture files themselves. * Exhausted batches stay local; polling does not reset their retry budget. Later captures for the same cycle and tool form a new batch with their own retry budget; they do not include or retry the exhausted files. Collect retained files manually after the owning launcher/collector has stopped. A collector crash after publication but before cleanup may leave duplicate dumps on replay; exactly-once delivery is deliberately not required. * Node-local scratch is assumed to have enough space for text dumps. Each rank/tool capture retains its output cap, but there is no aggregate staging quota or pre-capture directory scan. At the default 1 MiB cap, py-spy plus GDB use at most 8 MiB per cycle on a four-worker node or 16 MiB on an eight-worker node. Failed uploads can accumulate across cycles. Local write failures are logged after worker termination has been requested. * An ownership lock prevents two collectors using the same local staging root; a second launcher using the same root runs without dumps. Existing captures also disable dumps on that node, rather than being uploaded into a new job or deleted. Collect them or select another local root. Concurrent launchers on one node need distinct roots and destinations. * The collector is not automatically restarted. Uploads do not delay training restart. Shutdown wakes the collector immediately for one final collection pass (including pending retries within their attempt limit). It grants a bounded drain interval, then sends SIGKILL without waiting indefinitely for a stuck filesystem operation. A live old writer is never replaced. Node loss can lose unpublished captures. Existing NVRx logs contain concise capture/upload outcomes and destination paths. Each capture summary includes ``cycle``, ``global_rank``, ``local_rank``, worker ``pid``, all attempted tools' statuses, and the local cycle ``dir``. The existing log header supplies the hostname. Dump block headers independently include ``host``, ``cycle``, ``global_rank``, ``local_rank``, and ``pid``, so the aggregated file can be correlated with a rank in that cycle's application log without consulting the launcher log. Dump contents remain in the diagnostic files. A ``.partial`` left by an interrupted upload is not a published snapshot. Known Issues ------------ The launcher and the rank monitors do not coordinate worker teardown with timeout captures. A launcher can therefore send SIGTERM, and SIGKILL after ``--workers-stop-timeout``, to workers whose captures are still running. Depending on the capture phase the signal reaches, the published dump can be incomplete or incorrect. The debugger and the launcher's worker teardown can also stall. A stall lasts at most until the rank monitor's capture deadline (``--ft-timeout-dump-capture-timeout``), when the monitor kills the debugger. GDB and CUDA-GDB are affected most often. Examples: * A rank finishes its capture early and its monitor terminates it. The local launcher detects the exit and sends SIGTERM to the remaining local workers, some of which are still being captured. * A launcher detects a failure on a peer node through the opened rendezvous round and starts stopping its local workers while their captures are running. Addressing this requires coordination between the launcher and the rank monitors. Before starting a local teardown, the launcher should detect and wait for any local captures in progress. A rank monitor should not start a capture while a local teardown is in progress. Configuration ------------- Timeout dumps are configured with the options below. Each corresponds to a field of ``TimeoutDumpConfig``, the ``timeout_dump`` field of ``FaultToleranceConfig``; tool options configure that tool's entry in ``tools``, whose order is the execution order. .. list-table:: Shared fields :header-rows: 1 :widths: 25 20 25 30 * - Field - Default - Option - Meaning * - ``enabled`` - ``false``, or ``true`` when ``--ft-timeout-dump-tools`` is given - ``--ft-timeout-dump-enabled`` (``true``/``false``) - Enable or disable diagnostics * - ``tools`` - every tool, with defaults - ``--ft-timeout-dump-tools`` (comma-separated names) - Ordered tool entries; all must be executable when enabled * - ``sections`` - ``[step]`` - ``--ft-timeout-dump-sections`` (comma-separated) - Workload section names that trigger capture on timeout * - ``local_root`` - ``/tmp`` - ``--ft-timeout-dump-local-root`` - Node-local parent of ``nvrx-diags/`` * - ``capture_timeout`` - ``30`` seconds - ``--ft-timeout-dump-capture-timeout`` - Total capture grace across all tools, including cleanup * - ``max_capture_bytes`` - ``1048576`` - ``--ft-timeout-dump-max-capture-bytes`` - Output cap per tool, including headers * - ``poll_interval`` - ``30`` seconds - ``--ft-timeout-dump-poll-interval`` - Collection and retry interval; filesystem I/O can extend the cadence * - ``upload_attempts`` - ``3`` - ``--ft-timeout-dump-upload-attempts`` - Total attempts per batch, including the first upload * - ``shutdown_grace`` - ``10`` seconds - ``--ft-timeout-dump-shutdown-grace`` - Maximum launcher wait for the collector to finish its final pass .. list-table:: Tool entry fields :header-rows: 1 :widths: 25 20 25 30 * - Tool and field - Default - Option - Meaning * - ``pyspy``: ``executable`` - ``py-spy`` - ``--ft-timeout-dump-pyspy`` - Executable name or path * - ``gdb``: ``executable`` - ``gdb`` - ``--ft-timeout-dump-gdb`` - GDB executable name or path * - ``gdb``: ``command_file`` - ``null`` - ``--ft-timeout-dump-gdb-command-file`` - User-supplied command file; required when gdb is selected * - ``cuda-gdb``: ``executable`` - ``cuda-gdb`` - ``--ft-timeout-dump-cuda-gdb`` - CUDA-GDB executable name or path * - ``cuda-gdb``: ``command_file`` - ``null`` - ``--ft-timeout-dump-cuda-gdb-command-file`` - User-supplied command file; required when cuda-gdb is selected * - ``cuda-gdb``: ``mode`` - ``batch`` - ``--ft-timeout-dump-cuda-gdb-mode`` - ``batch`` or ``stdin`` command delivery