cpu-gpu-performance（中央处理器 图形处理器 性能）

Table of Contents

• When to Use
• Required TodoWrite Items
• Step 1: Establish Current Baseline
• Step 2: Narrow the Scope
• Step 3: Instrument Before You Optimize
• Step 4: Throttle and Sequence Work
• Step 5: Log Decisions and Next Steps
• Output Expectations

CPU/GPU Performance Discipline

When To Use

• At the beginning of every session (auto-load alongside token-conservation).
• Whenever you plan to build, train, or test anything that could pin CPU cores or GPUs for more than a minute.
• Before retrying a failing command that previously consumed significant resources.

When NOT To Use

• Simple operations with no resource impact
• Quick single-file operations

Required TodoWrite Items

1. cpu-gpu-performance:baseline
2. cpu-gpu-performance:scope
3. cpu-gpu-performance:instrument
4. cpu-gpu-performance:throttle
5. cpu-gpu-performance:log

Step 1: Establish Current Baseline

• Capture current utilization: Note which hosts/GPUs are already busy.

- uptime - ps -eo pcpu,cmd | head - nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv

• Record any CI/cluster budgets (time quotas, GPU hours) before launching work.
• Set a per-task CPU minute / GPU minute budget that respects those limits.

Step 2: Narrow the Scope

• Avoid running "whole world" jobs after a small fix. Prefer diff-based or tag-based selective testing:

- pytest -k - Bazel target patterns - cargo test <module>

• Batch low-level fixes so you can validate multiple changes with a single targeted command.
• For GPU jobs, favor unit-scale smoke inputs or lower epoch counts before scheduling the full training/eval sweep.

Step 3: Instrument Before You Optimize

• Pick the right profiler/monitor:

- CPU work: - perf - intel vtune - cargo flamegraph - language-specific profilers - GPU work: - nvidia-smi dmon - nsys - nvprof - DLProf - framework timeline tracers

• Capture kernel/ops timelines, memory footprints, and data pipeline latency so you have evidence when throttling or parallelizing.
• Record hot paths + I/O bottlenecks in notes so future reruns can jump straight to the culprit.

Step 4: Throttle and Sequence Work

• Use nice, ionice, or Kubernetes/Slurm quotas to prevent starvation of shared nodes.
• Chain heavy tasks with guardrails:

- Rerun only the failed test/module - Then (optionally) escalate to the next-wider shard - Reserve the full suite for the final gate

• Stagger GPU kernels (smaller batch sizes or gradient accumulation) when memory pressure risks eviction; prefer checkpoint/restore over restarts.

Step 5: Log Decisions and Next Steps

Conclude by documenting the commands that were run and their resource cost (duration, CPU%, GPU%), confirming whether they remained within the per-task budget. If a full suite or long training run was necessary, justify why selective or staged approaches were not feasible. Capture any follow-up tasks, such as adding a new test marker or profiling documentation, to simplify future sessions.

Output Expectations

• Brief summary covering:

- baseline metrics - scope chosen - instrumentation captured - throttling tactics - follow-up items

• Concrete example(s) of what ran (e.g.):

- "reran pytest tests/test_orders.py -k test_refund instead of pytest -m slow" - "profiled nvidia-smi dmon output to prove GPU idle time before scaling"