# Aqua Working Rules (for agents and humans) Distilled from the Walltime Chronicles agent guide and cheatsheet on 2026-09-22. Paraphrased rather than copied; follow the source links for the authoritative wording. - Entry point: - Agent guide: - Cheatsheet: ## System model Aqua is a shared PBS cluster. Login nodes are the SSH entry point for lightweight prep and inspection only; real work runs on compute nodes inside a scheduler allocation, interactive or batch. Submitting a job is not running it — queue time is normal and is not evidence of failure. Filesystems differ in purpose, performance, quota, and retention, so a temporary compute location is never the only copy of anything that matters. A command that works in an interactive shell can still fail unchanged inside a batch job, because the environment is not the same. ## Action boundaries Ask for explicit approval, per specific action, before: - Any queue mutation: submit, cancel/delete, requeue, alter, hold, release — including when it happens indirectly through a script, API, MCP tool, or SSH helper. - Any new allocation: interactive sessions, test jobs, arrays, replacement or dependency jobs. - Any retry or recovery loop. Approval to submit once is not approval to resubmit. - Deleting or overwriting files, recursive permission changes, storage-attribute changes. - Modifying shared environments, installing software, starting persistent services, changing credentials, or uploading anything to an external service. - Closing an allocation, and sending a support ticket. An approval request must carry the exact command or script, the resource request (CPUs, memory, GPUs, walltime), the job count or array range or job IDs, and the purpose and consequences. Propose concretely — show the script, working directory, and affected IDs — then ask. Never: - Weaken permission prompts, sandboxes, or access controls to get past a restriction. - Treat a filled-in config, an account capability, or a past task's approval as permission now. - Change permissions, HSM attributes, or timestamps to dodge retention policy. - Read credential stores, private keys, tokens, full environment dumps, or sensitive records. - Create persistent watchers or recurring tasks unasked. - Resubmit after a timeout without first inspecting scheduler state. - Bundle an unrequested repair into a diagnostic check. - Fabricate job IDs, output contents, resource measurements, or checks that were not run. No approval needed, inside the authorized scope: reading files, preparing and editing scripts, lightweight local checks, and explaining or reviewing existing work. ## Running work Keep training, benchmarks, inference, substantial preprocessing, and heavy tests off login nodes; a small dataset does not make a workload appropriate there. Bound thread counts, workers, and concurrent tasks to the allocation rather than to the physical CPU count, and account for nested parallelism. Reserve only what is needed, say when an allocation can be released, and never hold a reservation with dummy jobs or an idle interactive session. Prefer an approved PBS wrapper or a script that has already succeeded; otherwise build from the documented Aqua examples. Request resources and let PBS place them unless a documented need says otherwise, leave scheduler-provided GPU visibility alone, and route PBS output, application logs, results, caches, and temporaries to their intended locations. Source environment setup in the same shell that runs the program. For Python with `uv`, follow the uv-on-Aqua guide for environment and cache placement. Reading and searching also load the shared system: keep metadata operations narrow, prefer bounded log excerpts over whole logs, and check availability, access, and storage tier before bulk reads. Recursive walks and repeated listings are expensive even when read-only. ## Verification and diagnosis Monitor only authorized runs, with bounded read-only queries, a reasonable polling interval, back-off on errors, and a clear statement of when monitoring stops. None of these prove success: the job vanishing from the queue, a zero exit code, a checkpoint file, a log that stopped growing, or a wrapper that returned cleanly while skipping a task or swallowing an application error. Distinguish submitted, queued, running, failed, completed, and unknown. | Observation | What to check | | --- | --- | | Submission rejected | The scheduler message, against queue limits, account requirements, syntax | | Still queued | Job state; waiting alone is not a broken job | | Exited early | Init, working directory, input paths, environment, permissions, first error | | Hit a limit | Confirm the exit reason, then propose a justified request or recovery | | Log stopped growing | Buffering, application phase, output path, scheduler state | | "Done" but results thin | Compare the work actually performed and outputs against the requested config | | Status unretrievable | Report it as unknown and preserve the last observation with its time | Preserve original evidence — logs, checkpoints, caches — while diagnosing. Do not try to repair shared infrastructure; prepare a concise ticket instead (command, job ID, observation time, error excerpt, checks performed), redacted, and let the user approve sending it. Close out with: the action taken and job IDs; the verified scheduler and application state with a timestamp; the evidence, including whether expected outputs exist; where the scripts, logs, checkpoints, and results are; and the next step or anything still needing approval. ## Command reference Every `qsub` command and every PBS script header needs the project code (RPID) via `-P`. Submission and control: | Command | Purpose | | --- | --- | | `qsub