v0.11.4 Build-environment fix release. Every GPU job that used a JAX engine — the multinomial, bivariate/continuous, and CNN modeling runners — was failing on the cluster, and the failure was not in the modeling code at all: torch's PyPI wheels had silently migrated to CUDA 13, which collides with the CUDA-12 wheels jax[cuda12] requires. This release pins the whole GPU stack to CUDA 12, fixes a second (independent) bug where SLURM never actually bound a GPU to array-job tasks, and floors three Dependabot advisories. No behavioural changes to processing or analysis; no data-integrity fixes. v0.11.3 results are unaffected. ⚠️ Who is affected Anyone running GPU modeling jobs (main_univariate_dispatcher / main_model_selection_dispatcher with the JAX engines). These were failing outright. Anyone who re-syncs their environment. torch now resolves from PyTorch's CUDA-12.8 index rather than PyPI (see Upgrading). Anyone with a deployed cluster checkout. A git pull alone is not sufficient — see Upgrading. Fixed The CUDA-13 / CUDA-12 cuDNN collision that broke every JAX GPU job. torch's PyPI wheels now default to a CUDA 13 build, dragging in nvidia-cudnn-cu13, nvidia-cublas 13, nvidia-cusparselt-cu13, nvidia-nccl-cu13 and cuda-toolkit 13. jax[cuda12] needs the CUDA-12 wheels. Both sets install libcudnn.so.9 into the same nvidia/cudnn/ namespace directory, so the cu13-built cuDNN wins the file and JAX's cu12 process cannot create a cuDNN handle. The job died at the very first GPU op (jnp.array) with: Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED RET_CHECK failure (gpu_compiler.cc:2641) dnn_support != nullptr torch and torchvision are now pinned to PyTorch's cu128 index (torch 2.11.0+cu128, torchvision 0.26.0+cu128), so the entire GPU stack stays on CUDA 12 and no *-cu13 wheel is installed at all. The pin is scoped to Linux (marker = "sys_platform == 'linux'"). There is no CUDA on macOS and the cu128 index ships no macOS wheel, so an unscoped pin makes the package uninstallable there; macOS and Windows fall back to PyPI, where no cu12/cu13 collision can occur. The CUDA-12 pin is only meaningful on Linux anyway — that is where jax[cuda12] and the GPU cluster live. Two notes on the diagnosis, because both cost real time: XLA's accompanying Possibly insufficient driver version: 575.57.8 is a red herring — driver 575 is perfectly adequate for CUDA 12. The message is XLA's generic fallback whenever cuDNN init fails for any reason. CUDA 13 is not a viable target here. The cluster is heterogeneous: some GPU nodes run driver 575 (CUDA 12.9), others 580 (CUDA 13.0). CUDA 12 runs on both; a CUDA-13 stack would fail whenever SLURM landed the job on a 575 node. CUDA 12 is the only safe common target. torchvision promoted to a direct dependency. It was transitive-only, so the [tool.uv.sources] cu128 pin did not apply to it and it kept resolving from PyPI — a build whose compiled ops do not match a cu128 torch. Anything importing it (sam2 box-prompt masks, vocalocator, QLVM training) failed with: RuntimeError: operator torchvision::nms does not exist This was invisible to the multinomial job (it never imports torchvision) and was only caught by running the full suite against the actual locked environment. SLURM never bound a GPU to array-job tasks (--gpus vs --gres). The modeling scripts requested #SBATCH --gpus=1. That accounts a GPU — it duly appears in AllocTRES as gres/gpu=1 — but it does not reliably bind the device into an array task's cgroup. CUDA_VISIBLE_DEVICES was left unset, and the JAX engines died with CUDA_ERROR_NO_DEVICE on a node that visibly had an idle GPU. Both scripts now use #SBATCH --gres=gpu:1, which actually binds it. The gotcha is documented inline so it does not get "fixed" back. Security Three Dependabot advisories closed by flooring two transitive dependencies ([tool.uv] constraint-dependencies): | alert | package | was | now | issue | |---|---|---|---|---| | #54 | mistune | 3.2.1 | 3.3.3 | quadratic-time DoS in parse_link_text (CWE-400) | | #53 | soupsieve | 2.8 | 2.8.4 | unbounded memory on comma-separated selector lists | | #52 | soupsieve | 2.8 | 2.8.4 | ReDoS via catastrophic backtracking in the attribute-selector VALUE regex | Both are docs-only transitive dependencies (mistune via nbsphinx/jupyterlab, soupsieve via furo/nbsphinx/jupyterlab), and usv-playpen never parses untrusted Markdown or user-supplied CSS selectors — so practical exposure is nil. They are floored anyway because uv's lock is sticky: regenerating uv.lock for an unrelated change (the torch pin, in this very release) leaves transitive deps untouched, so a patched version is only ever picked up if it is explicitly pinned. Cluster scripts (housekeeping — no behavioural change) model_selection_behavior.sh and univariate_modeling_behavior.sh moved from other/cluster/modeling/ to other/cluster/usv_playpen/, alongside every other cluster job script. The now-empty modeling/ directory is gone. XLA log spam silenced. TF_CPP_MIN_LOG_LEVEL=3 + GLOG_minloglevel=2 suppress XLA's ERROR-level device_type: DEVICE_TYPE_INVALID Triton-GEMM messages, which otherwise bury the real output in .err. Verified on the cluster: cosmetic only — no change to the computation, the results, or the runtime. GPU provenance line. Every job now logs its node, CUDA_VISIBLE_DEVICES, and nvidia-smi -L. An empty CUDA_VISIBLE_DEVICES is the immediate tell for the binding bug above, and recording which GPU ran the job matters on a heterogeneous cluster (A100-40G vs L40S-46G). Informational only — the pyGAM analysis types (onset/params/category) are CPU-only and are never failed by it. Upgrading uv sync --extra gpu --group docs On Linux, torch and torchvision now come from https://download.pytorch.org/whl/cu128 instead of PyPI: expect torch 2.11.0+cu128 and torchvision 0.26.0+cu128, and every nvidia-*-cu13 wheel to disappear. On macOS and Windows nothing changes — they continue to resolve from PyPI (CPU/MPS builds), since the CUDA pin is Linux-scoped. On a deployed cluster checkout, git pull alone is not enough — the environment still holds the mismatched PyPI torchvision until you re-sync: git pull uv sync --extra gpu # <- required; installs torchvision 0.26.0+cu128 Without it, any job importing torchvision (sam2, vocalocator, QLVM training) still hits torchvision::nms does not exist. If you copy the modeling job scripts by path, note they now live under other/cluster/usv_playpen/, and adopt --gres=gpu:1 in any local copy you maintain. Full suite green (2184 passed). Full changelog: v0.11.3...v0.11.4