
We analyze a federated, peer-to-peer LLM training architecture that uses delta compression,BitTorrent-style chunked model distribution, and hierarchical merging to coordinate trainingacross thousands of consumer GPUs. The architecture is internally coherent and contains severalnon-trivial engineering decisions worth documenting; it is also, for the intended use case oftraining frontier-scale language models, the wrong shape of the problem. We characterize sevenconcrete failure modes – bandwidth, straggler effect, FedAvg convergence under non-IID data,the consumer-VRAM ceiling, total cost of training, the security envelope of the delta-validationrules, and data provenance – each paired with a reproducible Python script. The conclusion isthat for frontier-scale models the centralized cluster is faster, cheaper, and safer by enough thatdistributed federated training is economically and mathematically dominated. We close with ashort list of regimes where federated training remains the right tool.
