Abstract Acyl homoserine lactone (AHL)-based quorum sensing commonly refers to cell density-dependent regulatory mechanisms found in bacteria. However, beyond bacteria, this cell-to-cell communication mechanism is poorly understood. Here we show that a methanogenic archaeon, Methanosaeta harundinacea 6Ac, encodes an active quorum sensing system that is used to regulate cell assembly and carbon metabolic flux. The methanogen 6Ac showed a cell density-dependent physiology transition, which was related to the AHL present in the spent culture and the filI gene-encoded AHL synthase. Through extensive chemical analyses, a new class of carboxylated AHLs synthesized by FilI protein was identified. These carboxylated AHLs facilitated the transition from a short cell to filamentous growth, with an altered carbon metabolic flux that favoured the conversion of acetate to methane and a reduced yield in cellular biomass. The transcriptomes of the filaments and the short cell forms differed with gene expression profiles consistent with the physiology. In the filaments, genes encoding the initial enzymes in the methanogenesis pathway were upregulated, whereas those for cellular carbon assimilation were downregulated. A luxI–luxR ortholog filI–filR was present in the genome of strain 6Ac. The carboxylated AHLs were also detected in other methanogen cultures and putative filI orthologs were identified in other methanogenic genomes as well. This discovery of AHL-based quorum sensing systems in methanogenic archaea implies that quorum sensing mechanisms are universal among prokaryotes.