source code of the paper: Can AI Dream of Unseen Galaxies? Conditional Diffusion Model for Galaxy Morphology Augmentation. 🌌 GalaxySD We fine-tuned sd-1.5 specialized for galaxy image generation by galaxy images with annoted morphological description based on GZ2. The galaxy morphological description dataset in natural language insteal of vote fractions will release soon. Before all experiments, you need to unzip the zipped folder and then follow the below instructions. Our project HOMEPAGE. 🧠 Arcitecture Schematic diagram of our model and downstream tasks in our paper. Please see the schematic diagram in our homepage or paper. 🛠️ Git and create environment git clone https://github.com/chenruiRae/GalaxySD.gitcd GalaxySD conda create -n galaxysdconda activate galaxysdpip install -r requirements.txtNow you have set up the workspace and could fine-tune a GalaxySD model. ⚙️ Customize configurations For example, full fine-tuning training configurations are in `GalaxySD/cfgs/train/examples/fine-tuning_galaxy.yaml`. You could customize it before using. The parameters that must be modified to ensure the pipeline run well and corresponding descriptions in `fine-tuning_galaxy.yaml` are in the following table. The fine-tuning tool we used is HCP-Diffusion. Training Parameter Description Example pretrained_model_name_or_path Pretrained model name in hugging-face / downloaded local path stable-diffusion-v1-5/stable-diffusion-v1-5 img_root Image path a folder of .jpg files caption_file Caption path a folder of .txt files whose filenames are same as corresponding images. resume Continue the previous training by filling this part or start a new training by set it to null By setting these and the rest parameters in configuration, you could start full fine-tuning. Before inference, you must modify the inference configurations in GalaxySD/cfgs/infer/text2img_galaxy_full.yaml. Inference Parameter Description Example pretrained_model Pretrained model name in hugging-face / downloaded local path stable-diffusion-v1-5/stable-diffusion-v1-5 condition Control the generation type: i2i image: 'galaxy_cond.jpg' 🚀 Get started Training bash ./sub_gal_train_full.sh Inference Fill model name and steps and give prompts in `infer_script_full.sh`. You could use the model weights in 🤗HF (doi:10.57967/hf/6479)bash ./infer_script_full.shIf you wanna view a summary of generation, uncomment the last line of `infer_script_full.sh` and keep the prompts in `create_summary.py` consistent with those in inference script. 🔗 Project Resources 🏠 Homepage 🤗 GalaxySD Model Weights 🛠️ Trained Galaxy Embedding Tool 🗂️ Training Dataset 📊 A D-ETG Catalog