# depmap_omics  This repository contains code that processes data for the biannual [DepMap](https://www.depmap.org) data release. State of the pipeline for each release can be found under the "Releases" tab in this repo. ## Table of Contents - [Getting Started](#quickstart) - [Installation](#installation) - [Repository File Structure](#file-structure) - [Running the Pipeline](#running-pipeline) - [Uploading and Preprocessing](#upload-preprocess) - [Running Terra Pipelines](#running-terra-pipelines) - [Downloading and Postprocessing](#downloading-postprocessing) - [QC, Grouping and Uploading](#qc-grouping-uploading) ## Getting Started The processing pipeline relies on the following tools: - [python](https://www.learnpython.org/) - [R](https://www.codecademy.com/learn/learn-r) - [jupyter](https://jupyter.org/index.html) - [WDL](https://software.broadinstitute.org/wdl/documentation/) - [gcp](https://cloud.google.com/sdk/docs/quickstart-macos) - [docker](https://docs.docker.com/get-started/) - [Terra](https://software.broadinstitute.org/firecloud/documentation/) - [The Terra Convention: The dos and donts for maintaining a cleaner terra.](https://docs.google.com/document/d/1zTtaN-Px64f8JvgydZNdBbzBpFWyZzEpshSNxQh43Oc/edit#heading=h.dz5wh0l4bu9g) - [dalmatian](https://github.com/broadinstitute/dalmatian) ### Installatiion `git clone http://github.com/BroadInstitute/depmap_omics.git && cd depmap_omics` `pip install -e .` ### :warning: This repository needs other repos Some important data and code from the [genepy Library](https://github.com/broadinstitute/genepy). Use the instructions in the genepy page to install the package. ### :warning: You need the following R and python packages 1. You will need to install jupyter notetbooks and google cloud sdk - install [Google Cloud SDK](https://cloud.google.com/sdk/docs/downloads-interactive). - authenticate my SDK account by running `gcloud auth application-default login` in the terminal, and follow the instrucion to log in. 2. and GSVA for ssGSEA in R `R` run `R -e 'if(!requireNamespace("BiocManager", quietly = TRUE)){install.packages("BiocManager")};BiocManager::install(c("GSEABase", "erccdashboard", "GSVA", "DESeq2"));'` 3. For Python use the requirements.txt file `pip install -r requirements.txt` ### :warning: Follow instructions [here](documentation/getting_started.md) to set up Terra and obtain access to services required for running the pipeline. ## Repository File Structure __ccle_tasks/__ Contains a notebook for each of the different additional processing that the CCLE team has to perform as well as one-off tasks run by the omics team __data/__ Contains important information used for processing, including terra workspace configurations from past quarters __depmapomics/__ Contains the core python code used in the pipeline and called by the processing jupyter notebooks __\*\_pipeline/__ Contains some of the workflows' wdl files and script files used by these workflows __temp/__ Contains the temp file that can get removed after processing (should be empty) __documentation/__ Contains some additional files and diagrams for documenting the pipelines __tests/__ Contains automated pytest functions used internally for development __jupyter notebooks:__ `RNA_CCLE.ipynb` contains the DepMap processing pipelines for Expression and Fusion (from RNAseq data), and `WGS_CCLE.ipynb` contains the DepMap processing pipelines for Copy number and Mutations (from WGS/WES data) ## Pipeline Walkthrough The processing pipelines are encapsulated in two jupyter notebooks (`RNA_CCLE.ipynb` and `WGS_CCLE.ipynb`). Each is divided into four steps: uploading, running Terra pipelines, local postprocessing, and uploading. Here is a detailed walkthrough (_Note that the steps that are "internal only" are run as part of DepMap's data processing, but not meant for external users to reproduce due to various dependencies that are unique to our team at the Broad. The "internal only" functions below can be found in the [depmap_omics_upload repo](https://github.com/broadinstitute/depmap_omics_upload)_): ### 1. Uploading and Preprocessing (internal only) Currently, sequenced data for DepMap is generated by the Genomics Platform (GP) at the Broad who deposits them into several different Terra workspaces. Therefore, the first step of this pipeline is to look at these workspaces and - identify new samples by looking at the bam files and compare them with bams we have already onboarded - remove duplicates and ones with broken file paths - onboard new samples and new versions of old cell lines if we find any ### 2. Running Terra Pipelines We are using Dalmatian to send requests to Terra, so before running this part, external users need to make sure that the dalmatian `WorkspaceManager` object is initialized with the right workspace and that the functions are taking the correct workflow names as inputs. You can then run the RNAseq and/or WGS pipelines on your samples. **For a more in-depth documentation on what our pipelines contain, including the packages, input references, and parameters, please refer to this [summary of DepMap processing pipeline](documentation/DepMap_processing_pipeline.md).** ### 3. Downloading and Postprocessing (sections under **on local** in the notebooks) This step will do a set of tasks: - clean the workspaces by deleting large useless files, including unmapped bams. - retrieve from the workspace interesting QC results. - copy realigned bam files to our own data storage bucket (internal only). - download the outputs from Terra pipelines. The main postprocessing steps for each pipeline are as followed: #### Copy Number `copynumbers.py` contains the main postprocessing function `postProcess()` responsible for postprocessing segments and creating gene-level (relative and absolute) CN files and genomic feature table. Gene mapping information is retrieved from BioMart version `may2021`. The function also applies the following filters to segment and CN data: * Remove chrY segments from cell lines where their chrY segment count is bigger than 150 * Mark samples that have more than 1500 segments as QC fails and remove them * Remove genes whose Entrez ID is NaN in BioMart in the gene-level matrices _Internal only: `dm_omics.cnPostProcessing()` calls the above function on both WES and WGS data, merges them, renames the indices into ProfileIDs, and upload them to taiga._ Note: to get the exact same results as in DepMap, be sure to run `genecn = genecn.apply(lambda x: np.log2(1+x))` to the genecn dataframe in the CNV pipeline #### Mutation `mutations.py` contains `postProcess()`, the function responsible for postprocessing aggregated MAF files, genotyped mutation matrices (hot spot and damaging), binary guide mutation matrices, and structural variants (SVs). _Internal only: `dm_omics.mutationPostProcessing()` calls the above function on both WES and WGS data, merges them, renames the indices into ProfileIDs, removes genes whose hugo symbol is not in biomart, generates individual mutation datasets for variant types, and uploads them to taiga. It also generates and uploads a binary matrix for germline mutations._ #### Expression `expressions.py` contains the main postprocessing function responsible for postprocessing aggregated expression data from RSEM, which removes duplicates and QC failures, renames genes, filters and log transforms values, and generates transcrip-level, gene-level, and protein-coding gene-level expression data files. Gene mapping information is retrieved from BioMart version `may2021`. Optionally, in addition, it also generates Single-sample GSEA (ssGSEA) data. _Internal only: `dm_omics.expressionPostProcessing()` is a wrapper for the above function. It renames the indices into ProfileIDs and uploads the files to taiga._ #### Fusion Functions that postprocess aggregated fusion data can be found in `fusions.py`. We want to apply filters to the fusion table to reduce the number of artifacts in the dataset. Specifically, we filter the following: * Remove fusions involving mitochondrial chromosomes, or HLA genes, or immunoglobulin genes * Remove red herring fusions (from STAR-Fusion annotations column) * Remove fusions recurrent in CCLE (>= 25 samples) * Remove fusions that have (SpliceType=" INCL_NON_REF_SPLICE" AND LargeAnchorSupport="No" AND FFPM 0.005% - except if it is either - in TCGA > 3 times - OR in Cosmic > 10 times - AND in a set of known cancer regions. - OR exist in >5% of the CCLE samples - except if they are in TCGA >5 times #### RNA Once the expression files are saved, we do the following validations: - mean, max, var... - comparison to previous release: same mean, max, var... - we QC on the amount of genes with 0 counts for each samples After QC, data is uploaded to taiga for all portal audiences according to release dates in Gumbo. @[jkobject](https://www.jkobject.com) @gkugener @gmiller @5im1z @__[BroadInsitute](https://www.broadinstitute.org) If you have any feedback or run into any issues, feel free to post an issue on the github repo.