Chapter 5 Create Profiles

5.1 Confirm Environment Configuration

You CAN, if you choose, make your backends with the same machine that you have used to make CSVs and run DCP for CellProfiler. However, we typically do not, since backend creation can take a long time so it is desirable to use 2 machines - a small inexpensive machine with only a few CPUs for all the steps before backends and a larger machine with at least as many CPUs as (the number of plates in your batch +1) for backend creation. Both machines can be turned off when not in use and when off will generate only minimal charges.

To make a new backend machine, follow the identical instructions as the in the section below, only with a larger Instance Type (such as an m4.10xlarge).

Since backend creation also typically requires a large amount of hard disk space, which you pay for whether or not the machine is on, we recommend attaching a large EBS volume to your backend creation machine only when needed, then detaching and terminating it when not in use.

5.2 Add a large EBS volume to your machine

Follow the AWS instructions for creating and attaching the EBS volume. Two critical factors to note- the volume must be created in the same subnet as your backend creation machine, and should be approximately 2X the size as the analysis files in your batch - this is most easily figured by navigating to that location in the S3 web console and selecting "Actions -> Calculate total size".

Once the volume is created and attached, ensure the machine is started and SSH connect to it.

Create a temp directory which is required when creating the database backed using cytominer-database (discussed later).

mkdir ~/ebs_tmp

Get the name of the disk and attach it.

# check the name of the disk

#> xvda    202:0    0     8G  0 disk
#> └─xvda1 202:1    0     8G  0 part /
#> xvdba    202:80   0   100G  0 disk

# check if it has a file system
sudo file -s /dev/xvdba
# ...likely not, in which case you get:
#> /dev/xvdf: data

# if no file system, then create it
sudo mkfs -t ext4 /dev/xvdba

# mount it
sudo mount /dev/xvdba /home/ubuntu/ebs_tmp/

# change perm
sudo chmod 777 ~/ebs_tmp/

If you are starting from here, make sure the following steps have been completed on your ec2 instance and/or session before proceeding

5.3 Create Database Backend

Run creation of sqlite backend as well as aggregation of measurements into per-well profiles. This process can be very slow since the files are read from s3fs/EFS. We recommend first downloading the CSVs files locally on an EBS volume attached to the ec2 instance you are running on, and then ingesting.

To do so, first recreate the analysis output folder structure on the EBS volume:

mkdir -p ~/ebs_tmp/${PROJECT_NAME}/workspace/software

cd ~/ebs_tmp/${PROJECT_NAME}/workspace/software

if [ -d pycytominer ]; then rm -rf pycytominer; fi

git clone

cd pycytominer

git checkout jump

python3 -m pip install -e .

The command below first calls cytominer-database ingest to create the SQLite backend, and then pycytominer's aggregate_profiles to create per-well profiles. Once complete, all files are uploaded to S3 and the local cache are deleted. This step takes several hours, but metadata creation and GitHub setup can be done in this time. ingests and indexes the database.

pyenv shell 3.8.10

mkdir -p  ../../log/${BATCH_ID}/
parallel \
--max-procs ${MAXPROCS} \
--ungroup \
--eta \
--joblog ../../log/${BATCH_ID}/collate.log \
--results ../../log/${BATCH_ID}/collate \
--files \
--keep-order \
python3 pycytominer/cyto_utils/ ${BATCH_ID}  pycytominer/cyto_utils/ingest_config.ini {1} \
--temp ~/ebs_tmp \
--remote=s3://${BUCKET}/projects/${PROJECT_NAME}/workspace :::: ${PLATES} does not recreate the SQLite backend if it already exists in the local cache. Add --overwrite flag to recreate.
For pipelines that use FlagImage to skip the measurements modules if the image failed QC, the failed images will have Image.csv files with fewer columns that the rest (because columns corresponding to aggregated measurements will be absent). The ingest command will show a warning related to sqlite: expected X columns but found Y - filling the rest with NULL. This is expected behavior.
There is a known issue where if the alphabetically-first CSV failed QC in a pipeline where "Skip image if flagged" is turned on, the databases will not be created. We are working to fix this, but in the meantime we recommend either not skipping processing of your flagged images (and removing them from your data downstream) or deleting the alphabetically-first CSVs until you come to one where the pipeline ran completely.

This is the resulting structure of backend on S3 (one level below workspace) for SQ00015167:

└── backend
    └── 2016_04_01_a549_48hr_batch1
        └── SQ00015167
            ├── SQ00015167.csv
            └── SQ00015167.sqlite

At this point, the user needs to use the profiling template to use pycytominer to annotate the profiles with metadata, normalize them, and feature select them.

5.4 Create Metadata Files

First, get metadata for the plates. This should be created beforehand and uploaded into S3.

This is the structure of the metadata folder (one level below workspace):

└── metadata
    └── platemaps
        └── 2016_04_01_a549_48hr_batch1
            ├── barcode_platemap.csv
            └── platemap
                └── C-7161-01-LM6-006.txt

2016_04_01_a549_48hr_batch1 is the batch name – the plates (and all related data) are arranged under batches, as seen below.

barcode_platemap.csv is structured as shown below. Assay_Plate_Barcode and Plate_Map_Name are currently the only mandatory columns (they are used to join the metadata of the plate map with each assay plate). Each unique entry in the Plate_Map_Name should have a corresponding tab-separated file .txt file under platemap (e.g. C-7161-01-LM6-006.txt).


The tab-separated files are plate maps and are structured like this: (This is the typical format followed by Broad Chemical Biology Platform)

plate_map_name  well_position broad_sample  mg_per_ml mmoles_per_liter  solvent
C-7161-01-LM6-006 A07 BRD-K18895904-001-16-1  3.12432000000000016 9.99999999999999999 DMSO
C-7161-01-LM6-006 A08 BRD-K18895904-001-16-1  1.04143999999919895 3.33333333333076923 DMSO
C-7161-01-LM6-006 A09 BRD-K18895904-001-16-1  0.347146666668001866  1.11111111111538462 DMSO
  • plate_map_name should be identical to the name of the file (without extension).
  • plate_map_name and well_position are currently the only mandatory columns.

The external metadata is an optional file tab separated .tsv file that contains the mapping between a perturbation identifier to other metadata. The name of the perturbation identifier column (e.g. broad_sample) should be the same as the column in platemap.txt.

The external metadata file should be placed in a folder named external_metadata within the metadata folder. If this file is provided, then the following should be the folder structure

└── metadata
    ├── external_metadata
    │   └── external_metadata.tsv
    └── platemaps
        └── 2016_04_01_a549_48hr_batch1
            ├── barcode_platemap.csv
            └── platemap
                └── C-7161-01-LM6-006.txt

5.5 Set up GitHub

Once and only once - fork the profiling recipe to your own user name (Each time you make a new project - you may want to keep your fork up to date

Once per new PROJECT, not new batch - make a copy of the template repository into your preferred organization with a project name that is similar OR identical to its project tag on S3 and elsewhere.

5.6 Make Profiles

5.6.1 Optional - set up compute environment

These final steps are small and can be done either in your local environment or on your node used to build the backends. Conda and Git LFS are currently required. For now, the backend creation VMs don’t contain conda or git lfs, but in the meantime these are the commands used to install both for a Linux ecosystem (otherwise, search for your own OS).

wget -O ~/
bash ~/ -b -p $HOME/miniconda
export PATH="/home/ubuntu/miniconda/bin:$PATH"
source .bashrc
conda init bash
source .bashrc

curl -s | sudo bash

If not using the same machine + tmux as for making backends, where your environment variables are already set, set them up

5.6.2 Set new environment variables

Specifically, ORG and DATA should be the GitHub organization and repository name used when creating the data repository from the template. USER should be your GitHub username. CONFIG_FILE will be the name of the config file used for this run, so something that makes it distinguishable (ie, batch numbers being run at this time) is helpful.


5.6.3 If a first batch in this compute environment, make some directories

mkdir -p ~/work/projects/${PROJECT_NAME}/workspace/{backend,software}

5.6.4 Add your backend files

aws s3 sync s3://${BUCKET}/projects/${PROJECT_NAME}/workspace/backend/${BATCH_ID} ~/work/projects/${PROJECT_NAME}/workspace/backend/${BATCH_ID} --exclude="*" --include="*.csv"

5.6.5 If a first batch in this compute environment, clone your repository

cd ~/work/projects/${PROJECT_NAME}/workspace/software
git clone${ORG}/${DATA}.git
#depending on your repo/machine set up you may need to provide credentials here
cd ${DATA}

5.6.6 If a first batch in this project, weld the recipe into the repository

git submodule add${USER}/profiling-recipe.git profiling-recipe
git add profiling-recipe
git add .gitmodules
git commit -m 'finalizing the recipe weld'
git push
#depending on your repo/machine set up you may need to provide credentials here
git submodule update --init --recursive

5.6.7 If a first batch in this compute environment, set up the environment

cp profiling-recipe/environment.yml .
conda env create --force --file environment.yml

5.6.8 Activate the environment

conda activate profiling

5.6.9 If a first batch in this project, create the necessary directories


5.6.10 Download the load_data_CSVs

aws s3 sync s3://${BUCKET}/projects/${PROJECT_NAME}/workspace/load_data_csv/${BATCH_ID} load_data_csv/${BATCH_ID}
gzip -r  load_data_csv/${BATCH_ID}

5.6.11 Download the metadata files

aws s3 sync s3://${BUCKET}/projects/${PROJECT_NAME}/workspace/metadata/${BATCH_ID} metadata/platemaps/${BATCH_ID}

5.6.12 Make the config file

cp profiling-recipe/config_template.yml config_files/${CONFIG_FILE}.yml
nano config_files/${CONFIG_FILE}.yml

The changes you will likely need to make for most small use cases following this handbook- for aggregate set perform to false, for annotate sub-setting external set perform to false, in feature_select set gct to true, and finally at the bottom set the batch(es) and plates names

For large batches with many DMSO wells and external metadata ala the JUMP project - set perform under external to true, set file to the name of the external metadata file and set merge_column to the name of the compound identifier column in platemap.txt and external_metadata.tsv.

5.6.13 Set up the profiles

Note that the “find” step can take a few seconds/minutes

mkdir -p profiles/${BATCH_ID}
find ../../backend/${BATCH_ID}/ -type f -name "*.csv" -exec profiling-recipe/scripts/ {} \;
rsync -arzv --include="*/" --include="*.gz" --exclude "*" ../../backend/${BATCH_ID}/ profiles/${BATCH_ID}/

5.6.14 Run the profiling workflow

Especially for large number of plates, this will take some time. Output will be logged to the console as different steps proceed.

python profiling-recipe/profiles/  --config config_files/{$CONFIG_FILE}.yml

5.6.15 Push resulting files back up to GitHub

git add *
git commit -m 'add profiles for batch _'
git push

5.6.16 Push resulting files up to S3

parallel aws s3 sync {1} s3://${BUCKET}/projects/${PROJECT_NAME}/workspace/{1} ::: config_files gct profiles quality_control