CytoTable

Diagram showing data flow relative to this project.

Summary

CytoTable enables single-cell morphology data analysis by cleaning and transforming CellProfiler (.csv or .sqlite), cytominer-database (.sqlite), and DeepProfiler (.npz), and other sources such as IN Carta output data at scale. CytoTable creates parquet files for both independent analysis and for input into Pycytominer. The output files (such as Parquet and anndata file formats) have a documented data model, including referenceable schema where appropriate (for validation within Pycytominer or other image-based profiling projects).

📔🧑‍🏫 Check out our demonstration notebook for a one-stop reference of how to use CytoTable. CytoTable mise en place (general overview)

📓☁️ Consider also checking out our demonstration notebook covering how to use cloud-based data sources with CytoTable. CytoTable from the cloud (cloud-based overview)

The name for the project is inspired by:

• Cyto: “1. (biology) cell.” (Wiktionary: Cyto-)

• Table:

  • “1. Furniture with a top surface to accommodate a variety of uses.”

  • “3.1. A matrix or grid of data arranged in rows and columns.”
    (Wiktionary: Table)

Installation

Install CytoTable from PyPI or from source:

# install from pypi
pip install cytotable

# install directly from source
pip install git+https://github.com/cytomining/CytoTable.git

Contributing, Development, and Testing

We test CytoTable using ubuntu-latest and macos-latest GitHub Actions runner images.

Please see contributing.md for more details on contributions, development, and testing.

Relationship to other projects

CytoTable focuses on image-based profiling data harmonization and serialization. At scale, CytoTable transforms data into file formats which can be directly integrated with:

Please let us know how you use CytoTable (we’d love to add your project to this list)!

• Pycytominer for the bioinformatics pipeline for image-based profiling.

• coSMicQC for quality control.

• CytoDataFrame for interactive visualization of profiles with single cell images.

References

• Pycytominer

• cytominer-database

• DeepProfiler

• CellProfiler

• cytominer-eval

Contents:

• Overview
  • Presets and Manual Overrides
  • Data Sources
    • Data Source Locations
    • Data Source Types
  • Data Destinations
    • Data Destination Locations
    • Data Destination Types
  • Data Transformations
    • Data Chunking
    • Data Concatenations
    • Data Joins
  • Pagination
• Tutorial
  • CellProfiler CSV Output to Parquet
• Examples
  • CytoTable mise en place (general overview)
    • Phase 1: Cells are imaged by microscopes, with optional fluorescence staining
    • Phase 2: CellProfiler segments cells and measures numeric features
    • Phase 3: CytoTable harmonizes the feature datasets that CellProfiler generates
  • CytoTable from the cloud (using cloud-based data sources)
• Presentations
  • Presentation - ParslFest 2025
  • Poster - SBI² 2023
  • Poster - SciPy 2023
• Contributing
  • Code of conduct
  • Security
  • Quick links
  • Process
    • Bug reporting
    • Suggesting enhancements
    • Your first code contribution
    • Pull requests
    • Git commit messages
  • Development
    • Overview
    • Getting started
    • Code style
    • Linting
    • Testing
  • Documentation
    • Documentation Linting
    • Documentation Builds
  • Attribution
  • Publishing Releases
    • Release Locations
    • Release Publishing Process
• Code of Conduct
  • Our Pledge
  • Our Standards
  • Enforcement Responsibilities
  • Scope
  • Enforcement
  • Enforcement Guidelines
    • 1. Correction
    • 2. Warning
    • 3. Temporary Ban
    • 4. Permanent Ban
  • Attribution
• Architecture
  • Technical Architecture
    • Workflows
    • Data Technologies
  • Data Architecture
    • Sources
    • Data structure
    • Relationships
• Python API
  • Convert
    • _run_export_workflow()
    • convert()
    • _concat_join_sources()
    • _concat_source_group()
    • _get_table_columns_and_types()
    • _get_table_keyset_pagination_sets()
    • _infer_source_group_common_schema()
    • _join_source_pageset()
    • _prepare_join_sql()
    • _prep_cast_column_data_types()
    • _set_tablenumber()
    • _prepend_column_name()
    • _source_pageset_to_parquet()
  • Sources
    • _build_path()
    • _file_is_more_than_one_line()
    • _filter_source_filepaths()
    • _gather_sources()
    • _get_source_filepaths()
    • _infer_source_datatype()
  • Utils
    • Parsl_AppBase_init_for_docs()
    • _arrow_type_cast_if_specified()
    • _cache_cloudpath_to_local()
    • _column_sort()
    • _default_parsl_config()
    • _duckdb_reader()
    • _expand_path()
    • _extract_npz_to_parquet()
    • _gather_tablenumber_checksum()
    • _generate_pagesets()
    • _get_cytotable_version()
    • _natural_sort()
    • _parsl_loaded()
    • _sqlite_mixed_type_query_to_parquet()
    • _unwrap_source()
    • _unwrap_value()
    • _write_parquet_table_with_metadata()
    • evaluate_futures()
    • find_anndata_metadata_field_names()
    • map_pyarrow_type()
  • Presets
    • config
  • Exceptions
    • CytoTableException
    • DatatypeException
    • NoInputDataException
    • SchemaException