CytoTable

Diagram showing data flow relative to this project.

Summary

CytoTable enables single-cell morphology data analysis by cleaning and transforming CellProfiler (.csv or .sqlite), cytominer-database (.sqlite), and DeepProfiler (.npz), and other sources such as IN Carta data output data at scale. CytoTable creates parquet files for both independent analysis and for input into Pycytominer. The Parquet files will have a unified and documented data model, including referenceable schema where appropriate (for validation within Pycytominer or other projects).

The name for the project is inspired from:

• Cyto: “1. (biology) cell.” (Wiktionary: Cyto-)

• Table:

  • “1. Furniture with a top surface to accommodate a variety of uses.”

  • “3.1. A matrix or grid of data arranged in rows and columns.”
    (Wiktionary: Table)

Installation

Install CytoTable from PyPI or from source:

# install from pypi
pip install cytotable

# install directly from source
pip install git+https://github.com/cytomining/CytoTable.git

Contributing, Development, and Testing

We test CytoTable using ubuntu-latest and macos-latest GitHub Actions runner images.

Please see contributing.md for more details on contributions, development, and testing.

References

• pycytominer

• cytominer-database

• DeepProfiler

• CellProfiler

• cytominer-eval

Contents:

• Overview
  • Presets and Manual Overrides
  • Data Sources
    • Data Source Locations
    • Data Source Types
  • Data Destinations
    • Data Destination Locations
    • Data Destination Types
  • Data Transformations
    • Data Chunking
    • Data Concatenations
    • Data Joins
  • Pagination
• Tutorial
  • CellProfiler CSV Output to Parquet
• Contributing
  • Code of conduct
  • Security
  • Quick links
  • Process
    • Bug reporting
    • Suggesting enhancements
    • Your first code contribution
    • Pull requests
    • Git commit messages
  • Development
    • Overview
    • Getting started
    • Code style
    • Linting
    • Testing
  • Documentation
    • Documentation Linting
    • Documentation Builds
  • Attribution
  • Publishing Releases
    • Release Locations
    • Release Publishing Process
• Code of Conduct
  • Our Pledge
  • Our Standards
  • Enforcement Responsibilities
  • Scope
  • Enforcement
  • Enforcement Guidelines
    • 1. Correction
    • 2. Warning
    • 3. Temporary Ban
    • 4. Permanent Ban
  • Attribution
• Architecture
  • Technical Architecture
    • Workflows
    • Data Technologies
  • Data Architecture
    • Sources
    • Data structure
    • Relationships
• Python API
  • Convert
    • _to_parquet()
    • convert()
    • _concat_join_sources()
    • _concat_source_group()
    • _get_table_columns_and_types()
    • _get_table_keyset_pagination_sets()
    • _infer_source_group_common_schema()
    • _join_source_pageset()
    • _prepare_join_sql()
    • _prep_cast_column_data_types()
    • _set_tablenumber()
    • _prepend_column_name()
    • _source_pageset_to_parquet()
  • Sources
    • _build_path()
    • _file_is_more_than_one_line()
    • _filter_source_filepaths()
    • _gather_sources()
    • _get_source_filepaths()
    • _infer_source_datatype()
  • Utils
    • Parsl_AppBase_init_for_docs()
    • _arrow_type_cast_if_specified()
    • _cache_cloudpath_to_local()
    • _column_sort()
    • _default_parsl_config()
    • _duckdb_reader()
    • _expand_path()
    • _gather_tablenumber_checksum()
    • _generate_pagesets()
    • _get_cytotable_version()
    • _natural_sort()
    • _parsl_loaded()
    • _sqlite_mixed_type_query_to_parquet()
    • _unwrap_source()
    • _unwrap_value()
    • _write_parquet_table_with_metadata()
    • evaluate_futures()
  • Presets
    • config
  • Exceptions
    • CytoTableException
    • DatatypeException
    • NoInputDataException
    • SchemaException