Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
Evaluating the Quality of Cell Counting Methods: Experimental Design and Statistical Analysis
Summary
Cell counting is a fundamental measurement in biotechnology and is especially important for novel cell-based therapies, where cell count-based measurements are used to monitor the manufacturing processes of cellular therapeutic products and to release and appropriately dose the therapeutic. With a wide diversity in cell types, cell counting methods and purposes for cell counting, it is difficult to envision any one approach to counting that can meet all needs in biotechnology. In this project we aim to develop widely applicable tools to evaluate the performance of cell counting methods in the absence of a reference material, improving confidence in cell counting and assisting in the selection and development of cell counting methods that are fit for their intended purpose.
This work was developed in close collaboration with the Statistical and Engineering Division at NIST.
Description
Related Standards/Tools
COMET (Counting Method Evaluation Tool)
- A Shiny Application that executes the statistical analysis and reporting outlined in ISO 20391-2:2019.
- BETA VERSION NOW AVAILABLE: https://cell-counting.shinyapps.io/COMET/
Overview
Challenge
- Different counting methods can give different results
- How can you evaluate the quality of a counting method in the absence of a reference material?
Objective
- Improve confidence in cell counts
- Provide analytical metrics to quantify cell counting method performance
- Facilitate comparison between cell counting methods
Approach
- Dilution series experimental design and statistical framework to quantify cell counting performance, independent of measurement platforms and in the absence of a reference material and/or reference measurement
- Utilize the fundamental property of proportionality of cell count to dilution as an internal control to evaluate cell counting quality
A lack of suitable reference materials
An ideal cell count achieves high levels of both precision and accuracy. While a typical cell count may have good precision, its level of accuracy is often difficult to assess, due to a lack of appropriate reference materials. The figure below depicts the precision and accuracy of the ideal cell count, a typical cell count, and an unacceptable cell count. (See figure below
Evaluating the quality of cell counting methods in the absence of reference materials
While true accuracy cannot be assessed for a typical cell count when there is a lack of an appropriate reference material, other critical aspects of measurement quality can be evaluated. Here we have developed a method for evaluating aspects of the quality of a cell counting method through the use of a dilution series experimental design and statistical analysis. From the experimental design, a set of quality indicators is derived to assess the performance of a cell counting method. These quality indicators are based on precision of the method and proportionality of counts to dilution. This approach is relevant in aspects of selecting, optimizing, validating, and monitoring cell counting measurements that are fit-for-purpose.
Precision and The Principle of Proportionality: Proportionality to dilution serves as an internal control to evaluate method performance
High confidence in cell counting implies that the measurement is both accurate and precise. Precision of a cell count measurement indicates the closeness of agreement between replicate measurements of the same material under repeatable conditions and is critical in assuring confidence in measurement results. The concept of proportionality may be used as an internal reference when dilution fractions are well-controlled, and deviation from proportionality (quantified by residuals) serves as an alternative approach to the direct evaluation of accuracy. While applying the principle of proportionality does not provide the accuracy of the cell count, deviation from proportionality indicates that a systematic measurement error has occurred to reduce the overall measurement confidence.
Key Concepts
- Proportionality must hold true for an accurate cell counting process
- Deviation from proportionality indicates measurement error
- This approach does not require a reference material and is measurement platform independent
Using experimental design and statistical analysis, quality indicators that describe precision (coefficient of variation (%CV)) and deviation from proportionality (proportionality index (PI)) can be evaluated to assess aspects of the quality of cell counting method. Importantly, the quality indicators are specific for the cell preparation investigated and evaluate the overall quality of the cell counting method (or measurement process) including sample preparation, sample handling, data acquisition, and data processing.
The Dilution Series Experimental Design
The following schematic illustrates an example of a typical dilution series experimental design.
The design incorporates elements such as random sampling, replication and independent dilution - to create a statistically robust data set for evaluating a method’s proportionality and precision. An important aspect of the dilution series experimental design is the incorporation of dilution integrity. Here we recommend that samples are generated by independent dilution from the stock solution, and further recommend verification of the dilution step by obtaining measured dilution fractions (i.e. using a calibrated scale to measure the volumes of solution pipetted when creating each independent dilution). By maintaining a high degree of dilution integrity, proportionality to dilution can be used as an internal control for measurement quality. Any deviation from proportionality would more likely be due to systematic error in the measurement instead of random or systematic error introduced by pipetting.
Statistical Analysis to Evaluate Counting Method Performance
Using the dilution fraction experimental design, quality indicators (%CV, R2, PI) can be evaluated for a wide range of cell counting methods and can be used to determine the level of proportionality and precision achieved. These properties can also be used as a basis for comparison between counting methods. These performance metrics characterize the entire cell count measurement process, including the measurement platform, method specific factors such as dilution steps and sampling, and the specific cell preparation measured.
Related Publications
Practical application of cell counting method performance evaluation and comparison derived from the ISO Cell Counting Standards Part 1 and 2. Y Huang, J Bell, D Kuksin, S Sarkar, LT Pierce, D Newton, J Qiu & LLY Chan. Cell & Gene Therapy Insights 2021; 7(9), 937–960 (DOI: 10.18609/cgti.2021.126)
Standards Landscape in Cell Counting: Implications for Cell & Gene Therapy. Sarkar, Pierce, Lin-Gibson, Lund. (Cell & Gene Therapy Insights, 2019).
Evaluating the quality of a cell counting measurement process via a dilution series experimental design. Sarkar, Lund, Vyzasatya, Vanguri, Elliott, Plant, Lin-Gibson.(Cytotherapy, 2017)
Understanding and managing sources of variability in cell measurements. Lin-Gibson, Sarkar, Elliott, Plant. (Cell and Gene Therapy Insights, 2016)
Related Webinars
Nexcelom Webinar on Demand: Applying ISO Cell Counting Standards for Cell and Gene Therapy Products