Examples of Discussing Reproducibility | How to Summarize Values from Multiple Measurements

What Is Reproducibility?

Reproducibility is a property that indicates how closely the results agree when measurements or experiments are repeated under the same conditions.
If nearly the same measured value is obtained each time, reproducibility is considered high, whereas if the measured values vary greatly, reproducibility is considered low.
In chemistry experiments, reproducibility is an important indicator for evaluating the stability of experimental procedures and measurement methods.

For example, if the titration volumes are nearly the same when a titration is performed three times, the endpoint judgment and burette operation can be considered stable.
On the other hand, if the absorbance value changes greatly with each measurement, insufficient sample mixing, contamination of the cell, instrument noise, or other factors may have affected the results.
In a discussion of reproducibility, the variation in measured values is explained in relation to the operating conditions.

Main Items to Include in the Results

When summarizing values obtained from multiple measurements, it is important to clearly show the individual measured values, mean, standard deviation, and number of measurements.
In addition, organizing the relative standard deviation, maximum and minimum values, presence or absence of outliers, and difference from the theoretical value makes it possible to discuss reproducibility more specifically.

Main Items to Include in the Results

• Each measured value
• Number of measurements
• Mean
• Standard deviation
• Relative standard deviation
• Maximum and minimum values
• Range of measured values
• Presence or absence of outliers
• Whether outliers were excluded
• Theoretical or literature value
• Difference between the mean and theoretical value
• Measurement conditions
• Causes of variation
• Evaluation of reproducibility
• Points for improvement

Meaning of Taking Multiple Measurements

The purpose of taking multiple measurements is to identify variation and random errors that cannot be evaluated from a single measurement.
No matter how carefully an operation is performed, measured values contain reading errors, instrument noise, temperature changes, sample inhomogeneity, and other sources of variation.
By taking multiple measurements, it is possible to determine whether the measured values are stable.

Even if a single measured value is close to the theoretical value, it may have occurred by chance.
Conversely, even if one value deviates greatly, if the other measured values agree well, it is possible that a particular error occurred during that measurement.
Repeated measurements are fundamental for evaluating the reliability of results.

Meaning of the Mean

The mean is a representative value of multiple measurements.
It is calculated by adding all measured values and dividing by the number of measurements.
When repeated measurements contain random errors, using the mean can reduce the effect of variation in individual measured values to some extent.

However, reproducibility cannot be evaluated from the mean alone.
Even when two sets of measurements have the same mean, one set may consist of closely grouped values while the other may contain widely scattered values.
Therefore, the mean must be presented together with the standard deviation or relative standard deviation.

Evaluating Reproducibility Using Standard Deviation

Standard deviation indicates how widely the measured values are distributed around the mean.
When the standard deviation is small, the measured values are close to the mean, and reproducibility is considered high.
When the standard deviation is large, the measured values are widely scattered from the mean, suggesting that reproducibility may be low.

Standard deviation is useful for expressing reproducibility numerically.
However, the magnitude of the standard deviation also depends on the unit and magnitude of the measured values.
When comparing data with different magnitudes, relative standard deviation makes the comparison easier.

Comparing Variation Using Relative Standard Deviation

Relative standard deviation is obtained by dividing the standard deviation by the mean and expresses the variation as a proportion of the mean.
It is usually expressed as a percentage and is also called RSD.
It allows relative variation to be compared even when the magnitudes of the measured values differ.

For example, even if the standard deviation is the same value of 0.1, the meaning of the variation differs greatly when the mean is 1.0 compared with when the mean is 100.0.
Relative standard deviation makes it possible to evaluate what percentage of the mean the variation represents.
It is useful when comparing reproducibility.

Causes of Variation in Measured Values

Causes of variation in measured values include variation in sample preparation, differences in pipetting operations, reading of titration endpoints, instrument noise, temperature changes, insufficient sample mixing, contaminated equipment, and differences in judgment among operators.
Which factors have the greatest effect varies depending on the experiment.

In a discussion of reproducibility, rather than simply writing that “the values varied,” specifically describe which operation may have contributed to the variation.
Consider causes according to the measurement method, such as endpoint judgment for titration, cells and bubbles for absorbance measurements, and drying conditions or moisture absorption for mass measurements.

Discussion When Reproducibility Is High

When reproducibility is high, the values obtained from multiple measurements agree well, suggesting that the measurement procedure and instrument conditions were stable.
Small standard deviations and relative standard deviations can be used as evidence of good reproducibility.
However, whether the results are close to the theoretical value must be evaluated separately.

Results with high reproducibility indicate high precision in the experimental procedure.
However, if the concentration of the standard solution is incorrect or the instrument calibration is shifted, the mean may deviate from the theoretical value even when reproducibility is high.
Reproducibility and accuracy should be considered separately.

Discussion When Reproducibility Is Low

When reproducibility is low, the values obtained from multiple measurements vary greatly.
Possible causes include inconsistent operating conditions, an inhomogeneous sample, an unstable measuring instrument, or differences in reading judgment.
Large standard deviations or relative standard deviations provide evidence of low reproducibility.

When reproducibility is low, the reliability of the mean also decreases.
If the measured values vary greatly, even if the mean happens to be close to the theoretical value, the experimental conditions cannot be considered stable.
Remeasurement or a review of the operating conditions is necessary.

Difference Between Reproducibility and Accuracy

Reproducibility indicates how close the measured values are to one another.
Accuracy, on the other hand, indicates how close the measured values are to the theoretical or true value.
High reproducibility and high accuracy are not the same thing.

For example, even if nearly the same value is obtained every time, if that value deviates greatly from the theoretical value, the result has high reproducibility but low accuracy.
Conversely, even if the mean is close to the theoretical value, if the measured values vary greatly, the result may appear accurate but has low reproducibility.
In a report, it is important to distinguish between reproducibility and accuracy.

Random Error and Reproducibility

Random error is an error that occurs irregularly from one measurement to another.
Slight differences in pipetting operations, reading errors, instrument noise, and small temperature changes are examples of random errors.
The larger the random error, the more likely the measured values are to vary, resulting in lower reproducibility.

The effect of random error can be reduced to some extent by performing multiple measurements and calculating the mean.
However, if the variation between measurements is very large, considerable uncertainty remains even in the mean.
To improve reproducibility, it is important to standardize operations and stabilize the measurement environment.

Systematic Error and Reproducibility

Systematic error is an error that causes measured values to consistently deviate in the same direction.
Errors in the concentration of a standard solution, calibration errors in balances or pipettes, zero-point shifts in instruments, and insufficient blank correction can be causes.
Because systematic errors shift measured values in the same direction, they may not appear clearly in the standard deviation.

Therefore, systematic error may be present even when reproducibility is high.
If the measured values agree well every time but deviate greatly from the theoretical value, systematic error must be considered.
It is important not to judge the correctness of the results based on reproducibility alone.

How to Summarize Results When There Is an Outlier

Among values obtained from multiple measurements, there may be a measured value that deviates greatly from the others.
Such a value is called an outlier.
The presence of an outlier can greatly change the mean and standard deviation and also affect the evaluation of reproducibility.

A clear reason is required to exclude an outlier.
Examples include records showing that bubbles entered during measurement, the titration endpoint was clearly exceeded, the sample was spilled, or the concentration was prepared incorrectly.
If the cause is unknown, the value should not be readily excluded; instead, it should be discussed as part of the results including the outlier, or the measurement should be repeated.

Precautions When the Number of Measurements Is Small

When the number of measurements is small, the mean and standard deviation are strongly affected by a single measured value.
With only two or three measurements, one operational error can have a large effect on the overall results.
When the number of measurements is small, it should be considered that uncertainty remains in the evaluation of reproducibility.

Increasing the number of measurements makes it possible to evaluate the effects of random errors more reliably.
However, because there are limitations on experimental time and sample quantity, the necessary number of measurements varies depending on the experiment.
In a report, it is important to state the number of measurements and describe what can be concluded within that range.

How to Summarize Measured Values

Values obtained from multiple measurements are easier to understand when the individual measured values are organized in a table and the mean and standard deviation are shown.
If necessary, the relative standard deviation, relative error, and difference from the theoretical value should also be shown.
When summarizing results, use consistent units and pay attention to significant figures.

In a report, if only the mean is emphasized and the individual measured values are omitted, it becomes difficult to understand the degree of variation.
By showing the individual values together with the mean and standard deviation, the reader can more easily evaluate the reproducibility of the measurements.

Significant Figures and Reproducibility

When summarizing values obtained from multiple measurements, attention must be paid to significant figures.
If the mean is reported to more decimal places than justified by the variation indicated by the standard deviation, it may appear that the measurement was performed with greater precision than was actually achieved.
It is common to adjust the decimal place of the mean to match that of the standard deviation.

For example, if the standard deviation is approximately 0.1, writing the mean as 10.12345 provides no experimental meaning for the lower digits.
Measurement results should be reported using a number of digits appropriate to their reproducibility.
Proper handling of significant figures is important for accurately communicating the reliability of the results.

Discussion of Reproducibility in Titration Experiments

In titration experiments, agreement among multiple titration volumes can be used to evaluate reproducibility.
If the titration volumes agree well, the endpoint judgment, burette reading, dropping rate, and amount of sample collected can be considered stable.
On the other hand, if the titration volumes vary greatly, the operation near the endpoint or differences in judging the color change may be responsible.

In titration, a difference of even one drop near the endpoint may affect the result.
Particularly when the amount of sample is small, a difference of one drop represents a relatively large error.
To improve reproducibility, it is important to add titrant slowly near the endpoint and judge the color change using the same criterion each time.

Discussion of Reproducibility in Absorbance Measurements

In absorbance measurements, reproducibility can be evaluated from the degree of agreement among absorbance values obtained by measuring the same sample multiple times.
If the values agree well, the condition of the cell, blank correction, instrument stability, and sample homogeneity can be considered good.
If the values vary, contamination of the cell, bubbles, insufficient sample mixing, instrument noise, and other factors may be responsible.

In absorbance measurements, the orientation of the cell, water droplets or fingerprints on its surface, and small bubbles can also affect the measured value.
To repeat measurements under the same conditions, it is necessary to keep the cell clean, remove bubbles, and mix the sample thoroughly.

Discussion of Reproducibility in Mass Measurements

In mass measurements, the stability of the mass is checked when the same sample is measured repeatedly.
If the variation is small, the condition of the balance, drying state of the sample, and weighing procedure can be considered stable.
If the variation is large, moisture absorption by the sample, insufficient drying, static electricity, air currents, or contamination of the container may be responsible.

When measuring the mass of a product, the measured value may change due to insufficient drying or moisture absorption.
Whether the sample was dried to constant mass, cooled in a desiccator, and weighed with the balance draft shield closed also affects reproducibility.

Discussion of Reproducibility in pH Measurements

In pH measurements, the condition of the electrode, calibration, temperature, stirring condition of the sample, and cleaning of the electrode affect reproducibility.
Even when measuring the same sample, the values may vary if they are read before the electrode has stabilized.
It is important to calibrate with standard solutions and thoroughly clean the electrode before measurement.

pH is also affected by temperature and ionic strength.
If the measured values vary, the sample temperature may not have been constant, the electrode may have been contaminated, or calibration with standard solutions may have been insufficient.
In pH measurements, record the value after it has stabilized.

How to Show Reproducibility in a Graph

When values obtained from multiple measurements are shown in a graph, the standard deviation may be displayed as error bars in addition to the mean.
Error bars make it possible to visually understand the variation at each measurement point.
Smaller error bars indicate higher reproducibility, while larger error bars indicate greater variation in the measured values.

When comparing a control group and an experimental group, check not only the difference between the means but also the size of the error bars.
Even if the means differ, if the error bars are large and overlap considerably, it may be difficult to conclude that there is a clear difference.
In graphs, it is important to show both the mean and the variation.

Sources of Error Affecting Reproducibility

Major sources of error that reduce reproducibility include variation in sample preparation, differences in operator technique, errors in reading instruments, changes in temperature and humidity, insufficient instrument stability, sample inhomogeneity, differences in reaction time, insufficient stirring, and contaminated equipment.
These factors change the measured value from one measurement to another and increase the standard deviation.

In a discussion of reproducibility, divide the entire experiment into stages and consider at which stage variation was likely to occur.
Dividing the experiment into the sample preparation stage, reaction stage, post-treatment stage, measurement stage, and data processing stage makes it easier to identify specific causes.

When the Results Can Be Considered Good

From the perspective of reproducibility, results can be considered good when values obtained from multiple measurements agree well and the standard deviation and relative standard deviation are small.
In this case, the measurement procedure can be considered stable and the effect of random errors small.
Furthermore, if the mean is close to the theoretical or literature value, the results can also be more readily considered good in terms of accuracy.

However, if the mean deviates greatly from the theoretical value, systematic error may be present even when reproducibility is high.
Good results are not simply values that agree with one another, but values that fall within a reasonable range and can also be explained based on the operating conditions.

Example Discussions When the Experiment Did Not Go Well

When reproducibility is low, consider the cause based on the variation in measured values, outliers, the magnitude of the standard deviation, and the difference between the mean and theoretical value.
Organizing possible causes into operational errors, sample inhomogeneity, instability of the measuring instrument, and changes in environmental conditions leads to a more specific discussion.

How to Write Points for Improvement

In a discussion of reproducibility, it is important to describe not only the causes of variation but also what can be done to make the measured values more consistent in the future.
Points for improvement are easier to write when organized into sample preparation, equipment operation, measurement conditions, number of measurements, and data processing.

Improvements to Experimental Procedures

• Thoroughly mix the sample before collecting it
• Use pipettes and volumetric flasks correctly
• Use the same amount of sample each time
• Keep the reaction time consistent
• Keep the reaction temperature constant
• Keep stirring conditions consistent
• Allow the instrument to stabilize before measurement
• Thoroughly clean and dry the equipment

Improvements to Data Organization

• Increase the number of measurements
• Show individual measured values in a table
• Present the mean and standard deviation together
• Calculate the relative standard deviation
• Check the cause of outliers
• If an outlier is excluded, state the reason
• Adjust significant figures according to the standard deviation
• Also evaluate the difference from the theoretical value

Difference Between a Superficial Discussion and a Good Discussion

In a discussion of reproducibility, simply writing “measured three times,” “calculated the mean,” or “the values were close” results in a superficial discussion.
A good discussion explains the variation in measured values in relation to the standard deviation, operating conditions, outliers, and the difference from accuracy.

Examples of Expressions That Can Be Used in Reports

The following expressions can be used when writing the results and discussion of reproducibility and repeated measurements.
Adjust the necessary parts according to your own experimental results.

• By performing multiple measurements, the variation and reproducibility of the measured values were evaluated.
• Because the measured values were clustered around the mean, reproducibility was considered relatively high.
• Because the standard deviation was small, the effect of random errors was considered small.
• Because the relative standard deviation was small, the variation relative to the mean was considered small.
• The measured values showed large variation, and reproducibility was insufficient.
• Possible causes of the variation include sample preparation, reading of the measuring instrument, and inconsistencies in operating conditions.
• The presence of an outlier has a large effect on the mean and standard deviation.
• A clear basis, such as an operational error, is required to exclude an outlier.
• High reproducibility indicates precision but does not indicate accuracy.
• When summarizing results, the standard deviation and relative standard deviation should be presented in addition to the mean.

Points to Check When Discussing Reproducibility

Checking the following points before writing the report makes it easier to write the discussion.

• Is the number of measurements stated?
• Are individual measured values shown?
• Has the mean been calculated?
• Is the standard deviation shown?
• Is the relative standard deviation used when necessary?
• Have the causes of variation in the measured values been considered?
• Has the presence or absence of outliers been checked?
• Is there a basis for excluding outliers?
• Has the adequacy of the number of measurements been considered?
• Are reproducibility and accuracy distinguished?
• Are the significant figures appropriate for the standard deviation?
• Do the points for improvement correspond to the causes of variation?

Summary

Reproducibility is an important concept that indicates how closely results agree when experiments or measurements are repeated under the same conditions.
By performing multiple measurements, variation and random errors that cannot be identified from a single measurement can be evaluated.
When summarizing measured values, presenting the individual measured values, mean, standard deviation, and relative standard deviation makes it easier to evaluate reproducibility.

When reproducibility is high, the measured values are clustered around the mean and the standard deviation is small.
On the other hand, when reproducibility is low, the measured values vary greatly and the reliability of the mean also decreases.
Causes of variation include sample preparation, pipetting operations, endpoint judgment, instrument noise, temperature changes, and sample inhomogeneity.
If an outlier is present, it is important not to exclude it readily but to investigate its cause.

In a report, rather than simply writing that “multiple measurements were performed” or “the mean was calculated,” organize and discuss the variation in measured values, standard deviation, relative standard deviation, outliers, random error, systematic error, the difference between reproducibility and accuracy, and points for improvement.
Discussing reproducibility is important for evaluating the reliability of experimental results and making measurement procedures more stable.