Activity: Consider Generalizability

Consider This

Humans naturally generalize from experience, which can be useful in daily life but can also lead to overgeneralization. In science, generalization is not something to avoid entirely. Often times it is the goal. However, scientific claims should only extend as far as the evidence allows. A study may provide strong evidence within a specific sample or setting while still having limited generalizability to other populations, contexts, or conditions. Good scientific communication makes those boundaries clear.

Consider a time when you heard a broad statement made that didn't reflect the nuance or complexity of reality. Then consider the statements below. Are they too broad/too general?

"Cats are unfamiliar and do not like to cuddle."
"Scientists love nerdy stuff like Dungeons and Dragons."
"Bioinformaticians are really strong on the computer science discilpine but tend to be weaker in molecular biology."

These statements are not necessarily correct or incorrect outright - there are definitely scientists out there who like Dungeons and Dragons! But there are also definitely scientists out there who do not. These are called faulty generalizations.

If you want to learn more about common fallacies in logic and reasoning, you should! It is an important component of Philosophy, which is incredibly important for science. Consider this list from the Purdue Online Writing Lab as a starting point.

We want to ensure that the generalizations we make in science meet the evidence that is available. There are certain broad generalizations we can make in science that are supported by significant amounts of evidence, research performed by many multiples of independent research groups and labs, and consensus from the scientific community in these areas of expertise:

"Smoking cigarettes is detrimental to human health."
"Exposure to radiation increases risk for certain cancers."

However, we can generally only make limited generalizations based on one particular study or set of experiments (unless it is massive in scale). Consider a study performed by your lab where you compared gene expression with RNA-sequencing for 30 mice treated with a eczema drug compared to 30 controls. You find some evidence that the drug performed slightly better than controls. Which statement below best reflects the evidence?

The results show that the eczema drug SkinGreat is an effective treatment for eczema.
The results show that the eczema drug SkinGreat performs slightly better than controls.
The results show that the eczema drug SkinGreat performs slightly better than controls in a study comparing treated mice (n=30) against controls (n=30).
The results provide evidence that the eczema drug SkinGreat performs slightly better than controls in a study comparing treated mice (n=30) against controls (n=30).

What are the differences between these statements? How do small changes in wording change the expressed impact?

Relationship to Generalizability

Generalizability is defined as the ability to apply the results of a study to different settings, groups, and situations. Ultimately we would like to generate findings that remain consistent across various contexts (types of inquiry, measurement, and reproduction) [ScienceDirect].

Generalizability can be achieved by careful and thoughtful communication of results in the context of prior and concurrent work from other researchers.

A study is not “bad” if its findings are narrow. Problems arise when researchers claim broader applicability than their design supports.