Which of the following is a way to measure the quality of a research publication

An artist's interpretation of a paper published in Science that compared human and machine learning. Credit: Danqing Wang

What's the best measure of research output?

21 March 2016

Nature Index

Danqing Wang

An artist's interpretation of a paper published in Science that compared human and machine learning.

EXPLAINER

When ranking countries based on their output of high-quality research, weighted fractional count (WFC) is often used as the Nature Index's primary metric. And for good reason. WFC reflects the size of the contribution a country's researchers have made to every study published in the 68 top-tier journals included in the index. This measure also takes into account the higher proportion of astronomy papers in the index. It seems astronomers love to write papers. The index's unweighted measure of contribution is fractional count (FC).

For every paper included in the index, the FC and WFC are split among authors based on their affiliation. Take a recent paper published in Science, which created a computer model that captures humans' unique ability to learn, and had three authors from three different universities, two in the USA and one in Canada. For this paper, each affiliation received an FC of 0.33, and because it wasn't published in an astronomy journal, they received the same WFC. Adding the WFC of a country's institutions presents a picture of that nation's performance over a designated period of time.

In a recent post, we published a graph of the top and bottom 10 countries ranked by change in their WFC. It revealed a compelling narrative. Since 2012 China's contribution to high-quality research has soared, while traditional stronghold, the United States, appears to have lost its mojo. (Although it's worth noting that the USA's total WFC is miles ahead of anyone else, China included.)

But what does it mean when a country's WFC drops? Does that suggest its research performance is slipping?

Not necessarily. When assessing the output of a country's top-quality research, it is prudent to also consider article count - the total number of studies that a country's researchers have contributed too, regardless of the size of that contribution.

Consider this next graph. It shows the change in article count for the countries in the graph above. While the United States and Japan's article count followed a similar downward trajectory to their WFC, the article count of all the other countries grew between 2012 and 2015 - including the eight countries that experienced a drop in their WFC.

As article count isn't a weighted metric, it shouldn't be directly compared to WFC. When considering the change in a country's total number of papers versus the contribution it made to those papers, it best to compare AC with FC.

An interesting trend emerges when a country's article count goes up, but its fractional count dwindles. It suggests that while the country's researchers have contributed to a larger total number of studies, the proportion of their contribution has become smaller.

The reverse can be observed in countries with an increase in their FC but a fall in their AC. In those countries, researchers contributed to fewer studies but received more of the credit for the ones that were published.

These two scenarios at least partly reflect patterns of collaboration, and their significance will depend on a broader research context. What is certain, however, is that neither WFC, FC or AC alone can reveal the state of a country's high-quality natural science output. The three metrics should be considered together.

Module 4: Study Design Measures

How To Create a Research Methodology

When formulating methodology, it is critical to consider the types of methods that will most accurately and efficiently answer the research questions. For example, if one wishes to assess the efficacy of an educational intervention, the “change in knowledge” must be measured. This lends itself to a pre-test/post-test methodology in which the researcher will determine the knowledge of study participants on a specific topic prior to the intervention and then again after the educational intervention has been implemented. However, in order to determine the actual impact of an intervention, a pre-test/post-test methodology must always be compared with a control group. For more information about this, please see the section on Assessing Behavioral Changes: The Importance of Having a Baseline For Comparison.

Measures are the items in a research study to which the participant responds. Research measures include survey questions, interview questions, or constructed situations. When constructing interviews and surveys, it is important that the questions directly relate to the research questions. Furthermore, it is important that the surveys and interviews are not extremely time-consuming (ideally within a 20-30 minute limit). Simple questions that avoid ambiguity will lead to more accurate responses, particularly if a translator is being used. Lastly, before creating a survey, a thorough literature review should be conducted to determine if a similar study has already been conducted. If so, previous surveys should be used to yield standardized measures for comparison. Irrespective of the form that these measures take, there are several important design elements that are required to make the study effective.

Study measures should:

• Take into account the characteristics of the participant.
• Use informal language that a layperson will understand.
• Be respectful of the cultural context in which the participant has shaped his or her worldview.
• Provide neither too much nor too little information. Too much information can be an unnecessary distraction, while too little information leads to ambiguity and potential misinterpretation of the study measure.
• Be brief and specific.
• Avoid negations, as they can lead to mistakes and can be difficult to understand.
• Avoid double-barreled questions (questions that ask two questions in one, such as “Do you support the government’s decision to cut spending to police training and after school programs?”). If a participant would answer “yes” to one part but “no” to another, requesting a single answer of “yes or no” to two questions invalidates the measure.
• Use multiple questions to assess the same construct. For example, simply asking a person if they feel “good” about themselves as a measure of self-esteem is not specific and leaves room for individual interpretation. Instead, asking that person several questions about body image, self-worth, and self-evaluation can help paint a better picture of how they are really feeling.(1),(2),(3)

Self-Report

Self-report measures are essential in situations where the researcher is asking about a participant’s self-concept or seeking to study the specificities of a participant’s experience. Self-reporting may also be useful for logistical reasons, as it is often the simplest research method to implement and requires the fewest resources. Self-report measures, whether in survey form or in interview form, are susceptible to bias and therefore must be designed carefully by the researcher. The researcher also must be cautious about what kinds of conclusions are drawn from these measures. Some potential issues with self-report are:

Social desirability bias: Participants are usually uncomfortable or unwilling to share information that does not reflect well on them in their social environment, even if they know their responses are entirely anonymous. For example, participants may understate or overstate the extent to which they experience a certain feeling, depending on how socially appropriate or desirable they believe that feeling to be. Researchers must do their best to make it abundantly clear that anonymity will be preserved for the participant, and honesty must be encouraged. Researchers should also lead with less intimidating questions to make the participant feel more comfortable before asking anything that might be more difficult to answer honestly. Another option is to structure the question in such a way as to normalize the behavior: “As you know, many people do X… To what extent do you do X?”

Self-evaluation biases: Participants will sometimes bend their answers on self-report measures to better reflect how they “think they should be” rather than how they actually are. This is similar to the social desirability bias, but is more difficult to overcome because anonymity is not the issue. Instead, bias results from the participant’s evaluation of him or herself. The researcher’s best course of action is to encourage honesty and normalize the behavior or feeling as reviewed above.

Forgetfulness: Sometimes researchers ask participants about their past experiences or feelings without considering the fact that human memory is very plastic. People’s recollections may be inaccurate, and it is important for a researcher to consider this when designing study measures.(4)(5)

Go To Module 5: Data Preparation and Analysis >>

Footnotes

(1) Eibach, R.  “Scale Questions: Simple Questions, Complex Answers”  Lecture at Yale University 10/2/08.

(2) Pelham, B. W.; Blanton, H. Conducting Research in Psychology: Measuring the Weight of Smoke, 3rd Edition.  Wadsworth Publishing (February 27, 2006).

(3) Trochim, W. M. K.  “Survey Research” Research Methods Knowledge Base 2nd Edition.

(4) Pelham, B. W.; Blanton, H. Conducting Research in Psychology: Measuring the Weight of Smoke, 3rd Edition.  Wadsworth Publishing (February 27, 2006).

(5) Eibach, R.  “Scale Questions: Simple Questions, Complex Answers”  Lecture at Yale University 10/2/08.

What feature of a research study's design relates to the procedures that researchers take to prevent bias and increase accuracy?

Which is the best source for answering a foreground question quizlet?

What is the term in the chapter that represents an answerable inquiry into a specific concern or issue?

What is the best description of a peer reviewed journal article quizlet?