How AI Identifies Words That Predict Higher Citation Rates of Research Articles

Artificial intelligence can now estimate which research publications will be successful and which will not. New research goes even further: it aims to show which words in an abstract contribute most to this estimate and also explains the reasons behind it. This enables authors of scientific studies and evaluators to better understand the factors that determine a text’s impact. 

Why a Simple Yes-or-No Answer Is Not Enough 

For the author of a research article, it is not enough to know only whether their article will succeed or not. They also need a clear explanation of what specifically contributed to the assessment, whether positively or negatively. Machine learning models are highly accurate, but their decision-making is often complex and opaque. Common explanation methods, such as LIME or SHAP, usually work retrospectively and create a simplified substitute for the original model. Such an interpretation can be inaccurate and misleading for the author. 

A New Approach: SMER 

A team from the Faculty of Informatics and Statistics at Prague University of Economics and Business and Lawrence Berkeley National Laboratory (Lucie Dvořáčková, Marcin P. Joachimiak, Michal Černý, Adriana Kubecová, Vilém Sklenák, Tomáš Kliegr) introduced the SMER (Self Model Rated Entities) method. This method estimates whether an article will belong among above-average cited articles and, at the same time, identifies which specific words in the abstract contribute most to this estimate. 

How does it work? 

The model works with so-called word embeddings – mathematical representations (vectors) of words that capture their semantic relationships. 

• From each abstract, it creates an average of the vectors of the words it contains, thereby obtaining a concise mathematical representation of the text that is used to train a logistic regression model. The model then learns to recognize when an article is influential, meaning cited. 

• The strength of a specific word is determined by applying logistic regression to its individual vector. The resulting score shows how strongly the given word is associated with high citation rates within this model. 

The combination of averaging and a linear model makes it possible to achieve perfect fidelity. As a result, the explanation is exactly aligned with what the model actually calculates. 

What the Experiments Showed 

The researchers tested the study on the large biomedical sample CORD-19 (an open database of scientific texts on COVID-19 and related topics). In several stages, they verified the accuracy of the prediction and the quality of the interpretation. 

• Predicting article impact: The model was trained on articles from one period and tested on later articles. In this realistic scenario, it achieved an AUC of around 0.77, meaning that it was able to distinguish between above-average and below-average cited articles significantly better than chance. (AUC is the area under the ROC curve; a value of 1.0 indicates perfect accuracy, while 0.5 corresponds to a random guess.) 

• Comparison with traditional metrics: The AI results matched the reality of the scientific world. Articles that the model identified as the most promising often appeared in more prestigious journals and were written by authors with higher H-indexes; however, this information was not part of the training and served only as a retrospective check of the results. 

• Comparison of explanation methods: In a test based on so-called AOPC (monitoring the decline in model performance when the most important terms are removed), SMER proved to be the most accurate approach to identifying important words, surpassing even established approaches such as LIME and SHAP. 

• Specific findings: Among the highest-rated terms were names of virus variants and drugs, such as “omicron,” “remdesivir,” and “molnupiravir.” 

• New words outside the training set: Thanks to word embeddings (often translated into Czech as “vektorové reprezentace slov”), the model can also evaluate newly emerging terms that were not included in the logistic regression training, provided that a vector representation is available for them. 

Why This Is Groundbreaking 

Word-level explanations help not only authors in the writing process, but also institutions and editors who want to understand how content is assessed. 

The new SMER method also has practical potential, but the study verifies it mainly on scientific abstracts from the field of COVID-19. 

• Optimizing abstracts: Authors can see which words the model associates with higher citation rates. However, this does not mean that a more “packed” abstract indicates a higher-quality study. The model assesses only what it has learned from the training data, namely the relationship between the content of the abstract and citation rates. 

• Identifying trends: The model can recognize newly emerging expressions, such as names of virus variants or drugs. This may also be useful for reviewers or editors as a quick signal of which topics are beginning to emerge in the field. 

• Transparent assessment: Because the explanation corresponds exactly to the model’s calculation, it is easier to understand its decision. However, it is important to remember that this is still a model-based estimate based on the abstract, not a definitive assessment of the article’s quality.