Introduction
Background/rationale: Peer review is a key part of the journal publishing process. In other words, it is difficult for a journal to be recognized as legitimate unless it publishes peer reviewed research, because peer review is a process wherein manuscripts or scientific findings are evaluated by peer experts and their quality is formally confirmed. Reviewers help authors to improve the quality of the manuscript and editors to make reasonable decisions by providing opinions on the scholarly validity and novelty of manuscripts [1,2].
Peer review began in Europe in the 17th century, but the current form of peer review only began in the 19th century, and the full-scale introduction began in the mid-20th century, after World War II [1]. Initially, single-blind peer review was introduced. In the single-blind system, the identity of the reviewer is not disclosed, so there are few restrictions on critical reviews, but there are concerns about biased reviews. Since then, double-blind peer review has become more common; the double-blind system has few restrictions on critical reviews, as the reviewer’s identity is not disclosed, and concerns about biased reviews are reduced because the author’s identity is not disclosed to the reviewer; however, it is difficult to use the authors’ research information available from previously published articles by authors. Tripleblind review has also been introduced, but the fundamental problems of peer review have not been solved [3].
In recent years, some studies have been conducted with the goal of solving the problems of traditional peer review. Tennant et al. [4] described how blockchain technology can be used to inspire and motivate reviewers by preventing forgery and alterations and by providing rewards. The development of a peer review system that rewards reviewers with tokens through blockchain has been explored in several studies [5-8]. Furthermore, proposals have been made for a governance framework [9] and a peer review ecosystem to decentralize academic publications and to put in place a token economy using blockchain technology [8].
Assigning reviewers to review submitted manuscripts is a very difficult task for editors. Methods for solving the reviewer assignment problem include topic-based methods, bidding-based methods, and knowledge-based methods [10]. Kou et al. [11] developed a reviewer assignment system that applied a reviewer profile automatically extracted from published papers using the reviewer’s research topic. Goldsmith and Sloan [12] considered bidding for the selection of reviewers based on the editor’s preference. Sun et al. [13] applied knowledge rules and mathematical decision models. In addition, Karimzadehgan and Zhai [14] used an integer linear programming method that reflected multiple subtopics.
Furthermore, Schmidt et al. [15] argued that open peer review (OPR) can improve review quality, enhance transparency and accountability, and highlight the role of reviewers. In fact, the number of journals in the Directory of Open Access Journals that have adopted OPR has increased from 20 in 2016 to 147 in 2020. Some journal publishers are also attempting various forms of OPR [3].
Objectives: The aim of this study was to systematically analyze and categorize the problems of peer review, and then to present the design and development of an OPR system using a blockchain and reviewer recommendation technologies.
Problem-solving Model for Peer Review
Ross-Hellauer [2] summarized the problems of traditional peer review into six types: 1) unreliability and inconsistency, 2) delay and expense, 3) unaccountability and risks of subversion, 4) social and publication biases, 5) lack of incentives, and 6) wastefulness. We analyzed these six types of problems and developed a model to solve them.
Fig. 1 is a schematic diagram of the model that explains the causes and paths towards solving the problems of the traditional peer review system. The upper left part of the diagram shows the submitter’s point of view. It is necessary to prevent unfair/biased reviews and fraudulent activities by enhancing transparency in the peer review process. The lower left part of the diagram shows the editor’s point of view. It is necessary to ensure the selection of optimal reviewers and to prevent delayed/poor reviews to enhance the reliability of peer review.
More specifically, it can be said that delayed/poor reviews, which are related to problems 1) and 2) listed above, are due to a lack of reviewers and low motivation of reviewers, while unfair/biased reviews, which are related to problems 3) and 4), are due to the confidentiality of the review process and interests of the reviewers. Although 5) and 6) have not been addressed by traditional peer review, they seem to be related to the above problems as well. Specifically, 5) is related to problems 1) and 2), while 6) is related to problems 3) and 4).
The right part of Fig. 1 shows measures and solutions, we suggest, to solve the problems. Measures to resolve the problem of unfair/biased reviews include a) permanent recording of the review history, b) disclosure of the review process, and c) consideration of interests in the selection of reviewers. As measures to resolve the problem of delayed/poor reviews, d) contract-based review and rewards, e) open recruitment of reviewers, and f) reflection of expertise information in reviewer selection will be helpful.
First of all, blockchain technology can be used to implement some of those measures. In particular, blockchain can be used to solve problems a) and d). Blockchain can permanently record the review history, and it enables smart contracts and token rewards.
Next, problems b) and e) can be resolved by introducing OPR. Problems b) and e) are deeply related to open identity, open participation, open interaction, open pre-review manuscript, and open final-version commenting among the seven areas of OPR presented in Ross-Hellauer’s article [2].
Finally, problems c) and f) can be addressed by introducing knowledge-based reviewer recommendations. By managing expertise information in the candidate reviewer pool and comparing it with the subjects of submitted manuscripts, the best reviewers can be recommended. Information on interests such as co-author and colleague relationships can also be gathered and reflected in the selection of reviewers.
In summary, an OPR system using blockchain and reviewer recommendation technologies would help to solve the problems of traditional peer review.
Development of an OPR System
Conceptual design
In this study, an OPR system was developed that can help solve the traditional peer review problems presented above. The system is divided into on-chain and off-chain parts, as shown in Fig. 2. The on-chain part developed using blockchain includes a smart contract-based review management module (hereafter, “review management module”), a review disclosure management module, and a token bank module. The off-chain part consists of a submission management module and a reviewer recommendation module. A service broker is placed between the on-chain part and the off-chain part for mutual data interchange.
The functions of each module in the on-chain part are as follows. First, the review disclosure management module determines whether and when to disclose each of the six disclosure items. According to Table 1, the editor-in-chief of the journal can decide whether and when each item will be disclosed. The review management system performs rounds of peer review under the conditions chosen in the review disclosure management module. Each review round includes functions such as review contract, writing and revising review reports, and reviewer assessment. The review management module manages the execution status of the review contracts, records the scores of the reviewer’s contributions, and transfers them to the token bank. The token bank takes over the reviewer’s contribution scores and rewards the reviewer with tokens.
In the off-chain part, the submission module accepts a manuscript from the submitter and transfers it to the review management module through the service broker. The submission module also transfers the selected reviewers, who are recommended by the reviewer recommendation module, to the review management system.
Process design
The review procedure for the OPR system developed in this study is designed as shown in Fig. 3. When a submitter submits a manuscript to the submission management module in the off-chain part, the editor-in-chief assigns an editor. When the editor is determined, the submitted manuscript is automatically transferred to the review management system in the on-chain part. In the review management system, when the editor sets the review period for the manuscript to be reviewed and starts recruiting reviewers, candidate reviewers who are interested apply for review. When the editor finishes recruiting reviewers, the list of candidate reviewers is transferred to the submission management module. When the editor selects a candidate reviewer from the reviewer pool and requests a reviewer recommendation, the reviewer recommendation module displays a list of reviewers recommended. In the list of reviewers, not only is the reviewer’s expertise sorted in the order most similar to that of the submitted manuscript, but it also displays when there is a co-author relationship or colleague relationship with the authors. The editor selects the most suitable candidate reviewers from the list of candidate reviewers and sends a request for a review contract to the review management module. If all candidate reviewers accept the contract, the review begins. When the review is over, the reviewers write and submit the review report. When the review is over, the editor clicks the “finish review” button, reviews the review report, corrects any revisions, and submits. The submitter submits the revised manuscript reflecting the review report and assesses the reviewers. Editors can adjust the reviewer’s assessment scores. The scores are transferred to the token bank and saved as tokens.
Development result
Fig. 4 presents the configuration of the developed system. The blockchain used in the development is a private blockchain called Hyperledger Fabric, which utilizes open-source code developed in a project led by the Linux Foundation. The on-chain part was developed in the Go language on Ubuntu 16.04, which is a Linux operating system, and the off-chain part uses Java language on CentOS 7.7. The service broker is implemented based on Apache Kafka 2. The off-chain part was de-veloped by embedding it in ACOMS, the submission management system of Journal of Information Science Theory and Practice, an international journal published by KISTI (Korea Institute of Science and Technology Information), available at https://www.jistap.org/journal/intro.do?journalSeq=J000043. Fig. 5 shows the overall workflow of the full peer review process of the Journal of Information Science Theory and Practice. Fig. 6 shows an example of the user interface of the on-chain modules and the off-chain modules on the journal homepage.
Discussion
Private blockchain versus public blockchain
Blockchains are divided into public and private. In this study, the review management system was developed using Hyperledge Fabric, a private blockchain. Several review management systems have been developed based on public blockchains such as Bitcoin [16] and Ethereum [8,9,17], but systems using private blockchains have not yet been reported. Unlike a public blockchain, a private blockchain requires approval from a central organization to participate, so processing speed is fast, security is excellent, and it can be optimized for specific communities. It is believed that in an environment in which academic societies publish journals, such as in Korea, private blockchains are more suitable than public ones.
Gradual OPR acceptance model
The system developed in this study allows publishers to select among the six OPR items (Table 1) suggested by Ross-Hellauer [2] according to the policies of the publisher, so that OPR can be adopted step by step. Namely, when publishers begin to adopt OPR, at first, they can adopt a low level of OPR and gradually develop to a higher level .
Knowledge-based reviewer recommendation
In this study, a module for recommending reviewers was developed that can help editors assign reviewers. In this module, reviewers can be recommended after examining their expertise and completing the process of excluding interests. The procedure for selecting a reviewer is as follows. First, candidate reviewers are arranged in order of expertise similarity, after calculating the degree of expertise similarity between the attribute set extracted from the papers of the candidate reviewers using TextRank and the attribute set of the submitted manuscript. Second, conflict of interest checks are performed by matrix multiplication by expressing co-author and colleague relationships as a 2-mode network matrix. Third, the editor selects the best reviewer by referring to the list of final candidate reviewers. This module is expected to help editors solve the problems of traditional peer review, both by shortening the time to find reviewers and by helping them select the best reviewers.
Service broker
In the system described in this study, manuscript submission and reviewer selection are performed in the off-chain part, while smart contracts, peer review, assessment, and rewards are in the on-chain part. Between the two parts, a service broker intervenes. The presence of a service broker makes the process slightly more complicated, but provides an opportunity to link the on-chain part to various services such as conferences and research proposal review, as shown in Fig. 7.
Conclusion
In this study, an OPR system using blockchain and knowledge-based reviewer recommendation technologies was developed. Blockchain technology will be able to increase the transparency and reliability of peer review by permanently recording the review history, enabling the selection of OPR items, performing smart contracts, and rewarding tokens. The knowledge-based reviewer recommendation technology will enable the selection of reviewers according to expertise while avoiding conflicts of interest, thereby lightening the editor’s workload. The OPR system developed in this study is equipped with devices to resolve the problems of traditional peer review. If this system is applied to actual journals in the future, it will be a model of the peer review ecosystem in the future open science era.
In addition, establishing an assessment model for reviewers and using it widely to assess researchers will contribute to converting the current publisher-led scholarly publishing paradigm to a researcher-centered autonomous scholarly publishing paradigm.
PDF Links
PubReader
ePub Link
Full text via DOI
Download Citation 
CrossRef TDM
E-Mail 
Print 
