| Home | KCSE | Sitemap | Contact Us |  
top_img
Science Editing > Volume 5(2); 2018 > Article
Lukman, Dimyati, Rianto, Subroto, Sutikno, Hidayat, Nadhiroh, Stiawan, Haviana, Heryanto, and Yuliansyah: Proposal of the S-score for measuring the performance of researchers, institutions, and journals in Indonesia

Abstract

This study aimed to propose a tool for measuring the research performance of researchers, institutions, and journals in Indonesia based on bibliometrics. Specifically, the output of this measurement tool, referred to as the S-score, is described, as well as its implementation on the main database portal in Indonesia. The S-score was developed by a focus group discussion. The following 8 evaluation items for journal accreditation were analyzed in the development process: journal title, aims and scope; publisher; editorial and journal management; quality of articles; writing style; format of PDF and e-journal; regularity; and dissemination. The elements of the S-score are as follows: number of journal article documents in Scopus, number of non-journal-article in Scopus, number of citations in Scopus, number of citations in Google Scholar, the h-index in Scopus, and the h-index in Google Scholar. The S-score yields results ranging from S1 to S6. The above metrics were implemented on the Science and Technology Index, a database portal in Indonesia. The measurement tool developed through the focus group discussion was successfully implemented on the database portal. Its validity and reliability should be monitored consistently through regular assessments of S-scores. The S-score may be a good example of a metric for measuring the performance of researchers, institutions, and journals in countries where most journals are not indexed by Scopus.

Introduction

Universities in Indonesia are growing rapidly. As of February 6, 2017, there were 4,498 colleges with 268,322 lecturers (Table 1) [1]. The Ministry of Research, Technology and Higher Education, as well as most universities and research institutions in Indonesia, used the Scopus and Web of Science databases to evaluate the research performance of Indonesian researchers as of 2017 [2]. Indonesia has not developed its own measurement tool for research performance that can be used by policy-makers to evaluate researchers, institutions, researchers, and journals in various research fields [3]; therefore, a new measurement tool for assessing research performance in Indonesia is required. This study aims to propose a new metric, referred to as the S-score, to measure the research performance of researchers, institutions, and journals in Indonesia. This study proposes specific models, indicators, and metrics, and provides the results of the implementation of these metrics on a database portal. The results will be useful for countries where many journals are not indexed in international citation databases, such as Scopus or Web of Science.

Measurement Tools

This tool for measuring research performance was proposed through a focus group discussion in 2017. The expert group identified indicators based on the research products of researchers and their citation frequency in Scopus and Google Scholar. The indicators and evaluation items were proposed for measuring the performance of not only authors but also of institutions. Journals’ performance was measured by taking data from Indonesian journals indexed in Scopus, the Indonesia national journal accreditation system, and citation frequency in Google Scholar. Based on these evaluation items, the S-score was proposed.
Data on research products from all lecturers and researchers in Indonesia were collected and entered into the Science and Technology Index (SINTA, http://sinta2.ristekdikti.go.id/author) portal. The data collected from journals were matched through the Indonesian national accredited journal database (http://arjuna.ristekdikti.go.id/). The performance of researchers, institutions, and journals was presented in SINTA.

Results

Indicators and evaluation items

Table 2 presents the following evaluation items, contents, and scores used for journal accreditation by the Indonesian government: journal title, aims and scope; publisher; editorial and journal management; quality of articles; writing style; format of PDF and e-journal; regularity; and dissemination [4]. An explanation of constants and how they are weighted to calculate the S-score is presented in Table 3: A, number of journal article documents in Scopus; B, number of non-journal-article documents in Scopus; C, number of citations in Scopus; D, number of citations in Google Scholar; and N, current divisor. The measurement items used to evaluate the performance of researchers, institutes, and journals are presented in Fig. 1.

S-score and metrics for performance measurement

The S-score, which incorporated metrics for authors and institutions, was formulated by weighting factors, as shown in Table 3. The indicators and items for measuring research performance were formulated in two scenarios, which were expressed as formula 1 and 2. The first scenario of the S-score formula for researchers, without the h-index, was as follows:

(1)

(A×40)+(B×15)+(C×4)+(D×1)N
The second scenario of the S-score formula for researchers, with the h-index, was as follows:V

(2)

(A×40)+(B×15)+(C×1)+(D×4)+(E×16)+(F×4)N
The S-score formula for institutions was as follows:

(3)

(A×5)+(B×2)+(C×2)+(D×1)10
If we use the maximum SINTA score of a researcher on January 31, 2017 as the divisor (N), its value would be 102. The corresponding graph is shown in Fig. 2. The results of the measurements of author performance for all authors and institutions can be accessed at http://sinta2.ristekdikti.go.id. The results of the performance measurement will dynamically depend on the author registration process and data verification. The S-score for measuring the performance of journals is classified into a range of S1–S6, as measured by the values of the 8 criteria used for the accreditation of Indonesian journals, as presented in Table 2. The results of the classification by scores (S1–S6) can be seen in Table 4.

Integration of metadata into the model

The emergence of electronic journals and open access publishing has improved accessing digital data for each article, so that such data can be measured and made interoperable with various databases and indexers such as Scopus, Web of Science, Google Scholar, and others. Currently, thousands of Indonesian journals have been published in electronic form, and most journals are open access [5]. This study proposed a new model for integrating existing data in the Scopus and Google Scholar databases to measure research performance in Indonesia through the Scopus API (application programming interface) data retrieval mechanism, which can be incorporated into SINTA as shown in Fig. 3. A model of mapping and visualizing research performance based on data already entered into SINTA databases can be seen in Fig. 4. Mapping performance evaluation results in such a way can show profiles and ratings of researchers, institutions, and journals.

Implementation and data verification

The resulting indicators, formulas, and models were then tested against registered lecturers with a national lecturer number (NIDN) at universities and researchers listed as functional investigators at research institutes. Lecturers and researchers simply fill out the registration form at SINTA (http://sinta2.ristekdikti.go.id/author/) by filling in the lecturer’s identification number (NIDN/unique number of employee), name, title, Scopus identifier and Google Scholar identifier. Lecturers and researchers who sign up can correct any discrepancies in publication data and the impact after synchronization, metadata in Google Scholar and Scopus, as shown in Fig. 5. As of June 30, 2017, registered authors have verified as many as 32,218 publications based on 25,472 data points harvested from 1,424 institutions. The number of journals evaluated was 995 of the 1,807 journals registered through the Arjuna portal (http://arjuna2.ristekdikti.go.id/). Information on the implementation of journal performance metrics in SINTA is available from: http://sinta2.ristekdikti.go.id/journals. Journal profiles have been created on that site based on citation performance, with h-index values taken from Google Scholar; this information is available at http://sinta2.ristekdikti.go.id/journals/detail?id=664.

Meaning of New Metrics and Its Limitations

This study proposed a new metric, referred to as the S-score, to measure the performance of researchers, institutions, and journals based on a set of evaluation items. The S-score was successfully implemented in SINTA as a way to present performance results. Most evaluation items were designed through a focus group discussion. The opinions of an expert group are an important component of the development of new metrics or a novel evaluation system in a country because the evaluation system should be based on that country’s academic circumstances. Most scholarly journals published in Indonesia are not listed in Scopus or Web of Science. Therefore, a new local system incorporating locally appropriate metrics is required. This is a good example of such a system, and it was possible to incorporate the S-score in the Indonesian portal system (SINTA). Therefore, these new metrics could enrich the Indonesian database portal.
There were some limitations to this study. It was difficult to identify and match researchers through Google Scholar because some researchers did not have photos and the author’s name in the profile was not always the same as that in articles. Affiliations in Google Scholar and Scopus were often different from the current affiliations. Therefore findig out a precise number of publication from an institution is difficult and the data integrity should be pursued continuously. Another problem in the implementation of the S-score in SINTA is normalization; it is challenging to make comparisons among authors, disciplines, or institutions because research products vary according to the research category. Key items for evaluating performance were developed in this study and implemented in the SINTA portal based on a variety of items, including the number of citations in Google Scholar (Table 3); however, Google Scholar cannot be used as a key item because the quality of data was not consistent.

Conclusion

In the future, SINTA needs to adopt a normalized measurement method [6]. Normalization takes into account the following parameters: the average number of citations per publication (excluding self-citations); the percentage of publications without a citation, the average number of journal citations, and the performance of research units in related fields around the world. Measurements of research performance to be implemented within the SINTA must consider interdisciplinary and transdisciplinary areas, as some measures may not be adequate for such areas.
The h-index and some of its modifications are useful for quantifying the performance of researchers, similarly to the rankings of universities, and this is necessary in a global environment [7,8]. Rankings will be announced periodically (4 times a year) because real-time rankings may confuse policy-makers if rankings serve as a tool for rewards and penalties based on research performance.
The journal classification (S1–S6) should be considered as a determinant of Indonesian journals. This ranking system may be adopted by other local countries. Table 2 contains broadly applicable items for journal evaluation. The S-score of researchers and institutions is also based on the number of publications, citations, and h-index; therefore, this system will serve as a model to be referred to by other countries.
A new metric, referred to as the S-score, for measuring the performance of researchers, institutions, and journals in Indonesia was proposed and successfully integrated into the SINTA portal. Its validity and reliability should be monitored consistently through regular assessments of S-scores. This may be a good example of a metric suitable for measuring the performance of researchers, institutions, and journals in a country where most journals are not indexed in Scopus.

Conflict of Interest

No potential conflict of interest relevant to this article was reported.

Acknowledgments

This study was supported by a research grant provided by the Insentif Sistem Inovasi Nasional (Insinas) Programme of the Ministry of Research, Technology, and Higher Education of Indonesia Government (grant no. 62/P/RPL-LIPI/INSINAS-2/VI/2017).

Indicators and their evaluation items in Indonesia.
/upload/thumbnails/se-5-2-135f1.gif

Fig. 1.

Maximum Science and Technology Index score of researchers on January 31, 2017.
/upload/thumbnails/se-5-2-135f2.gif

Fig. 2.

Model of data integration in Indonesia.
/upload/thumbnails/se-5-2-135f3.gif

Fig. 3.

Model of performance mapping model in Indonesia.
/upload/thumbnails/se-5-2-135f4.gif

Fig. 4.

Mechanism of author verification and data collection in Indonesia.
/upload/thumbnails/se-5-2-135f5.gif

Fig. 5.

Table 1.

Number of universities, students, and lecturers in Indonesia in 2017
Universities
Students
Lecturer
Public Private Total Public Private Total Public Private Total
Universities 122 3,132 3,252 1,573,188 2,818,535 4,391,723 71,758 164,117 235,875
Religious universities 77 990 1,057 305,289 150,606 455,895 12,100 10,287 22,827
Service universities 177 0 182 107,028 0 107,028 9,619 0 9,619
Total 381 4,122 4,498 1,985,505 2,969,141 4,954,646 93,477 174,845 268,322

Data from Pangkalan Data Perguruan Tinggi. Kemenristekdikti [Internet]. Jakarta: Ristekdikti; 2017 [1].

Table 2.

Evaluation items for journal accreditation by the Indonesia government
Evaluation item Content Maximum score
Journal title, aims, and scope Journal title is meaningful, precise, and short so that it is easily referenced. 3
Aims and scope of journal should be lucid and unique. The research field should be indicated.
Publisher The publishing institute (professional organizations, universities, research and development institutes, and/or institutes authorized for it) have the status of a legal entity, thus able to guarantee the continuity of funds and legal protection 4
Editorial and journal management Reviewer 5
Management of quality of articles 2
Editorial board 3
Author guidelines 2
Quality of editing and formatting 2
E-journal management system (e-submission system) 3
Quality of article It fits the scope of the journal 4
Regional boundaries (international, regional, national, local) 6
Scientific originality of works 6
Contribution to the advancement of science 3
Citation 5
Primary reference source (journal, proceedings) ratio to other resources 4
Completeness of references 5
Analysis method 3
Conclusion 3
Writing style Representative article titles (straightforward and informative) 12
Inclusion of authors and affiliations (complete and consistent) 1
Abstract 2
Keyword 1
Structured description 1
Utilization of supporting documents (tables, figures, or supplements) 1
Reference citation style 1
Reference management (applications like Mendeley, etc.) 2
Terminology and language 2
Format of PDF and e-journal Format of PDF 8
Layout 1
Typography 1
PDF document resolution 2
Number of pages per volume 2
Journal website design 1
Regularity Regular publication 2
System of publishing order (volume, issue) 2
Page numbering 1
Retrieval in journal website (article, author) 1
Dissemination Count of unique visitors 4
Indexed in international databases (Scopus, Web of Science, DOAJ, etc.) 5
Unique identifier of articles (DOI) 2
Total 100

DOAJ, Directory of Open Access Journals.

Table 3.

Explanation of constants used to calculate the S-score suggested for measuring the performance of authors and institutes in Indonesia
Constant explanation Assessment weight Additional information
A No. of journal article documents in Scopus 40
B No. of non-journal-article documents in Scopus 15
C No. of citation in Scopus 4 If the no. of citations exceeds 1,000, then 1,000 will be used
D No. of citations in Google Scholar 1 If the no. of citations exceeds 1,000, then 1,000 will be used
E h-index in Scopus 16
F h-index in Google Scholar 4
N The divisor is a numerical result of a statistical calculation that takes into ccount the maximum SINTA score of a lecturer or researcher The current divisor value used is 102 (January 31,2017)

SINTA, Science and Technology Index.

Table 4.

Accreditation classification of journals published in Indonesia adopted by the portal Science and Technology Index
Classification Accredited score
S1 > 85 or indexed in Scopus
S2 71 – 85
S3 61 – 70
S4 51 – 60
S5 41 – 50
S6 31 – 40c

References

1. Pangkalan Data Perguruan Tinggi. Kemenristekdikti [Internet]. Jakarta: Ristekdikti 2017 [cited 2017 Apr 8]. Available from: http://forlap.ristekdikti.go.id/


2. Lukman L, Rianto Y, Nadhiroh IM. Development of national indicator performance evaluation indicators and research mapping systems with Scientometrics model. Jakarta: LIPI; 2017.


3. Lukman L, Yaniasih Y, Maryati I, Silalahi MA, Sihombing A. The strength of 50 Indonesian institutions: Scopus indexed publication profile. Jakarta: Ministry of Research, Technology and Higher Education; 2016.


4. Dikti D. Accreditation guidelines for scientific journal. Jakarta: Directorate General of Higher Education; 2014.


5. Lukman L. Index introduction: Scopus, Thomson Reuters, DOAJ, Sinta for librarians, lecturers and researchers. Jakarta: College Library Association; 2017.


6. Waltman L, Calero-Medina C, Kosten J, et al. The Leiden ranking 2011/2012: data collection, indicators, and interpretation. J Am Soc Inf Sci Technol 2012;63:2419–32. https://doi.org/10.1002/asi.22708
crossref

7. Hirsch JE, Buela-Casal G. The meaning of the h-index. Int J Clin Health Psychol 2014;14:161–4.
crossref

8. Meyers MA, Quan H. The use of the h-index to evaluate and rank academic departments. J Mater Res Technol 2017;6:304–11. https://doi.org/10.1016/j.jmrt.2017.09.004
crossref

Editorial Office
The Korea Science & Technology Center 2nd floor,
22 Teheran-ro 7-gil, Gangnam-gu, Seoul 06130, Korea
TEL : +82-2-3420-1390   FAX : +82-2-563-4931   E-mail : kcse@kcse.org
About |  Browse Articles |  Current Issue |  For Authors and Reviewers
Copyright © Korean Council of Science Editors. All rights reserved.