Performance comparison between Elecsys Anti-SARS-CoV-2 and Anti-SARS-CoV-2 S and Atellica IM SARS-CoV-2 Total and SARS-CoV-2 IgG assays
Article information
Abstract
Background
Although serological severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) tests from several manufacturers have been introduced in South Korea and some are commercially available, the performance of these test kits has not yet been sufficiently validated. Therefore, we compared the performance of Elecsys Anti-SARS-CoV-2 (ACOV2) and Anti-SARS-CoV-2 S (ACOV2S) and Atellica IM SARS-CoV-2 Total (COV2T) and SARS-CoV-2 IgG (sCOVG) serological tests in this study.
Methods
A total of 186 patient samples were used. For each test, we analyzed the positive rate of serological antibody tests, precision, linearity, and agreement among the four assays.
Results
The positive rates of COV2T, sCOVG, and ACOV2S were high (81.7%–89.2%) in total, with those for ACOV2S being the was the highest, while those of ACOV2 were as low as 44.6%. This may be related to the high completion rate of vaccination in Korea. The repeatability and within-laboratory coefficients of variation were within the claimed allowable imprecision; however, further research is needed to establish an allowable imprecision at low concentrations. COV2T showed a linear fit, whereas sCOVG and ACOV2S were appropriately modeled with a nonlinear fit. Good agreement was found among COV2T, sCOVG, and ACOV2S; however, the agreement between ACOV2 and any one of the other methods was poor.
Conclusions
Considering the different antigens used in serological SARS-CoV-2 antibody assays, the performance of the tested assays is thought to show no significant difference for the qualitative detection of antibodies to SARS-CoV-2.
Introduction
Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has been in the spotlight as the causative agent of coronavirus disease 2019 (COVID-19) since pneumonia patients were confirmed to be infected with a novel coronavirus in December 2019 [1,2]. The manifestations of SARS-CoV-2 infection can be various, spanning from an asymptomatic status to lethal complications [3,4]. As COVID-19 has been declared a global pandemic and spreads rapidly, accurate diagnosis is very important to control the pandemic.
Although a variety of laboratory tests are used for a differential diagnosis, real-time reverse transcription-PCR (RT-PCR) has been considered the gold standard to identify SARS-CoV-2 in clinical practice [5-7]. On the other hand, in serological testing, such as testing for SARS-CoV-2, antibody tests have been known to raise heterogeneity issues and limit identifying SARS-CoV-2 in the early stages of illness [8]. Antibody tests are primarily used to determine if a person has previously been infected with COVID-19 [9].
SARS-CoV-2 has four main structural proteins; spike (S), envelope (E), membrane (M), and nucleocapsid (N) proteins [10]. It has been reported that the S protein binds to the angiotensin-converting enzyme 2 (ACE2) to enter the host cell present on the surface of alveolar type II cells of the lung and epithelial cells of the oral mucosa [11-13]. The S protein has a subunit S1 harboring receptor-binding domains (RBD) which interacts with ACE2 [14,15]. These proteins, especially the S and N proteins, have been used as target antigens for the development of serological SARS-CoV-2 detection tests [16].
The antigens of Atellica IM SARS-CoV-2 Total (COV2T) and SARS-CoV-2 Immunoglobulin (Ig) G (sCOVG) antibody tests (Siemens Healthcare Diagnostics Inc., Tarrytown, NY, USA) target the S protein, while Elecsys Anti-SARS-CoV-2 (ACOV2) and Anti-SARS-CoV-2 S (ACOV2S) antibody tests (Roche Diagnostics International, Rotkreuz, Switzerland) target the N and S proteins, respectively. Although serological SARS-CoV-2 tests from several manufacturers have been introduced in South Korea and some of them are commercially available, the performance of each test kit has not yet been sufficiently validated. Therefore, we compared the performance of ACOV2, ACOV2S, COV2T, and sCOVG serological tests in this study.
Methods
Ethical statements: This study was approved by the Institutional Review Board of the Kosin University Gospel Hospital and informed consent was waived (KUGH 2022-03-038).
1. Preparation of patient samples
We collected leftover serum samples from 186 admitted patients in our hospital. After completing the requested tests from the inpatient clinics, each sample leftover was divided into new tubes and given a new code for the analyses.
2. Serological testing of antibodies to SARS-CoV-2
We used ACOV2 (Roche Diagnostics International), ACOV2S (Roche Diagnostics International), COV2T (Siemens Healthcare Diagnostics Inc.) and sCOVG (Siemens Healthcare Diagnostics Inc.) for the detection of antibodies to SARS-CoV-2. The sCOVG and ACOV2S test kits are immunoassays for the in vitro quantitative detection of IgG and total antibodies to SARS-CoV-2 in human serum and plasma, respectively, while the ACOV2 test qualitatively detects total antibodies to the SARS-CoV-2. The COV2T test is for the in vitro qualitative and semi-quantitative detection of total antibodies, including IgG and IgM antibodies, to SARS-CoV-2. The COV2T, sCOVG, and ACOV2S assays each use recombinant proteins representing the RBD of the S antigen in a sandwich assay format. On the other hand, an antibody to the N recombinant protein is used in the ACOV2 assay [16,17].
In this study, the COV2T and sCOVG tests and the ACOV2 and ACOV2S tests were performed on an Atellica IM automated analyzer (Siemens Healthcare Diagnostics Inc.) and a Cobas 8000 e801 automated analyzer (Roche Diagnostics International), respectively.
3. Positive rate of serological tests in enrolled patients
The enrolled patient group was divided into SARS-CoV-2 RT-PCR positive and negative groups. We calculated the percentage of serological SARS-CoV-2 positive test patients in two PCR testing groups and in the total patients group.
4. Precision analysis
Precision was determined according to the protocols from Clinical and Laboratory Standards Institute (CLSI) EP15-A3 [18]. Quality control materials were used in the precision evaluation. These controls consisted of two concentrations (low and high). The tests were performed at five runs per day for 5 days to calculate the repeatability and within-laboratory precision.
5. Linearity analysis
The protocol from CLSI EP06-A was used for linearity analysis [19]. We prepared two sets of five samples with equally spaced concentrations for the experiments. High and low concentrations of serum samples were prepared and they were mixed proportionally at 0:4, 1:3, 2:2, 3:1, and 4:0. We randomly measured the concentrations of the samples twice. The results were interpreted using the flowchart of EP06-A, which is based on a polynomial evaluation [19].
6. Agreement evaluation
To analyze the correlation between the assays, we used Pearson’s correlation coefficients and Passing-Bablok regression for the slope and intercept. Bland-Altman analysis was performed for the bias (mean difference) and 95% limits of agreement. Concordance rates and κ-values values were determined for the agreements between the methods based on the qualitative test results (positive vs. negative). Each serological SARS-CoV-2 antibody test used its own cutoff value that was claimed by its manufacturer to interpret the test results as positive or negative. The cutoffs for ACOV2, ACOV2S, COV2T, and sCOVG are 0.80 U/mL, 1.0 cutoff index (COI), 1.00 index, and 1.00 index (U/mL), respectively. When the measured serological test values were equal to or greater than the cutoff values, positive tests were reported.
7. Statistical analysis
The precision, linearity, Passing-Bablok regression, and Bland-Altman analyses were performed using Analyze-it for Microsoft Excel version 6.01.1 (Analyze-it Software Ltd., Leeds, UK). The remaining statistical analyses were performed using SPSS version 25 (IBM Corp., Armonk, NY, USA). Pearson correlation was used for the correlation between methods and kappa statistics were used for the agreement analysis. The κ-values are interpreted as follows: 0.00–0.20 for slight agreement; 0.21–0.40 for fair agreement; 0.41–0.60 for moderate agreement; 0.61–0.80 for substantial agreement; and 0.81–1.00 for almost perfect agreement [20]. The differences were considered statistically significant at a p<0.05.
Results
We prospectively analyzed the serum samples from 186 patients (66.2±15.2 years, male/female ratio=1.33), including 115 SARS-CoV-2 RT-PCR positive and 71 PCR negative patients. The 186 tested samples had a sampling date of 5 days or earlier after PCR testing.
1. Positive rate of serological tests in enrolled patients
The patient group was divided into 115 SARS-CoV-2 positive PCR and 71 negative PCR groups. The positive rates of COV2T and ACOV2S were the highest in SARS-CoV-2 RT-PCR positive and negative groups (87.0% and 94.4%, respectively) (Table 1). ACOV2S showed the highest positive rate (89.2%) in total, while the positive rates of ACOV2 were the lowest in SARS-CoV-2 RT-PCR positive and negative groups and in total (43.5%, 46.5%, and 44.6%, respectively).
2. Precision analysis
The repeatability and within-laboratory precision of COV2T, sCOVG, ACOV2, and ACOV2S tests are shown in Table 2. The repeatability and within-laboratory precision of the low concentration material of COV2T and ACOV2S are not presented because all values were below the measuring range (<0.6 U/mL and <0.4 U/mL, respectively). The CVs in the samples of low and high concentration materials ranged from 2.4% to 92.2% and from 1.0% to 2.5%, respectively.
3. Linearity analysis
Since ACOV2 is an immunoassay for the in vitro qualitative detection of antibodies to SARS-CoV-2, linearity analysis was not performed. The measuring intervals of COV2T, sCOVG, and ACOV2S were 0.64–73.56 U/mL, 0.53–141.15 U/mL, and 0.46–174.33 U/mL, respectively. COV2T showed excellent linearity within each analytical measurement range (Fig. 1). As analyzed using the flowchart of CLSI EP06-A, no 2nd or 3rd order polynomial fits were statistically better than a linear fit at the 5% significance concentration for COV2T. However, the 2nd order polynomial fits were statically better than a linear fit for sCOVG (p=0.048) and ACOV2S (p=0.001).
4. Agreement study
In Table 3, the correlation between sCOVG and ACOV2S was the best among the compared methods (r=0.833, p<0.001). The measured values of ACOV2S were greater than those of sCOVG (slope=142.7, intercept=–33.84) and the difference between the two paired methods proportionally increased with increasing concentrations (Fig. 2). COV2T was moderately correlated with sCOVG and ACOV2S (r=0.496, p<0.001 and r=0.413, p<0.001, respectively) (Table 3). The difference between ACOV2S and COV2T measurements showed a proportional increase to the average of the paired methods (Fig. 2). A proportional increase in the difference between COV2T and sCOVG measurements was observed. However, the two-pronged measurements were plotted against the average measurements over concentrations of around 40,000 U/mL (Fig. 2).
The concordance rate between methods was highest between COV2T and ACOV2S (96.8%), where the agreement was almost perfect (κ=0.839, p<0.001) (Table 4). Substantial agreements were observed between COV2T and sCOVG and between sCOVG and ACOV2S (r=0.750, p<0.001 and r=0.700, p<0.001, respectively). The concordance rate and agreement between ACOV2 and either one of the other methods were relatively worse compared to those between other combinations of compared groups (concordance rate and κ-value: 51.5% and 0.096, 53.2% and 0.124, and 51.1% and 0.098 between COV2T and ACOV2, sCOVG and ACOV2, and ACOV2 and ACOV2S, respectively).
Discussion
After all, clinical complications and prognosis are the main concerns in SARS-CoV-2 patients. Therefore, accurate measurement of antibodies to SARS-CoV-2 should help to predict and monitor the clinical manifestations during treatments. Unfortunately, several medications have been introduced worldwide, however, the validity of the treatments seems to be not perfectly proven yet [21-23]. This would make serological antibody tests more valuable for the better prognosis of patients. From this perspective, the performance comparison of this study would provide some insight into serological SARS-CoV-2 antibody tests.
When this study began, a majority of Koreans had completed vaccination for COVID-19 and every admitting patient had to be tested with SARS-CoV-2 RT-PCR to identify if a patient was currently infected with SARS-CoV-2; omicron was the dominant coronavirus variant in South Korea [24]. As a result, almost all enrolled patients were assumed to have experienced either real coronavirus infections or vaccination before admission. Although we collected samples from patients with SARS-CoV-2 positive and negative RT-PCR results, the positive rate of the serological test was highest in the negative RT-PCR group (94.4%) (Table 1). The positive rates of serological tests spanned from 79.1% to 94.4% and those of ACOV2 were relatively low (43.5%–46.5%). Considering that ACOV2 uses a nucleocapsid protein as an antigen and most vaccines used in Korea for inoculation were targeting spike protein in SARS-CoV-2 [25,26], it is assumed that more than half of enrolled patients had not recently been exposed to SARS-CoV-2 and that it is likely that the high positive rates in serological tests were caused by vaccinations. Practically, the seroprevalence of N-specific antibodies has been reported to remain as low as 1.59% after a vaccine campaign in Japan [27].
In Table 2, repeatability and within-laboratory coefficient of variations (CVs) are within the allowable imprecision claimed by manufacturers, except for those in the low concentration control in the sCOVG antibody test. The upper repeatability and within-laboratory CVs claimed by the manufacturer were 12.0% and 15% within the 0.80–2.00 U/mL concentration interval, respectively. The mean of low concentration material was 0.019 U/mL in this study, which is at least a 42.1 times lower concentration. In a previous study, materials over 2.0 U/mL were used for imprecision and the CVs were lower than the manufacturer’s claim [28]. Therefore, further evaluation needs to be performed to establish the imprecision at lower concentrations.
Although nonlinearity was determined in sCOVG and ACOV2S, the degree of nonlinearity of these two methods seemed visually different. The sCOVG test was relatively more linear than the ACOV2S test. When the difference plot was recalculated again when setting the allowable nonlinearity to 15%, which is claimed by manufacturers across the measuring interval, the concentration of nonlinearity was at the left end in both methods. After removing the left endpoint, the degree of nonlinearity was alleviated. As a result, the remaining 4 points were located within the allowable nonlinearity interval and the p-value of the 2nd order polynomial equation increased from 0.001 to 0.018 for the ACOV2S test (data not shown), which indicates that the degree of nonlinearity was more alleviated than before. Likewise, after removing left endpoint of the sCOVG test, the p-value of the 2nd order polynomial equation was changed from 0.048 to 0.747 (data not shown), which is more appropriate for a linear fit. Instead of establishing linearity, the range of the measuring interval should be reduced. Although COV2T has been claimed to be a semi-quantitative test, its linearity was good.
In the Passing-Bablok analysis, the slopes of regression lines were 0.041 and 142.7 between COV2T and sCOVG and between sCOVG and ACOV2S, respectively. This indicates that the absolute concentration of reported test results should be greatly different depending on the methods. Although the compared three methods detect different antibodies, test harmonization issues need to be raised. Significant differences between plasma- and serum-based SARS-CoV-2 antibody tests have already been reported [29].
The correlation and agreement were best between sCOVG and ACOV2S antibody tests (r=0.833, κ=0.839, p<0.001) (Tables 2, 3). Although the cutoffs of positive for anti-SARS-CoV-S in sCOVG and ACOV2S are not markedly different (0.80 U/mL and 1.0 U/mL, respectively), the difference in the absolute concentration between them showed a proportionally increasing trend as the concentration of antibodies increased. This might happen because constant CV% was affected across the measuring interval, which would show a linear slope. This could be caused by a calibration error in one method [30,31]. Since the manifestation of a regression line between ACOV2S and COV2T is different from the other paired methods, it is not easy to confirm if this is due to a proportional constant error, such as a calibration error.
Serological SARS-CoV-2 antibody tests require a careful interpretation in clinical practice because the sensitivity of the serological tests varies depending on the days after disease onset [32]. The cumulative seroprevalence for IgM and IgG increased from 44% and 56% on day 7 after symptom onset to over 95% on day 20 for IgM and day 16 for IgG, respectively [33]. The enrolled patient samples did not contain clinical information except for the age, sex, and SARS-CoV-2 RT-PCR results in this study. Furthermore, the majority of admitted patients had already completed vaccination when samples were obtained from patients. Therefore, the diagnostic validity of the assays could not be evaluated due to the lack of linked clinical information.
In conclusion, the positive rates of COV2T, sCOVG, and ACOV2S were as high as 81.7%–89.2% in total, and that for ACOV2S was the highest, while those of ACOV2 were as low as 44.6%. This may be related to the high completion rate of vaccination in Korea. The precision results were allowable based on the claimed allowable imprecision, however, further study needs to be conducted to establish the allowable imprecision at a lower concentration level. COV2T showed a linear fit, on the other hand, sCOVG and ACOV2S were appropriately modeled with a nonlinear fit. The agreements among COV2T, sCOVG, and ACOV2S were good, however, those between ACOV2 and either one of the other methods were poor, which is assumed to be due to the different antigens in methods. Considering the different adopted antigens that the serological SARS-CoV-2 antibody assays use, the performance of tested assays is thought to show no significant difference for the qualitative detection of antibodies to SARS-CoV-2.
Notes
Conflicts of interest
Seri Jeong and Hyunyong Hwang are editorial board members of the journal but were not involved in the peer reviewer selection, evaluation, or decision process of this article. No other potential conflicts of interest relevant to this paper were reported.
Funding
This research was supported by Roche Diagnostics and Siemens Healthineers companies in 2022. The funders had no role in study design, analysis, interpretation or publication. We declare that we have no financial conflicts of interest.
Author contributions
Conceptualization: HH. Data curation: SJ, YRH, HH. Formal analysis: HH. Funding acquisition: HH. Investigation: SJ, YRH. Methodology: HH. Project administration: HH. Supervision: HH. Validation: HH. Visualization: SJ, YRH. Writing - original draft: SJ, YRH. Writing - review & editing: HH. Approval of final manuscript: all authors.