Next Article in Journal
Aptamer-Based Sensors for Thrombin Detection Application
Previous Article in Journal
Luminescent Sensors Based on the Assembly of Coinage Metal Nanoclusters
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Chemosensors
Volume 10
Issue 7
10.3390/chemosensors10070254
chemosensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Ultra-Fast and Sensitive Screening for Antibodies against the SARS-CoV-2 S1 Spike Antigen with a Portable Bioelectric Biosensor

by
Sofia Mavrikou
1,*,
George Marios Papaioannou
1,
Vasileios Tsekouras
1,
Kyriaki Hatziagapiou
2,3,
Elizabeth Barbara Tatsi
2,
Filippos Filippatos
2,
Christina Kanaka-Gantenbein
2,
Athanasios Michos
2 and
Spyridon Kintzios
1,*
1
Laboratory of Cell Technology, Department of Biotechnology, Agricultural University of Athens, EU-CONEXUS European University, 11855 Athens, Greece
2
First Department of Pediatrics, National and Kapodistrian University of Athens, “Aghia Sophia” Children’s Hospital, Thivon 1, 11527 Athens, Greece
3
Physiotherapy Department, Faculty of Health and Care Sciences, State University of West Attica, Agiou Spiridonos 28, Egaleo, 12243 Athens, Greece
*
Authors to whom correspondence should be addressed.
Chemosensors 2022, 10(7), 254; https://doi.org/10.3390/chemosensors10070254
Submission received: 1 June 2022 / Revised: 24 June 2022 / Accepted: 28 June 2022 / Published: 30 June 2022
(This article belongs to the Section (Bio)chemical Sensing)
Browse Figures
Versions Notes

Abstract

:
As a consequence of the progress of the global vaccination against the COVID-19 disease, fast, accurate and affordable assays are needed for monitoring the efficiency of developing immunity against the coronavirus at the population level. In this context, we herewith report the proof-of-concept development of an innovative bioelectric biosensor for the ultra-detection (in less than three minutes) of IgG antibodies against the SARS-CoV-2 S1 spike antigen. The biosensor comprises a disposable set of screen-printed electrodes upon which are immobilized cells engineered to bear the S1 protein on their surface. When anti-S1 antibodies are presented to the engineered cell population, a rapid, specific, and selective change of the cell membrane potential occurs; this is in turn recorded by a bespoke portable potentiometer. End results are communicated via Bluetooth to a smartphone equipped with a customized user interface. By using the novel biosensor, anti-S1 antibodies could be detected at concentrations as low as 5 ng/mL. In a preliminary clinical trial, positive results were derived from patients vaccinated or previously infected by the virus. Selectivity over other respiratory viruses was demonstrated by the lack of cross-reactivity to antibodies against rhinovirus. After further clinical validation and extension to also screen IgM, IgA and possible neutralizing antibodies, our approach is intended to facilitate the mass and reliable detection of antibodies in the early stages following vaccination and to monitor the duration and level of acquired immunity both in a clinical and self-testing environment.

Graphical Abstract

1. Introduction

Following the coronavirus disease (COVID-19) outbreak, novel vaccines have been rapidly developed and deployed in several countries worldwide. Although available data so far indicates that vaccines are indeed highly effective at inducing protection against infection by SARS-CoV-2, limiting the severity of disease in infected patients [1], individual responses to vaccination may considerably vary due to factors such as age, sex, and others [2]. Numerous studies have demonstrated that the accumulation of the spike S1 glycoprotein antigen and its corresponding antibodies in the plasma of vaccinated patients is the only reliable indication of messenger RNA (mRNA) translation in the host body. Ogata et al. [3] have shown that S1 antigens appear as early as day one post-vaccination with a peak level detectable on day five. In correspondence, the levels of anti-S1 IgG antibodies peak within 2–3 weeks and may be detectable up to four months later, contrary to the levels of both IgM and IgA antibodies, which decline until week 8 after their initial peak. In fact, anti-S1 IgG levels may increase up to 100× two weeks after the first injection with an mRNA-based vaccine [4]. That said, a combined IgG, IgM/IgA test would provide more sensitivity than testing for a single antibody type [5]. The level of anti-nucleocapsid antibodies, associated with natural infection by the virus, does not change over time.
Although high-throughput serological assays for SARS-CoV-2 antibodies have been made commercially available over the last two years, they are not sensitive enough to provide meaningful information on antibody levels during the first week after vaccination [6], having a limit of detection in the range of 150–200 μg/mL for IgG and often lacking established positive controls and reference standards [7]. In addition, they are considered less reliable than plaque reduction neutralization tests (PRNTs), which are considered the gold standard for assay specificity [8], in part due to their limited range of measured antibody levels. On the other hand, the majority of rapid and portable assays and kits for the determination of SARS-CoV-2 antibodies relate to self-tests aimed at detecting infection by the virus by testing for patient immunoglobins, thus allowing for comparative discrimination between early infection stages (IgM, 4–10 days) or late ones (IgG, 11 days or later). In this respect, they are efficient since antibody levels are usually sufficiently high to detect at the early stages of infection, with median IgM detection occurring five days and IgG detection after a median of 14 days after symptom onset, followed by a significant decline after two or three months. In addition, the overwhelming majority of POC antibody tests target anti-nucleocapsid (anti-NP) antibodies, which offer limited information about the host response to immunity development [9,10].
Furthermore, should a rapid, low-cost and easy-to-use anti-S1 antibody assay with sufficient sensitivity and reliability become available for the general population, it would contribute to a faster return to pre-pandemic standards of socialization and a reduction in social distancing and other measures. In the present study, we report the proof-of-concept development of a cell-based assay for the ultra-fast, ultra-sensitive detection of anti-S1 antibodies against SARS-CoV-2. The working principle of the assay is based on the already established technology of molecular identification through membrane engineering, according to which mammalian cells engineered by electroinserting membrane-bound biorecognition elements respond to the binding of the respective corresponding analytes with a very specific and measurable change in their membrane potential [11]. This approach has been recently used for the detection of the SARS-CoV-2 S1 protein in both laboratory and clinical settings using either anti-S1 antibodies [12] or the angiotensin-converting enzyme 2 (ACE2) [13] as the membrane-bound biorecognition elements. In two independent, subsequent clinical trials, the application of this novel bioelectric biosensor approach confirmed the detection of the virus in positive samples with a 92.8% success rate compared to RT-PCR [14,15]. Therefore, in the present study, we reversed the previous approach by using S1 antigens as the biorecognition elements, which were electroinserted in mammalian cells to determine anti-S1 antibodies in standard solutions and samples derived from a small group of either vaccinated or previously infected patients. The perspective of a very sensitive, fast, and specific test for anti-SARS-CoV-2 antibodies is demonstrated at the proof-of-concept. No false-negative results were recorded. No cross-specificity was observed for antibodies against rhinovirus, another significant respiratory virus. In addition, the Internet of Things (IoT) compatibility of the sensor bears promise for its possible application for mass population seroprevalence monitoring, public health and vaccination strategy planning. Finally, the configuration of the biosensor module could enable its future adaptation for self-testing to individually access the level of immunity and support decisions for vaccination schemes.

2. Materials and Methods

2.1. Cell Culture and Growth Conditions

The cells (SK-N-SH neuroblastoma cells, ATCC HTB-11TM) were cultured under standard conditions at 37 °C and 5% CO2, in 1× Minimum Essential Medium (MEM) with Earle’s balanced salt (Biowest, Nuaillé, France). In the culture medium, 10% of fetal bovine serum (FBS) (Thermo Fisher Scientific, Waltham, MA, USA) was added, as well as 2 mM of L-glutamine, 0.1 mM nonessential amino acids, 1 mM sodium pyruvate (Biowest, Nuaillé, France) and 1 Uμg-1 antibiotic (penicillin/streptomycin). Cells were subcultured twice per week in a 1:10 ratio each time. The dissociation of the cells from the culture flask was accomplished after a five-minute treatment with a solution of Trypsin-EDTA (0.05% trypsin, 0.02% EDTA) (Biowest, Nuaillé, France).

2.2. Sensor Fabrication from Membrane-Engineered SK-N-SH Cells (SK-N-SH/S1 Spike Protein)

The membrane-engineered SK-N-SH cells were fabricated with the electroinsertion of SARS-CoV-2 S1 spike protein into the cell membrane. Briefly, neuroblastoma cells were dissociated from the culture flask with Trypsin-EDTA solution and were collected after a 4 min, 1000 rpm centrifugation. The preferable final cell density was 2.5 × 106 cells/mL. The pellet from the centrifugation (SK-N-SH cells) was resuspended in phosphate-buffered saline (PBS) (pH 7.4) with three (3) concentrations (0, 0.5, and 5.0 μg/mL) of recombinant human coronavirus SARS-CoV-2 Spike Glycoprotein S1 (ab273068, Abcam plc, Cambridge, UK).
SARS-CoV-2 S1 spike protein solutions were incubated with the cells for 20 min at 4 °C. Afterward, the solutions were inserted into electroporator cuvettes (Eppendorf Eporator, Eppendorf AG, Hamburg, Germany) and were treated twice with an electric field at 1800 V/cm for 5 ms. Finally, the cells were transferred to a petri dish (60 × 15 mm) with 3 mL of cell culture medium and incubated overnight in culture chamber. The day after the electroporation, the membraned engineered cells (SK-N-SH/S1 spike protein) were mechanically detached from the surface of the petri dish with PBS and collected in Eppendorf tubes (Figure 1).

2.3. Measurements

As reported previously [11,16] the membrane potential of membrane-engineered cells alters when analytes bind to the electroinserted bioactive molecules/receptors, producing bioelectric changes in the cell membranes, measurable with the use of an appropriate potentiometer. In our study, a multichannel custom-made potentiometer was used for the recording of the cell membrane potential changes according to the BERA principle (Bioelectric Recognition Assay) [17,18]. For each measurement, a disposable sensor strip (iMiCROQ S.L., Tarragona, Spain) containing eight screen-printed carbon electrodes (working electrode: carbon, reference: Ag/AgCl) was used.
Membrane-engineered cell suspensions (50,000 cell/electrode) were added at the top of the electrodes. Then, 10 μL of the sample (serum spiked with the Recombinant Anti-SARS-CoV-2 Spike Glycoprotein S1 antibody (ab273073, Abcam plc, Cambridge, UK) or unknown patient sample) were added. The response of the cells was measured immediately as a time series of potentiometric measurements (in volts) through austom-made software. The duration of each measurement was 3 min, and each time 360 values were recorded.

2.4. Serum Specimens Collection

Patients (4 volunteers) participating in the study (Table 1) tested positive for RT-PCR (Real-Time Polymerase Chain Reaction) for SARS-CoV-2, using nasopharyngeal swabs. All samples were collected by a standardized procedure, using Dacron swabs (FIRATMED, 8870000244, Albacete, Spain), at the time of hospital admission, and >2–3 weeks prior venous blood collection for antibody detection with the biosensor. We also included 3 healthy volunteers, all immunized with the second dose of mRNA vaccines, one month before blood sample collection. Healthy volunteers had never experienced any COVID-19 compatible symptoms or had ever been tested positive for SARS-CoV-2.
Venous blood samples from all volunteers were collected in 5 mL volume in EDTA Vacutainer® tubes (Beckton Dickinson) for antibody identification and measurement. Plasma samples were separated from the EDTA blood after centrifugation at 1200× g for 15 min and stored at −80 °C until the analysis. All plasma samples, prior to biosensor analysis, were validated with the Roche Elecsys ® Anti-SARS-CoV-2 and Elecsys ® Anti-SARS-CoV-2 S immunoassays run on the Roche cobas e 411 analytical unit, following the manufacturer’s instructions (COI ≥ 1.0 and >0.8 U/mL, respectively, was considered positive). The Roche Elecsys® Anti-SARS-CoV-2 immunoassay uses a recombinant protein representing the nucleocapsid (N) antigen and provides detection of antibodies (including IgG) (https://diagnostics.roche.com, accessed on 15 May 2020), while the Elecsys® Anti-SARS-CoV-2 S immunoassay was used to detect IgG antibodies against the receptor-binding domain (RBD) of S1 subunit of SARS-CoV-2 spike protein [19]. Both are validated for use on human serum and plasma.

2.5. Data Analysis and Experimental Design

At the first experimental procedure we optimized the electroinserted antigen concentration (S1 protein) and fabricated standard curves of the biosensor’s responses to 6 different concentrations of the SARS-CoV-2 S1 antibody (0.005, 0.01, 0.05, 0.1, 0.5, 1 μg/mL).
Then, we investigated putative cross-reactivity with other pathogens antibodies (Mouse monoclonal antibody specific for Rhinovirus VP3 Antibody, MAB12382, Native Antigen Company, Kidlington, UK) in spiked human blood serum samples from non-vaccinated healthy individuals (Table 2).
The experiment was repeated on three different days, and all the results are expressed as a ratio to control (mean ± SD) according to the following equation:
N o r m a l i z e d   B i o s e n s o r   R e s p o n s e = m e a n   f i n a l   v a l u e   o f   s a m p l e m e a n   f i n a l   v a l u e   o f   c o n t r o l
Differences between the means were tested for statistical significance using ANOVA and t-test.

3. Results

Optimizaton of Membrane-Engineered SK-N-SH—S1 Protein Cells Responses against the SARS-CoV-2 S1 Antibody
A 10 μL sample volume of several concentrations of the anti-S1 spiked serum (0, 0.005, 0.01, 0.05, 0.1, 0.5, and 1 μg/mL) was added to 50,000 membrane-engineered cells onto the electrode surface. When plain PBS (0 μg/mL S1) was used for the cell-membrane engineering process, the responses recorded (Figure 2A) after the addition of anti-S1 antibodies had no statistically significant differences in comparison with serum samples free of antibodies (control). In contrast, when 0.5 μg/mL and 5 μg/mL of S1 were used for increasing cell specification, the results obtained displayed an increase in the cell membrane potential (Figure 2B,C). Upon exposure to increasing concentrations of SARS-CoV-2 anti-S1 antibody, the SK-N-SH/S1 membrane-engineered cells presented dose-dependent responses relative to the anti-S1 concentrations used (up to 0.1 μg/mL).
The lowest anti-S1 concentration detected by the SK-N-SH/S1 membrane engineered cells (Figure 2C), which presented statistically significant differences from the control, was 0.005 μg/mL. Overall, the bioelectric signal produced in the case of cells membrane-engineered with 0.5 μg/mL (Figure 2B) was relatively weak in comparison with the cellular responses obtained with the 5 μg/mL S1 (C). A semi-linear concentration-dependent pattern was observed with the use of 5 μg/mL of S1 for cell membrane engineering, at a concentration range between 0.05 and 0.1 μg/mL of anti-S1. At higher concentrations, the signals reach a plateau and start to decline. This phenomenon has been previously observed in several assays regarding the bioelectrical identification of molecules through membrane engineering [13,20,21]. It could be attributed to the saturation of the system when the concentration of the analyte exceeds a certain concentration.
Membrane-engineered cells with 5 μg/mL S1 antigen were used to assess the biosensor’s accuracy and specificity. Serum samples from healthy donors (10), vaccinated individuals (3), and COVID-19 patients (4) were used. The normalized cell biosensor’s responses against serum samples derived from vaccinated individuals and patients are presented in Figure 3. The biosensor was able to detect antibodies from all positive samples, with a statistical significance of p < 0.001. In addition, the biosensor successfully identified all negative samples (serum from non-vaccinated healthy individuals) with no false-positive results.
The bioelectric biosensor based on SK-N-SH/S1 spike protein cells, membrane-engineered with 0.5 μg/mL (Figure 4A) and 5 μg/mL (Figure 4B) of SARS-CoV-2 S1 protein, was also evaluated for cross-reacting with antibodies from the outer capsid protein 3 of another respiratory virus (rhinovirus). Six different concentrations of the rhinovirus VP3 antibody (0.005, 0.01, 0.05, 0.1, 0.5, 1 μg/mL) spiked into human serum were tested. As the concentration of the S1 antigen inserted into the membrane of SK-N-SH cells increased, a decrease in the normalized biosensor signal was observed. The normalized biosensor responses were significantly higher than the control (serum samples from healthy non-vaccinated individuals), as well as the responses against similar concentrations of the anti-S1 protein antibodies. Furthermore, no differentiation between the applied concentrations of the VP3 antibodies could be observed.

4. Discussion

The use of membrane-engineered cells for the detection of viral antibodies has already been reported in the past by Perdikaris et al. [22], who developed a biosensor for the detection of anti-HBs antibodies based on immobilized Vero cells bearing the respective antigen (HBsAg). As biorecognition elements, the customized cells responded in a very rapid (45 s) and reproducible way. A relationship between changes in [Ca2+]cyt and the attachment of HBV particles to cells membrane-engineered with anti-HBs was also established in that report.
Following a similar approach in the present study, we demonstrated at the proof-of-concept that (a) mammalian cells engineered to bear the SARS-CoV-2 S1 spike protein antigen are able to detect the corresponding anti-S1 antibody by changing their membrane-potential (b) said change is very sensitive (with a primarily established LOD of 5 ng/mL), selective (at least against anti-rhinovirus antibodies) and ultra-fast (three minutes) (b) it is feasible to develop a portable biosensor based on the investigated approach for the practical determination of antibodies in both vaccinated and past infected patient samples. It was also determined that higher concentrations of the target anti-S1 antibody did not result in a higher response of the biosensor; this might be due to a saturation of the biosensor’s response due to increased competition among membrane-bound antigens and antibodies at high concentrations in the vicinity of the engineered cell membrane. This putative condition could lead to a lower antibody-antigen binding rate compared to antigens interacting with antibodies at lower concentrations, i.e., a type of “hook-effect” [23] may be responsible for this particular pattern of response. Point-of-care (POC) applications in in vitro diagnostics (IVD) are especially vulnerable to the hook effect. The hook effect is a common phenomenon documented in various immunological and serological assays that leads to false-negative results. It is a frequently occurring event that is notoriously difficult to detect in clinical laboratories, where the reporting of erroneous results can have serious medical implications [24]. A false-negative result from the hook effect can result in incorrect clinical decisions with consequential suboptimal patient care [25,26].
Advanced optical sensor technologies (Table 3) such as photonic [27] and surface plasmon resonance sensors [28,29] are able to provide results within 10 min, whereas more traditional approaches require more time (LFA, ELISA, 15–60 min) [30,31].On the other hand, electrochemical sensors require a mean response time of approximately 30 min [32,33,34,35,36]. Taking into account the above-mentioned, our biosensor is able to detect SARS-CoV-2 antibodies in an ultra-rapid manner (3 min) and at quite low concentrations (5 ng/mL).
Most electrochemical sensors for SARS-CoV-2 antibody detection (listed in Table 3) present LOD levels ranging from 37 to 200 ng/mL, whereas optical sensors can get as low as 12.75 ng/mL. The sensitivity of the novel biosensor is considerably high when compared with the LOD levels reported with commonly used serology tests (20–180 mg/mL) [7], therefore demonstrating its potential application for monitoring seroprevalence of anti-S1 antibodies even during the phase of their declining concentration in the patient’s blood, i.e., relatively late after the last vaccination or infection event. This type of information is crucial in order to determine the level of population and/or individual immunity and the necessity of interventive actions, such as recurrent/booster vaccine doses. As it stands, the present study cannot guarantee the use of the novel biosensor for the quantitative determination of the antibody titer in clinical samples; however, the concentration-dependent response observed in our experiments, albeit in a very limited range of concentrations, justifies for further investigation of the range of potential linear response against various antibody concentrations.
Regarding selectivity and cross-reactivity of the presented approach, the SK-N-SH cell line is affected by rhinovirus 1. The virus affects neuronal cells by inducing the expression of proinflammatory cytokines such as IL-1β, IL-6 and TNF-α. Furthermore, rhinoviruses and associated anti-bodies can bind to different types of cellular membrane glycoproteins, in particular the intercellular adhesion molecule 1 (ICAM-1) [40]. They are also known to be able to enter neuroblastoma cells, such as SK-N-SH, which were used in the present study [41]. More interestingly, it has been previously shown that ICAM-1 cell adhesion events are associated with changes in calcium influx in SK-N-SH cells followed by changes in cell membrane potential [42]. In our study, human blood serum samples from non-vaccinated healthy individuals were spiked with mouse monoclonal antibodies specific for Rhinovirus VP3 Antibody to investigate putative cross-reactivity with other pathogens’ antibodies. The very high bioelectric signal produced by the interaction between cells membrane-engineered to the S1 spike antigen and anti-rhinovirus antibodies is clearly and readily distinguishable from the lower-intensity signal related to the response of the same cells against anti-SARS-CoV-2 antibodies indicating that anti-rhinovirus antibodies stimulate false-positive results.
Achievement of population immunity (also known as herd immunity) is a definite goal of the current global vaccination program against COVID-19. Besides vaccination, immunity can also be developed through previous infection [43]. However, the level of acquired immunity depends on the individual response as determined by various factors, including age, sex, underlying disease, etc. For example, De Giorgi et al. [44] reported that, although 91.4% of 116 convalescent patients had detectable anti-SARS-CoV-2 IgG levels up to 11 months after symptom recovery, 25% of the donors had neutralizing levels that dropped to an undetectable titer over time. This phenomenon has been confirmed in further studies [45,46]. An improved, more accurate knowledge of the pattern of population-level immunological response is critical for the design of an efficient vaccination strategy and the estimation of the risk of reinfection. Therefore, tools facilitating the derivation of information about the long-term kinetics of anti-SARS-CoV-2 antibodies would be quite helpful to be able to determine the level of cumulative protection against the virus, a process complicated both by the different types of vaccines and vaccination rates and the emergence of virus variants. In addition, it should be said tools become available as portable or handheld devices tailor-made for home testing, a considerable milestone would be achieved in supporting individual strategies and decisions regarding social distancing and revaccination. Novel materials such as molecularly imprinted polymer nanoparticles can be utilized to increase test sensitivity and the ability to withstand harsh environmental conditions [47].
In conclusion, even though the novel biosensor has been tested at the proof-of-concept, the results presented in this report are interesting enough to justify its further clinical validation and extension to also screen IgM and IgA antibodies, possibly also neutralizing (Neu) antibodies, as well as testing over a broader range of anti-SARS-CoV-2 antibody concentrations (above 1 μg/mL) and possible cross-reactive antibody species. It should be emphasized that, due to the current set-up of the biosensor, test results can be processed instantly, classified, and recorded via mobile internet to a central server where an e-certificate can be automatically issued. In addition, cloud-based result data can be used as a valuable source of information and statistics for government agencies responsible for public health planning as well as the pharmaceutical industry. In this way, and after its due optimization, our approach is intended to facilitate the mass and reliable detection of antibody detection in the early stages following vaccination and to monitor the duration and level of acquired immunity, as well as support individual and personalized self-testing in the future.

Author Contributions

Conceptualization, S.K.; methodology, S.K. and S.M.; formal analysis, S.K., S.M., K.H., V.T., G.M.P., E.B.T. and F.F.; investigation S.M., G.M.P. and S.K.; data curation, S.M., K.H., V.T. and S.K.; writing—original draft preparation, S.K., K.H., G.M.P., V.T. and S.M.; writing—review and editing, S.K.; supervision, S.K., C.K.-G. and A.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Scientific Committee of the “Aghia Sophia” Children’s Hospital.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to medical confidentiality and privacy.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Baden, L.R.; El Sahly, H.M.; Essink, B.; Kotloff, K.; Frey, S.; Novak, R.; Diemert, D.; Spector, S.A.; Rouphael, N.; Creech, C.B.; et al. Efficacy and Safety of the mRNA-1273 SARS-CoV-2 Vaccine. N. Engl. J. Med. 2021, 384, 403–416. [Google Scholar] [CrossRef] [PubMed]
  2. Arkhipova-Jenkins, I.; Helfand, M.; Armstrong, C.; Gean, E.; Anderson, J.; Paynter, R.A.; Mackey, K. Antibody Response After SARS-CoV-2 Infection and Implications for Immunity: A Rapid Living Review. Ann. Intern. Med. 2021, 174, 811–821. [Google Scholar] [CrossRef] [PubMed]
  3. Ogata, A.F.; Cheng, C.-A.; Desjardins, M.; Senussi, Y.; Sherman, A.C.; Powell, M.; Novack, L.; Von, S.; Li, X.; Baden, L.R.; et al. Circulating Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Vaccine Antigen Detected in the Plasma of mRNA-1273 Vaccine Recipients. Clin. Infect. Dis. 2021, 74, 715–718. [Google Scholar] [CrossRef] [PubMed]
  4. Bradley, B.T.; Bryan, A.; Fink, S.L.; Goecker, E.A.; Roychoudhury, P.; Huang, M.L.; Zhu, H.; Chaudhary, A.; Madarampalli, B.; Lu, J.Y.C.; et al. Anti-SARS-CoV-2 Antibody Levels Measured by the AdviseDx SARS-CoV-2 Assay Are Concordant with Previously Available Serologic Assays but Are Not Fully Predictive of Sterilizing Immunity. J. Clin. Microbiol. 2021, 59, e0098921. [Google Scholar] [CrossRef]
  5. Sethuraman, N.; Jeremiah, S.S.; Ryo, A. Interpreting Diagnostic Tests for SARS-CoV-2. JAMA 2020, 323, 2249–2251. [Google Scholar] [CrossRef]
  6. Padoan, A.; Bonfante, F.; Cosma, C.; Chiara, C.D.; Sciacovelli, L.; Pagliari, M.; Bortolami, A.; Costenaro, P.; Musso, G.; Basso, D.; et al. Analytical and clinical performances of a SARS-CoV-2 S-RBD IgG assay: Comparison with neutralization titers. Clin. Chem. Lab. Med. 2021, 59, 1444–1452. [Google Scholar] [CrossRef]
  7. Tré-Hardy, M.; Wilmet, A.; Beukinga, I.; Favresse, J.; Dogné, J.M.; Douxfils, J.; Blairon, L. Analytical and clinical validation of an ELISA for specific SARS-CoV-2 IgG, IgA, and IgM antibodies. J. Med. Virol. 2021, 93, 803–811. [Google Scholar] [CrossRef]
  8. Jääskeläinen, A.J.; Kuivanen, S.; Kekäläinen, E.; Ahava, M.J.; Loginov, R.; Kallio-Kokko, H.; Vapalahti, O.; Jarva, H.; Kurkela, S.; Lappalainen, M. Performance of six SARS-CoV-2 immunoassays in comparison with microneutralisation. J. Clin. Virol. 2020, 129, 104512. [Google Scholar] [CrossRef]
  9. Kim, S.; Hao, Y.; Miller, E.A.; Tay, D.M.Y.; Yee, E.; Kongsuphol, P.; Jia, H.; McBee, M.; Preiser, P.R.; Sikes, H.D. Vertical Flow Cellulose-Based Assays for SARS-CoV-2 Antibody Detection in Human Serum. ACS Sens. 2021, 6, 1891–1898. [Google Scholar] [CrossRef]
  10. Wang, J.J.; Zhang, N.; Richardson, S.A.; Wu, J.V. Rapid lateral flow tests for the detection of SARS-CoV-2 neutralizing antibodies. Expert Rev. Mol. Diagn. 2021, 21, 363–370. [Google Scholar] [CrossRef]
  11. Kokla, A.; Blouchos, P.; Livaniou, E.; Zikos, C.; Kakabakos, S.E.; Petrou, P.S.; Kintzios, S. Visualization of the membrane engineering concept: Evidence for the specific orientation of electroinserted antibodies and selective binding of target analytes. J. Mol. Recognit. 2013, 26, 627–632. [Google Scholar] [CrossRef] [PubMed]
  12. Mavrikou, S.; Moschopoulou, G.; Tsekouras, V.; Kintzios, S. Development of a Portable, Ultra-Rapid and Ultra-Sensitive Cell-Based Biosensor for the Direct Detection of the SARS-CoV-2 S1 Spike Protein Antigen. Sensors 2020, 20, 3121. [Google Scholar] [CrossRef] [PubMed]
  13. Mavrikou, S.; Tsekouras, V.; Hatziagapiou, K.; Tsalidou, A.; Bakakos, P.; Rovina, N.; Koutsoukou, A.; Michos, A.; Nikola, O.; Koniari, E.; et al. Angiotensin-Converting Enzyme 2 (ACE2) As a Novel Biorecognition Element in A Cell-Based Biosensor for the Ultra-Rapid, Ultra-Sensitive Detection of the SARS-CoV-2 S1 Spike Protein Antigen. Chemosensors 2021, 9, 341. [Google Scholar] [CrossRef]
  14. Apostolou, T.; Kyritsi, M.; Vontas, A.; Loizou, K.; Hadjilouka, A.; Speletas, M.; Mouchtouri, V.; Hadjichristodoulou, C. Development and performance characteristics evaluation of a new Bioelectric Recognition Assay (BERA) method for rapid Sars-CoV-2 detection in clinical samples. J. Virol. Methods 2021, 293, 114166. [Google Scholar] [CrossRef]
  15. Mavrikou, S.; Tsekouras, V.; Hatziagapiou, K.; Paradeisi, F.; Bakakos, P.; Michos, A.; Koutsoukou, A.; Konstantellou, E.; Lambrou, G.I.; Koniari, E.; et al. Clinical Application of the Novel Cell-Based Biosensor for the Ultra-Rapid Detection of the SARS-CoV-2 S1 Spike Protein Antigen: A Practical Approach. Biosensors 2021, 11, 224. [Google Scholar] [CrossRef]
  16. Moschopoulou, G.; Vitsa, K.; Bem, F.; Vassilakos, N.; Perdikaris, A.; Blouhos, P.; Yialouris, C.; Frosyniotis, D.; Anthopoulos, I.; Mangana, O.; et al. Engineering of the membrane of fibroblast cells with virus-specific antibodies: A novel biosensor tool for virus detection. Biosens. Bioelectron. 2008, 24, 1033–1036. [Google Scholar] [CrossRef]
  17. Kintzios, S.; Pistola, E.; Konstas, J.; Bem, F.; Matakiadis, T.; Alexandropoulos, N.; Biselis, I.; Levin, R. The application of the bioelectric recognition assay for the detection of human and plant viruses: Definition of operational parameters. Biosens. Bioelectron. 2001, 16, 467–480. [Google Scholar] [CrossRef]
  18. Kintzios, S.; Pistola, E.; Panagiotopoulos, P.; Bomsel, M.; Alexandropoulos, N.; Bem, F.; Ekonomou, G.; Biselis, J.; Levin, R. Bioelectric recognition assay (BERA). Biosens. Bioelectron. 2001, 16, 325–336. [Google Scholar] [CrossRef]
  19. Michos, A.; Tatsi, E.-B.; Filippatos, F.; Dellis, C.; Koukou, D.; Efthymiou, V.; Kastrinelli, E.; Mantzou, A.; Syriopoulou, V. Association of total and neutralizing SARS-CoV-2 spike -receptor binding domain antibodies with epidemiological and clinical characteristics after immunization with the 1st and 2nd doses of the BNT162b2 vaccine. Vaccine 2021, 39, 5963–5967. [Google Scholar] [CrossRef]
  20. Shin, S.; Choi, M.; Shim, J.; Park, S. Hook effect detection and detection-range-controllable one-step immunosensor for inflammation monitoring. Sens. Actuators B Chem. 2020, 304, 127408. [Google Scholar] [CrossRef]
  21. Moschopoulou, G.; Kintzios, S. Application of “membrane-engineering” to bioelectric recognition cell sensors for the ultra-sensitive detection of superoxide radical: A novel biosensor principle. Anal. Chim. Acta 2006, 573–574, 90–96. [Google Scholar] [CrossRef]
  22. Perdikaris, A.; Alexandropoulos, N.; Kintzios, S. Development of a Novel, Ultra-rapid Biosensor for the Qualitative Detection of Hepatitis B Virus-associated Antigens and Anti-HBV, Based on “Membrane-engineered” Fibroblast Cells with Virus-Specific Antibodies and Antigens. Sensors 2009, 9, 2176–2186. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  23. Ross, G.M.S.; Filippini, D.; Nielen, M.W.F.; Salentijn, G.I.J. Unraveling the Hook Effect: A Comprehensive Study of High Antigen Concentration Effects in Sandwich Lateral Flow Immunoassays. Anal. Chem. 2020, 92, 15587–15595. [Google Scholar] [CrossRef] [PubMed]
  24. Namburi, R.; Kancherla, V.; Ponnala, A. High-dose hook effect. J. Dr. NTR Univ. Health Sci. 2014, 3, 5–7. [Google Scholar] [CrossRef]
  25. Hillström, A.; Hagman, R.; Tvedten, H.; Kjelgaard-Hansen, M. Validation of a commercially available automated canine-specific immunoturbidimetric method for measuring canine C-reactive protein. Vet. Clin. Pathol. 2014, 43, 235–243. [Google Scholar] [CrossRef]
  26. Yadav, Y.K.; Fatima, U.; Dogra, S.; Kaushik, A. Beware of “hook effect” giving false negative pregnancy test on point-of-care kits. J. Postgrad. Med. 2013, 59, 153–154. [Google Scholar] [CrossRef]
  27. Cognetti, J.S.; Steiner, D.J.; Abedin, M.; Bryan, M.R.; Shanahan, C.; Tokranova, N.; Young, E.; Klose, A.M.; Zavriyev, A.; Judy, N.; et al. Disposable photonics for cost-effective clinical bioassays: Application to COVID-19 antibody testing. Lab A Chip 2021, 21, 2913–2921. [Google Scholar] [CrossRef]
  28. Basso, C.R.; Malossi, C.D.; Haisi, A.; de Albuquerque Pedrosa, V.; Barbosa, A.N.; Grotto, R.T.; Araujo Junior, J.P. Fast and reliable detection of SARS-CoV-2 antibodies based on surface plasmon resonance. Anal. Methods 2021, 13, 3297–3306. [Google Scholar] [CrossRef]
  29. Jiang, M.; Dong, T.; Han, C.; Liu, L.; Zhang, T.; Kang, Q.; Wang, P.; Zhou, F. Regenerable and high-throughput surface plasmon resonance assay for rapid screening of anti-SARS-CoV-2 antibody in serum samples. Anal. Chim. Acta 2022, 1208, 339830. [Google Scholar] [CrossRef]
  30. González-González, E.; Garcia-Ramirez, R.; Díaz-Armas, G.G.; Esparza, M.; Aguilar-Avelar, C.; Flores-Contreras, E.A.; Rodríguez-Sánchez, I.P.; Delgado-Balderas, J.R.; Soto-García, B.; Aráiz-Hernández, D.; et al. Automated ELISA On-Chip for the Detection of Anti-SARS-CoV-2 Antibodies. Sensors 2021, 21, 6785. [Google Scholar] [CrossRef]
  31. Huang, C.; Wen, T.; Shi, F.-J.; Zeng, X.-Y.; Jiao, Y.-J. Rapid Detection of IgM Antibodies against the SARS-CoV-2 Virus via Colloidal Gold Nanoparticle-Based Lateral-Flow Assay. ACS Omega 2020, 5, 12550–12556. [Google Scholar] [CrossRef] [PubMed]
  32. de Puig, H.; Timilsina, S.S.; Rainbow, J.; Jolly, P.; Najjar, D.; Durr, N.; Alter, G.; Li, J.Z.; Yu, X.G.; Walt, D.R.; et al. Simultaneous detection of SARS-CoV-2 RNA and host antibodies enabled by a multiplexed electrochemical sensor platform. medRxiv 2021. [Google Scholar] [CrossRef]
  33. Yakoh, A.; Pimpitak, U.; Rengpipat, S.; Hirankarn, N.; Chailapakul, O.; Chaiyo, S. Paper-based electrochemical biosensor for diagnosing COVID-19: Detection of SARS-CoV-2 antibodies and antigen. Biosens. Bioelectron. 2021, 176, 112912. [Google Scholar] [CrossRef] [PubMed]
  34. Lorenzen, A.L.; Dos Santos, A.M.; Dos Santos, L.P.; da Silva Pinto, L.; Conceição, F.R.; Wolfart, F. PEDOT-AuNPs-based impedimetric immunosensor for the detection of SARS-CoV-2 antibodies. Electrochim. Acta 2022, 404, 139757. [Google Scholar] [CrossRef]
  35. Russell, S.M.; Alba-Patiño, A.; Vaquer, A.; Clemente, A.; de la Rica, R. Improving the Quantification of Colorimetric Signals in Paper-Based Immunosensors with an Open-Source Reader. Sensors 2022, 22, 1880. [Google Scholar] [CrossRef]
  36. Zeng, J.; Duarte, P.A.; Ma, Y.; Savchenko, O.; Shoute, L.; Khaniani, Y.; Babiuk, S.; Zhuo, R.; Abdelrasoul, G.N.; Charlton, C.; et al. An impedimetric biosensor for COVID-19 serology test and modification of sensor performance via dielectrophoresis force. Biosen.s Bioelectron. 2022, 213, 114476. [Google Scholar] [CrossRef]
  37. Dzimianski, J.V.; Lorig-Roach, N.; O’Rourke, S.M.; Alexander, D.L.; Kimmey, J.M.; DuBois, R.M. Rapid and sensitive detection of SARS-CoV-2 antibodies by biolayer interferometry. Sci. Rep. 2020, 10, 21738. [Google Scholar] [CrossRef]
  38. Li, J.; Wang, J.L.; Zhang, W.L.; Tu, Z.; Cai, X.F.; Wang, Y.W.; Gan, C.Y.; Deng, H.J.; Cui, J.; Shu, Z.C.; et al. Protein sensors combining both on-and-off model for antibody homogeneous assay. Biosens. Bioelectron. 2022, 209, 114226. [Google Scholar] [CrossRef]
  39. Calvo-Lozano, O.; Sierra, M.; Soler, M.; Estévez, M.C.; Chiscano-Camón, L.; Ruiz-Sanmartin, A.; Ruiz-Rodriguez, J.C.; Ferrer, R.; González-López, J.J.; Esperalba, J.; et al. Label-Free Plasmonic Biosensor for Rapid, Quantitative, and Highly Sensitive COVID-19 Serology: Implementation and Clinical Validation. Anal. Chem. 2022, 94, 975–984. [Google Scholar] [CrossRef]
  40. Bochkov, Y.A.; Gern, J.E. Rhinoviruses and Their Receptors: Implications for Allergic Disease. Curr. Allergy Asthma Rep. 2016, 16, 30. [Google Scholar] [CrossRef] [Green Version]
  41. Abdullah, H.; Heaney, L.G.; Cosby, S.L.; McGarvey, L.P.A. Rhinovirus upregulates transient receptor potential channels in a human neuronal cell line: Implications for respiratory virus-induced cough reflex sensitivity. Thorax 2014, 69, 46. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  42. Bouillon, M.; Audette, M. Retinoic acid-stimulated intercellular adhesion molecule-1 expression on SK-N-SH cells: Calcium/calmodulin-dependent pathway. Cancer Res. 1994, 54, 4144–4149. [Google Scholar] [PubMed]
  43. WHO. Coronavirus Disease (COVID-19): Herd Immunity, Lockdowns and COVID-19. Available online: https://www.who.int/news-room/questions-and-answers/item/herd-immunity-lockdowns-and-covid-19 (accessed on 12 May 2022).
  44. De Giorgi, V.; West, K.A.; Henning, A.N.; Chen, L.N.; Holbrook, M.R.; Gross, R.; Liang, J.; Postnikova, E.; Trenbeath, J.; Pogue, S.; et al. Naturally Acquired SARS-CoV-2 Immunity Persists for Up to 11 Months Following Infection. J. Infect. Dis. 2021, 224, 1294–1304. [Google Scholar] [CrossRef]
  45. Harvey, R.A.; Rassen, J.A.; Kabelac, C.A.; Turenne, W.; Leonard, S.; Klesh, R.; Meyer, W.A., III; Kaufman, H.W.; Anderson, S.; Cohen, O.; et al. Association of SARS-CoV-2 Seropositive Antibody Test With Risk of Future Infection. JAMA Intern. Med. 2021, 181, 672–679. [Google Scholar] [CrossRef] [PubMed]
  46. Kang, C.K.; Kim, M.; Lee, S.; Kim, G.; Choe, P.G.; Park, W.B.; Kim, N.J.; Lee, C.-H.; Kim, I.S.; Jung, K.; et al. Longitudinal Analysis of Human Memory T-Cell Response According to the Severity of Illness up to 8 Months After Severe Acute Respiratory Syndrome Coronavirus 2 Infection. J. Infect. Dis. 2021, 224, 39–48. [Google Scholar] [CrossRef] [PubMed]
  47. McClements, J.; Bar, L.; Singla, P.; Canfarotta, F.; Thomson, A.; Czulak, J.; Johnson, R.E.; Crapnell, R.D.; Banks, C.E.; Payne, B.; et al. Molecularly Imprinted Polymer Nanoparticles Enable Rapid, Reliable, and Robust Point-of-Care Thermal Detection of SARS-CoV-2. ACS Sens. 2022, 7, 1122–1131. [Google Scholar] [CrossRef] [PubMed]
Chemosensors 10 00254 g001 550
Figure 1. Membrane-engineered cells preparation and biosensor setup.
Figure 1. Membrane-engineered cells preparation and biosensor setup.
Chemosensors 10 00254 g001
Chemosensors 10 00254 g002 550
Figure 2. Optimization of biosensor’s responses against 6 different spiked concentrations of the SARS-CoV-2 S1 antibody (0.005, 0.01, 0.05, 0.1, 0.5, 1 μg/mL) in human blood serum. Responses of membrane-engineered SK-N-SH cells with (A) Plain PBS, (B) 0.5 μg/mL S1 protein, and (C) 5 μg/mL S1 protein. Results are presented as normalized biosensor responses (n = 24). a: non-statistically significant different results. b: statistically significant different results (p < 0.001).
Figure 2. Optimization of biosensor’s responses against 6 different spiked concentrations of the SARS-CoV-2 S1 antibody (0.005, 0.01, 0.05, 0.1, 0.5, 1 μg/mL) in human blood serum. Responses of membrane-engineered SK-N-SH cells with (A) Plain PBS, (B) 0.5 μg/mL S1 protein, and (C) 5 μg/mL S1 protein. Results are presented as normalized biosensor responses (n = 24). a: non-statistically significant different results. b: statistically significant different results (p < 0.001).
Chemosensors 10 00254 g002
Chemosensors 10 00254 g003 550
Figure 3. Normalized biosensor responses against human blood serum samples derived from three vaccinated donors (V1, V2, V3) and four COVID-19 patients. (P1, P2, P3, P4) As a control an average of ten (10) healthy individuals that were not previously vaccinated was used. Membrane-Engineered SK-N-SH cells with 5 μg/mL S1 protein were used as the biorecognition element (n = 24). b: statistically significant different results (p < 0.001).
Figure 3. Normalized biosensor responses against human blood serum samples derived from three vaccinated donors (V1, V2, V3) and four COVID-19 patients. (P1, P2, P3, P4) As a control an average of ten (10) healthy individuals that were not previously vaccinated was used. Membrane-Engineered SK-N-SH cells with 5 μg/mL S1 protein were used as the biorecognition element (n = 24). b: statistically significant different results (p < 0.001).
Chemosensors 10 00254 g003
Chemosensors 10 00254 g004 550
Figure 4. Normalized biosensor responses against 6 different concentrations of the rhinovirus VP3 antibody (0.005, 0.01, 0.05, 0.1, 0.5, 1 μg/mL) in human blood serum. Responses of membrane-engineered SK-N-SH cells with (A) 0.5 μg/mL S1 protein. (B) 5 μg/mL S1 protein. Results are presented as normalized biosensor responses (n = 24). b: statistically significant different results (p < 0.001).
Figure 4. Normalized biosensor responses against 6 different concentrations of the rhinovirus VP3 antibody (0.005, 0.01, 0.05, 0.1, 0.5, 1 μg/mL) in human blood serum. Responses of membrane-engineered SK-N-SH cells with (A) 0.5 μg/mL S1 protein. (B) 5 μg/mL S1 protein. Results are presented as normalized biosensor responses (n = 24). b: statistically significant different results (p < 0.001).
Chemosensors 10 00254 g004
Table
Table 1. Patients’ personal and medical history.
Table 1. Patients’ personal and medical history.
PatientSexAge (Years)Date of PCR PositivityDate of Venous Blood SampleUnderlying Diseases
1Female9123 April 202011 May 2020Type 2 diabetes mellitus
Hypertension
Atrial fibrillation
Hemodialysis
2Male80 19 April 202011 May 2020Hypertension
Atrial fibrillation
Hemodialysis
Chronic obstructive pulmonary disease (COPD)
3Female5023 April 202011 May 2020Hemodialysis
4Male8623 April 202011 May 2020Hypertension
Hemodialysis
Table
Table 2. Information regarding healthy donors (1–10) and vaccinated donors (V1–3,).
Table 2. Information regarding healthy donors (1–10) and vaccinated donors (V1–3,).
DonorAge (Years)SexDate of Venous Blood Sample
169Male1 October 2020
258Male2 October 2020
346Female1 October 2020
445Male1 October 2020
540Female1 October 2020
638Female2 October 2020
736Female2 October 2020
822Female1 October 2020
922Male1 October 2020
1020Male2 October 2020
V148FemaleFebruary 2021
V235FemaleFebruary 2021
V324MaleFebruary 2021
Table
Table 3. Summary of sensors for SARS-CoV-2 antibodies detection. (Y indicates that clinical validation has been performed. N indicates that the system has not been tested against clinical samples).
Table 3. Summary of sensors for SARS-CoV-2 antibodies detection. (Y indicates that clinical validation has been performed. N indicates that the system has not been tested against clinical samples).
SensorResponse TimeLimit of DetectionClinical ValidationReference
Multiplexed electrochemical (EC) sensor platform60 minNo quantitative assayY[32]
Biolayer interferometry immunosorbent assay (BLI-ISA)20 min0.037 µg/mLY[37]
Paper-based electrochemical biosensor30 min0.14 ng/mLY[33]
Surface plasmon resonance10 minNo quantitative assayY[28]
Automated ELISA On-Chip60 minNo quantitative assayY[30]
PEDOT-AuNPs-based impedimetric immunosensor30 minNo quantitative assayY[34]
High-throughput surface plasmon resonance assay5–6 min0.057 μg/mLY[29]
Disposable photonics1–5 minNo quantitative assayY[27]
Protein sensors-based on allosteric enzymes45 minNo quantitative assayY[38]
Label-Free plasmonic biosensor15 min12.75 ng/mLY[39]
Paper-Based immunosensors Several minutesNo quantitative assayY[35]
Impedimetric biosensor30 min200 ng/mLN[36]
Colloidal gold nanoparticle-based lateral-flow (AuNP-LF)15 minNo quantitative assayY[31]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Mavrikou, S.; Papaioannou, G.M.; Tsekouras, V.; Hatziagapiou, K.; Tatsi, E.B.; Filippatos, F.; Kanaka-Gantenbein, C.; Michos, A.; Kintzios, S. Ultra-Fast and Sensitive Screening for Antibodies against the SARS-CoV-2 S1 Spike Antigen with a Portable Bioelectric Biosensor. Chemosensors 2022, 10, 254. https://doi.org/10.3390/chemosensors10070254

AMA Style

Mavrikou S, Papaioannou GM, Tsekouras V, Hatziagapiou K, Tatsi EB, Filippatos F, Kanaka-Gantenbein C, Michos A, Kintzios S. Ultra-Fast and Sensitive Screening for Antibodies against the SARS-CoV-2 S1 Spike Antigen with a Portable Bioelectric Biosensor. Chemosensors. 2022; 10(7):254. https://doi.org/10.3390/chemosensors10070254

Chicago/Turabian Style

Mavrikou, Sofia, George Marios Papaioannou, Vasileios Tsekouras, Kyriaki Hatziagapiou, Elizabeth Barbara Tatsi, Filippos Filippatos, Christina Kanaka-Gantenbein, Athanasios Michos, and Spyridon Kintzios. 2022. "Ultra-Fast and Sensitive Screening for Antibodies against the SARS-CoV-2 S1 Spike Antigen with a Portable Bioelectric Biosensor" Chemosensors 10, no. 7: 254. https://doi.org/10.3390/chemosensors10070254

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop