ALL Metrics
-
Views
-
Downloads
Get PDF
Get XML
Cite
Export
Track
Research Article

Annotation and curation of human genomic variations: an ELIXIR Implementation Study

[version 1; peer review: 2 approved with reservations]
PUBLISHED 08 Oct 2020
Author details Author details
OPEN PEER REVIEW
REVIEWER STATUS

This article is included in the ELIXIR gateway.

This article is included in the EMBL-EBI collection.

Abstract

Background: ELIXIR is an intergovernmental organization, primarily based around European countries, established to host life science resources, including databases, software tools, training material and cloud storage for the scientific community under a single infrastructure.
Methods: In 2018, ELIXIR commissioned an international survey on the usage of databases and tools for annotating and curating human genomic variants with the aim of improving ELIXIR resources. The 27-question survey was made available on-line between September and December 2018 to rank the importance and explore the usage and limitations of a wide range of databases and tools for annotating and curating human genomic variants, including resources specific for next generation sequencing, research into mitochondria and protein structure.
Results: Eighteen countries participated in the survey and a total of 92 questionnaires were collected and analysed. Most respondents (89%, n=82) were from academia or a research environment. 51% (n=47) of respondents gave answers on behalf of a small research group (<10 people), 33% (n=30) in relation to individual work and 16% (n=15) on behalf of a large group (>10 people). The survey showed that the scientific community considers several resources supported by ELIXIR crucial or very important. Moreover, it showed that the work done by ELIXIR is greatly valued. In particular, most respondents acknowledged the importance of key features and benefits promoted by ELIXIR, such as the verified scientific quality and maintenance of ELIXIR-approved resources.
Conclusions ELIXIR is a “one-stop-shop” that helps researchers identify the most suitable, robust and well-maintained bioinformatics resources for delivering their research tasks.

Keywords

ELIXIR, survey, database, bioinformatics tools

Introduction

ELIXIR is an intergovernmental organization, primarily centred in Europe, established to host resources, such as databases, software tools, training material and cloud storage for the scientific community, under a single infrastructure. ELIXIR started in 2013 and now includes 22 state members and over 220 research Institutions. By providing a one-stop-shop for the scientific community, it aims to help researchers identify the most suitable bioinformatics resources (and the appropriate training material and workshops) to deliver their research task. Moreover, ELIXIR recognizes and facilitates sharing of data and exchange of expertise between its members, with the goal of agreeing on best practice.

In 2018, ELIXIR commissioned an international survey on the usage of databases and tools for annotating and curating human genomic variants with the aim of improving ELIXIR resources. Here we present the result of the survey.

Methods

A 27-question survey (Extended data) was designed and agreed upon by several ELIXIR members from various European countries. The survey was constructed around six main themes (detailed in “Questionnaire structure”) aimed at exploring the usage of ELIXIR resources and tools. Six ELIXIR nodes/partners informed the construction of the survey. Two researchers/research centres were identified by each of the six ELIXIR nodes to participate in a pilot test. The survey was modified based on the feedback received. The final questionnaire was made available on-line using the Webropol survey website between September and December 2018. Responses were uploaded to a Finland-based university server. Members of the Finland ELIXIR node accessed the responses and stored them in a .csv file for analysis. Information about participation was displayed on the survey and completion of the survey was taken as confirmation of participation.

Considering the absence of identifying information in data published here, and the non-sensitive nature of the survey, no ethical approval was sought for this study. No information presented here can be used to identify survey participants.

Participants

For each country, an ELIXIR representative was asked to recruit prospective survey respondents from academia, clinical diagnostics, clinical research, industry and the Government within their country. No additional eligibility criteria were set. Participation to the survey was advertised using several means, such as the mailing lists of universities, research centers, private companies and research institutes within each European country. Prospective participants were asked to fill the anonymized on-line questionnaire. A three-month deadline was given for completing the on-line questionnaire. Reminders to complete the survey were published in the regular newsletters at the recruited institutions. This approach did not allow calculation of the response rate.

Data collected

The survey collected two types of data:

- quantitative data: this included category ranking metrics, as well as general frequency of use of the tools and databases surveyed. Whenever possible, a list of possible choices was provided for selection, to allow proper survey analysis and interpretation.

- qualitative data: this included participant comments.

Questionnaire structure

The survey was divided in six sections (the full questionnaire with the list of tools, databases and resources that were surveyed is presented in Extended Data):

Section 1 - Background information

Section 2 - Resources for annotating and curating human genomic variants (9 questions)

Section 3 - Next generation sequencing (3 questions), (If no, skip this section)

Section 4 - Mitochondria (4 questions), (If no, skip this section)

Section 5 - Proteins (5 questions), (If no, skip this section)

Section 6 - ELIXIR (3 questions)

Sections 1, 2 and 6 were open to all participants, whereas sections 3 to 5 were only open to participants who worked or had an interest in working in these specific fields. Accordingly, respondents were asked to skip the sections that were not pertinent to their work or that of their group.

The complete list of tools and databases surveyed is presented in Extended data.

Data analysis

Data analysis was performed using RStudio (version 0.98.1062) and R (R version 3.4.4)1. Quantitative data are presented as absolute and relative frequency. Qualitative data were analysed per theme.

Results

A total of 92 questionnaires were available for analysis. Data were collected from 18 European countries and the Unites States (Extended data: Figure S1). Finland contributed to the majority of answers (30%, n=28), followed by the United Kingdom (11%, n=10). The large majority of respondents (89%, n=82) identified themselves as belonging to academia or a research environment (Extended data: Table S1). The large majority (51%, n=47) of respondents gave answers to the questionnaire in relation to their work and that of a small group of (<10) people. This was followed by answers given in relation to individual work (33%, n=30) and the work of a large group of (>10) people (16%, n=15). As the survey was sent to colleagues by the organisers, the responders may well not be representative of the wider community. Additionally, most of the responders were from European countries and the responses could be substantially different elsewhere, such as from the USA.

Key resources

Section 2 of the survey aimed at identifying key data resources and tools used by the scientific community primarily in ELIXIR member states. Respondents were asked to rank 52 resources (listed in Extended data: Table S2). A total of 2055 responses were collected from 88% (82) responders. In total, 22 resources were considered critical/very important by more than 50% of respondents who knew the resources and considered them relevant to their work (Figure 1). Three ELIXIR resources, Ensembl2, the Ensembl Variant Effect Predictor (VEP)3 and UniProt4, were within the top ten. Several additional resources were listed by respondents and are presented in Extended data: Tables S3 and S4.

99744b7d-2945-4e40-802a-509f749d37b1_figure1.gif

Figure 1. List of 22 resources considered critical/very important by >50% of respondents (Q4).

Guidelines

A total of 78 responses were collected from 50 (54%) respondents on the standards and guidelines used for the interpretation of sequence variants. 66% (n=33) of respondents followed the American College of Medical Genetics (ACMG) standards and guidelines for the interpretation of sequence variants5, 22% (n=11) national guidelines, and 8% (n=4/50) in house guidelines. 4% (n=2) answered “other guidelines” and listed the European Leukemia Net (ELN) guidelines6 and the “Sequence Ontology base classification for variants”7. With regards to the human genome reference assembly version, approximately one third (38%, n=31/81) of respondents used both GRCh37 and GRCh38, 44% (n=36/81) GRCh37 only, 15% (n=12/81) GRCh38 alone, and 2% (n=2/81) GRCh37, GRCh38 and older versions (Figure 2).

99744b7d-2945-4e40-802a-509f749d37b1_figure2.gif

Figure 2. Usage of the Human Reference Genome build (Q7). Data are presented as percentages.

Data/data formats

Four types of data/data formats used in human genome variation analysis (VCF/BCF, BAM, FASTQ, FASTA) were considered critical/very important by the majority of respondents (Extended data: Table S5). Additional data formats that were listed as critical/very important were: BED (cited 4 times), gVCF (cited twice), PLINK8, IMPUTE9 and Hail MatrixTable. Genome-wide association studies (GWAS) and rare disease studies were listed as the most important topics and analytical operations in human genome variation analysis performed by respondents to this survey (Extended data: Table S6).

Important resource features

One of the main goals of this survey was to identify user requirements in the use of resources for human genome variation annotation and curation. A total of 777 responses on the importance of basic features of databases and tools were analysed. The majority of respondents (>83%) considered critical/very important a resource’s free/low cost, open license to academia, the ability to assess the quality of its data, good user documentation and its availability in English. An easy-to-use web browser and privacy policy were deemed critical/very important by >77% of respondents (Table 1).

Table 1. Importance of basic features in variation analysis and annotation databases and tools (Q10).

Critical
N (%)
Very important
N (%)
Useful
N (%)
Total
N
Language, in English 45 (52)30 (35)11 (13)86
Cost-free / Low cost of use 37 (43)34 (40)15 (17)86
Easy to assess the quality of each data resource 35 (40)42 (48)10 (11)87
Open licenses or public statement of open terms of
use for academia
31 (37)40 (48)13 (15)84
Good user documentation 30 (35)38 (45)17 (20)85
Easy-to-use web/browser access 25 (33)27 (36)23 (31)75
Publicly available privacy policy (security around
personal data and cookies are described)
17 (24)32 (44)23 (32)72
Availability for commercial use 5 (19)9 (33)13 (48)27
Widely used 14 (18)34 (44)30 (38)78
Training and customer service available for use 10 (14)24 (33)39 (53)73
Language, in local languages (non-English) 0 (0)1 (5)21 (95)22
TOTAL251311215777

A total of 1173 responses were analysed on the importance of technical features in variation analysis, annotation tools and databases. Good curation of the database was the top scoring technical feature, ranked critical/very important by 92% of respondents. The following five technical features were considered critical/very important by >75% of respondents: i) fit for purpose data used by the resource, ii) the scientific coverage and comprehensiveness of the resource; iii) scalability to high-throughput analysis; iv) availability of datasets for download; and v) ability to analyse large datasets/queries (Table 2).

Table 2. Importance of technical features in variation analysis and annotation tools and databases.

Critical
N (%)
Very important
N (%)
Useful
N (%)
Total
N
Good curation of database38 (46)38 (46)7 (8)83
Ability to analyse large data sets/queries33 (39)31 (36)21 (25)85
Data fit for purpose (e.g. complete set, clinically relevant,
up-to-date data
32 (38)40 (48)12 (14)84
Scalability to high-throughput analysis28 (35)35 (44)16 (20)79
Datasets available for download26 (33)35 (44)18 (23)79
Scientific coverage and comprehensiveness of the resource25 (31)43 (53)13 (16)81
Programmatic access (e.g. linux/unix)21 (30)25 (36)24 (34)70
Run locally20 (28)22 (31)29 (41)71
Data in large number of global populations available20 (24)35 (42)29 (35)84
Database links to variants-related scientific literature18 (21)35 (41)32 (38)85
Population-specific data available14 (18)28 (36)35 (45)77
Response time of key web pages and search functions13 (18)29 (39)32 (43)74
Open interfaces (e.g. REST)10 (16)14 (23)38 (61)62
Good data visualization options12 (15)27 (33)42 (52)81
Multiple data sharing formats (e.g. plain text, FASTA, XML,
RDF, Dublin Core, tsv, JSON)
9 (12)28 (36)41 (53)78
TOTAL3194653891173

Next Generation Sequencing

The next section of the survey focused on Next Generation Sequencing (NGS). 66 (72%) respondents worked with NGS and ranked 6 sequencing methods listed in Extended data: Table S7. Whole exome, panel DNA, whole genome and RNA sequencing were used by more than 60% of respondents. Methylation sequencing was the least used technique (28% of respondents) but was the technique the majority of respondents would like to use (23% of respondents). Several other techniques were listed among “Other”, such as single-cell RNA sequencing, ribosome profiling, chromatin conformation techniques (3C, 4C, HiC, capture HiC), assay for transposase-accessible chromatin using sequencing (ATAC-SEQ), targeted RNA sequencing, fusion genes on cDNA, microbiome sequencing, GRO-seq.

The next section of the survey explored the use of resources related to the study of mitochondria. Only 15 (16%) respondents worked with mitochondrial DNA (mtDNA) alone or in combination with nuclear genes involved in mitochondrial functionality. Of these, 7 used NGS alone, 2 NGS and Sanger sequencing and 6 could not specify. Additional methods listed under “Other methods” were whole genome sequencing (WGS), whole exome sequencing (WES), RNA-seq and mtDNA included in WGS or WES.

Mitochondrial resources

Among the mitochondria-dedicated databases, MITOMAP10 was ranked as critical/very important by 65% of respondents, followed by HmtDB (Human Mitochondrial Database)11 by 44% and HmtVar (Human Mitochondrial Variants Database)12 by 33%. With the exception of MITOMAP, MitoCarta13 and HmtDB, more than half of the respondents were unaware of all other databases (Extended data: Table S8).

Among the six tools for mitochondria-related research that were ranked, MToolBox14, HaploGrep215, MitoTip16 and MitImpact217 were equally reported as critical/very important by 18% of respondents (Extended data: Table S9). The following mitochondrial databases/tools not included in the survey were listed among “Other” and ranked critical/very important: MitoFates18 and an in house built annotation pipeline. mvTool (part of MSeqDR.org) and MitoMaster (part of MITOMAP) were listed among “Other tools” used for mitochondria-related research.

Protein structure

The next section of the survey explored the use of resources related to the study of protein structures. 21 out of 92 (22.8%) respondents worked with protein structure, whereas 5 (5.4%) did not but would like to. The results of the survey are presented based on 182 answers from these 26 respondents. Databases of experimentally determined protein structure and complexes, such as PBD19 and PDBe20, were considered critical/very important by 65% (n=17/26) of respondents, followed by tools that report on the structural consequences of variants (62%, n=16/26). All 7 tools/databases that were surveyed were ranked critical/very important by over 45% of respondents (Table 3). The following tools/databases were not surveyed but cited by respondents as key resources for structural modelling of variants: Pymol21, Rosetta22, KiNG23, Modeller24, PolyPhen-225, SNP3D26, SNPs&GO27, SAAPdab/SAAPpred28, HOPE29 and Yasara30.

Table 3. List of protein-related resources for variant analysis there were ranked in the survey (Q21).

Critical
N (%)
Very
important
N (%)
Useful
N (%)
Not
relevant
N (%)
Not
known
N (%)
Total
N
Critical/Very
important
N (%)
Databases of experimentally-
determined protein structure
including complexes (e.g.PDB/ PDBe)
14 (54%)3 (12%)7 (27%)02 (8%)2617 (65%)
Tools to predict protein structure
from sequence (e.g. Phyre2 /
SWISSMODEL)
6 (23%)6 (23%)10 (38%)3 (12%)1 (4%)2612 (46%)
Molecular graphics viewers (e.g.
PyMol, Chimera)
5 (19%)9 (35%)7 (27%)1 (4%)4 (15%)2614 (54%)
Tools that report the structural
consequences of variants (e.g. loss of
a salt bridge)
4 (15%)12 (46%)8 (31%)2 (8%)02616 (62%)
Databases of predicted protein
structures
3 (12%)10 (38%)10 (38%)3 (12%)02613 (50%)
Databases of predicted complexes2 (8%)9 (35%)10 (38%)4 (15%)1 (4%)2611 (42%)
Tools to predict the structure of
protein complexes (i.e. protein
docking)
2 (8%)10 (38%)8 (31%)4 (15%)2 (8%)2612 (46%)
TOTAL365960171018295

Key limiting factors to the use of tools for modelling the structural consequences of variants were the lack of expertise in the area (54%, n=13/24), difficulties in using the tools (42%, n=10/24), and difficulties in interpreting the results (38%, n=9/24) (Figure 3). Lack of high throughput capability was listed among “Other limitations”. The difficulty to translate protein dynamics into structural models was also cited.

99744b7d-2945-4e40-802a-509f749d37b1_figure3.gif

Figure 3. Limitations to the use of tools to model the structural consequences of variants were ranked in Q23.

ELIXIR

The last section of the questionnaire covered questions related to ELIXIR as a platform that coordinates the tools and databases surveyed in this study and the benefits that ELIXIR, as a European intergovernmental organization, offers to the scientific community. The majority of respondents (52%, n=42/92) were not aware that the tools and databases surveyed in Q25 are part of ELIXIR (Extended data: Figure S2). However, the long-term sustainability of ELIXIR core resources and the verified scientific quality of datasets were considered critical/very important by >76% (n=68) of participants who answered this question. The international standards for describing and saving data, the verified quality and maintenance of ELIXIR resources and facilities to find the right research tools were also considered critical/very important by the majority of respondents (>60%) (Table 4). Additional challenges that were identified by respondents regarding the annotation and curation of human genetic variants were related to the quality of data: incorrect entries, conflicting annotation, limited access to expert curation and classification of variants obtained in routine diagnostic setting and limited number genotype-phenotype annotations. An important caveat is the ascertainment bias that the survey was primarily sent to colleagues of the authors and this probably is not a representative sample of the community involved in the annotation and curation of human genomic variations.

Table 4. Importance of the benefits that ELIXIR offers to the scientific community (Q26).

Critical
N (%)
Very
important
N (%)
Useful
N (%)
Not
relevant
N (%)
Don't
know
N (%)
Total
N
Critical/ Very
important
N (%)
Long-term sustainability of
ELIXIR core resources
38 (43%)30 (34%)11(13%)4 (5%)5 (6%)8868 (77%)
Verified scientific quality of
datasets
32 (36%)36 (40%)14 (16%)2 (2%)5 (6%)8968 (76%)
Verified quality and maintenance
of ELIXIR resources
31 (36%)32 (37%)15 (17%)3 (3%)6 (7%)8763 (72%)
International standards for
describing and saving data
28 (32%)38 (44%)14 (16%)3 (3%)4 (5%)8766 (76%)
Facilities to find the right
research tools
24 (27%)37 (42%)16 (18%)7 (8%)5 (6%)8961 (69%)
Training in data analysis and
current best practices
18 (20%)30 (34%)33 (37%)3 (3%)5 (6%)8948 (54%)
Network of supercomputing
services
10 (12%)30 (35%)28 (33%)11 (13%)6 (7%)8540 (47%)
Pan-European infrastructure of
knowledge and support
9 (10%)31 (36%)29 (34%)7 (8%)10 (12%)8640 (47%)
Promotion of industry
collaboration
5 (6%)15 (17%)39 (45%)17 (20%)10 (12%)8620 (23%)
TOTAL1952791995761791474

Conclusions

This survey shows that the scientific community in ELIXIR member states considers several resources supported by ELIXIR crucial or very important Moreover, it shows that the work done by ELIXIR is greatly valued. In particular, most respondents acknowledged the importance of key features and benefits promoted by ELIXIR, such as the verified scientific quality and maintenance of ELIXIR-approved resources.

Data availability

Underlying data

Open Science Framework: ELIXIR, https://doi.org/10.17605/OSF.IO/SWX4231.

Extended data

Open Science Framework: ELIXIR, https://doi.org/10.17605/OSF.IO/SWX4231.

This project contains the following extended data:

  • - Copy of the online survey

  • - Figure S1. Number of responses from the countries that participated in the survey

  • - Figure S2. Results for Q25: “Before the survey, were you aware that the resources listed below are part of ELIXIR?

  • - Tables S1. Answers to Q2: Place of work. More than one answer was allowed per participant

  • - Table S2. List of Databases and Tools surveyed in Q4

  • - Table S3. Additional Resources listed by Respondents under “Other” in Q4 and considered Critical or Very Important

  • - Table S4. Other resources listed by Respondents in Q5

  • - Table S5. The importance of data formats in human genome variations analysis (Q8)

  • - Table S6. List of the topics and analytical operations in human genome variation analysis considered most important in the Respondents’ work (Q9)

  • - Table S7. Answers to Q14 (Do you use these sequencing methods?)

  • - Table S8. Mitochondrial databases (Q18)

  • - Table S9. Mitochondrial tools (Q18)

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Comments on this article Comments (0)

Version 1
VERSION 1 PUBLISHED 08 Oct 2020
Comment
Author details Author details
Competing interests
Grant information
Copyright
Download
 
Export To
metrics
Views Downloads
F1000Research - -
PubMed Central
Data from PMC are received and updated monthly.
- -
Citations
CITE
how to cite this article
David A, Barbié V, Attimonelli M et al. Annotation and curation of human genomic variations: an ELIXIR Implementation Study [version 1; peer review: 2 approved with reservations] F1000Research 2020, 9(ELIXIR):1207 (https://doi.org/10.12688/f1000research.24427.1)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
track
receive updates on this article
Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?
Key to Reviewer Statuses VIEW
ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions
Version 1
VERSION 1
PUBLISHED 08 Oct 2020
Views
10
Cite
Reviewer Report 18 Aug 2022
Osamu Ogasawara, Bioinformation and DDBJ Center, National Institute of Genetics, Shizuoka, Japan 
Approved with Reservations
VIEWS 10
The authors conducted a systematic survey of the usage of databases and tools for annotating and curating human genomic variants with the aim of improving ELIXIR resources. Eighteen European countries and the United States participated in the survey and a ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Ogasawara O. Reviewer Report For: Annotation and curation of human genomic variations: an ELIXIR Implementation Study [version 1; peer review: 2 approved with reservations]. F1000Research 2020, 9(ELIXIR):1207 (https://doi.org/10.5256/f1000research.26946.r145730)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
Views
14
Cite
Reviewer Report 22 Jan 2021
Lina Ma, National Genomics Data Center, Beijing Institute of Genomics, Chinese Academy of Sciences, Beijing, China;  China National Center for Bioinformation, Beijing, China 
Approved with Reservations
VIEWS 14
The authors systematically analysed the results and statistics of the questionnaire for an international survey on the usage of ELIXIR databases and tools, which are mainly devoted to annotating and curating human genomic variants. This survey would benefit both ELIXIR ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Ma L. Reviewer Report For: Annotation and curation of human genomic variations: an ELIXIR Implementation Study [version 1; peer review: 2 approved with reservations]. F1000Research 2020, 9(ELIXIR):1207 (https://doi.org/10.5256/f1000research.26946.r76824)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.

Comments on this article Comments (0)

Version 1
VERSION 1 PUBLISHED 08 Oct 2020
Comment
Alongside their report, reviewers assign a status to the article:
Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions
Sign In
If you've forgotten your password, please enter your email address below and we'll send you instructions on how to reset your password.

The email address should be the one you originally registered with F1000.

Email address not valid, please try again

You registered with F1000 via Google, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Google account password, please click here.

You registered with F1000 via Facebook, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Facebook account password, please click here.

Code not correct, please try again
Email us for further assistance.
Server error, please try again.