The BioMart community portal: an innovative alternative to large, centralized data repositories

1The methods of data generation and processing that are utilized in biomedical sciences have radically changed in recent years. With the advancement of new high-throughput technologies, data have grown in terms of quantity as well as complexity. However, the significance of the information that is hidden in the newly generated experimental data can only be deciphered by linking it to other types of biological data that have been accumulated previously. As a result there are already numerous bioinformatics resources and new ones are constantly being created. Typically, each resource comes with its own query interface. This poses a problem for the scientists who want to utilize such resources in their research. Even the simplest task such as compiling results from a few existing resources is challenging due to the lack of a complete, up to date catalogue of already existing resources and the necessity of constantly learning how to navigate new query interfaces. A different challenge is faced by collaborating groups of scientists who independently generate or maintain their own data. Such collaborations are seriously hampered by the lack of a simple data management solution that would make it possible to connect their disparate, geographically distributed data sources and present them in a uniform way to other scientists. The BioMart project has been set up to address these challenges.

2BioMart is an open source data management system, which is based on a data federation model. Under this model, each data source is managed, updated and released independently by their host organization while the BioMart software provides a unified view of these sources that are distributed worldwide. The data sources are presented to the user through a unified set of graphical and programmatic interfaces so that they appear to be a single integrated database. To navigate this database and compile a query the user does not have to learn the underlying structure of each data source but instead use a set of simple abstractions: datasets, filters and attributes. Once a user's input is provided, the software distributes parts of the query to individual data sources, collects the data and presents the user with the unified result set.

3The BioMart software is data agnostic and its applications are not limited to biological data. It is cross-platform and supports many popular relational database managements systems, including MySQL, Oracle, PostgreSQL. It also supports many third party packages such as Taverna, Galaxy, Cytoscape and biomaRt, which part of the Bioconductor library.

4The BioMart project currently maintains two independent code bases: one written in Java and one written in Perl. For more information about the architecture and capabilities of each of the packages please refer to previous publications. The latest version of the Java based BioMart software has been significantly enhanced with new additions to the existing collection of graphical user interfaces (GUIs). It has also been re-engineered to provide better support and extensibility for data analysis and visualization tools. The first of the BioMart tools based on this new framework has already been implemented and is accessible from the BioMart Community Portal.

5The BioMart project adheres to the open source philosophy that promotes collaboration and code reuse. Two good examples of how this philosophy benefits the scientific community are provided by two independent research groups. The INRA group based in Toulouse, France has recently released a software package called RNAbrowse (RNA-Seq De Novo Assembly Results Browser). The Pfizer group based in La Jolla, USA has just announced the release of OASIS: A Web-based Platform for Exploratory Analysis of Cancer Genome and Transcriptome data (www.oasis-genomics.org). Both of these software packages are based on the BioMart software.

6The BioMart community consists of a wide spectrum of different research groups that use the BioMart technology to provide access to their databases. It currently comprises 30 scientific organizations supporting 38 database projects that contain over 800 different biological datasets spanning genomics, proteomics, model organisms, cancer data, ontology information and more. The BioMart community is constantly growing and since the last publication, 11 new database projects have become available. As new BioMart databases become available locally they also become gradually integrated into the BioMart Community Portal. The main function of the portal is to provide a convenient single point of access to all available data that is distributed worldwide (Figure 1). All BioMart databases that are included in the portal are independently administered and funded. Table 1 provides a detailed list of all BioMart community resources as of March 2015.

7The current version of the BioMart Community Portal operates two different instances of the web server: one implemented in Perl and the other in Java. Both servers support complex database searches and although they use different types of GUIs, they share the same navigation and query compilation logic based on selection of datasets, filters and attributes. The Java version of the portal also includes a section for specialized tools, which consists of the following: Sequence retrieval, ID Converter and Enrichment Analysis. Sequence retrieval allows easy querying of sequences while the ID Converter tool allows users to enter or upload a list of identifiers in any format (currently supported by Ensembl), and retrieve the same list converted to any other supported format. The enrichment tool supports enrichment analysis of genes in all species included in the current Ensembl release. For each of those species a broad range of gene identifiers is available. Furthermore, the tool supports cross species analysis using Ensembl homology data. For instance, it is possible to perform a one step enrichment analysis against a human disease dataset using experimental data from any of the species for which human homology data is available. Finally, the enrichment tool facilitates analysis of BED files containing genomic features such as Copy Number Variations or Differentially Methylated Regions. The output is provided in tabular and network graphic format (Figure 2).

8The BioMart Community Portal handles queries from several interfaces such as: PERL APIJava APIWeb interfacesURL based accessRESTful web serviceSPARQL

9PERL API

10Java API

11Web interfaces

12URL based access

13RESTful web service

14SPARQL

15For more detailed description of all the interfaces please refer to earlier publications. In the section below we provide a description and compare the REST-based web service, which is implemented in Perl and its counterpart, which is implemented in Java. It is worth noting that the web service maintains the same query interface both in Perl and Java implementations. For example, the web service query (Figure 3A) can be run against java-based server as follows: curl –data-urlencode query@query.xml http://central.biomart.org/martservice/resultsor its Perl-based counter-part as belowcurl –data-urlencode query@query.xml http://www.biomart.org/biomart/martservice

16curl –data-urlencode query@query.xml http://central.biomart.org/martservice/results

17or its Perl-based counter-part as below

18curl –data-urlencode query@query.xml http://www.biomart.org/biomart/martservice

19By default, query sets the attribute processor to ‘TSV’ requesting tab-delimited results (Figure 3B). Alternatively, by setting processor to ‘JSON’, would return JSON formatted results (Figure 3C), which are readily consumable by third-party web-based clients saving overhead of parsing and format translations. Please note that JSON format is only available in the java version.

20A simple way to compile a web service query for later programmatic use is to use one of the web GUIs and generate the query XML using REST/SOAP button. After following the steps outlined by the GUI and clicking the ‘results’ button, the user needs to click the REST/SOAP button, save the query and run it as described above. Alternatively a user can take advantage of the programmatic access to all the metadata defining marts, datasets, filters and attributes. The access to the metadata served by the Java and Perl BioMart servers is provided using the following webservice requests:

21Java (central.biomart.org) registry information:http://central.biomart.org/martservice/portalavailable marts:http://central.biomart.org/martservice/martsdatasets available for a config:http://central.biomart.org/martservice/datasets?config=snp_configattributes available for a dataset:http://central.biomart.org/martservice/attributes?datasets=btaurus_snp&config=snp_configfilters available for a dataset:http://central.biomart.org/martservice/filters?datasets=btaurus_snp&config=snp_config

22registry information:

23http://central.biomart.org/martservice/portal

24available marts:

25http://central.biomart.org/martservice/marts

26datasets available for a config:

27http://central.biomart.org/martservice/datasets?config=snp_config

28attributes available for a dataset:

29http://central.biomart.org/martservice/attributes?datasets=btaurus_snp&config=snp_config

30filters available for a dataset:

31http://central.biomart.org/martservice/filters?datasets=btaurus_snp&config=snp_config

32Perl (www.biomart.org) registry information:http://www.biomart.org/biomart/martservice?type=registrydatasets available for a mart:http://www.biomart.org/biomart/martservice?type=datasets&mart=ensemblattributes available for a dataset:http://www.biomart.org/biomart/martservice?type=attributes&dataset=oanatinus_gene_ensemblfilters available for a dataset:http://www.biomart.org/biomart/martservice?type=filters&dataset=oanatinus_gene_ensemblconfiguration for a dataset:http://www.biomart.org/biomart/martservice?type=configuration&dataset=oanatinus_gene_ensembl

33registry information:

34http://www.biomart.org/biomart/martservice?type=registry

35datasets available for a mart:

36http://www.biomart.org/biomart/martservice?type=datasets&mart=ensembl

37attributes available for a dataset:

38http://www.biomart.org/biomart/martservice?type=attributes&dataset=oanatinus_gene_ensembl

39filters available for a dataset:

40http://www.biomart.org/biomart/martservice?type=filters&dataset=oanatinus_gene_ensembl

41configuration for a dataset:

42http://www.biomart.org/biomart/martservice?type=configuration&dataset=oanatinus_gene_ensembl

43Please note that the granularity between mart and dataset has been improved in the Java version through the introduction of multiple dataset configs. This facilitates the end-users to browse various views of the same dataset, which are presented through the portal either using a different GUI or subsets of data.

44Given the coverage of the current BioMart datatsets, many relevant biological questions can be answered. For example, a researcher who has detected potentially pathogenic variants in FGFR2 (ENSG00000066468) from exome sequencing patients may be interested if the same variants have been previously described and if they were associated with the same or similar diseases. To answer this, integrated data from Ensembl can be queried as shown in Table 2 to display all known variants annotated within FGFR2 that are predicted as pathogenic by SIFT and Polyphen. The genomic position outputs can be compared to the researcher's variants and the phenotype data used to assess candidacy for their cases. For example, the first batch of results shows a C->G variant at position 121520160 on chromosome 10 that is associated with Apert syndrome (OMIM:176943).

45Another common use case that BioMart is used for is to analyse a list of genes to establish whether they are associated with particular protein functions, pathways or diseases more often than would be expected by chance (enrichment analysis). For example, a researcher may have discovered that AURKA, AURKB, AURKC, PLK1, CDK1 and CDK4 are differentially expressed in their experiment and used BioMart's enrichment tool with its default settings to analyse these genes. The results show that these genes are enriched for involvement in the cell cycle, kinase activity and mitotic nuclear division amongst others. Many other real usage examples are documented in our previous paper and the BioMart special issue in Database: the journal of biological databases and biocuration (www.oxfordjournals.org/our_journals/databa/biomart_virtual_issue.html).

46Since its conception as a data-mining interface for the Human Genome Project BioMart has rapidly grown to become an international collaboration involving a large number of different groups and organizations both in academia and in industry. It has been successfully applied to many different types of data including genomics, proteomics, model organisms, cancer data, etc., proving that its generic data model is widely applicable. BioMart has also provided a first successful solution for the unprecedented data management needs of the International Cancer Genome Consortium proving that the federated model scales well with the amounts of data generated by Next Generation Sequencing.

47There are a number of important factors that contributed to the BioMart's success and its adoption by many different types of projects around the world as their data management platform. BioMart's ability to quickly deploy a website hosting any type of data, user-friendly GUI, several programmatic interfaces and support for third party tools has proved to be an attractive solution for data managers who were in need of a rapid and reliable solution for their user community. BioMart has also proven to be a platform of choice for many smaller organizations that lack the necessary resources to embark on the development of their own data management solution. As a result, more and more database projects have become accessible through the BioMart interface. The arrival of these new resources coupled with the data federation technology provided by the BioMart software has galvanized the creation of the BioMart Community Portal. The federated model has proven to be very cost-effective since all development and maintenance of individual databases is left to the individual data providers. It also has proven to be very scalable as the internet and database traffic is handled by the local BioMart servers. As a result the BioMart Community Portal service has grown impressively not only in terms of available data but also the level of service. The BioMart community portal now averages over million requests per our services per day. Building on this level of service and the wealth of information that has become accessible through the BioMart interface, the BioMart Community Portal has effectively introduced a new, more scalable and much more cost-effective alternative to the large data stores maintained by specialized organizations.