the most useful R function of the week: unnest from tidyr

There are many great functions in CRAN and BioConductor, and certainly saying that unnest from the tidyr package is the best is a big exaggeration. However this function solved a big problem in data formatting that made me waste a lot of time in the past, that I was surprised no one had implemented a function for it yet.

Imagine we have a dataframe like the following:

> mygenes
Entrez  symbols
7841    MOGS,CDG2B,CWH41,DER7,GCS1
4248    MGAT3,GNT-III,GNT3

The first column contains the Entrez of each gene. This columns is fine, as it contains only one value per row, and it is easy to query or join with other dataframes. The second column, however, contains a comma-separated list of gene names, all associated to the same Entrez IDs. This column is a mess to deal with, because we need to use grepl to query it, and we can’t join it with other dataframes as long as it is in this form.

The unnest function from tidyr allows to convert this data frame in a “tidier” form, containing one row for each combination gene symbol and alias:

> library(tidyr)
> library(dplyr)
> mygenes %>% 
    mutate(symbols=strsplit(as.character(symbols), ",")) %>% 

   Entrez symbols
1    7841    MOGS
2    7841   CDG2B
3    7841   CWH41
4    7841    DER7
5    7841    GCS1
6    4248   MGAT3
7    4248 GNT-III
8    4248    GNT3
9    5728    PTEN
10   5728     BZS
11   5728    CWS1
12   5728     DEC
13   5728    GLM2
14   5728    MHAM
15   5728  MMAC11
16   5728    TEP1

This code makes use of the %>% and some functions from the dplyr package, but it is still R!

Having the dataframe in this long form makes it a lot easier to deal with it. For example, let’s imagine that somebody asks us to get the Entrez IDs for the list of gene symbols DER7 and DEC. We would just have to do a simple subset on the dataframe:

> unn %>% 
    mutate(symbols=strsplit(as.character(symbols), ",")) %>% 
    unnest(symbols) %>% 
    subset(symbols %in% c("DER7", "DEC"))

   Entrez symbols
4    7841    DER7
12   5728     DEC

This is just a silly example, which may have been solved with some application of apply and grepl, but in the real world there are a lot of more complex applications for it. For example, here is some code I used to split Blat output into one line per exon (or blat alignment block):

Continue reading

Posted in news | 2 Comments

a formula interface for GeneOntology analysis

clusterProfiler is a nice R library for doing GO and KEGG enrichment analysis. It has a simple interface and it can produce some clear plots using the ggplot engine. Today I contributed a formula interface to clusterProfiler, making it easier to do enrichment of multiple groups of genes.

Let’s imagine you have a dataframe in which one column contains a list of Entrez Ids, while the other columns encode some grouping variables:

> mygenes 
Entrez    group othergroup
1         A     good
100       A     bad
1000      A     good
100101467 B     bad
100127206 B     good
100128071 B     good

The new formula interface allows to do a GO analysis on each of the groups. For example, we can group them  by the column “group”, and compare the classification of the two groups:

> install_github(c("GuangchuangYu/DOSE", "Guangchuang/clusterProfiler"))
> library(clusterProfiler)
> mygenes <- data.frame(Entrez=c('1', '100', '1000', '100101467',
                            '100127206', '100128071'),
                  group = c('A', 'A', 'A', 'B', 'B', 'B'),
                  othergroup = c('good', 'bad', 'good', 'bad', 'good', 'good'))
> GO.enrichment <- compareCluster(Entrez~group, data=mygenes, fun='groupGO')
> print(summary(GO.enrichment))
  Cluster         ID          Description Count GeneRatio     geneID
1       A GO:0016020             membrane     2       2/5   100/1000
2       A GO:0005576 extracellular region     3       3/5 1/100/1000
3       A GO:0005581      collagen trimer     0       0/5           
4       B GO:0016020             membrane     1       1/3  100127206
5       B GO:0005576 extracellular region     0       0/3           
6       B GO:0005581      collagen trimer     0       0/3

> plot(GO.enrichment, plotAll=T)  clusterProfiler example

In this case group A is enriched in membrane and extracellular region, while group B is only enriched in membrane genes. The groupGO function used here doesn’t provide p-values – we should have used enrichGO instead. I guess that the 3 Entrez ids in group A correspond to 5 genes in Gene Ontology, so that’s why the plots shows a total of 5 in group A. See clusterProfiler’s documentation for better examples.

The formula interface allows also multiple grouping. For example:

GO.enrichment <- compareCluster(Entrez~group+othergroup, data=mygenes, fun='groupGO')
  Cluster         ID          Description Count GeneRatio    geneID
1   A.bad GO:0016020             membrane     1       1/2       100
2   A.bad GO:0005576 extracellular region     1       1/2       100
3  A.good GO:0016020             membrane     1       1/3      1000
4  A.good GO:0005576 extracellular region     2       2/3    1/1000
5   B.bad GO:0016020             membrane     0       0/2          
6   B.bad GO:0005576 extracellular region     0       0/2          
7  B.good GO:0016020             membrane     1       1/2 100127206
8  B.good GO:0005576 extracellular region     0       0/2          

Of course this example is not much interesting, since it is only 6 randomly chosen genes. However with bigger datasets the formula interface can be much more powerful.

Now that I think of it, it would be better if compareCluster would return a dataframe with multiple columns, instead of merging them into a single column called Cluster. This would be make it possible to plot the results using facets or something more fancy. It would be something like this:

  group othergroup         ID          Description Count GeneRatio    geneID
1     A        bad GO:0016020             membrane     1       1/2       100
2     A        bad GO:0005576 extracellular region     1       1/2       100
3     A       good GO:0016020             membrane     1       1/3      1000
4     A       good GO:0005576 extracellular region     2       2/3    1/1000
5     B        bad GO:0016020             membrane     0       0/2          
6     B        bad GO:0005576 extracellular region     0       0/2          
7     B       good GO:0016020             membrane     1       1/2 100127206
8     B       good GO:0005576 extracellular region     0       0/2

However this would probably require introduce some retro-incompatibility in the library, and it is not a big deal as the Cluster column can be easily split using the separate function from tidyr.

Posted in news | 1 Comment

Tidy data and VCF

I think that 2014 has been the year in which my R programming style has changed the most. This is because a lot of innovative and nice libraries have been released, like dplyr, magrittr and tidyr. I started in January as a ddply enthusiast, and now instead my code is full %>% instructions and dplyr functions.

If you missed these libraries, a good starting point is the article “Principles of Tidy Dataset“, in which the author Hadley Wickham suggests some best practices for organising a dataset in a “tidy” format before doing any analysis. These practices will be already familiar to you if you have experience with the reshape/reshape2 packages, and if you used ggplot2 in the past. However, it is good to read a good summary as in the article.

Inspired by this article, I wrote a post on Biostar to discuss how a popular format in bioinformatics – the VCF – in a tidy format. Here is the link to the discussion.

The VCF in a tidy format would like more or less as below. On one hand, it would be a bit too redundant, and many columns would be replicated multiple times, making the file more sensible to typos introduced by the users and occupying more disk space. On the other hand, it would be easier to read, more flexible and able to accommodate other informations, like the population of each individual or more info about the genotype quality.

> vcf %>%
    gather(individual, value, -c(X.CHROM:FORMAT)) %>%
    separate(value, into=strsplit('GT:GQ:DP:HQ', ':')[[1]], ':', extra='drop') %>%
    separate('GT', into=c('allele1', 'allele2'), '[|/]') %>%
    gather(allele, genotype, -c(X.CHROM:individual, GQ:HQ)) %>%
    arrange(X.CHROM, POS, ID, individual) %>% 
    select(-INFO, -FORMAT, -FILTER) %>%  # let's omit this for better visualization
    subset(ID!='microsat1')              # let's omit this for better visualization

 X.CHROM  POS          ID     REF ALT QUAL individual GQ DP    HQ  allele genotype
 20       14370   rs6054257   G   A   29   NA00001    48  1 51,51 allele1        0
 20       14370   rs6054257   G   A   29   NA00001    48  1 51,51 allele2        0
 20       14370   rs6054257   G   A   29   NA00002    48  8 51,51 allele1        1
 20       14370   rs6054257   G   A   29   NA00002    48  8 51,51 allele2        0
 20       14370   rs6054257   G   A   29   NA00003    43  5   .,. allele1        1
 20       14370   rs6054257   G   A   29   NA00003    43  5   .,. allele2        1
 20       17330           .   T   A    3   NA00001    49  3 58,50 allele1        0
 20       17330           .   T   A    3   NA00001    49  3 58,50 allele2        0
 20       17330           .   T   A    3   NA00002     3  5  65,3 allele1        0
 20       17330           .   T   A    3   NA00002     3  5  65,3 allele2        1
 20       17330           .   T   A    3   NA00003    41  3  <NA> allele1        0
 20       17330           .   T   A    3   NA00003    41  3  <NA> allele2        0
 20       1110696 rs6040355   A G,T   67   NA00001    21  6 23,27 allele1        1
 20       1110696 rs6040355   A G,T   67   NA00001    21  6 23,27 allele2        2
 20       1110696 rs6040355   A G,T   67   NA00002     2  0  18,2 allele1        2
 20       1110696 rs6040355   A G,T   67   NA00002     2  0  18,2 allele2        1
 20       1110696 rs6040355   A G,T   67   NA00003    35  4  <NA> allele1        2
 20       1110696 rs6040355   A G,T   67   NA00003    35  4  <NA> allele2        2
 20       1230237         .   T   .   47   NA00001    54  7 56,60 allele1        0
 20       1230237         .   T   .   47   NA00001    54  7 56,60 allele2        0
 20       1230237         .   T   .   47   NA00002    48  4 51,51 allele1        0
 20       1230237         .   T   .   47   NA00002    48  4 51,51 allele2        0
 20       1230237         .   T   .   47   NA00003    61  2  <NA> allele1        0
 20       1230237         .   T   .   47   NA00003    61  2  <NA> allele2        0

Posted in methodology | Leave a comment

farewell to Barcelona

I am posting this a bit late (since I already moved 6 months ago); anyway, the news is that I left my lab in Barcelona, and moved to London!

prbb from avda aiguader

The institute where I did my PhD: the PRBB in Barcelona. On the other side of the building there is the beach.

I am satisfied about my time in Barcelona, where I did my master thesis and my PhD on network theory applied to human population genetics. However, it was time to move and try new experiences.

a picture taken from the 4th floor of the PRBB

a picture taken from the 4th floor of the PRBB

Apart from the change of city, I also changed my field of work, as I am now working on cancer genetics. My new group is a young group recently moved from Italy to London, famous for research on the systems-level properties of cancer genes , for a database called Network of Cancer Genes, and involved in a consortium for the sequencing of hepatocellular carcinoma. I will keep you informed of the proceedings!

Posted in news | 1 Comment

my first PyPI package: vcf2networks

My first Python package is in PyPI!! I guess that now I can officially call myself a python programmer.

VCF2Networks is a python script to calculate genotype networks from population genetics data. Genotype networks are a method used in systems biology to study the “innovability” of a given phenotype, by representing all the genotypes associated with the phenotype as a graph, and studying some properties of this graph, such as the average path length and the average degree. For more info, you can look at the slides of the “Origins of Evolutionary Innovations” book club in this blog. The script in VCF2Networks allows to take any dataset of genotypes stored in the VCF format, and calculate many of these properties.

In principle, I am planning to submit an application note about the script to a bioinformatics-oriented journal. So, if you have some little time to lent me, and you want to test it, any feedback will be very useful for me. At the moment, the major issue is to simplify the installation, because this package depends on numpy and python-igraph, and these two modules require some terrible C libraries that must be installed separately. If you are aware of any way to distribute a binary package of a python module that depends on C libraries, your suggestion will be really welcome.

Posted in news | Tagged , , , | 1 Comment

The presentation of my PhD defence

That’s it! Last week I defended my PhD thesis!! I have gone through it, and survived to tell!

I don’t feel very different from before, apart from being relieved :-). Now the future is possibly more difficult than before, because I have to look for a job position and finish a lot of things.

While I was preparing the slideshow, I realized that there are not many examples of presentations for a PhD defence online. This is bad, because you need all forms of help to prepare this presentation.The PhD defence is the last thing that you do as a PhD student, so you want to do it perfectly. It is also the moment when you describe many years of your work to the your colleagues and family. Thus, it is bad that there are few examples of slideshows for PhD defence online.

Here is the presentation that I have prepared for my defence. I hope that it will be useful to other people as an example for their defences.

I think that, for this type of presentation, the first slide to make is the “summary of the talk” slide, like the “Topics” slide I have. Usually I don’t like to have such summary slides in my presentation, but for the Thesis defence it is very important, because it gives you a feeling of security when you present. Having a well defined structure allows you to know when you can stop to drink some water or to check if everybody is following, and to know exactly what to say in each slide of the talk.

Posted in news, papers, slideshows, talks | 2 Comments

my poster featured in the “Better Posters” blog!

My ECCB2012 poster has been featured in the Better Posters blog. Check the article here:

I am glad because betterposters is one of my favorite blogs. It’s a blog about designing and improving posters for scientific conferences, and it contain many tips and examples of how scientific posters can be improved.

The poster featured there is the poster of the “Post-its”, which I briefly described in the article of the “best practices“.

Here are some other comments and tips from my experience of using post-its to get feedback during a conference:

Continue reading

Posted in news | 2 Comments

Two books on Human Evolution and on the concept of Race, that should be read together

I have decided that, from time to time, I will post some book recommendation here on this blog. This is the first of this series, dedicated to a pair of books on the evolution of the human genome and the concepts of races / human populations.

fatal invention    10000 years explosion

Continue reading

Posted in book club, news | Leave a comment

my attempt at following every possible Best Practice in Bioinformatics

I have just uploaded my first paper to arXiv. The title is “Human Genome Variation and the concept of Genotype Networks“, and presents a first, preliminary application of the concept of Genotype Networks to human sequencing data. I know that the title may sound a bit pretentious, but we wanted to  pay a tribute to a great article by John Maynard Smith, to which the work presented is inspired.

Nevertheless, in this blog post I am not going to discuss the contents of the paper, but only on how I did this work. This was a project that I did in my last year of my PhD, and I have made an extra effort in trying to follow every best practice rules I knew.

I started my PhD in the pre-bedtools and pre-vcftools era of bioinformatics, and I saw the evolution of this field, from a spare group of people in nodalpoint to the rise of Biostar and Seqanswers. During this time, I have read and followed a lot of discussions about “what is the best way to do bioinformatics”, from whether to use source control, to testing, and much more. For the last project as a PhD student, I wanted to apply all the practices that I had learn, to determine if it was really worth to spend time learning them.

Premise: dates and times of the project

My PhD fellowship supports a three months stay in another laboratory in Europe. I decided to do it in prof. Andreas Wagner’s group in Zurich.

The decision to go to Wagner’s group was motivated by a book that he had recently published, entitled “The Origins of Evolutionary Innovations”. Previous to the start of this project I had read some articles by Andreas Wagner, and found them very interesting, so the opportunity to stay in his lab was very exciting. However, in light of what I learned during this time, I have admit that before December 2011, I didn’t understand most of the concepts present in the book. Thus, we can say that for this project, I started from zero.

I started thinking of this project in December 2011. I did the first practical implementation in the three months of the stay in Zurich, from May to August 2012. The first preliminary results came in January 2013, and the first manuscript in April 2013. We submitted to ArXiv in August 2013. During this period of time, I have also worked on three other projects, wrote my thesis, and taught at the Programming for Evolutionary Biology workshop in Leipzig.

I started working on this project in December 2011, and finished in August 2013. The log only shows the activity of code changes.

I started working on this project in December 2011, and finished in August 2013. This figure only shows the activity of code changes.


Note: this blog article is very long, you may want to download as PDF and read it more comfortably.

Continue reading

Posted in methodology, news, papers | 9 Comments

Thesis deposited!! Here is my preface

I have just deposited my PhD thesis! If everything goes well, I will defend it within a few months. It took me a while, but I am there at last!

I can not post the thesis online yet, but in the meanwhile, I would like to at least post the preface I wrote for it.

My thesis is dedicated to detection and characterization of signatures of selection in the human genome. Thus, the preface is about the ethical problems faced in human population genetics, and narrates a little story about a mistake made by an earlier anthropologist. I hope that you will enjoy it. As for me, I am going to make a short break to celebrate.



Toward the end of the 18th century, the German anthropologist Johan Friedrich Blumenbach wrote a book on the origins of mankind, with the aim of demonstrating that all humans belong to the same species. The society of the 18th century was much more segregated than our modern society, and some people, including renowned scientists, believed that blacks and American Indians did not belong to the same species as the white man. Blumenbach, who was a strong opponent of racist theories, decided to write a book to demonstrate that all human have a common origin, and that there are no scientific basis for any discrimination.

Eventually, Blumenbach succeeded in his noble intentions, but a little design mistake in his book led to a misinterpretation that he would not have desired. In his book, Blumenbach listed the human populations in the following order: American, Mongolian, Caucasian, Malaysian and African. Since he believed that the human species originated in the Caucasian region, he explicitly put the Caucasian population in the middle, as a way to remind his readers that all human beings have a common origin. Unfortunately, this tiny detail was interpreted as a prove that the Caucasian was the purest of all human races. People believed that if even him, the most egalitarian scientist of the time, positioned white people at the center of the Geometry of races, it was because these had a special importance.

This error is representative of how delicate is to work in the field of Human Population Genetics. If Blumenbach had decided to list the populations in a different order, for example, by placing Caucasians in the second position, events like the Jim Crow’s laws in the United States and even the Nuremberg laws in Germany would not have had the same scientific justification they had. A whole life spent to demonstrate that all people are equal has gone forgotten because of a bad decision in listing the names of some populations. Blumenbach was a strong champion of equality, but his mistake affected the life of innocent people.

This thesis is dedicated to Johan Friedrich Blumenbach, with the hope that learning from his mistake will protect me from making similar errors. The work presented here describes new methods to analyze human population genetics data, and specifically, to detect genes and alleles that have given a selective advantage to a human population. Nevertheless, these “selective advantages” are only relative to events to which our ancestors have been exposed in the past. The only reason why we study them is to understand how our genome works, with the aim of designing better medicines and improve our health conditions.

The field of Human Population Genetics is in a delicate position at this moment. We live in times of cheap genome sequencing, and we can expect that, in the close future, genome sequencing will become a component of our daily lives. Moreover, the appearance of new communication media has made science more accessible to everybody – with good and bad implications. This means that the research that is being written right now by population geneticists will soon be read by not only by scientists, but also by people moved by other interests. It is difficult to predict how our work will be interpreted, as it was difficult, in the 18th century, to predict how a mistake in listing populations would have had such negative impact.

I hope that those who will read this thesis will do it with a positive mind. I have tried with all my efforts to avoid any concept that may be misinterpreted, but my lack of experience may have not allowed me to find all the potential flaws. I hope that the people who will read this thesis will be savvy when they encounter mistakes, and that they will be stimulated to learn more about this subject. Eventually, they will discover that despite the errors that scientists can make, Blumenbach was right in his intentions: all humans beings belong to the same species, and there are no scientific basis for any form of discrimination.


(This preface is inspired by the chapter “The Geometer of Race” in the book “I have Landed” by S.J.Gould, 2003, and by the book “Fatal Invention” by Dorothy Roberts, 2011)

Posted in news | Leave a comment