A slightly modified version of the InParanoid 4.1 software (4 (link),8 (link)) was used for computing InParanoid 8. The difference is in how the two BLAST (9 (link)) passes are run. In the distributed version 4.1, the two BLAST passes are run after each other—the first run to find all homologs between two species, and then a second run is launched per query sequence to make accurate alignments with only the homologs found in pass 1. We have not yet found a BLAST setting that simultaneously makes accurate alignments and efficiently avoids false low-complexity matches. The default composition-based score adjustment (10 (link)) in BLAST does the latter, and is used in pass 1, but it often truncates the alignments which may cause InParanoid to miss true orthologs (8 (link)), hence it is not used in pass 2.
In order to improve computational throughput, we ran both BLAST passes in parallel and after both were done, extracted matches from pass 2 for homologs found in pass 1; see Figure1 . This can also save total real runtime as only two BLAST runs are launched instead of thousands of tiny runs per species comparison, which causes a lot of input/output (I/O) overhead. There are some drawbacks however: the pass 2 computation and results become much larger, and the infrastructure and work required to synchronize, supervise and load balance the increased number of computational jobs is considerable. We opted for this solution mainly because it offers a higher degree of parallelization.
To speed up the parallel pass 2, the BLAST parameter z (effective database size) was changed from 1 to 5 000 000. This can reduce BLAST's ability to find matches, and it therefore happens at low frequency that homologs found in pass 1 are not reported in the parallel pass 2. To handle this, a quality control and repair step was added where missing pairs were rerun with a normal pass 2. It also catches other problems, such as failed or truncated runs, and repairs them too. Hence the quality control is currently more rigorous for the parallel method than for the serial. The computations were run on a Linux cluster with around 300 8-core nodes, and took in total about 113 core years of which nearly all was spent on running BLAST (blastall 2.2.18). Only about 0.5% of the computation time was spent on running the InParanoid orthology detection algorithm, implemented in Perl. Even though the procedure was modified, the parameters were effectively the same as by default. Compared to the previous release, the number of pairwise species comparisons increased by 650%, from 4950 to 37 128.
In order to improve computational throughput, we ran both BLAST passes in parallel and after both were done, extracted matches from pass 2 for homologs found in pass 1; see Figure
To speed up the parallel pass 2, the BLAST parameter z (effective database size) was changed from 1 to 5 000 000. This can reduce BLAST's ability to find matches, and it therefore happens at low frequency that homologs found in pass 1 are not reported in the parallel pass 2. To handle this, a quality control and repair step was added where missing pairs were rerun with a normal pass 2. It also catches other problems, such as failed or truncated runs, and repairs them too. Hence the quality control is currently more rigorous for the parallel method than for the serial. The computations were run on a Linux cluster with around 300 8-core nodes, and took in total about 113 core years of which nearly all was spent on running BLAST (blastall 2.2.18). Only about 0.5% of the computation time was spent on running the InParanoid orthology detection algorithm, implemented in Perl. Even though the procedure was modified, the parameters were effectively the same as by default. Compared to the previous release, the number of pairwise species comparisons increased by 650%, from 4950 to 37 128.