Mosquito Genomic Classifier

2025

Abstract

A partitioned Random Forest ensemble for classifying mosquito species from genomic data. GSoC 2025, MalariaGEN.

View source

1   A matrix that never fits

The Anopheles species that carry malaria can be nearly impossible to separate by eye, and the distinction is not academic. Species differ in behaviour and in insecticide resistance, so control decisions depend on getting the identification right. Genomic markers give a basis for that call which morphology does not. I worked on this during GSoC 2025 with the MalariaGEN team under Wellcome Sanger Tree of Life.

The obstacle turned out to be the shape of the data rather than the model. Each genome in the Vector Observatory dataset carries millions of SNP markers, and the matrix implied by every sample against every marker is not something you hold in memory. Subsampling the markers or the samples would have fixed that by throwing away the signal I was trying to learn from. So the question became how to train on all of it without ever loading all of it.

2   Partitioning twice over

The answer was to stop treating the genome as a single input. Genotypes are read from compressed, chunked Zarr arrays instead of one array in memory, encodings are standardised and missing calls imputed, and the genome is divided into roughly 150 regions so that preparation and fitting only ever touch one partition at a time. Each partition trains its own Random Forest on a balanced subset, scored with stratified cross-validation. That takes memory out of the way.

What I did not anticipate is that the split does a second job. Once every region has its own model and its own validation score, those scores say something about the region itself: some parts of the genome separate these species and some do not. A partitioning I adopted to fit in memory turned into a way of asking where the signal lives, and it drops an assumption a single whole-genome model makes quietly, that every region contributes equally.

Partitioned ensemble over a genome too large to load at once Genotypes are read from Zarr in chunks. The genome is split into about 150 partitions, each with its own Random Forest. Partitions are ranked by validation F1 and only the highest-ranked ones vote on the species call. Zarr store chunked SNP reads Partition 001 · Random Forest Partition 002 · Random Forest Partition 150 · Random Forest Majority vote top 50 to 60 partitions vote chunks ranked by validation F1 ranked below the cut: trained, then unused at inference
Figure 1.No model sees the whole genome. Around 150 partitions are trained independently, ranked by validation F1, and only the highest-ranked ones vote on the species call, which keeps both training and inference inside memory. Greyed partitions are trained but unused at inference.

3   How many partitions should vote

That raises a question a single model never poses. At prediction time, which of the 150 models get a say? Ranking partitions by validation F1 gives an order but not a cutoff. The ensemble-size experiment was strongest around the top 50 to 60 partitions, and adding further lower-ranked regions did not reliably improve on it. A weak region still votes, which is what makes leaving it in expensive. So the prediction interface exposes the choice rather than burying it: take the top N, or name specific partitions.

Cross-validation scores for the leading partitions ranged from 98.47% to 99.98%. Those numbers describe this dataset under this evaluation. They are not a promise about new populations or different sequencing conditions, and a partition that separates species cleanly here has not been shown to do so elsewhere. What ships is a Python API and command-line interface with input validation and batch processing over local or remote Zarr, plus notebooks covering loading, prediction and the partition-selection plots, so the ensemble can be used without reproducing the training experiments.

Related reading