During summer 2025, I got to work with the Sanger Institute under Google Summer of Code, with the MalariaGEN team. The mission was simple to say but hard to do. Correctly identify malaria carrying mosquito species from their genomic fingerprints. These species look the same, but their behavior and insecticide resistance can be totally different, so classification really matters for real world malaria control.
The big challenge was scale. The Vector Observatory dataset is huge, and each genome has millions of SNP markers. You cannot just load everything into memory and train one giant model. So I went full nerd mode and built a divide and conquer pipeline.
I stored the data in Zarr so we could read small chunks from disk, not the whole genome. Then I standardized genotype encoding and handled missing values with imputation. After that, I partitioned the genome into about 150 segments, trained a Random Forest model per segment on a balanced subset, and ranked the segments by F1 score.
For prediction, I built an ensemble that uses only the best partitions. The sweet spot was the top 50 to 60 partitions. Each model votes, and the final class comes from majority voting. This helped reduce noise from weaker regions and kept inference fast.
The result was really strong. Cross validation scores for the top partitions were between 98.47 and 99.98 percent. I also packaged everything as an open source Python library with a CLI and notebooks, so researchers can actually use it without digging through training code.