During the summer of 2025, I had the incredible opportunity to contribute to the global fight against malaria through my Google Summer of Code project with the Sanger Institute, a world-renowned hub for genomic research. I worked under the MalariaGEN team, a collaborative effort focused on using genomic data to understand and combat malaria. Their Vector Observatory project collects and curates vast amounts of genomic information from malaria-carrying mosquitoes, particularly the Anopheles gambiae complex, to support public health strategies worldwide.
Problem Statement
Malaria remains a global health crisis, transmitted primarily by mosquitoes of the Anopheles gambiae complex in Africa. This group includes at least eight species that are visually indistinguishable, yet their differences are critical for effective malaria control. These “cryptic species” vary in behavior: some prefer biting humans, increasing malaria transmission, while others target animals, posing less risk. Some rest indoors, making them vulnerable to indoor spraying, while others rest outdoors. Additionally, certain species have developed resistance to common insecticides, complicating control efforts. Misidentifying these species can lead to ineffective strategies, like using the wrong insecticides or targeting the wrong habitats, wasting resources and failing to curb malaria spread.
My GSoC project aimed to address this by building a machine learning tool to classify these mosquito species accurately using their genomic “fingerprints.” By analyzing subtle DNA differences, known as Single Nucleotide Polymorphisms (SNPs), the tool would enable researchers and public health officials to identify species quickly and reliably, paving the way for targeted, data-driven malaria control interventions.
Understanding Genomic Structure
The genomic data I worked with is structured as an n × 2 matrix, where n represents the number of SNPs (genetic markers) across the genome, and the “2” reflects the diploid nature of the mosquito’s DNA. In diploid organisms like Anopheles gambiae, each individual inherits two sets of chromosomes (one from each parent).
This pair-wise representation captures the genetic diversity and is essential for distinguishing species based on their unique SNP patterns.
To prepare this data for analysis, I followed two critical initial steps:
- Standardizing Genotype Encoding: The raw data often contained genotypes like (0,0) (0,1), (1,0) and (1,1) which needed unification.
- Imputation for Completeness: Genomic datasets often have missing values due to sequencing gaps or errors (represented by -1). This involved estimating missing genotypes based on the frequency of values of that SNP in rest of dataset, ensuring the data was fully populated for downstream machine learning tasks without introducing significant bias.
Despite these preparations, there were 2 genomic structure posed key challenges:
- Massive Data Scale: The MalariaGEN Vector Observatory dataset spans terabytes, with each mosquito genome containing tens of millions of SNPs (distributed across 5 chromosomes 2L, 3L, 2R, 3R, X). Loading a single genome into memory on standard hardware is impractical, let alone processing 22K for classification, necessitating innovative data-handling strategies.
- Subtle Genetic Signals: The Anopheles gambiae complex species are genetically similar, with distinguishing SNPs scattered across millions of positions. Identifying these predictive markers amid noise was like searching for needles in a haystack, requiring sophisticated feature selection to avoid overfitting.
Partition
To tackle the massive scale of the genomic data, I adopted a “divide and conquer” strategy by partitioning the genome into manageable segments. Each partition represents a chunk of the chromosome, typically around 1 million base pairs allowing the system to process the data in smaller, memory-efficient pieces rather than attempting to handle the entire terabyte-scale dataset at once. This approach was critical given the limitations of standard hardware, where loading a full genome was impossible.
The partitioning process began with the use of the Zarr data format, which stores data in compressed, chunked arrays on disk. This enabled random access to specific partitions without loading the entire genome into memory. For each partition, I applied the standardized encoding and imputation steps described earlier, ensuring that every segment was consistently prepared for analysis. This modular design not only reduced memory usage but also allowed parallel processing of different partitions, significantly speeding up the data pipeline.
Partitioning also facilitated the identification of species-specific genetic signals. By analyzing each segment independently, the system could focus on localized SNP patterns that were most predictive of species identity, avoiding the noise from less informative regions. This laid the foundation for training specialized machine learning models on each partition, a key step in building a robust classification tool tailored to the complex genomic structure of the Anopheles gambiae complex.
Training
With the genome partitioned into 150 segments, the next step was to train machine learning models to classify the mosquito species. However, even with partitioning, the dataset still contained 22,000 data points, which was substantial and posed computational challenges. To make training manageable and effective, I selected a balanced random subset of 1,000 samples from across the 150 partitions. This subset ensured representation from all species, addressing the natural class imbalance where some species were rarer than others.
I used the Random Forest algorithm for training, chosen for its ability to handle high-dimensional genomic data and its robustness against overfitting. Each of the 150 partitions had its own dedicated model, trained on the 1,000-sample subset specific to that segment. To validate the models and ensure reliable performance, I implemented Stratified K-Fold Cross-Validation. This technique divided the data into folds while maintaining the same species distribution in each fold, providing a fair assessment of the model’s accuracy and preventing bias toward more common species.
The training process was automated with a loop that iterated over each partition, fitting a model and evaluating its performance using metrics like the F1-score. This approach allowed the system to learn localized genetic patterns unique to each chromosomal segment, setting the stage for combining these models into a powerful ensemble for final predictions.
Prediction
While we managed to reduce computation in training by using a smaller, balanced dataset, achieving fast and accurate predictions remained the key goal. The trained models from the 150 partitions provided a wealth of localized insights, but a single, reliable species classification was needed for practical use. To accomplish this, I developed an ensemble prediction engine that combined the strengths of the top-performing models. I evaluated each model’s performance on the validation set, ranking the 150 partitions based on their F1-scores to identify the most predictive segments. From this ranking, I selected the top 50 models (why 50? will discuss in next section), which demonstrated the highest accuracy in distinguishing species. The prediction process then involved running a new mosquito genome through these top 50 models, with each model casting a “vote” for the species it identified. The final classification was determined by a majority vote, ensuring a robust and consensus-driven result.
This ensemble approach minimised the impact of noise or weaker models, leveraging the collective wisdom of the best partitions. To optimize speed, the prediction engine processed each partition in parallel where possible, using the pre-computed Zarr data format to quickly access relevant genomic segments.
Hypothesis Validation
To ensure the effectiveness of my approach, I validated two critical hypotheses that underpinned the project’s design.
- The partition-based strategy would effectively isolate predictive genomic regions
- the models were learning meaningful biological signals rather than spurious patterns.
To test the first hypothesis, I plotted the cumulative F1-score of the ensemble as I incrementally added partitions from highest to lowest rank. The results were revealing: performance peaked with the top 50-60 partitions, achieving near-perfect accuracy, but began to decline as less informative partitions introduced noise. This confirmed that focusing on a “Top-N” ensemble strategy was optimal, validating the partitioning and model selection process.
For the second hypothesis, I examined a top-performing model’s feature importances, which highlighted the specific SNPs it prioritized for classification. These SNPs aligned with known biological regions associated with species differentiation, demonstrating that the model was capturing genuine genetic signals. This validation reinforced the reliability of the trained models and their ability to support accurate species identification in real-world applications.
Results and Impact
The culmination of my GSoC 2025 project delivered several significant outcomes that have the potential to transform malaria control efforts. The classifier achieved remarkable accuracy, with the top 50 partitions yielding cross-validation scores ranging from 98.47% to 99.98%, proving its effectiveness in distinguishing the cryptic species of the Anopheles gambiae complex.
I packaged the tool into a comprehensive, open-source Python library, complete with pre-trained models, a CLI, and a programmatic API. This makes it accessible to researchers and health officials worldwide, even those without deep technical expertise. To further enhance usability, I developed a series of five detailed Jupyter notebooks, offering step-by-step guidance on installation, data loading, and prediction processes.
The project’s success lies in its innovative approach—breaking down the massive genomic dataset into manageable partitions and leveraging an ensemble of models to deliver accurate, rapid results. All code, models, and documentation are publicly available at malariagen/vector-taxon-classifier-prediction
Thanks for Reading :)