Combining self-attention and convolution to understand OCT images

Optical Coherence Tomography (OCT) is a widely used imaging modality for diagnosing retinal diseases, with artificial intelligence (AI) increasingly supporting clinical decision-making. A prominent AI model in this domain, called RetFound is a large Vision Transformer (ViT-H/24) model pretrained on nearly one million OCT scans that has shown promise in tasks such as age-related macular degeneration (AMD) detection. However, its substantial size (over 300 million parameters) and high computational requirements pose challenges for real-world deployment. In this study, we hypothesize that ViTs alone are inefficient for OCT interpretation and propose a novel hybrid model that combines self-attention with convolutional layers. This architecture leverages both global context and local structure while remaining lightweight and training-efficient. We demonstrate the effectiveness of this approach on down stream binary classification tasks with our model, AUROC of 0.51, outperforming RetFound (0.49). On a subsequent multiclass task, our model achieves an AUROC of 0.85, closely matching RetFound's 0.87 with both outperforming other benchmark models. Additionally, the model demonstrates competitive image reconstruction, indicating a stronger grasp of underlying OCT structures.