To measure directly residual stresses by digital image correlation using hole drilling, the deformation pattern that is governed by the residual stresses is used to affine transform the image captured after the object is deformed. If the values of trial residual stress components are properly chosen, the image after affine transformation will have a maximum similarity to the original image. This turns the residual stress measurement issue into a pure numerical computational process, which leads to the direct output of residual stresses. Validation tests have proved the viability of the approach. The proposed concept and principle could be extended to other specific measurement tasks with known deformation patterns.