This study provides a broad phylogenetic analysis for the Eubrachyura, with the inclusion of three new Panopeidae mitochondrial genomes: Eurypanopeus depressus (flatback mud crab) (15,854bp), Panopeus herbstii (Atlantic mud crab) (15,812bp) and Rhithropanopeus harrisii (Harris, or ‘white-fingered’ mud crab) (15,892bp). These new mitogenomes were analyzed alongside all available brachyuran mitochondrial genomes (n = 113), comprising 80 genera from 29 families, to provide an updated phylogenetic analysis of the infra-order Brachyura (“true crabs”). Our analyses support the subsection Potamoida within the Eubrachyura as the sister group to Thoracotremata. The family Panopeidae aligns with the family Xanthidae to form the Xanthoidea branch, which is supported by current morphological and genetic taxonomy. A unique gene arrangement termed ‘XanGO’ was identified for the panopeids and varies relative to other members of the subsection Heterotremata (within the Eubrachyura) via a transposition of the trnV gene. This gene arrangement is novel and is shared between several Xanthoidea species, including Etisus anaglyptus (hairy spooner crab), Atergatis floridus (brown egg crab), and Atergatis integerrimus (red egg crab), suggesting that it is a conserved gene arrangement within the Xanthoidea superfamily. Our study further reveals a need for taxonomic revision of some brachyuran groups, particularly the Sesarmidae. The inclusion of panopeid mitogenomes into the greater brachyuran phylogeny increases our understanding of crab evolution and higher level Eubrachyuran systematics.