Being lack of the pre-earthquake reference information, a new method of damaged building detection of high-resolution remote sensing image based on optimized visual dictionary is proposed. Firstly, wavelet-JSEG (WJSEG) segmentation and a set of non-building screening rules are applied to extract the potential building set. Secondly, a visual dictionary model of earthquake damage is constructed by introducing spectral, texture and geometric morphological features to across the semantic gap between pixels and earthquake damage features. On this basis, a visual dictionary optimization strategy based on intra-class and inter-class penalty indexes is designed to further reduce redundant information and evidence conflict. Finally, the buildings are further classified into intact buildings, partially damaged buildings and ruins by random forest classifier. In two experiments, the overall accuracy of the proposed method reached more than 85%, which can provide key decision support information for post-earthquake emergency response and reconstruction.