Remote photoplethysmography (rPPG), as a contactless heart rate measurement technique, has been widely applied in health monitoring, affective computing, and driving safety. Existing supervised rPPG methods rely heavily on ground truth physiological signal labels, yet such reference labels are difficult to acquire and costly to annotate, which limits their practical deployment. To address this issue, this paper proposes a self-supervised learning method that integrates masked autoencoder (MAE) with contrastive learning (CL), termed PhysCMAE, to enable accurate video-based heart rate measurement when only limited reference physiological signal labels are available. First, multi-scale spatiotemporal maps are constructed from the temporal variations of pixel values in facial skin regions of interest. Next, an asymmetric encoder-decoder architecture based on a Vision Transformer is designed to perform masked autoencoder pretraining on the multi-scale spatiotemporal maps, thereby learning fine-grained physiological feature representations. Then, in the later stage of pretraining, a contrastive learning module is introduced, and a frequency-domain contrastive strategy is adopted to enhance the instance discriminability of physiological features. Finally, an rPPG prediction head is added, and the pretrained encoder is finetuned to map video physiological features to high-precision blood volume pulse (BVP) signals and perform heart rate estimation. In within-dataset experiments on two public datasets, UBFC-RPPG and PURE, the proposed method achieves mean absolute errors of 0.71 bpm and 0.59 bpm, respectively. In cross-dataset experiments, the mean absolute errors are 3.80 bpm and 4.88 bpm, representing error reductions of about 68.0% and 15.2%, respectively, compared with baseline methods. In cross-dataset experiments on the self-collected BSIPL-RPPG dataset, the mean absolute errors are 3.96 bpm and 6.69 bpm, respectively, achieving the second-best results among existing supervised learning methods. Experimental results demonstrate that, compared with existing self-supervised learning approaches, PhysCMAE can achieve higher heart rate estimation accuracy and stronger cross-dataset generalization under conditions of scarce physiological signal labels, providing an effective technical solution for the practical application of video-based rPPG technology.