As a non-invasive approach to monitor heart rate (HR), the photoplethysmography (PPG) signal provides a simple and accurate measurement in silence. However, difficulties are found when dealing with signals monitored during physical activities, because complex noise shares a very close frequency bin with HR. Traditional analogue filters can hardly derive HR in this situation. This paper is aimed to establish a joint framework for HR monitoring from PPG signals during physical activities. A combination of Short-time Fourier Transform (STFT) and spectral analysis was adopted as the principal part, with a medium filter as assisted. The time-frequency resolution in low frequency is enhanced by the size-fixed window function in STFT. Based on 12 datasets sampled at 25 Hz and recorded during different physical activities, the HR data derived via the proposed algorithm was analysed. The average absolute estimation error was 1.06 beats/min and the standard deviation was 0.69 beats/min. Compared with the true HR via ECG, the cross-correlation average was 0.9917 beats/min and the standard deviation was 0.0141 beats/min. Therefore, the proposed framework is proved reliable for HR monitoring from PPG during physical activities of high intensity. It can be applied to smart wearable devices for fitness tracking and health information tracking.