Phonocardiogram (PCG) is a recording of various heart sounds, including heart murmurs, and plays a critical role in diagnosing cardiovascular abnormalities. However, PCGs are inherently susceptible to noise, which can distort the signal and lead to inaccurate diagnoses. Although many studies have addressed this issue, most rely on synthetic noise sources like white or pink noise, potentially limiting real-world applicability. In this paper, we propose a deep learning–based multi-task learning model that leverages the Short-Time Fourier Transform (STFT) and heart murmur types for denoising. Along with synthetic noise, we incorporate real PCG background noise and conversational speech for more realistic training. Experimental results show robust denoising performance across most noise types without data augmentation, including substantial enhancements under challenging conditions such as SNR −5 dB. This approach enables more reliable PCG acquisition in extreme noise environments and has the potential to significantly enhance auscultation systems.