In this paper, we explore the use of seismicity data on a single-station basis in site response characterization. We train a supervised deep-learning model, SeismAmp, to recognize and separate seismic site response with reference to seismological bedrock (VS  = 3.45 km s−1) in a broad frequency range (0.2–20 Hz) directly from single-station earthquake recordings (features) in Japan. Ground-truth data are homogeneously created using a classical multistation approach—generalized spectral inversion at a total number of 1725 sites. We demonstrate that site response can be reliably separated from single-station seismograms in an end-to-end approach. When SeismAmp is tested at new sites in both Japan (in-domain) and Europe (cross-domain), it achieves the lowest standard deviation among all tested single-station techniques. We also find that horizontal-to-vertical spectral ratio (HVSR) is not the optimal use of single-station recordings. The individual components of each record carry salient information on site response, especially at high frequencies. However, part of the information is lost in HVSR. SeismAmp could lead to improved site-specific earthquake hazard prediction in cases where recordings are available or can be collected at target sites. It is also a convenient tool to remove repeatable site effects from ground motions, which may benefit other applications, for example, improving the retrieval of seismic source parameters. Finally, SeismAmp is trained on data from Japan, future studies could explore transfer learning for practical applications in other regions.