Using the Kiban-Kyoshin network (KiK-net) downhole array data in Japan, we analyze the accuracy (bias) and variability (precision) resulting from common site response modeling assumptions, and identify critical parameters that significantly contribute to the uncertainty in site response analyses. We perform linear and equivalent-linear site response analyses at 100 KiK-net sites using 3720 ground motions ranging from weak to strong in amplitude. We find that the maximum shear strain in the soil profile, the observed peak ground acceleration at the ground surface, and the predominant spectral period of the surface ground motion, are the best predictors of where the evaluated models become inaccurate and/or imprecise. The peak shear strains (γ max) beyond which linear analyses become inaccurate in predicting surface pseudo-spectral accelerations (PSA) are a function of vibration period, and are between γ max = 0.01% and 0.2% for periods less than 0.5 s. Equivalent-linear analyses become inaccurate at peak strains of approximately γ max = 0.4% over this range of periods.