Self-exited thermoacoustic instability is undesirable in gas turbines and other propulsion systems due to structural vibration, overheating and flame flashback. Such instability is characterized with large-amplitude limit cycle oscillations. It is typically resulting from the dynamic interaction between turbulence flow-flame-acoustics. A better understanding on the interaction, and an alternative design of effective damping/control approach to stabilize combustors are wanted. In this work, 3D numerical simulations are conducted on a swirling combustor to gain insight on the generation of nonlinear swirling flow-flame-coupled thermoacoustic instability. To capture the turbulence flow-combustion-acoustics interaction, the classical turbulence model for swirling flow RNG k-ε and eddy dissipation concept model with 2-steps are applied. The model is validated first with the experimental data available in the literature. Then it is applied to study the effect of swirling number SN. It is found that increasing the SN leads to the amplitude of combustion instability being increased. The dominant frequency is found to shift to a higher value with increased SN. To attenuate the selfexcited thermoacoustic instability, we propose an alternative passive control approach by implementing a heat exchanger at a pre-selected segment of combustor wall; its temperature TH could be varied. Numerical results show that properly setting TH can attenuate the thermoacoustic instability by approximately 20 dB. The present study contributes to developing a platform to simulate nonlinear thermoacoustic instability in a swirling combustor. It also opens up an alternative control means to prevent the onset or attenuate the undesirable limit cycle oscillations.