In this question, I get a good reaction why the answer is not good, so I want to delete it. However, the persons who made the comments spend some time on it; should I immediately remove the answer or should I let them know to remove it? It feels bad to remove an answer including their comments 'immediately'.
The entire purpose of comments is to induce the poster to modify a post (question or answer). You have two options — either make modifications or don't. Modification could also mean deleting.
Don't worry about the comments. Comments that become irrelevant after modification will probably get deleted.