We each have different opinions and thresholds for what we think is appropriate or not. That's in part why there is a voting system for the more drastic actions. In that sense, as long as you do what you believe to be correct, judgement calls can't technically be "wrong".
That said, I disagree with your judgement in this particular case. While the answer doesn't strictly speaking answer the question, and therefore you can make a argument for it being "not an answer", I wouldn't have flagged it as such. I personally don't have a problem with answers that don't directly address the question as long as they provide some relevant information. OPs often don't ask what they really want to know, and often don't know that they don't know something important. I personally consider answers that provide such information as being relevant, useful to the OP and the site, and not deserving of being removed.
There is more discussion of this issue here.