Flags, close votes and downvotes can all be applied unfairly. There are "checks and balances" in the system to reduce this.
A minimum reputation is required to close vote; it takes more than one close vote to close; and questions can be reopened.
Patterns of serial downvoting are detected by the system and reversed and failing that, behaviors can be reported and investigated by admins and possibly acted upon.
In the case of flags, you can directly express your disagreement, if you have moderator privileges (10K reputation or more).
Sometimes a little yellow number appears in the main bar, denoting flagged items needing moderator attention. Next to items there is a button [Flag or Disagree].
The choices there are:
I am flagging this answer because
it is spam: This answer is effectively an advertisement with no disclosure. It is not useful or relevant, but promotional.
it is offensive, abusive, or hate speech: This answer contains content that a reasonable person would deem inappropriate for respectful discourse.
it is not an answer: This was posted as an answer, but it does not attempt to answer the question. It should possibly be an edit, a comment, another question, or deleted altogether.
it has invalid flags: I do not agree with the existing flags on this answer, they are incorrect and should be ignored.
it is very low quality: This answer has severe formatting or content problems. This answer is unlikely to be salvageable through editing, and might need to be removed.
other (needs ♦ moderator attention): This answer needs a moderator's attention. Please describe exactly what's wrong.
The "invalid flags" choice seems to be designed exactly out of concern for the same thing that you've identified.