I am not a high-reputation member. I have had answers down-voted within SE.
I am more interested in why my contributions were down-voted than who down-voted. With no feedback, I have no way to improve.
I think down-votes could remain anonymous as long as the down-voter gives some feedback.
For me, an anonymous comment associated with a down-vote, or an up-voted comment associated with a down-vote are more use than no comment.
I wouldn't make anonymous comments on a down-vote mandatory. However, if folks down-vote silently, I'd value that response less than an anonymous comment, and an anonymous comment less than a comment.
If someone down-votes with nonsense, then I can put that down as a vandal, and ignore it. I might like to be able to ask a moderator to 'wash away' an anonymous comment which is vandalism.
If someone gives feedback that I disagree with, then I can deal with that.
If someone identifies errors I have made, then I would like to know, rather than a silent down-vote.
Whenever I down-vote, I either write comments, or up-vote the comments that expressed my view. I might even feel happier connecting the up-voted comments, or my comments to my down-vote.