Lets say someone works hard and asks on-topic questions to gain a high reputation, then when they achieve that, they troll around the whole site. Can they lose reputation because of this?
The answer is "Yes". The negative rep from downvotes goes straight into your rep.
Nonetheless, given that a downvote is -2 whereas an upvote is +10 (answer) /+5 (question) /+15 (accept) it takes much longer to loose all that rep than to gain it. Almost certainly you'll just be banned before you get down to 1. For an example, see this question.