You should be fine to remove the washer.
Although you can't just tighten the nuts so there is no slop in the spring part of the washer.