I mainly use the keyboard shortcuts, and so merely relabelling the button doesn't really help (I basically never look at it, and sometimes don't even need to scroll down far enough to expose it to decide that something is delete-worthy).
Do you think you could make even more changes to the styling of the (pseudo-) dialog to make it more obvious when you are retracting a close vote?
Alternatively, perhaps change the keyboard shortcut so that md does not even invoke the button when the action is different? (It might be convenient to have a new separate shortcut mr to retract, but I'm guessing that would be significantly harder to implement.)