Torah Temimah cites Bava Kamma 91b.
One might think that if he swore to harm himself and did not, he would be exempt — Scripture says “to harm or to benefit”: just as benefit is optional, so harm is optional.
The parallel of optional benefit includes optional self-directed harm within the verse’s scope for the oleh ve-yored.