Torah Temimah cites Nazir 62b.
One might think that if he swore to harm others and did not, he would be liable — Scripture says “to harm or to benefit”: just as benefit is optional, so harm is optional.
Harm to others that is not within his rights is excluded from the oleh ve-yored of shevu’as bitui by the parallel to optional benefit.