Torah Temimah cites Nedarim 16b.
One might think that if he swore to nullify a mitzvah and did not, he would be liable — Scripture says “to harm or to benefit”: just as benefit is optional, so harm is optional.
Oaths to violate a mitzvah are excluded from the oleh ve-yored of shevu’as bitui by the parallel of optional benefit.