Torah Temimah cites Shevuos 26b.
“And it was hidden” — excluding the intentional violator; “from him” — that the shevu’ah itself was forgotten.
The oleh ve-yored of shevu’as bitui is for one who forgot the oath (or its content), not for deliberate violation, which is handled by other penalties.