I think you can reshuffle the numbers as you described, but yes, if certain numbers are fixed you’ll end up with non-sequential numbers like in Jochem’s example.
Would it be possible for you to delay adding a number to the invoice? I’m thinking of a case where a user can review an invoice and then either send it or delete it. When the user reviews it, it is still unnumbered. When the user then sends it, after which you probably won’t allow the user to delete it anymore, it is assigned the next number.