What's to stop my opponent changing a match score after I've set it?

Nothing. But each time either of you change the score the match will be highlighted for the other to confirm the score in My matches. It's not obligatory to confirm but it would give your opponent a feel-good factor that you've agreed the score and neither of you can make a change without the other being given a reasonable opportunity to review it.

Whatever the score is, confirmed or not, is assumed to be valid.