Yeah, I'd say size is something you get past - though I did get myself a decent compact to always carry around for similar reasons, but having the choice I find I don't use it as much as I might have. Still enough that I'm glad I have it, but without it I'd regret shots that I otherwise decide are not worth taking. I always take the D750 when I know I'm going to be doing photography.

I do think in your position I probably would wait a year or two for the next generation before upgrading, (unless you're specifically hitting the boundaries that the A7iii definitely overcomes).