Consider the following argument:
Proposition 1: The language of physics (as an empirical science) is mathematics.
I think this should be uncontroversial to the majority of working physicists.
Proposition 2: That the mathematics used in physics will eventually be formalised in the way that mathematicians use mathematics.
There is no generally accepted formalism that makes sense of the Feynman Path Integral, though there are special cases that have been formalised. But I think most physicists would accept that this will be a matter of time & human ingenuity. When Newton invented calculus to investigate problems in dynamics, he was famously criticised by Berkeley for his fluxions, they were only put on a more certain (cumbersome) basis a century later.
Whereas its not a priori certain that a question wholly mathematical necessarily has a theory behind it, I think it is generally accepted in the community that the mathematics behind physics should - I would argue that its this certainty that allows physicists to take the shortcuts they do. I should clarify that formal here means that all foundational questions have been cleared up (I'm not going into Godel now).
Proposition 3: That these formalisms will form a self consistent whole.
Again, I don't think this should be controversial. Currently we have GR & QFT. I take it as generally accepted that there is a further theory that will combine both.
Proposition 4: That this theory will not be subject to Popperian falsifiability (though we cannot verify this).
Popper suggested that theories progress by falsification. I'm proposing once we reach a 'true' theory, by definition it will not be falsifiable by definition. Of course only some 'Oracle' can verify this by comparing the 'true' underlying mathematical reality to the one we've reached by our unaided efforts. (Note that I am putting the word true in quotes as I'm not sure what true means in these circumstances.)
Proposition 5: That a self-consistent whole is capable of further internal development.
I can't see how this can be controversial.
Proposition 6: That this internal development must proceed on aesthetic grounds—what mathematicians call mathematical intuition, elegance; and what physicists call physical intuition.
Given that the theory can no longer be tested, meaning that there can be no experimental evidence to force a change, the only development must be internal. I'm assuming physical/mathematical intuition can be characterised as a certain form of aesthetic. I don't see this as controversial, given some of the famous pronouncements by physicists & mathematicians of all stripes.
Where does this argument fall down?
I suggest at step 4, because there can be no final underlying mathematical reality. This seems a bit presumptuous, considering that the whole scientific project relies on this, but if there is, we go to step 5, which again states there isn't a final underlying mathematical reality.
As Dorfman points out below, its not a given that underlying reality can be expressed in mathematical form. Verlinde, inventor of Entropic Gravity has stated same in an interview. (I'd provide a link but I forget where I saw it).
The point of my argument is to demonstrate even accepting that there is will lead to contradictions.