What is S-matrix theory and what was its role in the development of modern physics?

S-matrix theory is the program of describing physics using only asymptotic states and transitions between these, the asymptotic states in flat space-time are particles coming in to a collision, and going out afterwards. It is today subsumed into the holographic principle--- it is holographic physics in flat space-time. But it predates holography by 40 years, and it is the central principle that gives rise to string theory.

There is an important physical detail to understand regarding this: particles in infinite plane waves are noninteracting (except in 1+1d) they are free field theory states, because the particles have negligible probability of finding each other in an infinite space. The S-matrix is defined to be the residual non-identity transformation between the past-asymptotic states and the future asymptotic states, it is mostly a delta function doing nothing.

There is also with a less singular delta function which contains the scattering information, which appears as deflections and phases when you superpose the asymptotic states into wave packets that collide. In math: S = I + iA, where I is delta functions for each incoming momentum, while A is the relativistically invariant amplitude when used between relativistically normalized asymptotic states. A has only an overall energy-momentum conserving delta function, so it is less singular than the I.

In S-matrix theory, you are supposed to extract every other observable from this asymptotic thing, the invariant amplitude A, at least in principle. So you aren't allowed to speak about any state at intermediate times, except inasmuch as you know how to build it up from superpositions of asymptotic free particles.

This was extremely counterintuitive, because the notion of things happening in space and time doesn't appear, only the asymptotic states are fully consistent to talk about. So the whole history of the world starts to look like an interlude between free cold particles that form the Earth, and free cold particles which fly out when it reaches heat death! It sounds completely crazy. Consider that you haveno idea how to build up anything like a dewer of He3 from asymptotic cold states.

The reason people took this seriously is because the S-matrix was made to get around the issues of short distances in quantum field theory, which introduces arbitrarily high-energy intermediates to describe any scattering process, and also to get around ambiguities of field definition. In Feynman diagrams, you integrate over arbitrarily localized collisions, and you need to deal with arbitrarily short distances. The idea in S-matrix theory is to integrate instead over arbitrary asymptotic states in intermediate expressions, so that you don't have to deal with arbitrarily localized objects. The S-matrix is also real observable quantities defined using real asymptotic particle states, so it doesn't depend on which fields you choose to declare fundamental and do a path integral over.

The idea was to provide quantum fields with an invariant formulation using observable processes, much like what Heisenberg gave quantum mechanics with the energy representation. It's another application of positivism, this time to relativistic physics, and in this form, the idea is also due to Heisenberg, although the mathematical S-matrix formalism is Wheeler's.

The main problem with the approach is that you usually end up easily reconstructing a sum over localized events just from extrapolations of the sum over asymptotic states to arbitrarily high momenta. For example, if you consider asymptotic electron states and photon states, they reconstruct a free photon field and free electron field. Then using the scattering of the photon and electron, you can build up an S-matrix perturbation theory, and it is just the same Feynman diagrams you get from the interacting theory of the Dirac field with the E&M field. The sum over high-energy photons and electrons just reproduces a localized field theory of photons and electrons, and the S-matrix is just the least detailed way of describing what is going on.

I'll call this "Feynman's chagrin": S-matrix theory, without extra physics, has a way of turning right back into field theory. It's what Feynman realized when he formulated S-matrix style diagrams, thought he had a radical new theory, and compared notes with Schwinger and realized he didn't. The perturbative contribution to scattering from a sum over intermediate asymptotic electron states of arbitrarilu high momentum turns into a particle propagator for an idealized electron between space-time points, but these point particle path sums are exactly the field correlators in an interacting field theory, expanded in powers of the interaction!

The same thing happens for S-matrix theories of Pion scattering, they turn into effective field theories, as laboriously shown by Weinberg in the 1960s. In general, when there are a finite number of asymptotic free particle states that you sum over, you reproduce a field theory by defining effective fields for these, and adding interactions locally is the way to satisfy causality conditions on the S-matrix.

This path for S-matrix theory sort of died in the early 1970s, because it was equivalent to effective field theory. In this context, the S-matrix physics is just a subset of field theory physics, and you can very nearly prove that the only S-matrix is some field theory, Weinberg gives an elegant exposition of this near-proof in his books, the main implicit assumption is that the asymptotic states are exhausted by a finite number of free particle states.

But there is another path for S-matrix theory---- when there are infinitely many families of particles in the asymptotic states. This is the case in idealizations of strong interaction physics, where you assume the pions and hadrons are stable in first approximation. This is the "Narrow Resonance approximation", it is described pedagogically in Feynman's classic monograph "Photon Hadron Interactions".

In the narrow resonance approximation, the strongly interacting particles lie on Regge trajectories, families of particles of arbitrarily high spin and mass, with a law relating the mass-squared to the spin. These families are the natural representation of bound states in scattering problems, and Geoffrey Chew postulated that Hadronic resonances (particles) lie on straight-line Regge trajectories, with mass-squared proportional to spin, and a universal slope. This was conjectured from the famous Chew-Frautschi plot.

Then S-matrix theory is the statement that all hadrons are composite (true), that they have no field theory constituents (false), and that they can be used to make a theory of pure Regge trajectories on asymptotic states, so that only composite particles appear in the formulation of the theory (revolutionary, inspiring, but perhaps only partly true for the strong interactions).

Constructing S-matrix theories for Regge trajectories is what took up the attention of about half of the theorists in the 1960s. There were several solid insights about scattering near the beam line from this:

But the main coup of the S-matrix theory was the discovery of a fully consistent leading order scattering amplitude for straight line Regge trajectories, the Veneziano amplitude. Since this amplitude scattered trajectories, it did not turn into field theory, it wouldn't Weinbergify into a field theory, no Feynman chagrin.

Instead there was Scherky triumph, because Scherk showed that exchange of the objects in Veneziano's model reproduced field theory only when you got rid of the higher excitations by making them infinitely massive. This means that this was a genuine generalization of quantum field theory, it was the radical new theory that Feynman thought he had in the early 1950s, the radical theory that Chew wanted in 1960.

This thoery is string theory, and the gravitational re-interpretation of string theory explains why it was natural to discover it this way. In gravity hugh energy objects are big floppy black holes, with internal motion, so that the asymptotic states of quantum gravity do not have a finite number of particles, but whole classes of highly boosted spinning black holes, which don't decay because they're going so fast. These family sums over asymptotic states can never produce a field theory, because any ultraviolet divergence is due to enormously extended black holes, or "infrared" strings.

The history of strings from this point onwards is well known, but the roots of this in S-matrix theory is unfairly buried. Part of the reason is pure politics. S-matrix stuff was big in the Soviet Union. Another understandable reason is that QCD is correct.

I wrote a little more on stackexchange: What are bootstraps?