We develop x-ray multi-modal intrinsic-speckle-tracking (MIST), a form of x-ray speckle-tracking that is able to recover both the position-dependent phase shift and the position-dependent small-angle x-ray scattering (SAXS) signal of a phase object. MIST is based on combining a Fokker-Planck description of paraxial x-ray optics, with an optical-flow formalism for x-ray speckle-tracking. Only two images need to be taken in the presence of the sample, corresponding to two different transverse positions of the speckle-generating membrane, in order to recover both the refractive and local-SAXS properties of the sample. Like the optical-flow x-ray phase-retrieval method which it generalises, the MIST method implicitly rather than explicitly tracks both the transverse motion and the diffusion of speckles that is induced by the presence of a sample. Application to x-ray synchrotron data shows the method to be efficient, rapid and stable.