More precisely, one should consider algebraic curves C of a given genus g, and their Jacobians J. There is a moduli space Mg of such curves, and a moduli space Ag of abelian varieties of dimension g, which are principally polarized. There is a morphism
which on points (geometric points, to be more accurate) takes C to J. The content of Torelli's theorem is that ι is injective (again, on points). The Schottky problem asks for a description of the image of ι.
It is discussed for g ≥ 4: the dimension of Mg is 3g − 3, for g ≥ 2, while the dimension of Ag is g(g + 1)/2. This means that the dimensions are the same (0, 1, 3, 6) for g = 0, 1, 2, 3. Therefore g = 4 is the first interesting case, and this was studied by W. Schottky in the 1880s. Schottky applied the theta constants, which are modular forms for the Siegel upper half-space, to define the Schottky locus in Ag. A more precise form of the question is to determine whether the image of ι essentially coincides with the Schottky locus (in other words, whether it is Zariski dense there).
If one describes the moduli space Ag in intuitive terms, as the parameters on which an abelian variety depends, then the Schottky problem asks simply what condition on the parameters implies that the abelian variety comes from a curve's Jacobian. The classical case, over the complex number field, has received most of the attention, and then an abelian variety A is simply a complex torus of a particular type, arising from a lattice in Cg. In relatively concrete terms, it is being asked which lattices are the period lattices of compact Riemann surfaces.
One of the major achievements of Bernhard Riemann was his theory of complex tori and theta functions. Using the Riemann theta function, necessary and sufficient conditions on a lattice were written down by Riemann for a lattice in Cg to have the corresponding torus embed into complex projective space. (The interpretation may have come later, with Solomon Lefschetz, but Riemann's theory was definitive.) The data is what is now called a Riemann matrix. Therefore the complex Schottky problem becomes the question of characterising the period matrices of compact Riemann surfaces of genus g, formed by integrating a basis for the abelian integrals round a basis for the first homology group, amongst all Riemann matrices.