By default MathGL save all primitives in memory, rearrange it and only later draw them on bitmaps. Usually, this speed up drawing, but may require a lot of memory for plots which contain a lot of faces (like cloud, dew). You can use quality function for setting to use direct drawing on bitmap and bypassing keeping any primitives in memory. This function also allow you to decrease the quality of the resulting image but increase the speed of the drawing.
The code for lowest memory usage looks like this:
int sample(mglGraph *gr)
{
gr->SetQuality(6); // firstly, set to draw directly on bitmap
for(i=0;i<1000;i++)
gr->Sphere(mglPoint(mgl_rnd()*2-1,mgl_rnd()*2-1),0.05);
return 0;
}