Let's start from 19th Century, and work backward.
The most common sort of small ensemble is probably the string quartet. It consists of two violins, [lowest note: G below Middle C], a viola, [lowest note: one octave below Middle C], and a Cello, [lowest note: two octaves below Middle C].
The string quartet has the interesting property that the four instruments sound almost just like each other; in fact, they're essentially four sizes of violins, with the larger ones sounding lower. We call this character homogeneity; they blend together perfectly.
Another group for which a lot of music is written is a wind quintet. Music for a wind quintet sounds beautiful and balanced, but the balance doesn't come from the nature of the instruments. A wind quintet consists of (typically a flute, an oboe, [lowest note Middle C in both cases], a clarinet, [lowest note B Flat, an octave and one more whole note below Middle C], a bassoon [lowest note: E Flat, a little less than two octaves below M. C.], and a French Horn, [wait, I have to look this up ... Lowest Note: Two octaves below M.C.].
The presence of the horn in the wind quintet is often a surprise for anyone learning about them for the first time. I don't know the history of the situation, but I imagine that someone must have tried it, and it must have sounded satisfactory.
Obviously, the Wind Quintet does not sound homogeneous by design, but they do sound wonderful. There are certainly other combinations, for instance lots of combinations with pianos: the piano quintet, consisting of a piano and a string quartet; the piano trio, consisting of a piano, and violin, and a cello.
There are also many different Quintets, consisting of a String quartet, and one instrument not from the quartet, e.g. a Clarinet Quintet (string quartet + clarinet).
In earlier times—say before the 17th Century—small ensemble were assumed to consist instruments of the same family; for instance a recorder consort: which would consist of several recorders all of the same style. There are bass recorders, tenor recorders, alto recorders, soprano recorders, and sopranino recorders. Often they were sold in a matched set, in a special case with a place for each recorder.
There were also what was known as a school of viols, consisting of viols of grades sizes, from a treble viol, down to a bass viol.
Recorder ensembles and Viol ensembles had extremely homogeneous sounds. In early music, I suppose pieces written for them were monophonic; that moving from chord to chord, without the complexity of Baroque Music.
Contorts with different sorts of instruments were called 'broken consorts'. If a consort consisted all ofthe same type of instrument, they were called 'whole', or 'closed' consorts—at least, according to Wikipedia.
I'm sure that in earlier posts I have linked to examples of many kinds of small ensemble, but I could be wrong. Someday, I hope to link to pieces for small ensembles on YouTube, and that would complete this post.
Archie