Open Mouth Hum is a variation of humming, a semi-occluded phonation. Open mouth humming can be interpreted in different ways, but the approach I like is described by Berton Coffin in his semiannual book The Overtones of Bel Canto.
Essentially, the vowel is sung behind the hand that is covering the open mouth. Soft palate is lowered to allow the sound to travel into nasal passages. Tongue takes its position for the vowel. Soft palate goes up and back once the hand is removed.
Coffin also called this exercise “vowel behind the hand”.
To execute the open mouth hum place the back of your hand (or the palm) over the slighly opened mouth and sing the vowel.
The idea came from G.B. Lamperti who said “do not hum but sing as though you are humming with your mouth open”. It is interpreted differently by different vocal pedagogues, and it seems that only Coffin adopted the “hand over mouth” approach.
Here is what he has to say about the exercise:
When vowels are sung againsts the palm or back of the hand with the mouth open, the soft palate is forced to open to the lower position. In this position the pillars of the fauces and the soft palate and uvula can be trained to move upward and downward, and the air stream can be taught to divide through both the nose and mouth in singing when necessary. This opening facilitates the high notes… Hold hand over the mouth long enough to hear and feel the high vibration in the turbinates then remove the hand, keeping that memory of vibration of the soft palate.”
– Berton Coffin in The Overtones of Bel Canto (p. 37)
Coffin would instruct to begin the sound with a small puff of air through the nose. Which helps place the voice where it needs to be. He instructed to “fly” the notes, that is use a great deal of air flow to keep the vowel going (especially in falsetto).
Coffin recommended to practice open mouth hum frequently. It is great was to strengthen overtones of the voice. It also helps to take off the dead weight in the passaggio area.
Open mouth hum is used in both falsetto and open vowel exercises.