The brothers were born into a family of paper manufacturers in Annonay, in Ardèche, France. Their parents were Pierre Montgolfier (1700–1793), and his wife Anne Duret (1701–1760), who had sixteen children. Pierre established his eldest son Raymond Montgolfier, later Raymond de Montgolfier (1730–1792) as his successor.
Joseph, the 12th child, possessed a typical inventor's temperament—a maverick and dreamer, and impractical in terms of business and personal affairs. Étienne had a much more even and businesslike temperament. As the 15th child, and particularly troublesome to his elder siblings, he was sent to Paris to train as an architect. However, after the sudden and unexpected death of Raymond in 1772, he was recalled to Annonay to run the family business. In the subsequent 10 years, Étienne applied his talent for technical innovation to the family business; papermaking was a high-tech industry in the 18th century. He succeeded in incorporating the latest Dutch innovations of the day into the family mills. His work led to recognition by the government of France as well as the awarding of a government grant to establish the Montgolfier factory as a model for other French papermakers.
Of the two brothers, it was Joseph who first contemplated building machines. Gillispie puts it as early as 1777 when Joseph observed laundry drying over a fire incidentally form pockets that billowed upwards. Joseph made his first definitive experiments in November of 1782 while living in the city of Avignon. He reported, some years later, that he was watching a fire one evening while contemplating one of the great military issues of the day—an assault on the fortress of Gibraltar, which had proved impregnable by both sea and land. Joseph mused on the possibility of an air assault using troops lifted by the same force that was lifting the embers from the fire. He believed that contained within the smoke was a special gas, which he called 'Montgolfier Gas', with a special property he called levity.
As a result of these musings, Joseph set about building a box-like chamber 1x1x1.3m (3 ft by 3 ft (0.91 m) by 4 ft) out of very thin wood and covering the sides and top with lightweight taffeta cloth. Under the bottom of the box he crumpled and lit some paper. The contraption quickly lifted off its stand and collided with the ceiling. Joseph then recruited his brother to balloon building by writing the prophetic words "Get in a supply of taffeta and of cordage, quickly, and you will see one of the most astonishing sights in the world" The two brothers then set about building a contraption 3 times larger in scale (27 times larger in volume). The lifting force was so great that they lost control of their craft on its very first test flight on 14 December 1782. The device floated nearly 2 kilometres (about 1.2 mi). It was destroyed after landing by the "indiscretion" of passersby.