Hcfyk

Before the widespread use of firearms, various kinds of plate armour were in widespread use. As muskets (and later, rifles) became widespread, full plate became less effective and fell out of use.

However, mortars and (later) cannon did exist, firing exploding shells. Unit formations would be changed depending on whether the enemy had artillery or not, because a tightly-packed formation would suffer greater casualties from a well-placed shell. Shrapnel was very much a feature of battles (named after Henry Shrapnel, of course).

With this in mind, why was body armour so rarely used? Even the most basic armour would have reduced casualties. More curiously still, effective steel helmets were replaced with caps or leather helmets such as shakos which are more decorative than protective. Steel was still relatively expensive, but not unreasonably so, and wrought iron plate was in mass production.

At the outbreak of WWI, the high number of casualties with head wounds led to swift adoption of helmets such as the Adrian helmet. Other body armour was tried as well, with varying degrees of success, but helmets were a clear win. The only innovative aspect of WWI helmets was that they were pressed from a single steel sheet which made them cheaper to manufacture. As far as their basic design went, they were functionally indistinguishable from helmets of the Middle Ages.

Failing to equip your troops to keep them alive could perhaps be understood if combat effectiveness was allowed to degrade during peacetime. However in a period covering the Napoleonic Wars, the American Civil War, and the maintenance of empires around the globe by the various European powers, this clearly is not the case.

Why did it take 100+ years for the value of effective helmets and body armour to be recognised? And how could the combined militaries of the world forget that it existed for 100+ years?

Updated following comment by Steve Bird: I've seen this question already, and I don't think it answers my question. As answers to that question say, the issue with shields is that using them effectively takes serious training. The same applies for fighting in full plate armour too - whilst I'm not a HEMA player, I'm sure it takes some practise to ignore sword strikes to your extremities and trust in your armour. This is not the case for helmets and light breastplates though - they're simple to put on, they give decent passive protection, and apart from getting used to the weight they don't affect anything else.

Also on another comment by Steve Bird, the scope may seem to be wide, but in fact it applies through the entire 19th century. Whilst the guns available are radically different, from muskets and smooth-bore cannons at the start of the century to recognisably modern rifles and artillery at the end of the century, the uniforms worn by soldiers were unchanged throughout. Headgear in particular is clearly a fashion statement and is not designed for personal protection. I'm curious why that's the case when keeping yourself alive would seem to be more of a priority - and even if keeping your troops alive is not your top priority, why anyone would start outfitting them with shakos or pith helmets (which do cost money) instead of a simple helmet.

Looking at this slightly backwards, you could ask what are the factors that have enabled the supply of body armor to the modern infantry soldier?

Modern ballistic materials, such as kevlar, are comparatively lightweight and flexible, so armor can be manufactured in a small range of sizes and self-tailored (with straps, velcro, elastic, etc) to fit and so does little to interfere with the soldiers ability to fight. The cost of training a modern soldier is high and there is also a public expectation that the government will protect them as well as they can be protected.

A modern soldier is transported to the battlefield (by ship/plane/vehicle or combination of those) and is driven around the battlefield in an APC or flown in a helicopter, so the extra weight of their armor not a long-term factor. Modern battlefield first-aid and medical care is such that even a badly wounded soldier stands a good chance of survival, so preventing an immediately fatal wound occuring in the first place also improves the soldiers chances of returning to service.

In comparison, a rigid metal breastplate/backplate is far less practical. Getting an exact fit for each soldier would be much harder and tailoring each to its owner would be more expensive. Potentially you might be able to manufacture an adjustable metal armor but that would drive up the cost (and probably the weight too). Without a good fit it would be uncomfortable to wear and might actually interfere with soldier's movement and, therefore, their ability to fight. As the campaign progressed, the soldiers would almost certainly lose weight, which would mean that a breastplate that was a good fit when they left home becomes a loose fit after a few months in the field.

Also the 19th Century infantry soldier was literally a foot soldier, they marched to and from the battlefield (in some cases over hundreds of miles). The soldier had to carry everything they owned, their weapon(s), ammunition, uniform(s), cooking utensils, blankets, etc. Including a set of rigid armor would add both weight and encumbrance. They wouldn't want to wear the armor the whole time so it would have to be slung from the backpack in some manner.

So would there be an actual benefit to the 19th Century soldier of having this extra armor?

Studies of casualties during the Napoleonic campaigns have shown that only about 30% of the French army's fatalities at the time were from battlefield injuries. The biggest dangers to the soldier were disease and malnutrition. Body armor would provide no protection against the first and its added burden would accelerate the second.

This situation didn't change much through the century, two-thirds of the fatalities in the American Civil war were due to disease. There were similar figures for the Crimean War:

It is worth noting that the majority of deaths were not a result of combat but rather that of disease. For example, 2,755 of the British Empire’s force were killed in action compared to the 17,580 who died of disease.

On the battlefield, the majority of wounds would have been from direct fire weapons; muskets, rifles, solid shot (cannonballs, grape and cannister shot). For example, from an analysis of the injuries at the Battle of Waterloo:

There were, over the course of the four-day campaign, around 100,000 casualties to care for. About 60% of wounds were caused by small-arms from low-energy transfer injuries fired by smooth bore muzzle-loading fusils, carbines and pistols...Wounds from heavy (3–12lbs) iron round shot were usually fatal if received centrally or on the head and neck. Many limbs were avulsed by round shot. Tangential strikes by these large balls could cause severe tissue internal disruption.

There were similar figures for the American Civil war, where small arms accounted for 51.8% of battle injuries, with cannon an additional 5.7%. It's probably worth noting that bladed weapons, such as sabers and bayonets (which body armor might have provided protection against) accounted for less than one percent of these battlefield injuries.

It would have been possible to manfacture armor plate that could withstand musket and early rifle rounds but these would require thick plating that would have been unreasonably heavy for an infantry soldier. No practical armor plate would be able to stop cannon-fired shot. By comparison to the modern battlefield, in the early part of the century there was far less shrapnel (shell fragments) flying about. Early in the century there were field howitzers firing explosive shells but these were comparatively rare and their method of fusing meant that many of the fired shells were ineffective. By the end of the century, guns firing high-explosive shells were the most common form of artillery. The shrapnel from these shells would be travelling at similar velocities to rifle rounds so the armor required to stop those would be prohibitively heavy.

If a soldier was wounded, the lack of effective medical care on the battlefield and the lack of hygiene in first aid in general would mean that a soldiers chances of a return to service were low. Even a minor wound could become infected and cause the loss of a limb or even death.

...while musket balls, grape shot and sabre slashes could be lethal enough, but even superficial wounds could be fatal. Field hospitals were notoriously unhygienic and without antibiotics common infections were deadly.
^source

So armor that prevented a fatal chest wound might only mean that the soldier ends up dying a slower, more painful death in a field hospital. From a brutal economic point of view, it was more cost effective to have the men die on the battlefield than it was to treat the wounded and have them die later.

So if you were in charge of 19th Century army and you wanted to improve survival rates for your men, it would be far more productive to spend your money on additional food and improving heath care than on body armor.

Quality of steel was not sufficient enough to be practicable for body armor and helmets

Historically speaking, use of armor, shields and helmets declined with advance of firearms. During the Napoleonic era, they were almost completely abandoned, except in heavy cavalry units that used them to protect themselves from cold weapons (swords, sabres, spears ...) not from firearms. Helmets and to a lesser extent body armor reappeared in WW1 and latter in WW2. Helmets did offer protection against shrapnel and glancing bullets, but body armor (like for example this Soviet body armor) was only issued to specialized units (combat engineers mostly) do to it being to cumbersome and unhandy .

Now, as you can see, body armor mentioned above protected only breast, and only from submachine gun ammunition, yet it weighted 3.5 kg . Mind you, this happened in 1940's with already well developed steel technology. But in 19th century, before Bessemer process, steel was produced only sporadically with much lesser quality then after that. Contrary to that, sheer energy of muskets is comparable to modern firearms like Ak-47 and M-16 . Muskets are of course much less precise then modern weapons, but to compensate for that they were usually fired en masse and from closer range. As a consequence of that, effective body armor would have to be at least 5-6 mm thick, with corresponding weight increase, to give protection for relatively limited area.

As for helmets, note that at that time and place most of incoming fire was direct, unlike WW1 and latter. This includes artillery fire (canister shoot and cannonballs filled with gunpowder). Helmets usually protect only forehead from direct fire, and rest is dedicated to protection from overhead shrapnel and flying debris. Military caps from that area did offer some protection from direct fire, but they were mostly dedicated to stopping saber and sword cuts from above (like this bearskin capof Napoleonic Imperial Guard) . Shrapnel only became major source of combat casualties with advent of static trench warfare in WW1 . At that time, quality of steel improved sufficiently that both sides could issue various types of helmets relatively effective for stopping it .

Like you wrote in your question, there was a period of time when bullets and shrapnel would pierce right through armor, so there was little point in putting bulky armor on.

In addition, armor was expensive, and not easy to put on and use. This required training. By contrast, the introduction of firearms made it so that, rather than extensively train a unit of pikemen, you could give rudimentary training to new foot soldiers and send them straight to battle. So from a cost and training perspective it made little sense.

There was also a twist you might have missed which occurred when hard helmets were re-introduced during WW1: the leather strap that kept them attached were too solid. When something blew up too closely, the helmet could get caught in the pressure wave and snap your neck or worse, and that is (part of) why they were often worn unbuckled.