Colonel E. Elmer Ellsworth was a well-known militia commander before the Civil War and a friend of Abraham Lincoln. Following the attack on Fort Sumter, Ellsworth quickly organized a regiment of New York City firemen as the First New York Zouaves (later the 11th New York Volunteers). After arriving in Washington, D.C., the regiment was dispatched to Alexandria, Virginia and, noticing a Confederate flag flying above the Marshall House hotel, Ellsworth and a few of his men climbed to the roof and removed the offending flag. As they descended the hotel staircase, Ellsworth was confronted and shot by the hotel keeper, who was immediately killed by Ellsworth's men. Colonel Ellsworth was the first Union officer to be killed in the war, and his death triggered an outpouring of grief in the North. Ellsworth had been born in 1837 near Albany, New York, and a call went out from a number of the prominent citizens of that city to avenge his death by raising a regiment made up of one man from each town and ward in the Empire State. Each soldier was to be unmarried, not less than 5' 8" in height and not more than thirty years old. The men selected for the regiment arrived in Albany from the four corners of New York State on August 8, 1861, and the regiment was quickly formed. The regiment joined the Army of the Potomac in late October 1861 in Virginia, and settled into camp with the assistance of the 83rd Pennsylvania Volunteer Infantry Regiment, beginning a relationship that would keep these two regiments together throughout the war. In the spring of 1862, the 44th New York Infantry participated in the Peninsula campaign, suffering significant casualties during the battles of the Seven Days including Hanover Court House, Gaines' Mill and Malvern Hill. Several men were captured at Gaines' Mill, including Private Julian Knowlton of Company A, who later was paroled.
    The 44th New York Infantry continued to see action with the Army of the Potomac during the remainder of 1862, again experiencing heavy losses at the battles of Second Bull Run and Fredericksburg, fighting side by side with the 83rd Pennsylvania. The regiment avoided the bloodbath at Antietam when it was held in reserve, and the 44th New York Infantry also suffered only minor casualties at Chancellorsville.
    The Ellsworth Avengers made their most important contribution to the Union cause during the battle of Gettysburg. The left of the Union line on July 2, 1863 ended just short of a low hill known locally as Little Round Top. If the Rebels could control Little Round Top, they would be able to enfilade the Union line with artillery and win the battle. Confederate troops under General John Bell Hood realized this and deviated from an attack on the Union left itself (which had been ordered earlier in the day by General Robert E. Lee) to try instead to take the hill. At about the same time, Union General Gouverneur K. Warren while on patrol discovered that Little Round Top was unoccupied but for a signal station and, also recognizing the importance of the position, he quickly dispatched messengers to find Union troops to seize the prominence. Descending Little Round Top, one of the messengers met the Union Fifth Corps, and Colonel Strong Vincent commanding the Third Brigade -- made up of the 16th Michigan, 44th New York, 83rd Pennsylvania and 20th Maine -- quickly comprehended the gravity of the situation and directed his Brigade to charge up Little Round Top on the "double quick."
    The Union troops barely beat the Rebels to the summit, with the 20th Maine holding the left, the 16th Michigan on the right, and the 44th New York and the 83rd Pennsylvania in the center and as usual standing together. The ensuing battle was extremely fierce, with the undermanned Union regiments facing wave after wave of Hood's Alabama, Texas and Georgia troops. The weakened 16th Michigan was in danger of collapsing, but reinforcements in the form of Colonel Patrick O'Rorke's 140th New York Infantry arrived in the nick of time. Finally, almost out of ammunition and seeing the Confederates massing for one more assault, Colonel Joshua Lawrence Chamberlain of the 20th Maine ordered an unlikely bayonet charge that surprised the Rebel attackers and saved the day. The Union victory came at a heavy price, however, with the 44th New York and the other three regiments of the Third Brigade suffering severe losses. Union Colonels Vincent and O'Rorke were both killed. Over 50% of the soldiers in Company A of the 44th New York were killed or wounded, including Private Knowlton who was badly wounded in the knee and spent the remainder of the war in and out of army hospitals.
    The service of the Ellsworth Avengers did not end at Gettysburg, and the regiment participated in General Ulysses S. Grant's 1864 campaign against General Lee, including the battles of the Wilderness, Laurel Hill and Cold Harbor. What remained of the original regiment was mustered out at the end of three years of hard service on October 11, 1864 at Albany, New York. Of the 1061 members of the Ellsworth Avengers who marched to the war from Albany on October 21, 1861, only 184 returned to Albany in 1864.
 
 

 Additional Notes of interest on Elmer Ellsworth supplied by Jim Nesbitt.

The actual story is even more exciting than what you have depicted. It's equally tragic!

 The unit, made up of the Volunteer Firemen from New York City, was gathered up through the efforts of a man who was the Secretary for the Fire Department. He was George Fash NESBITT, also a Brigadier General of the Army of New York and Chief Engineer for the State.

The following is an excerpt from notes on this great-great-grandfather of mine.

"George was the Secretary/Treasurer to the Volunteer Fire Department of the City of New York. His uniform collar pins are in the possession of the author (JFN). As such, he was somewhat responsible for the development and support of the ELLSWORTH Zoaves, which military unit was made up of members of the Volunteer Fire Department of the City of New York.
Colonel ELSWORTH was a militarist of great repute who decided that "the bravest men are those who serve on the New York City Fire Department". It was that choice that created the "Zoaves" which were to become the first military unit entering the American Civil War to lose an officer in combat. Colonel ELLSWORTH was to be the first Union Officer to be killed in that
war. E Elmer ELLSWORTH was killed in action on 24 MAY 1861 at the MARSHALL House Tavern in Alexandria, Virginia. The action in which he lost his life was in response to a direct, personal order from President LINCOLN who had come to regard ELLSWORTH as almost a member of his own family."
I have a photograph of a portion of the flag that Colonel ELLSWORTH caused to be removed from atop the MARSHALL House Tavern. I also have photographs of the uniform and accoutrement of Colonel ELLSWORTH that are on display at the Capitol Rotunda in Albany, New York. The tunic has the hole through which the ball passed which struck Colonel ELLSWORTH squarely in the heart.
Colonel ELLSWORTH was accompanied by his Adjutant to the roof of the MARSHALL House, NOT "by his men". He had gone ahead of "his men" on the expedition to the MARSHALL House because, it is written, he was concerned that, should his men see the flag atop the Tavern, they might raze the building and perhaps even the whole town.
The story is well recorded as reproduced in the Centennial Record (1876) produced by the Town of Mechanicville, New York from whence Colonel ELLSWORTH came. It is where he is buried as well, and I have photographs of the monument located there.

As a parting note. Colonel ELLSWORTH's body was displayed in the Capitol Rotunda in Washington, D.C. where it was viewed by thousands. It was upon this same catafalque that President LINCOLN's body was displayed, only a scant few years later. President LINCOLN mourned ELLSWORTH as if he were his own son. On a personal note, I find it interesting too, that the
NESBITT noted above was one of history's great lithographers and it was his company that produced the Wanted Posters for John Wilkes BOOTH, the slayer of Abraham LINCOLN, his friend

A History of the Forty-Fourth Regiment New York Volunteer Infantry, E. A. Nash (1910); Morningside Books reprint (1988)

 The Attack and Defense of Little Round Top, O. W. Norton (1913); Morningside Books reprint (1983)

New York in the War of the Rebellion 1861 to 1865, F. Phisterer (1912) Official Records of the Union and Confederate Armies
in the War of the Rebellion (1900)

Pension, military and medical files of Julian Knowlton, National Archives