Feldman was born in Brooklyn, New York City into a family of Russian-Jewish immigrants from Kiev. His father was a manufacturer of children's coats. As a child he studied piano with Vera Maurina Press, who, according to the composer himself, instilled in him a "vibrant musicality rather than musicianship." Feldman's first composition teachers were Wallingford Riegger, one of the first American followers of Arnold Schoenberg, and Stefan Wolpe, a German-born Jewish composer who studied under Franz Schreker and Anton Webern. Feldman and Wolpe spent most of their time simply talking about music and art.
In early 1950 Feldman went to hear the New York Philharmonic give a performance of Anton Webern's Symphony, op. 21. After this work, the orchestra was going to perform a piece by Sergei Rachmaninoff, and Feldman left immediately before that, disturbed by the audience's disrespectful reaction to Webern's work. In the lobby he met John Cage, who was at the concert and had also decided to step out. The two composers quickly became good friends, with Feldman moving into the apartment on the second floor of the building Cage lived in. Through Cage, he met sculptor Richard Lippold (who had a studio next door) and artists Sonia Sekula, Robert Rauschenberg, and others, and composers such as Henry Cowell, Virgil Thomson, and George Antheil.
With encouragement from Cage, Feldman began to write pieces that had no relation to compositional systems of the past, such as the constraints of traditional harmony or the serial technique. He experimented with non-standard systems of musical notation, often using grids in his scores, and specifying how many notes should be played at a certain time, but not which ones. Feldman's experiments with the use of chance in his composition in turn inspired John Cage to write pieces like the Music of Changes, where the notes to be played are determined by consulting the I Ching.
Feldman was commissioned to compose the score for Jack Garfein's 1961 film, Something Wild. However, after hearing the music for the opening scene, in which a character (played by Carroll Baker, incidentally also Garfein's wife) is raped, the director promptly withdrew his commission, opting to enlist Aaron Copland instead. The reaction of the startled director was said to be, "My wife is being raped and you write celesta music?"
Through Cage, Feldman met many other prominent figures in the New York arts scene, among them Jackson Pollock, Philip Guston and Frank O'Hara. He found inspiration in the paintings of the abstract expressionists, and throughout the 1970s wrote a number of pieces around twenty-minutes in length, including Rothko Chapel (1971, written for the building of the same name, which houses paintings by Mark Rothko) and For Frank O'Hara (1973). In 1977, he wrote the opera Neither with words by Samuel Beckett.
In 1973, at the age of 47, Feldman became the Edgard Varèse Professor (a title of his own devising) at the University at Buffalo. Prior to that time, Feldman had earned his living as a full-time employee at the family textile business in New York's garment district.
Later, he began to produce his very long works, often in one continuous movement, rarely shorter than half an hour in length and often much longer. These works include Violin and String Quartet (1985, around 2 hours), For Philip Guston (1984, around four hours) and, most extreme, the String Quartet II (1983, which is over six hours long without a break.) Typically, these pieces maintain a very slow developmental pace (if not static) and tend to be made up of mostly very quiet sounds. Feldman said himself that quiet sounds had begun to be the only ones that interested him. In a 1982 lecture, Feldman noted: "Do we have anything in music for example that really wipes everything out? That just cleans everything away?"
Feldman married the Canadian composer Barbara Monk shortly before his death. He died from pancreatic cancer in 1987 at his home in Buffalo, New York, after fighting for his life for three months.