The man, said to be dressed as a doctor, began shooting at the 972-bed Bronx-Lebanon Hospital, reports say.
At least three people are believed to have been shot.
Messages on social media spoke of doctors and nurses barricading themselves inside the building.
The New York Times quoted a fire department official as saying that three doctors had been shot. Their condition is not known.
Some reports say the gunman may have been a former employee but this has not been confirmed.
The 120-year-old hospital is in the Morris Heights district of the Bronx.
SOURCES: CNN, BBC, NYT