Israel's Benjamin Netanyahu - who earlier in the day encouraged French Jews to emigrate to Israel - and Palestinian President Mahmoud Abbas were also present and walked just a few steps from one another.
"In the same way that the civilised world stood today with France against terror, so it must stand with Israel against terror," Netanyahu said at a ceremony in a Paris synagogue.
After world leaders left the march, Hollande stayed to greet survivors of the Charlie Hebdo attack and their families, while hundreds of thousands of people marched slowly and in near-total silence through Paris streets.
"We're not going to let a little gang of hoodlums run our lives," said Fanny Appelbaum, 75, who said she lost two sisters and a brother in the Nazi concentration camp at Auschwitz. "Today, we are all one."
Zakaria Moumni, a 34-year-old Franco-Moroccan draped in the French flag, agreed: "I am here to show the terrorists they have not won - it is bringing people together of all religions."
The attacks have raised difficult questions of free speech, religion and security, and exposed the vulnerability of states to urban attacks.
The head of France's 550,000-strong Jewish community, Roger Cukierman, said Hollande had promised that Jewish schools and synagogues would have extra protection, by the army if necessary, after the killings. He also called for limits on hate speech and more control on suspected jihadists.
Hours before the march, a video emerged featuring a man resembling the gunman killed in the kosher deli. He pledged allegiance to the Islamic State insurgent group and urged French Muslims to follow his example.