Ahmad won 34 of the 54 votes in the election, which was held in Ethiopia's capital Addis Ababa on Thursday.
The result means a change in eadership for the first time since Cameroonian Hayatou took charge in 1988.
Ahmad, who appeared very emotional, becomes only the seventh Caf president in the body's 60-year history.
The result was greeted by cheers at the Caf congress as Ahmad was carried on the shoulders of jubilant supporters to the podium after the result was announced.
The 57-year-old said: "When you try to do something, you mean that you can do it. If I can't do it, I never stand."
The departure of Hayatou is a huge change for African football and the 70-year-old will also lose his Fifa position and his place on the ruling council of world football's governing body.
He had been challenged for the Caf presidency only twice before and both times won with landslide victories.
This time he won only 20 votes, ending his hopes of winning an eighth term in office that would have seen his presidency extend for more than three decades in total.
"His excellency Issa Hayatou has done a lot for African football (but) it is time for him to step back," said George Afriyie, vice president of the Ghana Football Association.
Hayatou was led from the auditorium by aides who brushed off requests for comment from journalists.
"Africa has made an emphatic decision that we are ready for change," said Liberian Football Association President Musa Bility.