Yohane Banda said the authorities had not made it clear that he was giving up David for good when he signed adoption papers earlier this month.
Madonna and her husband, Guy Ritchie, were granted a temporary order earlier this month, allowing them to take the baby away.
Mr Banda has previously defended the adoption, saying it was in the best interests of his motherless son, and criticised charities who are mounting a legal challenge.
But today he told reporters of his doubts. Speaking from Lipunga village, where he grows onions and tomatoes, Mr Banda said: "Our understanding was that they would educate and take care of our son just as they were doing at the orphanage."
"
He said that because he cannot read, he had not realised the significance of the court papers he was signing.
Mr Banda had left his son at an orphanage because his wife died shortly after childbirth. He lost two other children to malaria.
Madonna's Malawian lawyer, Alan Chinula, declined to comment but insisted the singer had followed all the correct procedures.
