Abstract: The D-meson production in high energy proton-proton(p-p) and proton-nucleus(p-A) collisions is an important approach to study the character of saturation gluon. By considering the strong coupling effects at leading order, the D meson production in high energy p-p and p-A collisions at LHC energies is studied in the framework of Color Glass Condensate(CGC). Using the dipole amplitude obtained from the KLR-AdS/CFT color dipole model through Fourier transform, and taking into account the cold nuclear matter effects, we investigate the D meson cross section in proton-lead (p-Pb) collisions at \sqrts\!=\!5.02 TeV in different centrality class. Then, we calculate the nuclear modification factor for D meson production and semi-leptonic decay processes in p-Pb collisions at \sqrts\!=\!8.16 TeV. It is found that the theoretical results considered the strong coupling effects fit better with the latest experimental data from ALICE and LHCb Collaboration than those obtained without considering the strong coupling effects. Finally, we also predict the results for the experiments at \sqrts\!=\!8.16 TeV of the Large Hadron Collider(LHC). It is shown that the theoretical results at \sqrts\!=\!8.16 TeV are larger than those results at \sqrts\!=\!5.02 TeV.