Nanoparticle applications in the petroleum industry have grown recently especially in EOR and well stimulation. Transport of nanoparticles in porous media involves different interaction mechanisms either between nanoparticles and porous media (adsorption and release from pore surface, single-particle plugging, and multi-particles plugging) or inter-nanoparticle (aggregation and gelation). This dissertation aims to provide mathematical and numerical framework for multiscale modeling of nanoparticles transport in porous media including all these different interaction mechanisms for the first time. First, mechanistic model based on Extended DLVO theory is developed to study the rate of deposition and release of nanoparticles in porous media at different temperature, ionic strength, and pH. Empirical equation has been derived to calculate zeta potential at different conditions. The effect of surface roughness has been included in the model using the effective height and density of the surface roughness distribution. Numerical model has been used to compare the theoretically calculated rates with several experimental data. The model shows good fitting with different experimental results. Secondly, although the nanoparticles are small, they can be retained in the porous media by three different damage mechanisms i.e. surface deposition, mono-particle plugging, and multi-particles plugging. We have developed a mathematical model that captures these different damage mechanisms. The model is validated with experimental data to obtain the model parameters. Sensitivity analysis is presented using the proposed numerical model. The preliminary numerical results demonstrate that nanoparticle size, concentration, injection rate and permeability are the dominant factors that control the degree of formation damage. Thirdly, Population Balance Equation (PBE) is used to model the kinetics of aggregation and gelation of nanoparticles in batch. Quadrature Method of Moments (QMOM) is used to convert the PBE with continuous distribution of nanoparticle size into a set moment equations for efficient computation. The modeled developed in this study is used to compare between the kinetics of aggregation and gelation of fumed silica and colloidal silica nanoparticles at the same conditions. The case studies presented show a unique behavior of fumed silica over colloidal silica nanoparticles for forming a gel network at significantly low concentration. Finally, QMOM-PBE is coupled with fluid flow model to predict the nanoparticles aggregation and interactions in porous media. Changes in nanoparticle size and shape due to inter–particle interactions (i.e. aggregation) may significantly affect particle mobility and retention in porous media. Model sensitivity analysis shows the influence of particle concentration and interstitial velocity gradient on inter–particle, and, consequently, particle–collector interactions. Also, filtration prevents the buildup of viscosity of the fluid in porous media compared to batch. However, in free media where filtration is negligible like fractures, viscous fluid or soft gel can be formed. This dissertation attempts to answer the critical questions pertaining the coupling of aggregation and in situ gelation on the nanoparticles transport in porous media. The model can be further extended to simulate different nanoparticles-based applications in oil and gas reservoirs such as acid diversion, water shutoff, conformance control, and hydraulic fracturing.