A general approach to optimal nonlinear filtering can be described by a recursive Bayesian approach. The key step in this approach is to determine the probability density function of the state vector conditioned on available measurements. However, an optimal solution to the Bayesian filtering problem can only be obtained exactly for a small class of problems such as linear and Gaussian cases. Therefore, in practice, approximate solutions, such as the extended Kalman filter, have been used.

An optimal nonlinear filtering in a recursive Bayesian approach is a two-step process which consists of the prediction and the update process. In the update process, the priori conditional state probability density function (PDF) from the prediction process is updated through Bayes' rule using measurements from sensors. The prediction of conditional state PDF can be made by solving the Fokker-Planck equation (FPE) that governs the time-evolution the conditional state PDF. However, it is extremely difficult to obtain an analytical solution of the Fokker-Planck equation with the exception of a few special cases. So far this estimation method has not been employed much in practice because of the high computational cost needed in solving the FPE numerically. In this dissertation, methods to improve the efficiency of the numerical method in solving the FPE are investigated to enhance the efficiency of the nonlinear filtering.

Two finite difference methods, namely i) the explicit forward method and ii) the alternating direction implicit (ADI) method, are used to solve the FPE numerically. Although the explicit forward method is much simpler to implement, the ADI method is preferred for its low computational cost. To reduce the computational cost further, as the first contribution of the dissertation, a moving domain scheme is developed to reduce the domain of integration required for solving the Fokker-Planck equation numerically. Simulation results show that the accuracy of the estimation is improved as compared with the Extended Kalman Filter, and at the same time the computational cost is significantly lower with the proposed moving grid scheme than the case without it.

Recently a nonlinear filtering algorithm using a direct quadrature method of moments was proposed, where the associated Fokker-Planck equation is solved efficiently via discrete quadrature based on moment constraints. For some problems, however, this approach showed the phenomenon similar to the "degeneracy'' in a particle filter, which is the concentration of weight on particular particles. The possible cause of the phenomenon is that only the weights are updated through the modified Bayes' rule. Therefore, in this dissertation, as another contribution, a new hybrid filter is proposed where the measurement update equations in the extended or the unscented Kalman filter are used along with the direct quadrature method of moments to solve the FPE. In this way the "degeneracy'' problem can be mitigated.

Then, new proposed filtering methods are applied to several challenging problems such as i) the bearing-only tracking problem, ii) the relative orbit position estimation problem, and iii) the orbit determination problem to demonstrate their advantages. Simulation results indicate that the performance of the proposed filters are better than existing nonlinear filtering methods, such as the Extended Kalman Filter especially with less measurement updates.