We consider a stochastic filtering problem for a partially observed modulated Hawkes process with an application to the High Frequency Trading (HFT) Problem. In this thesis, HFT problem is emphasized on the optimization with the price impact. The modulated Hawkes process consists three parts: marked point process, intensity process and baseline process where intensity process is driven by. We model the marked point process as the incoming of market orders and intensity as trade fill rate. In addition, we propose the trade intensity with different base states defined as the baseline process under the Hawkes process setting.

In HFT problem, we model a price impact function consisting of midprice and trade intensity where midprice is driven by market orders (marked point process) and information flow (an independent wiener process). Moreover, we define a value function to optimize the expected price impact function at an optimal stopping time within a fixed period of time. This is essentially the optimal stopping time (OST) problem. Under our problem formulation, the trade intensity is only partially observed through market orders. Hence, we conduct the conditional distribution of the trading intensity process using stochastic particle filtering (SPF) method. In addition, we propose an integrated Advanced Rao-Blackwell (ARB) algorithm to improve the current SPF algorithms. With ARB, we can filter the posterior distribution of trade intensity under the information filtration. In order to optimize the value function, we use this posterior filter for intensity and apply Snell envelope and dynamic programming principle to discretize the time for the objective function. Last, we employ Least-Square Monte Carlo to approximate the conditional expectation of the price impact function so we can resolve the OST problem.