Target detection in the background of sea clutter is an important part of sea surface radar signal processing. The detection of weak targets in sea clutter is based on the statistical characteristics of sea clutter, but the statistical persistence does not reflect the intrinsic dynamics of sea clutter. Therefore, the detection results are not ideal. Based on the chaotic characteristics of sea clutter, this dissertation reconstructs the phase space of the sea clutter, and particle swarm optimization (PSO) is applied to radial basis function(RBF) neural network kernel function parameters. In the optimization study, this method is validated by using IPIX radar to measure sea clutter with target in Dartmouth area. The results show that: PSO-RBF small target detection method has good predictability in the background of chaotic sea clutter. Compared with the general radial basis neural network, the improved algorithm not only has fast convergence speed but also has high recognition rate.