For the problem of offset detection of dies on matrix test trays, a vision-based die offset detection system is built, which mainly includes loading and off-loading channels, a 3D moving platform, and a vision detection module. This system takes a standard sample as the benchmark and builds an image correction algorithm combining template matching and affine transformation to achieve the generality between different inspections. A rectangular measurement algorithm for a specific area is designed, based on the grey scale gradient and the direction of the gradient; the system calculates the die offset using center coordinates of the line which connecting the top left and bottom right points of the die as the reference point. The experimental results show as follows: The offset detection error range of this system is -2. 145 to 4. 257 μm, the average time of algorithm execution is 72. 56 ms, and the operation speed is 20 mm/ s, and the average detection time for chips on matrix test trays with 5 × 12 is 64. 5 s/ tray, which can meet the demand of the actual processing process; the accuracy of this system is tested using the bias and linearity analysis method in measurement system analysis (MSA), the linearity of bias is significant. To conclude, the bias and linearity of the system meet the requirements.