A photonic crystal fiber (PCF) with circular air holes in the fiber cladding and elliptical air holes in the fiber core is proposed. According to calculation, both ultrahigh birefringence (larger than 0.01) and ultralow confinement loss (less than 0.001dB/km) can be achieved simultaneously over a large wavelength range for a PCF with only four rings of circular air holes in the fiber cladding. The confinement loss in this PCF can be effectively reduced while the birefringence almost remains the same. The proposed design of the PCF is a solution to the tradeoff between the birefringence and the confinement loss for the originally reported highly birefringent elliptical-hole PCF. Moreover, an approach to modify the effective index of fiber core is also suggested in this letter