Crop load and production estimation is a crucial aspect for modern farms during the whole growing period. This work presents a fruit recognition and tracking system to allow precise counting of fruits in real-time while traversing the orchard doing the appropriate farming tasks. A field test campaign was performed on Fuji apples to assess the effectiveness of a proposed solution for counting apples at near harvest growth stage. The results showed that the solution was able to correctly keep the counting error under 10%. Future improvements will be focused on multiple objects tracking performances and object detection model refinements to improve the solution effectiveness.