Chess move recognition with AI

Electronic boards that can record a chess game automatically already exist, but they're expensive: in Brazil, the cheapest model costs over R$ 4,000 (roughly USD 700). My thesis for my Telecommunications Engineering degree at IFSC came out of that gap: could a phone camera, computer vision, and a neural network trained to recognize chess pieces get a similar result?

The goal

The original idea was a full mobile app, filming the game in real time. After running into the complexity of making that work reliably, I narrowed the scope to a more direct proof of concept: given a sequence of photos of the board, one per move, the system returns the entire game in algebraic notation, the same notation used in chess books and websites (1. e4 e5 2. Nf3 Nc6 ...).

How it works

Every move is treated as a comparison between two board states, before and after. For each photo, the system locates the board and maps its 64 squares, uses a YOLOv11 model (trained on an image set I labeled myself) to detect each piece's position, and compares that state against the previous photo's to figure out which move connects the two. The python-chess library tests every legal move until it finds the one that produces exactly the new state. This also works as validation: if the detected position doesn't match any legal move, that signals a detection error, and the system asks for a new photo instead of reporting an incorrect move.

At the end of the game, every move gets concatenated into a single report:

Results

The model reached 99.1% mAP on the test set, but with an important caveat: the entire dataset was captured with a single physical board, under the same lighting and camera setup. Generalizing to any board would require a much more diverse dataset.

What's out of scope

This project is still a proof of concept, not a product: it works with photos taken manually between moves, not continuous video, and it assumes the camera is positioned close to 90° relative to the board. Turning this into an actual app would mean solving real-time detection and tolerance for much more varied camera angles.

The complete code is in the repository linked below, along with the full thesis.