Neurosymbolic AI for handwritten formulas
Reading a sequence of handwritten digits and operators and computing the result, comparing logic built into the model with logic added to the loss.
- Kind
- Research
- Context
- Learning and Reasoning course, KU Leuven
- Result
- With five images, a plain neural network falls to 7.1% accuracy; putting the logic in the architecture keeps it at 98.7%.
- Built with
- Python, PyTorch Lightning, Probabilistic logic programming, CNNs
- Status
The task
The input is a row of handwritten images of digits and operators, such as an image of a 3, a plus sign and a 5. The model has to output the value of the formula.
The only supervision is the final result of each formula. No model ever sees a label for an individual digit or operator.
Three models
- Neural baseline. A CNN that predicts the result directly from the images.
- Logic in the architecture. Separate networks recognise digits and operators, and a probabilistic logic program combines their outputs into a result. The logic is compiled once into AND-OR trees, so training backpropagates straight through the reasoning.
- Logic in the loss. A network predicts the result directly, with a logic-consistency penalty added during training only. At inference it is a plain neural network.
Result
Longer formulas break the plain neural network: it has to learn every combination from examples, and there are too many. Putting the rules of arithmetic in the architecture means the networks only need to recognise single symbols, so accuracy stays near-perfect as formulas get longer. Adding the rules to the loss helps on short formulas but does not carry over.
How much data each approach needs
With 3 images and only 300 training formulas, logic in the architecture already reaches 70.6%, against 9.1% for the plain network.
| Training formulas | 300 | 3,000 | 30,000 |
|---|---|---|---|
| Neural baseline | 9.1% | 22.6% | 86.0% |
| Logic in the architecture | 70.6% | 94.2% | 97.8% |
| Logic in the loss | 18.3% | 82.8% | 96.5% |
Mean of 3 runs.
A new task
I also designed a task from scratch: arithmetic modulo 10, with multiplication, division and correct operator precedence (1 + 2 × 3 = 7), written as a new logic program.