Machine learning algorithms, such as neural networks, are rapidly gaining popularity as tools for research in pure mathematics. In one application, a classifier is trained on a set of multiplication tables with the goal of detecting property P, where P may be an algebraic property such as commutativity or associativity. We present evidence that relatively small neural networks can solve these problems with high accuracy. We also analyze these models by varying the structure of the training data and show that they likely achieve this accuracy by detecting consequences of property P, rather than property P itself.