NLP-based Detection of Mathematics Subject Classification

Yihe Dong
Wolfram Research

Task

Automatically detect Mathematics Subject Classification (MSC), given text.

Examples

Alphanumeric combinations e.g. 14D24, 34L20.

Classes follow a hierarchy: natural groups according to first two and three digits.

14H: curves in algebraic geometry.

14J: surfaces in algebraic geometry.

Data

161471 arXiv papers with MSC labels, spanning 4575 classes.

Each paper contains multiple classes.

Challenges

4575 classes: large number of classes, comparably small number of papers.

Imbalanced class representation: example: 53D (symplectic geometry) appear more often than 70F (dynamics of a system of particles) by an order of magnitude.

Overlapping classes: Many classes share overlapping content. Example, both 55R (algebraic topology) and 18F (category theory) pertain to fiber bundles.

Hybrid Approach

Neural networks (NN): predict the three-digit prefixes of the most likely classes.

Necessary for learning general characteristics of classes.

K-nearest neighbors (KNN): predict the five-digit MSC starting with these top prefixes.

Necessary, as insufficient data and overlapping classes make models trained on five-digit classes inaccurate.

K-nearest neighbors (KNN)

Create vector representation of each document, use the MSC of the nearest document vectors.

K-nearest neighbors (KNN)

Terms-based vector representation

23852 vocabulary word representatives. Carefully selected based on frequency analysis.

Both singleton words and n-grams, e.g. “cyclotomic field.”

Word uniformization: singularization, stemming, etc. E.g.:

annihilate, annihilator, annihilating -> annihilat.

matrices -> matrix.

Document vector created by filling in weighted scores at indices corresponding to terms.

Latent semantic analysis

Truncated SVD:

Dimension reduction.

Consolidate terms that share semantic similarity.

Project from the 23852-dimensional feature space down to 150 dimensions.

M weighted, normalized, term-document matrix, with decomposition M->UD(V^T)

Dimensions: M: 23852 × 17027; U: 23852 × 150; (V^T): 150 × 17027; D: 150 x 150.

Given 23852-dimensional query vector v, project it down to 150 dimensions with (OverHat[v])==(D^(-1))(U^T)v, and look for the nearest column vectors in corpus to (OverHat[v])

K-nearest neighbors (KNN)

Terms-based vector representation

23852 vocabulary word representatives. Carefully selected based on frequency analysis.

Both singleton words and n-grams, e.g. “cyclotomic field.”

Word uniformization: singularization, stemming, etc. E.g.:

annihilate, annihilator, annihilating -> annihilat.

matrices -> matrix.

Document vector created by filling in weighted scores at indices corresponding to terms.

Latent semantic analysis

Truncated SVD:

Dimension reduction.

Consolidate terms that share semantic similarity.

Project from the 23852-dimensional feature space down to 150 dimensions.

M weighted, normalized, term-document matrix, with decomposition M->UD(V^T)

Dimensions: M: 23852 × 17027; U: 23852 × 150; (V^T): 150 × 17027; D: 150 x 150.

Given 23852-dimensional query vector v, project it down to 150 dimensions with (OverHat[v])==(D^(-1))(U^T)v, and look for the nearest column vectors in corpus to (OverHat[v])

(stackoverflow.org)

Neural networks

Two networks: two digit + three digit networks.

Aim for net simplicity to be faster, and less prone to overfitting

Why not:

RNN, CNN, or SVM?

Neural networks

Data preprocessing

Same vocabulary processing as in KNN.

Oversample less-represented classes to ameliorate class imbalance.

Remove few classes that have underwhelming representation.

Net architecture

Trainable embedding layer. Feature dimension 256 (128) for three-digit (two-digit) net.

Input: length-n sequence of word indices

Output: n x 256 (128) matrix.

Average pooling layer

Input: n x 256 (128) matrix.

Output: 256 (128) dimensional vector.

Linear layer

Input: 256 (128) dimensional vector.

Output: number of classes. 622 (68) for three-digit (two-digit) net.

Training

Loss function: Mean squared error.

optimizer: Adam.

Results

Recall rate (ratio between true positives and all actual positives) on 1000 test documents, each predicting 8 classes:

0.88 for five-digit classification.

0.90 for three-digit classification.

0.93 for the two-digit classification.

(wikipedia.org)

Questions?

Suggestions for improvements?