From 51136f9be75a9375ccd766ae86bd44fd36d120ff Mon Sep 17 00:00:00 2001 From: Martin Thoma Date: Sun, 29 Jan 2017 15:14:05 +0100 Subject: [PATCH] HASY: Add content --- publications/hasy/content.tex | 432 ++++++++++++++++++++++++++++++++++ 1 file changed, 432 insertions(+) create mode 100644 publications/hasy/content.tex diff --git a/publications/hasy/content.tex b/publications/hasy/content.tex new file mode 100644 index 0000000..8fe7951 --- /dev/null +++ b/publications/hasy/content.tex @@ -0,0 +1,432 @@ +%!TEX root = main.tex +\section{Introduction} +Publicly available datasets have helped the computer vision community to +compare new algorithms and develop applications. Especially +MNIST~\cite{LeNet-5} was used thousands of times to train and evaluate models +for classification. However, even rather simple models consistently get about +$\SI{99.2}{\percent}$ accuracy on MNIST~\cite{TF-MNIST-2016}. The best models +classify everything except for about 20~instances correct. This makes +meaningful statements about improvements in classifiers hard. Possible reason +why current models are so good on MNIST are +\begin{enumerate*} + \item MNIST has only 10~classes + \item there are very few (probably none) labeling errors in MNIST + \item every class has \num{6000}~training samples + \item the feature dimensionality is comparatively low. +\end{enumerate*} +Also, applications which need to recognize only Arabic numerals are rare. + +Similar to MNIST, \dbName{} is of very low resolution. In contrast to MNIST, +the \dbNameVersion~dataset contains \dbTotalClasses~classes, including Arabic +numerals and Latin characters. Furthermore, \dbNameVersion{} has much less +recordings per class than MNIST and is only in black and white whereas +MNIST is in grayscale. + +\dbName{} could be used to train models for semantic segmentation of +non-cursive handwritten documents like mathematical notes or forms. + +\section{Terminology} +A \textit{symbol} is an atomic semantic entity which has exactly one visual +appearance when it is handwritten. Examples of symbols are: +$\alpha, \propto, \cdot, x, \int, \sigma, \dots$ +%\footnote{The first symbol is an \verb+\alpha+, the second one is a \verb+\propto+.} + +While a symbol is a single semantic entity with a given visual appearance, a +glyph is a single typesetting entity. Symbols, glyphs and \LaTeX{} commands do +not relate: + +\begin{itemize} + \item Two different symbols can have the same glyph. For example, the symbols +\verb+\sum+ and \verb+\Sigma+ both render to $\Sigma$, but they have different +semantics and hence they are different symbols. + \item Two different glyphs can correspond to the same semantic entity. An example is +\verb+\varphi+ ($\varphi$) and \verb+\phi+ ($\phi$): Both represent the small +Greek letter \enquote{phi}, but they exist in two different variants. Hence +\verb+\varphi+ and \verb+\phi+ are two different symbols. + \item Examples for different \LaTeX{} commands that represent the same symbol are + \verb+\alpha+ ($\alpha$) and \verb+\upalpha+ ($\upalpha$): Both have the same +semantics and are hand-drawn the same way. This is the case for all \verb+\up+ +variants of Greek letters. +\end{itemize} + +All elements of the data set are called \textit{recordings} in the following. + + +\section{How HASY was created} +\dbName{} is derived from the HWRT dataset which was first used and described +in~\cite{Thoma:2014}. HWRT is an on-line recognition dataset, meaning it does +not contain the handwritten symbols as images, but as point-sequences. Hence +HWRT contains strictly more information than \dbName. The smaller dimension +of each recordings bounding box was scaled to be \SI{32}{\pixel}. The image +was then centered within the $\SI{32}{\pixel} \times \SI{32}{\pixel}$ bounding +box. + +\begin{figure}[h] + \centering + \includegraphics*[width=\linewidth, keepaspectratio]{figures/sample-images.png} + \caption{100 recordings of the \dbNameVersion{} data set.} + \label{fig:100-data-items} +\end{figure} + +HWRT contains exactly the same recordings and classes as \dbName, but \dbName{} +is rendered in order to make it easy to use. + +HWRT and hence \dbName{} is a merged dataset. $\SI{91.93}{\percent}$ of HWRT +were collected by Detexify~\cite{Kirsch,Kirsch2014}. The remaining recordings +were collected by \href{http://write-math.com}{http://write-math.com}. Both +projects aim at helping users to find \LaTeX{} commands in cases where the +users know how to write the symbol, but not the symbols name. The user writes +the symbol on a blank canvas in the browser (either via touch devices or with a +mouse). Then the websites give the Top-$k$ results which the user could have +thought of. The user then clicks on the correct symbol to accept it as the +correct symbol. On \href{http://write-math.com}{write-math.com}, other users +can also suggest which symbol could be the correct one. + +After collecting the data, Martin Thoma manually inspected each recording. This +manual inspection is a necessary step as anonymous web users could submit any +drawing they wanted to any symbol. This includes many creative recordings as +shown in~\cite{Kirsch,Thoma:2014} as well as loose associations. In some cases, +the correct label was unambiguous and could be changed. In other cases, the +recordings were removed from the data set. + +It is not possible to determine the exact number of people who contributed +handwritten symbols to the Detexify part of the dataset. The part which was +created with \href{http://write-math.com}{write-math.com} was created by +477~user~IDs. Although user IDs are given in the dataset, they are not +reliable. On the one hand, the Detexify data has the user ID 16925, +although many users contributed to it. Also, some users lend their phone to +others while being logged in to show how write-math.com works. This leads to +multiple users per user ID. On the other hand, some users don't register and +use write-math.com multiple times. This can lead to multiple user IDs for one +person. + +\section{Classes} +The \dbNameVersion~dataset contains \dbTotalClasses~classes. Those classes include the +Latin uppercase and lowercase characters (\verb+A-Z+, \verb+a-z+), the Arabic +numerals (\verb+0-9+), 32~different types of arrows, fractal and calligraphic +Latin characters, brackets and more. See \cref{table:symbols-of-db-0,table:symbols-of-db-1,table:symbols-of-db-2,table:symbols-of-db-3,table:symbols-of-db-4,table:symbols-of-db-5,table:symbols-of-db-6,table:symbols-of-db-7,table:symbols-of-db-8} for more information. + +\section{Data} +The \dbNameVersion~dataset contains \dbTotalInstances{} black and white images +of the size $\SI{32}{\pixel} \times \SI{32}{\pixel}$. Each image is labeled +with one of \dbTotalClasses~labels. An example of 100~elements of the +\dbNameVersion{} data set is shown in~\cref{fig:100-data-items}. + +The average amount of black pixels is \SI{16}{\percent}, but this is highly +class-dependent ranging from \SI{3.7}{\percent} of \enquote{$\dotsc$} to \SI{59.2}{\percent} of \enquote{$\blacksquare$} average +black pixel by class. + +The ten classes with most recordings are: +\[\int, \sum, \infty, \alpha, \xi, \equiv, \partial, \mathds{R}, \in, \square\] +Those symbols have \num{26780} recordings and thus account for +\SI{16}{\percent} of the data set. 47~classes have more than \num{1000} +recordings. The number of recordings of the remaining classes are distributed +as visualized in~\cref{fig:class-data-distribution}. + +\begin{figure}[h] + \centering + \includegraphics*[width=\linewidth, keepaspectratio]{figures/data-dist} + \caption{Distribution of the data among classes. 47~classes with + more than \num{1000} recordings are not shown.} + \label{fig:class-data-distribution} +\end{figure} + +A weakness of \dbNameVersion{} is the amount of available data per class. For +some classes, there are only 51~elements in the test set. + +The data has $32\cdot 32 = 1024$ features in $\Set{0, 255}$. +As~\cref{table:pca-explained-variance} shows, \SI{32}{\percent} of the features +can explain~\SI{90}{\percent} of the variance, \SI{54}{\percent} of the +features explain \SI{99}{\percent} of the variance and \SI{86}{\percent} of the +features explain \SI{99}{\percent} of the variance. + +\begin{table}[h] + \centering + \begin{tabular}{lccc} + \toprule + Principal Components & 331 & 551 & 882 \\ + Explained Variance & \SI{90}{\percent} & \SI{95}{\percent} & \SI{99}{\percent} \\ + \bottomrule + \end{tabular} + \caption{The number of principal components necessary to explain, + \SI{90}{\percent}, \SI{95}{\percent}, \SI{99}{\percent} + of the data.} + \label{table:pca-explained-variance} +\end{table} + +The Pearson correlation coefficient was calculated for all features. The +features are more correlated the closer the pixels are together as one can see +in~\cref{fig:feature-correlation}. The block-like structure of every 32th +feature comes from the fact the features were flattened for this visualization. +The second diagonal to the right shows features which are one pixel down in the +image. Those correlations are expected as symbols are written by continuous +lines. Less easy to explain are the correlations between high-index +features with low-index features in the upper right corner of the image. +Those correlations correspond to features in the upper left corner with +features in the lower right corner. One explanation is that symbols which have +a line in the upper left corner are likely $\blacksquare$. + +\begin{figure}[h] + \centering + \includegraphics*[width=\linewidth, keepaspectratio]{figures/feature-correlation.pdf} + \caption{Correlation of all $32 \cdot 32 = 1024$ features. The diagonal + shows the correlation of a feature with itself.} + \label{fig:feature-correlation} +\end{figure} + + +\section{Classification Challenge} +\subsection{Evaluation} +\dbName{} defines 10 folds which should be used for calculating the accuracy +of any classifier being evaluated on \dbName{} as follows: + +\begin{algorithm}[H] + \begin{algorithmic} + \Function{CrossValidation}{Folds $F$} + \State $D \gets \cup_{i=1}^{10} F_i$\Comment{Complete Dataset} + \For{($i=0$; $\;i < 10$; $\;i$++)} + \State $A \gets D \setminus F_i$\Comment{Train set} + \State $B \gets F_i$\Comment{Test set} + \State Train Classifier $C_i$ on $A$ + \State Calculate accuracy $a_i$ of $C_i$ on $B$ + \EndFor + \State \Return ($\frac{1}{10}\sum_{i=1}^{10} a_i$, $\min(a_i)$, $\max(a_i)$) + \EndFunction + \end{algorithmic} + \caption{Calculate the mean accuracy, the minimum accuracy, and the maximum + accuracy with 10-fold cross-validation} +\label{alg:seq1} +\end{algorithm} + +\subsection{Model Baselines} +Eight standard algorithms were evaluated by their accuracy on the raw image +data. The neural networks were implemented with +Tensorflow~\cite{tensorflow2015-whitepaper}. All other algorithms are +implemented in sklearn~\cite{scikit-learn}. \Cref{table:classifier-results} +shows the results of the models being trained and tested on MNIST and also for +\dbNameVersion{}: +\begin{table}[h] + \centering + \begin{tabular}{lrrr} + \toprule + \multirow{2}{*}{Classifier} & \multicolumn{3}{c}{Test Accuracy} \\%& \multirow{2}{*}{\parbox{1.2cm}{\centering HASY\\Test time}}\\ + & MNIST & HASY & min -- max\hphantom{00 } \\\midrule% & + TF-CNN & \SI{99.20}{\percent} & \SI{81.0}{\percent} & \SI{80.6}{\percent} -- \SI{81.5}{\percent}\\% & \SI{3.1}{\second}\\ + Random Forest & \SI{96.41}{\percent} & \SI{62.4}{\percent} & \SI{62.1}{\percent} -- \SI{62.8}{\percent}\\% & \SI{19.0}{\second}\\ + MLP (1 Layer) & \SI{89.09}{\percent} & \SI{62.2}{\percent} & \SI{61.7}{\percent} -- \SI{62.9}{\percent}\\% & \SI{7.8}{\second}\\ + LDA & \SI{86.42}{\percent} & \SI{46.8}{\percent} & \SI{46.3}{\percent} -- \SI{47.7}{\percent}\\% & \SI{0.2}{\second}\\ + QDA & \SI{55.61}{\percent} & \SI{25.4}{\percent} & \SI{24.9}{\percent} -- \SI{26.2}{\percent}\\% & \SI{94.7}{\second}\\ + Decision Tree & \SI{65.40}{\percent} & \SI{11.0}{\percent} & \SI{10.4}{\percent} -- \SI{11.6}{\percent}\\% & \SI{0.0}{\second}\\ + Naive Bayes & \SI{56.15}{\percent} & \SI{8.3}{\percent} & \SI{7.9}{\percent} -- \hphantom{0}\SI{8.7}{\percent}\\% & \SI{24.7}{\second}\\ + AdaBoost & \SI{73.67}{\percent} & \SI{3.3}{\percent} & \SI{2.1}{\percent} -- \hphantom{0}\SI{3.9}{\percent}\\% & \SI{9.8}{\second}\\ + \bottomrule + \end{tabular} + \caption{Classification results for eight classifiers. + % The test time is the time needed for all test samples in average. + The number of + test samples differs between the folds, but is $\num{16827} \pm + 166$. The decision tree + was trained with a maximum depth of 5. The exact structure + of the CNNs is explained in~\cref{subsec:CNNs-Classification}.} + \label{table:classifier-results} +\end{table} + +The following observations are noteworthy: +\begin{itemize} + \item All algorithms achieve much higher accuracy on MNIST than on + \dbNameVersion{}. + \item While a single Decision Tree performs much better on MNIST than + QDA, it is exactly the other way around for~\dbName{}. One possible + explanation is that MNIST has grayscale images while \dbName{} has + black and white images. +\end{itemize} + + +\subsection{Convolutional Neural Networks}\label{subsec:CNNs-Classification} +Convolutional Neural Networks (CNNs) are state of the art on several computer +vision benchmarks like MNIST~\cite{wan2013regularization}, CIFAR-10, CIFAR-100 +and SVHN~\cite{huang2016densely}, +ImageNet~2012~\cite{deep-residual-networks-2015} and more. Experiments on +\dbNameVersion{} without preprocessing also showed that even the +simplest CNNs achieve much higher accuracy on \dbNameVersion{} than all other +classifiers (see~\cref{table:classifier-results}). + +\Cref{table:cnn-results} shows the 10-fold cross-validation results for four +architectures. +\begin{table}[H] + \centering + \begin{tabular}{lrrrr} + \toprule + \multirow{2}{*}{Network} & \multirow{2}{*}{Parameters} & \multicolumn{2}{c}{Test Accuracy} & \multirow{2}{*}{Time} \\ + & & mean & min -- max\hphantom{00 } & \\\midrule + 2-layer & \num{3023537} & \SI{73.8}{\percent} & \SI{72.9}{\percent} -- \SI{74.3}{\percent} & \SI{1.5}{\second}\\ + 3-layer & \num{1530609} & \SI{78.4}{\percent} & \SI{77.6}{\percent} -- \SI{79.0}{\percent} & \SI{2.4}{\second}\\ + 4-layer & \num{848753} & \SI{80.5}{\percent} & \SI{79.2}{\percent} -- \SI{80.7}{\percent} & \SI{2.8}{\second}\\ + TF-CNN & \num{4592369} & \SI{81.0}{\percent} & \SI{80.6}{\percent} -- \SI{81.5}{\percent} & \SI{2.9}{\second}\\ + \bottomrule + \end{tabular} + \caption{Classification results for CNN architectures. The test time is, + as before, the mean test time for all examples on the ten folds.} + \label{table:cnn-results} +\end{table} +The following architectures were evaluated: +\begin{itemize} + \item 2-layer: A convolutional layer with 32~filters of size $3 \times 3 \times 1$ + is followed by a $2 \times 2$ max pooling layer with stride~2. The output + layer is --- as in all explored CNN architectures --- a fully + connected softmax layer with 369~neurons. + \item 3-layer: Like the 2-layer CNN, but before the output layer is another + convolutional layer with 64~filters of size $3 \times 3 \times 32$ + followed by a $2 \times 2$ max pooling layer with stride~2. + \item 4-layer: Like the 3-layer CNN, but before the output layer is another + convolutional layer with 128~filters of size $3 \times 3 \times 64$ + followed by a $2 \times 2$ max pooling layer with stride~2. + \item TF-CNN: A convolutional layer with 32~filters of size $3 \times 3 \times 1$ + is followed by a $2 \times 2$ max pooling layer with stride~2. + Another convolutional layer with 64~filters of size $3 \times 3 \times 32$ + and a $2 \times 2$ max pooling layer with stride~2 follow. A fully + connected layer with 1024~units and tanh activation function, a + dropout layer with dropout probability 0.5 and the output softmax + layer are last. This network is described in~\cite{tf-mnist}. +\end{itemize} + +For all architectures, ADAM~\cite{kingma2014adam} was used for training. The +combined training and testing time was always less than 6~hours for the 10~fold +cross-validation on a Nvidia GeForce GTX Titan Black with CUDA~8 and CuDNN~5.1. +\clearpage +\subsection{Class Difficulties} +The class-wise accuracy +\[\text{class-accuracy}(c) = \frac{\text{correctly predicted samples of class } c}{\text{total number of training samples of class } c}\] +is used to estimate how difficult a class is. + +32~classes were not a single time classified correctly by TF-CNN and hence have +a class-accuracy of~0. They are shown in~\cref{table:hard-classes}. Some, like +\verb+\mathsection+ and \verb+\S+ are not distinguishable at all. Others, like +\verb+\Longrightarrow+ and +\verb+\Rightarrow+ are only distinguishable in some peoples handwriting. +Those classes account for \SI{2.8}{\percent} of the data. + +\begin{table}[h] + \centering + \begin{tabular}{lcrlc} + \toprule + \LaTeX & Rendered & Total & Confused with & \\\midrule + \verb+\mid+ & $\mid$ & 34 & \verb+|+ & $|$ \\ + \verb+\triangle+ & $\triangle$ & 32 & \verb+\Delta+ & $\Delta$ \\ + \verb+\mathds{1}+ & $\mathds{1}$ & 32 & \verb+\mathbb{1}+ & $\mathbb{1}$ \\ + \verb+\checked+ & {\mbox {\wasyfamily \char 8}} & 28 & \verb+\checkmark+ & $\checkmark$ \\ + \verb+\shortrightarrow+ & $\shortrightarrow$ & 28 & \verb+\rightarrow+ & $\rightarrow$ \\ + \verb+\Longrightarrow+ & $\Longrightarrow$ & 27 & \verb+\Rightarrow+ & $\Rightarrow$ \\ + \verb+\backslash+ & $\backslash$ & 26 & \verb+\setminus+ & $\setminus$ \\ + \verb+\O+ & \O & 24 & \verb+\emptyset+ & $\emptyset$ \\ + \verb+\with+ & $\with$ & 21 & \verb+\&+ & $\&$ \\ + \verb+\diameter+ & {\mbox {\wasyfamily \char 31}} & 20 & \verb+\emptyset+ & $\emptyset$ \\ + \verb+\triangledown+ & $\triangledown$ & 20 & \verb+\nabla+ & $\nabla$ \\ + \verb+\longmapsto+ & $\longmapsto$ & 19 & \verb+\mapsto+ & $\mapsto$ \\ + \verb+\dotsc+ & $\dotsc$ & 15 & \verb+\dots+ & $\dots$ \\ + \verb+\fullmoon+ & {\mbox {\wasyfamily \char 35}} & 15 & \verb+\circ+ & $\circ$ \\ + \verb+\varpropto+ & $\varpropto$ & 14 & \verb+\propto+ & $\propto$ \\ + \verb+\mathsection+ & $\mathsection$ & 13 & \verb+\S+ & $\S$ \\ + \verb+\vartriangle+ & $\vartriangle$ & 12 & \verb+\Delta+ & $\Delta$ \\ + \verb+O+ & $O$ & 9 & \verb+\circ+ & $\circ$ \\ + \verb+o+ & $o$ & 7 & \verb+\circ+ & $\circ$ \\ + \verb+c+ & $c$ & 7 & \verb+\subset+ & $\subset$ \\ + \verb+v+ & $v$ & 7 & \verb+\vee+ & $\vee$ \\ + \verb+x+ & $x$ & 7 & \verb+\times+ & $\times$ \\ + \verb+\mathbb{Z}+ & $\mathbb{Z}$ & 7 & \verb+\mathds{Z}+ & $\mathds{Z}$ \\ + \verb+T+ & $T$ & 6 & \verb+\top+ & $\top$ \\ + \verb+V+ & $V$ & 6 & \verb+\vee+ & $\vee$ \\ + \verb+g+ & $g$ & 6 & \verb+9+ & $9$ \\ + \verb+l+ & $l$ & 6 & \verb+|+ & $|$ \\ + \verb+s+ & $s$ & 6 & \verb+\mathcal{S}+ & $\mathcal{S}$ \\ + \verb+z+ & $z$ & 6 & \verb+\mathcal{Z}+ & $\mathcal{Z}$ \\ + \verb+\mathbb{R}+ & $\mathbb{R}$ & 6 & \verb+\mathds{R}+ & $\mathds{R}$ \\ + \verb+\mathbb{Q}+ & $\mathbb{Q}$ & 6 & \verb+\mathds{Q}+ & $\mathds{Q}$ \\ + \verb+\mathbb{N}+ & $\mathbb{N}$ & 6 & \verb+\mathds{N}+ & $\mathds{N}$ \\ + \bottomrule + \end{tabular} + \caption{32~classes which were not a single time classified correctly by + the best CNN.} + \label{table:hard-classes} +\end{table} + +In contrast, 21~classes have an accuracy of more than \SI{99}{\percent} with +TF-CNN (see~\cref{table:easy-classes}). + +\begin{table}[h] + \centering + \begin{tabular}{lcr} + \toprule + \LaTeX & Rendered & Total\\\midrule + \verb+\forall + & $\forall $ & 214 \\ + \verb+\sim + & $\sim $ & 201 \\ + \verb+\nabla + & $\nabla $ & 122 \\ + \verb+\cup + & $\cup $ & 93 \\ + \verb+\neg + & $\neg $ & 85 \\ + \verb+\setminus + & $\setminus $ & 52 \\ + \verb+\supset + & $\supset $ & 42 \\ + \verb+\vdots + & $\vdots $ & 41 \\ + \verb+\boxtimes + & $\boxtimes $ & 22 \\ + \verb+\nearrow + & $\nearrow $ & 21 \\ + \verb+\uplus + & $\uplus $ & 19 \\ + \verb+\nvDash + & $\nvDash $ & 15 \\ + \verb+\AE + & \AE & 15 \\ + \verb+\Vdash + & $\Vdash $ & 14 \\ + \verb+\Leftarrow + & $\Leftarrow $ & 14 \\ + \verb+\upharpoonright+ & $\upharpoonright$ & 14 \\ + \verb+- + & $- $ & 12 \\ + \verb+\guillemotleft + & $\guillemotleft $ & 11 \\ + \verb+R + & $R $ & 9 \\ + \verb+7 + & $7 $ & 8 \\ + \verb+\blacktriangleright+ & $\blacktriangleright$ & 6 \\ + \bottomrule + \end{tabular} + \caption{21~classes with a class-wise accuracy of more than \SI{99}{\percent} + with TF-CNN.} + \label{table:easy-classes} +\end{table} + + +\section{Verification Challenge} +In the setting of an online symbol recognizer like +\href{http://write-math.com}{write-math.com} it is important to recognize when +the user enters a symbol which is not known to the classifier. One way to achieve +this is by training a binary classifier to recognize when two recordings belong to +the same symbol. This kind of task is similar to face verification. +Face verification is the task where two images with faces are given and it has +to be decided if they belong to the same person. + +For the verification challenge, a training-test split is given. The training +data contains images with their class labels. The test set +contains 32~symbols which were not seen by the classifier before. The elements +of the test set are pairs of recorded handwritten symbols $(r_1, r_2)$. There +are three groups of tests: +\begin{enumerate}[label=V\arabic*] + \item $r_1$ and $r_2$ both belong to symbols which are in the training set, + \item $r_1$ belongs to a symbol in the training set, but $r_2$ + might not + \item $r_1$ and $r_2$ don't belong symbols in the training set +\end{enumerate} + +When evaluating models, the models may not take advantage of the fact that it +is known if a recording $r_1$ / $r_2$ is an instance of the training symbols. +For all test sets, the following numbers should be reported: True Positive (TP), +True Negative (TN), False Positive (FP), False Negative (FN), +Accuracy $= \frac{TP+ TN}{TP+TN+FP+FN}$. + + +% \section{Open Questions} + +% There are still a couple of open questions about \dbNameVersion: + +% \begin{enumerate} +% \item What is the accuracy of human expert labelers? +% \item What is the variance between human experts labeling the samples? +% \end{enumerate} + + +\section{Acknowledgment} + +I want to thank \enquote{Begabtenstiftung Informatik Karls\-ruhe}, the Foundation +for Gifted Informatics Students in Karlsruhe. Their support helped me to write +this work.