Code for publication published in ICPR 2022

Use the last version (with Guided CTC and Weighted Label Prior) for better results

Early Recognition of Untrimmed Handwritten Gestures with Spatio-Temporal 3D CNN.

This extends the work of ICDAR 2021 : https://gitlab.inria.fr/IntuiDocEnLigne/reconnaissance-precoce-2d-avec-deep-learning

William Mocaër, Eric Anquetil, Richard Kulpa

The code is written in python with tensorflow 2.7.

Model

The package "Model" contains the model file ModelEarly2D.py

FittedModel

The package "FittedModel" contains the fitted model, containing weights, results on Test set and informations about weights

utils

Utils contains a parser for inkml partially developped by Robin Dahuron.

Databases

Each folder of database contains three files :

• CreateAndExportArtificalUntrimmedDataset - create the artificals datasets by concatenating files
• CreateAndProcessArtificalUntrimmedDataset - create the artificals datasets by concatenating files (same way) and preprocess to generate the representations
• Training - fit to optimize the losses
• EvaluationUntrimmed - the evaluation runs on test set
• VisualizingResult - export the results in image, as in qualitative results of the paper
• VisualizingResultWithConf - export the results in image, with temporal informations and confidence
• BoundedEval.py (in MTGSetB) : contains the code of the BOD (Bounded Online Detection) metric

Preprocessing.py are residuals from ICDAR 2021 work.

The databases ILGDB and MTGSetB, Untrimmed versions, should be available on the intuidoc website: https://www-intuidoc.irisa.fr/en/mtgsetb-and-ilgdb-untrimmed/ if not you can contact Eric Anquetil or William Mocaër.

ILGDB

ILGDB. The ILG database is a mono-stroke pen-based gestures dataset performed by 38 users. It contains 21 different gesture classes with a total of 1923 samples, 693 are used for training and 1230 for testing.

N. Renau-Ferrer, P. Li, A. Delaye and E. Anquetil, "The ILGDB database of realistic pen-based gestural commands," Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), Tsukuba, 2012, pp. 3741-3744.

MTGSetB

The MTGSetB dataset is composed of 45 di�erent multi-touch gestures regrouped into 31 rotation invariant gesture classes made by 33 users.

Zhaoxin Chen, Eric Anquetil, Harold Mouchère, Christian Viard-Gaudin. A graph modeling strategy for multi-touch gesture recognition. 14th International Conference on Frontiers in Handwriting Recognition (ICFHR-2014), Sep 2014, Crete island, Greece

requirements.txt and requierementIgrida.txt

this file contains all dependencies necessary to run the code