MaGRiTTE is an automated approach to learn embeddings that capture the structural information of files.

Its core architecture is composed of two stages based on two popular architectures used in representation learning: transformer encoders and convolutional autoencoders.

First, we encode file rows using a transformer encoder architecture specifically pre-trained to learn structural features.

Transformer models, as demonstrated by the large successes in natural language processing tasks, are successful in learning the inherent structure of sentences and languages. Therefore, we specifically train a transformer model to learn the "language" of data files, i.e., their structure, with file rows corresponding to "sentences" of such language.

But file structure is not limited to rows, as columns and combination of rows can also have structural significance (for example, with files containing different tables, or sequences of preamble lines). For this reason, the second stage of MaGRiTTE is designed to  learn a file-wise structural encoding.

To do so, we use a variational autoencoder architecture composed of several convolutional layers, trained to reconstruct the stack of row-wise embeddings using a fixed length vector for file-wise representation.

The intution for the use of convolutional autoencoding is that file-wise structure can be understood by the occurrence of bi-dimensional row-column local features, which convolutional filters have proven successful in detecting.

The overall MaGRiTTE architecture is summarized in the image below.

To pre-train the transformer architecture on structural features, we propose two core adaptations: a novel tokenization stage and specialized training objectives. To abstract the data content of a file, and train the transformer architecture on structural features, we introduce “pattern tokenization”: Assuming that structural properties are identifiable through special characters, we reduce all alphanumeric characters to a set of few general patterns. After tokenization, the rows of the input files are split on newline characters and a percentage of the special character tokens is masked before feeding it to the row encoder model. The  row-transformer model is then trained on two objectives, reconstructing the masked tokens, and identifying whether pairs of rows belong to the same file.

The row embeddings produced by the first stage are then used as the input for the file embedding stage of MAGRiTTE. In this stage, the encoder and decoder models are trained using the reconstrocution loss on the row embeddings feature maps.

Due to the convolutional nature of the second stage, we fix the amount of rows in the input files to a given number (selected as a result of hyperparameter tuning), and either truncate the exceeding rows from the input file, or padding them by random replication of the existing rows in case the original file does not have enough.

The file-wise embedding vector is obtained, after the training stage, as the innermost fixed-length feature vector, which is used as the input of the encoder stage of the autoencoder structure.

We evaluate the effectiveness of the learned structural representations on three tasks to analyze unseen data files:

1. Fine-grained dialect detection, i.e., identifying the structural role of characters within rows,
2. Line and cell classification, i.e.,identifying metadata, comments, and data within a file,
3. Table extraction, i.e., identifying the boundaries of tabular regions.

We compare the use of MAGRiTTE encodings with state-of-the-art approaches that were specifically designed for these tasks. In future work, we aim at using MAGRiTTE embeddings in generative fashion to perform structural data preparation, e.g., changing file dialects, or as metadata for structural indexing of files in data lakes.