flan-t5-text2sparql-naive

This model is a fine-tuned version of google/flan-t5-base on the lc_quad dataset. It achieves the following results on the evaluation set:

• Loss: 0.4105

Model description

T5 has performed well in generating SPARQL queries from natural text, but semi automated preprocessing was necessary (Banerjee et.al.). FLAN-T5 comes with the promise of being better than T5 across all categories, so a re-evaluation is needed. Our goal is to find out what kind of preprocessing is still necessary to retain good performance, as well as how to automate it fully.

This is the naive version of the fine-tuned LLM, blindly applying the same tokenizer both on the natural language question as well as the target SPARQL query.

Intended uses & limitations

This model performs very bad, do not use it! We wanted to find out whether preprocessing is still necessary or T5 can figure things out on its own. As it turns out, preprocessing is still needed, so this model will just serve as some kind of baseline.

An example:

Create SPARQL Query: What was the population of Clermont-Ferrand on 1-1-2013?

'SELECT ?obj WHERE  wd:Q2'}

Training and evaluation data

LC_QUAD 2.0, see sidebar.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	301	0.5173
0.6515	2.0	602	0.4861
0.6515	3.0	903	0.4639
0.4954	4.0	1204	0.4478
0.4627	5.0	1505	0.4340
0.4627	6.0	1806	0.4247
0.4404	7.0	2107	0.4177
0.4404	8.0	2408	0.4139
0.429	9.0	2709	0.4115
0.4201	10.0	3010	0.4105

Framework versions

• Transformers 4.18.0
• Pytorch 1.10.2+cu102
• Datasets 2.4.0
• Tokenizers 0.12.1