EN 
SV 01 Language model · experiment
2020Högskoleprovet
AI scores 1.3 on Högskoleprovet — better than 80% of Swedish test takers.
An AI model outscored 80% of Swedish test takers
Archived experiment · preliminary 2020 result.
Try the model yourself
ORD
Model thinks: lärare vid högskola
Can a neural network understand the language required for higher education?
The breakthrough in AI: Transformers
In September 2019, I woke up high as a kite after a collarbone operation. I look at my phone and in my feed, I see that there is a new model (Albert, A lite Bert) that beats the average person on tasks for Reading Comprehension in English. This made my high head spin even more. What were the possibilities? What if I was doing this? I have to do this!
The breakthrough came from a model created for machine translation and its continuations in the form of GPT and Bert. The new models completely shattered many of the leaderboards that researchers "compete" on for various tasks. I had done some experiments in English but it was Swedish that was most interesting and there were no good models.
In November, somewhat recovered, I started a journey to explore what the latest breakthroughs in Artificial Neural Networks for Language understanding meant for my language, Swedish. What fascinated me the most was not the high results on text and sentence classification, but that they managed to do tests on reading comprehension. Although the implementations are incredibly similar on a technical level, it is very exciting when it seems to have arisen a different type of reading comprehension than the one we have but which can still "reason" (pattern match?) over texts and answer questions. This was extremely exciting, what did it mean to me? How does it work? What does it output? Are there any similarities with how we do it?
What made the result possible?
WEB · PARLIAMENT · SUBTITLES · WIKIPEDIA
Opportunity
The new models were trained on enormous amounts of data compared with my earlier experiments, which mostly used Swedish Wikipedia—a fraction of the size of its English counterpart. The new models used 16GB, 38GB, even 140GB or more. Instead of training on isolated lines or sentences, they learned from full texts. I had to rebuild the entire dataset from scratch. In the end, it reached roughly 100GB.
Printed on 25 million sheets of A4 and laid end to end, that text would stretch from Stockholm to Seoul. At an average of 1.7 minutes per page, reading it continuously would take more than 80 years. And this was still considered small.
Fortunately, strong open resources were available. Common Crawl supplied a large part of the material; after cleaning one dump, I found about 16GB of useful Swedish text. The Riksdag’s public texts, OPUS movie subtitles, Wikipedia, and later OSCAR completed the corpus.
Training models of this size requires serious computing power. On my workstation with an RTX 2080, a good training run could have taken years. Google’s TensorFlow Research Cloud gave researchers access to Tensor Processing Units built specifically for this kind of work. That access made my “small-scale” experiments possible.
Open
NLP research was remarkably open. Models, data, and code moved quickly between researchers and companies. It was hectic to follow, but ideal for someone who learns by taking working systems apart.
I asked the researchers behind ALBERT if I could see how they solved RACE, a reading-comprehension benchmark based on Chinese school exams. Two days later, a couple of files appeared on GitHub. The work began.
How do you teach an AI Swedish?
Tokenization
En bilfärja lämnade lilla varholmen
[ "en", "bil", "##fär", "##ja", "lämnade", "lilla", "var", "##holmen" ]
Does it see words? Letters? The model can see "tokens", a token can be a letter, part of a word or a whole word. Dividing texts into tokens has its pros and cons. The biggest problem and advantage is that these models look at how all the tokens in a text affect each other ("Attention"). If all tokens were just letters in the example above, it would be 19 and in longer texts this would require a lot more compute; if we instead look at words it is only 5. If we save all words, we get a gigantic amount of tokens to store, and we miss some context that could be useful. With the current tokenization, we get 8, a better trade-off. For example, you can guess that ["any token", "##holmen"] is about a place by the water even if the model has never seen that combination before.
Difficulties
Swedish has a huge number of compound words. This quickly becomes a problem for current tokenization algorithms, which are not well adapted to them. A different inflection of one subword can change the tokenization of the entire compound. There are also duplicates depending on whether a fragment appears at the beginning or inside a word: "holme" and "##holme" are treated as completely different tokens. You can test this yourself in the demo below.
Solutions
There is ongoing research into more efficient attention for longer sequences, but tokenization itself can also be smarter. This is especially relevant for Germanic languages other than English. More compute may make some of these compromises less important over time, but better language-specific tokenization remains an interesting direction to explore.
01 · Text to data
Tokenization
Base model
12 × 12layers × attention headsGPT-3
96 × 96layers × attention headsContext window
2,048tokensThe model
Embeddings
The model begins with creating embeddings, a kind of representation of what a token means. As an example, it tries to place "agreements" and "contracts" close to each other, probably "lawyer" is in the immediate area if it is well trained.
Transformer
Then these representations go through several transformer layers. A transformer layer learns to ask questions ("attention") from each token on all other tokens. An example could be "who created the contract". A basic model often has 12 such "questions per layer". The answer to these questions is merged and goes through a neural network that creates a new representation of the token that can move on to the next layer. A base model often has 12 layers. If you increase heads and layers, you increase the number of patterns that the model can pick up. Often known models have different names, but it is usually variations on this or training method that differentiate them. For example GPT-3 has 96 heads and 96 layers and a slimmer variant of attention which makes it less expensive flops-wise to look at longer contexts(2048 tokens).
Model
the model learns Swedish
the model earns Swedish
earns → replaced
Learning Swedish
Task
To learn a model Swedish, you must have a task that it can practice on. A task where you can see if it is wrong. In order for it to learn as well as possible, you need extreme amounts of samples of that task. Therefore, it is best to come up with tasks that work on any text. Previously, this has been to remove tokens and then let the model guess which token should be there. Either in the middle of a sentence or guessing the next token in a text. If you guess a token in the middle of a text, it will be good at understanding texts. If you train it to guess the next token, it will be good at writing texts.
More efficient
One method developed by Clark et al. (Stanford, Google) is having a model that replaces tokens in texts with similar tokens. Then you have another model that guesses which word has been replaced. This leads to the model being given a more difficult task, which also means that it must look hard at each token, which provides better feedback. This leads to the model requiring significantly less resources to train and is significantly more energy-friendly and thus more environmentally friendly than previous methods. I chose this method.
Below you can see the output from the model. The larger the number, the more the model thinks it has been replaced.
Pre-training
Vocabulary
Sentence completion
Reading comprehension
Testing the model on Högskoleprovet
The goal is to train the model to be able to do tasks that are similar to the university entrance exam and then test it on the university entrance exam. This as a test of the generalization potential that the new models have. For example, a recurring question in the university entrance exam is, "what does the author say in the first paragraph?". Or the particular style of language that is used. I want to see if the model has such a general understanding that you should not have to train it to learn small tricks to get good results.
ORD
Definition
This task is more like some form of a given word, a combination of words or a phrase, choose the most similar in meaning. It is not as easy as just having a list of words, because here there are proverbs and other things.
Training
The datasets consists of words from Swedish wiktionary. I use a cosine similarity loss. Splits 80/10/10 and gets ~ 0.91 in cosine pearson
Reflektion
What is interesting about this task is that you can see how the model / method generalizes in addition to the list of synonyms that only consists of a word to cope with phrases and also words that are not in the list of synonyms. When I tested the model on 3 college exams, I got around 84% accuracy on the words. I also then to run the standard way for encoders with pooling and got around 89% at best. However, I do not use the result from there as I used 3 college exams as a dev set and this is cheating. What I want to do is train it to quite generally solve similar tasks as HP and only use HP as a final test. What is interesting besides generalization is that this task is probably a very good measure of how well the model divides tokens. Correct subword -> more generalization.
MEK
Definition
Read a text that has several empty spaces, in these places one or more words fits. Choose one from 4 options which is best.
Training
Here I did not train at all. The model already has a real talent for this from its basic understanding of Swedish! You can test above to see what the model spits out when you replace different words. Even if it thinks that the word you have replaced may be there, it will generally be more uncertain about the words around it. So by looking at a box around the words, you have a pretty good basis for whether the words fit. This gave 78% accuracy on 3 tests.
Reflection
There is a lot to gain here. Some by training it to find exchanged words in texts with extremely high quality and not random pages from the internet. You could then apply a classifier and train it to choose the right alternative. However, this goes a bit against the purpose, to test how well it generalizes.
LÄS
Definition
Read a text and a question. Choose the answer from 4 options whichever is best.
Training
Here comes the difficult part. The university entrance exam is probably the largest dataset available on reading comprehension in swedish that is somewhat easily available. Maybe it is available for national tests for high school or middle school or at a company that offers training materials for this. I chose to look at what material was available in English. I trained a model to translate from English to Swedish. That model had to translate reading comprehension tasks into Swedish. Then I trained a Swedish model to do the tasks. The code for training in Tensorflow I have shared in Huggingfaces github. The best result I got with this method was 1.0. I think there is a lot to be gained by using better data and using a larger model, unfortunately this was both resource-intensive in the form of compute but also in time. I hope to be able to continue experimenting with this in the future. There is a model in English, UnifiedQA. It is trained on T5, a model that is available in extreme sizes. They trained a model to do very broad reading comprehension tasks, I do tests with their 3B variant in the end and translate with a model that enters the Swedish texts in English to see where the limit seems to go.
Reflection
The biggest limitation here is data and model size. Reading comprehension requires a lot from the model. The more patterns the model can find the better the results. Having many examples of the patterns required to do the task also increases the chance. What I had the opportunity to do was basically a model with 12 layers based on my then knowledge. I started experimenting with getting into how to fit and optimize for larger models. Using UnifiedQA 11B would possibly increase the accuracy by about 5%.
If you look at the results below, it still seems that it might be better to train a smaller model in Swedish and still beat the big English with translation. It would be exciting to see if I continue! It should also be taken into account that the English tasks are easier compared to the Swedish ones.
05 · Evaluation