AI gets a better score than 80% of Swedes* on our test for language understanding

*WIP, preliminary results

Here is a few demos where you can try things on your own

ORD

Model thinks: lärare vid högskola

MEK

Model thinks: prokrastinera – aktivt

LÄS

Närmare skolans praktik Paradoxalt nog tycks det i många länder som om framväxten av en utbildningsforskning har inneburit en marginalisering av lärarnas roll i denna. Lärarna ses som forskningskonsumenter och förväntas implementera forskningens resultat. Dock handlar den universitetsbaserade akademiska forskningen sällan direkt om de frågor som lärarna arbetar med. Även om forskningen genomförs i klassrummen är forskningsfrågorna, och därmed resultaten, inte relaterade till lärares uppgifter och frågor. Det finns därför ett klart uttalat behov av att utveckla forskningsansatser som talar mer direkt till lärare och de professionella problem de har att hantera – forskning som kan ge lärare bättre teoretiska och praktiska redskap i sitt arbete. Det är mot en sådan bakgrund som den nya floran av praxisnära forskningsansatser kan förstås. De fyller det ”mellanrum” som har uppstått mellan forskningen och skolans verksamheter, och de utvecklas från två håll. Å ena sidan söker universitetsforskningen sig närmare skol- praktiken på olika sätt, å andra sidan börjar skolorna bli alltmer intresserade av olika former av forsknings- och utvecklingsprojekt (FoU). Beteckningen praxisnära forskning lanserades av Vetenskapsrådets utbildningsvetenskapliga kommitté som ett sätt att beteckna forskning som mer direkt skulle svara mot skolornas och lärarutbildningarnas behov. I USA används beteckningen practitioner research eller teacher research för forskning som bedrivs av lärare. Sådan praktikerforskning grundas i en kritik av synen på lärare som forskningskon- sumenter och implementerare av andras kunskap. En viktig skillnad mellan termerna praktikerforskning och praxisnära forskning är att den förra riktar uppmärksamheten mot vem det är som utför forskningen – medan den senare fokuserar den verksamhet där forskningsfrågorna uppstår. I början av 1990-talet utvecklades så kallade design experiments av Ann Brown och Allan Collins. De flyttade ut forskningen från universitetets laboratorier till naturliga skolmiljöer där teoretiskt grundade undervisningsdesigner prövades och vidareutvecklades. Designforskningen har därefter expanderat kraftigt och Jan van der Akker har föreslagit begreppet (design and) development research som samlingsterm för en familj av närbesläktade forskningsansatser som designforskning, designexperiment, formativ forskning och ”engineering research”. Gemensamt för dessa ansatser är att de är kollaborativa (lärare och forskare i samverkan), interventionistiska (man studerar inte den praktik som är, utan prövar specifika teorigrundade utformningar av undervisningen) och iterativa (cykliska förlopp där teori och praktik ömsesidigt påverkar varandra). På olika sätt överbryggar de gapet mellan teori och praktik, involverar lärare och syftar till att utveckla klassrumspraktiken. I sådana forskningsansatser går förfiningen av de praktiska produkterna och undervisningens utformning hand i hand med förfiningen av teoriutvecklingen. Design- och utvecklingsforskningen har vuxit fram som ett praxisnära alternativ inom ramen för den akademiska forskningen. Praktikerforskningen har däremot utvecklats i kontrast till akademisk forskning och utgått från lärares behov av att förstå, och helst lösa, de praktiska problem man står inför. Det talas också om klinisk forskning. Vården brukar användas som förebild för utbildningsområdet när det gäller att få till stånd en forskningsbaserad verksamhet. Den övervägande delen av forskningen inom vårdområdet är så kallad klinisk forskning som, något förenklat, kan beskrivas som forskning i anslutning till sjukdomar och deras botande och förebyggande, till skillnad från icke-klinisk disciplinär forskning om till exempel grundläggande biokemiska processer. Det är framför allt forskningsobjekten, det vill säga vad man forskar om, och syftet med forskningen – att lösa ett problem, utveckla ett botemedel och förbättra verksamheten – som är det specifika med klinisk forskning. Beteckningen klinisk forskning används även i över- förd betydelse inom olika verksamhetsområden för att tala om forskning som leder till kunskap som de inom yrket behöver för att förbättra sina professionella bedömningar. Anthony Kelly definierar klinisk utbildningsvetenskaplig forskning som forskning om lärande och undervisning ur ett lärarperspektiv. Det är därigenom en forskningsmodell med stora möjligheter att utveckla kunskaper som kan hjälpa lärarna i deras arbete. Kunskapen som genereras integreras i de professionella bedömningarna i de kliniska verksamheterna. Man kan se den kliniska forskningen som en brygga mellan grundläggande forskning och den professionella yrkesutövningen. Beteckningen klinisk forskning avser däremot inte de specifika metoder som används inom den medicinska kliniska forskningen i dag. Varje verksamhetsområde måste utveckla forskningsansatser som är relevanta i förhållande till de frågor och problem man har. För det mesta är lärarna hänvisade till sina egna erfarenheter och funderingar för att besvara sina frågor. Den forskning som bedrivs sysslar oftast med generella frågor om lärande och undervisning, medan studier av konkreta problem i lärandesituationen än så länge i stor utsträckning lyser med sin frånvaro. Det är mot den bakgrunden man ska förstå det stora intresset för den japanska lesson study-modellen, som är direkt utformad för att utveckla de pedagogiska verksamheterna i anslutning till konkreta lärandeproblem. Enligt James Hiebert, Ronald Gallimore och James Stigler erbjuder den japanska lesson study-traditionen en modell för utveckling av en kollektiv professionell kunskapsbas för lärare. Tillsammans utformar lärarna lektioner som prövas, observeras, analyseras och revideras. Genom detta lärarägda utvecklingsarbete görs lärarnas kunskaper publika – möjliga att dokumentera, diskutera och verifiera. Avslutningsvis vill jag betona att en forskningsgrundad skolutveckling förutsätter en balans mellan olika typer av forsknings- och utvecklingsarbete. Skolan behöver forskning av olika slag för att utvecklas. Den ena forskningsansatsen ska därför inte ställas mot den andra – det viktiga är att det finns rimliga proportioner mellan olika typer av forskningsansatser. Skolan behöver en motsvarighet till vårdens kliniska forskning. Begreppen praxisnära forskning och utvecklingsforskning är alternativ som förts fram för att beteckna en sådan forskning. Begreppen kan dock kritiseras för att vara alltför vida och för att inte ta fasta på den profes- sionsutvecklande aspekten. En del använder därför klinisk forskning i överförd bemärkelse. En annan väg vore att försöka hitta ett nytt begrepp. Eftersom beteckningen pedagogisk i dag är knuten till disciplinen pedagogik, som är en samhällsvetenskaplig disciplin snarare än att vara de pedagogiska yrkenas kunskapsgrund, behövs ett begrepp som fångar kärnan i läraryrket. Ett begrepp som liksom klinisk har grekiskt ursprung, och som fokuserar lärarprofessionens bildningsuppdrag, är paedeutisk. I sin grekiska ursprungsbetydelse rymmer paedeutisk såväl det vi kallar bildning som det som skapar bildning. Ingrid Carlgren

Model thinks: Avsaknaden av forskning som har konkret koppling till lärarnas undervisningssituation.

soon

Deep Neural Networks on Högskoleprovet a test on AI of the language understanding required for higher education

The breakthorugh in AI: Transformers

In September 2019, I woke up high as a kite after a collarbone operation. I look at my phone and in my feed, I see that there is a new model (Albert, A lite Bert) that beats the average person on tasks for Reading Comprehension in English. This made my high head spin even more. What were the possibilities? What if I was doing this? I have to do this!

The breakthrough came from a model created for machine translation and its continuations in the form of GPT and Bert. The new models completely shattered many of the leaderboards that researchers "compete" on for various tasks. I had done some experiments in English but it was Swedish that was most interesting and there were no good models.

In November, somewhat recovered, I started a journey to explore what the latest breakthroughs in Artificial Neural Networks for Language understanding meant for my language, Swedish. What fascinated me the most was not the high results on text and sentence classification, but that they managed to do tests on reading comprehension. Although the implementations are incredibly similar on a technical level, it is very exciting when it seems to have arisen a different type of reading comprehension than the one we have but which can still "reason" (pattern match?) over texts and answer questions. This was extremely exciting, what did it mean to me? How does it work? What does it output? Are there any similarities with how we do it?

Efficient Estimation of Word Representations in Vector Space | Attention Is All You Need | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding| ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

What made it possible to beat the average Swede?

Opportunity

The new models were trained on huge amounts of data in relation to the experiments I did before which were mostly on Swedish Wikipedia which is only a fraction of the size compared to english wikipedia. The models that came were trained on 16GB, 38GB, 140GB +. In the past, people often trained on lines or sentences, but now it was full texts. I had to rebuild my entire dataset from scratch. No small task to do alone. All in all, I came up with about 100GB. If you assume that 1MB is 250 A4 pages, you would come to Seoul, South Korea from Stockholm. Given that the text would result in 25 million sheets printed placing them one after the other. Assuming that it would take an average of 1.7 minutes to read a page for a human being, it would take over 80 years of constant reading. This is for the small models.

Fortunately, there are a lot of resources available. The largest resource is CommonCrawl which contains dumps of internet. After cleaning the data, I found about 16GB on my first dump. The Riksdag shares a lot of texts and this was added quickly and easily. OPUS and its subtitles from movies and Wikipedia went along. Later Oscar was added. All in all, I came up with about 100GB.

Training these new large models requires a lot of computing power. For my small workstation with an RTX2080 graphics card, it would probably take several years to train a model really well. To be able to train, a supercomputer was needed. Google has developed a new type of hardware, Tensor Processing Unit (TPU) that is specifically designed to train models. These are provided freely through the Tensorflow Research Cloud. As a base member of the Tensorflow research cloud, you get access to a bunch of TPUs. Enough to run "small-scale" experiments in this world. It is thanks to TPU access that I have been able to do this.

Open

Internationally, research in NLP (Natural Language Processing) is incredibly open. Everything runs like a running locomotive. It's terribly hectic to keep up with. Fortunately, a lot of models, data and code are shared. Even companies like Google share a lot and are a big part of why a lot of this is happening right now. This suits me perfectly who loves to pick things apart instead of endlessly repeating the basics until they stick without seeing the some nice end result.

What should I start disassembling? I ask those who did Albert, if I could see their code on how they solved RACE (Preparatory tasks for the Chinese college exam) and two days later a couple of files fell into Github. The grind begins ...

Huge thanks to: |

TensorFlow Research Cloud | OPUS |

OSCAR

How do you teach an AI Swedish?

Tokenization

En bilfärja lämnade lilla varholmen
[ "en", "bil", "##fär", "##ja", "lämnade", "lilla", "var", "##holmen" ]

Does it see words? Letters? The model can see "tokens", a token can be a letter, part of a word or a whole word. Dividing texts into tokens has its pros and cons. The biggest problem and advantage is that these models look at how all the tokens in a text affect each other ("Attention"). If all tokens were just letters in the example above, it would be 19 and in longer texts this would require alot more compute, if we instead look at words it is only 5. If we save all words, we get a gigantic amount of tokens to store, partly we miss some context that could be useful. With the current tokenization, we get 8, a better trade-off. For example, you can guess that ["any token", "##holmen"] is about a place by the water even if the model has never seen that combination before"

Diificulties
In Swedish, we have an insane amount of compound words, such as "longtimestorage" and "pretimevoting". This quickly becomes a problem with how current algorithms search for tokens, which are not at all adapted to this. The tokens that the model sees are often poorly adapted and a different inflection of a sub-word can throw around all the tokens that become of the whole compound word. There will also be duplicates if it is in the beginning or in a word, e.g. then "holme" and "##holme" are seen as completely different tokens. You can test yourself below in the demo.

Solutions
On the one hand, there is a lot of research on "Attention", which makes it difficult to look at longer sequences and some progress has been made. But then you can also have a smarter system for how to tokenize. There is some research on this and it is partly a problem that may be focused on in Germanic languages other than English. I myself have a lot of ideas that I would like to explore. I assume that more people may be working on developing different solutions already. Then it may simply and partly be the case that more computing power is required, which resolves itself over time and this only becomes a temporary quick fix to compensate for today's slow computers seen from a future perspective. Need m0ar compute.

Tokenization

Result:31

som

exempel

så

försöker

den

placera

avtal

och

kontrakt

nära

varandra

förmodligen

så

ligger

jurist

närområdet

den

är

bra

träna

##d

204

1007

275

2715

220

8713

2371

181

5063

1548

2275

6324

275

695

13313

26152

221

220

193

520

4763

120

Modellen

Embeddings
The model begins with creating embeddings, a kind of representation of what a token means. As an example, it tries to place "agreements" and "contracts" close to each other, probably "lawyer" is in the immediate area if it is well trained.

Transformer
Then these representations go through several transformer layers. A transformer layer learns to ask questions ("attention") from each token on all other tokens. An example could be "who created the contract". A basic model often has 12 such "questions per layer". The answer to these questions is merged and goes through a neural network that creates a new representation of the token that can move on to the next layer. A base model often has 12 layers. If you increase heads and layers, you increase the number of patterns that the model can pick up. Often known models have different names, but it is usually variations on this or training method that differentiate them. For example GPT-3 has 96 heads and 96 layers and a slimmer variant of attention which makes it less expensive flops-wise to look at longer contexts(2048 tokens).

Model

Layer:12

Head:7

Token:

Lära sig svenska

Task
To learn a model Swedish, you must have a task that it can practice on. A task where you can see if it is wrong. In order for it to learn as well as possible, you need extreme amounts of samples of that task. Therefore, it is best to come up with tasks that work on any text. Previously, this has been to remove tokens and then let the model guess which token should be there. Either in the middle of a sentence or guessing the next token in a text. If you guess a token in the middle of a text, it will be good at understanding texts. If you train it to guess the next token, it will be good at writing texts.

More efficient
One method developed by Clark et al. (Stanford, Google) is having a model that replaces tokens in texts with similar tokens. Then you have another model that guesses which word has been replaced. This leads to the model being given a more difficult task, which also means that it must look hard at each token, which provides better feedback. This leads to the model requiring significantly less resources to train and is significantly more energy-friendly and thus more environmentally friendly than previous methods. I chose this method.

Below you can see the output from the model. The larger the number, the more the model thinks it has been replaced.

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

Pre-training

Test its abilities on the university entrance exam

The goal is to train the model to be able to do tasks that are similar to the university entrance exam and then test it on the university entrance exam. This as a test of the generalization potential that the new models have. For example, a recurring question in the university entrance exam is, "what does the author say in the first paragraph?". Or the particular style of language that is used. I want to see if the model has such a general understanding that you should not have to train it to learn small tricks to get good results.

ORD

Definition
This task is more like some form of a given word, a combination of words or a phrase, choose the most similar in meaning. It is not as easy as just having a list of words, because here there are proverbs and other things.

Training
The datasets consists of words from Swedish wiktionary. I use a cosine similarity loss. Splits 80/10/10 and gets ~ 0.91 in cosine pearson

Reflektion
What is interesting about this task is that you can see how the model / method generalizes in addition to the list of synonyms that only consists of a word to cope with phrases and also words that are not in the list of synonyms. When I tested the model on 3 college exams, I got around 84% accuracy on the words. I also then to run the standard way for encoders with pooling and got around 89% at best. However, I do not use the result from there as I used 3 college exams as a dev set and this is cheating. What I want to do is train it to quite generally solve similar tasks as HP and only use HP as a final test. What is interesting besides generalization is that this task is probably a very good measure of how well the model divides tokens. Correct subword -> more generalization.

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks |

MEK

Definition
Read a text that has several empty spaces, in these places one or more words fits. Choose one from 4 options which is best.

Training
Here I did not train at all. The model already has a real talent for this from its basic understanding of Swedish! You can test above to see what the model spits out when you replace different words. Even if it thinks that the word you have replaced may be there, it will generally be more uncertain about the words around it. So by looking at a box around the words, you have a pretty good basis for whether the words fit. This gave 78% accuracy on 3 tests.

Reflection
There is a lot to gain here. Some by training it to find exchanged words in texts with extremely high quality and not random pages from the internet. You could then apply a classifier and train it to choose the right alternative. However, this goes a bit against the purpose, to test how well it generalizes.

LÄS

Definition
Read a text and a question. Choose the answer from 4 options whichever is best.

Training
Here comes the difficult part. The university entrance exam is probably the largest dataset available on reading comprehension in swedish that is somewhat easily available. Maybe it is available for national tests for high school or middle school or at a company that offers training materials for this. I chose to look at what material was available in English. I trained a model to translate from English to Swedish. That model had to translate reading comprehension tasks into Swedish. Then I trained a Swedish model to do the tasks. The code for training in Tensorflow I have shared in Huggingfaces github. The best result I got with this method was 1.0. I think there is a lot to be gained by using better data and using a larger model, unfortunately this was both resource-intensive in the form of compute but also in time. I hope to be able to continue experimenting with this in the future. There is a model in English, UnifiedQA. It is trained on T5, a model that is available in extreme sizes. They trained a model to do very broad reading comprehension tasks, I do tests with their 3B variant in the end and translate with a model that enters the Swedish texts in English to see where the limit seems to go.

Reflection
The biggest limitation here is data and model size. Reading comprehension requires a lot from the model. The more patterns the model can find the better the results. Having many examples of the patterns required to do the task also increases the chance. What I had the opportunity to do was basically a model with 12 layers based on my then knowledge. I started experimenting with getting into how to fit and optimize for larger models. Using UnifiedQA 11B would possibly increase the accuracy by about 5%.

If you look at the results below, it still seems that it might be better to train a smaller model in Swedish and still beat the big English with translation. It would be exciting to see if I continue! It should also be taken into account that the English tasks are easier compared to the Swedish ones.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | UnifiedQA: Crossing Format Boundaries With a Single QA System

AI gets a better score than 80% of Swedes* on our test for language understanding

ORD

MEK

LÄS

Deep Neural Networks on Högskoleprovet a test on AI of the language understanding required for higher education

The breakthorugh in AI: Transformers

What made it possible to beat the average Swede?

Opportunity

Open

How do you teach an AI Swedish?

Tokenization

Tokenization

Modellen

Model

Lära sig svenska

Pre-training

Test its abilities on the university entrance exam

ORD

MEK

LÄS

Result

Score: 0, 1.9% - 0/0

Loading...