Thought experiment.- Imagine yourself learning to ride a bike or maybe even a skateboard. Those skills help you prepare for other activities that require similar motor skills. When you grow up maybe it’s easier to ride a motorcycle. Transfer learning is similar in this way, we use a pre-trained model for a similar process.
When a feature of this transfer helps our cause, it’s considered a positive transfer and when it doesn’t it’s called negative transfer.
Why Transfer Learning
It has the potential to help models be more refined and avoid either over or underfitting. In comparison to deep learning, we might not need copious amounts of data to train our model, which in term helps save time, money and resources.
Why did TL become a thing
Recollecting data to train an AI model can be very time consuming and also it takes a lot of effort. So people sought ways to make it easier; one way was to “transfer” knowledge between tasks. A model that was trained to recognize red apples might have some insight on how to recognize early stages of certain types of cancer. In essence, it’s like when someone learns a new language and maybe even a third or fourth. As you go, it becomes easier to learn a new language. Sure dialects, tones and grammar might be similar or completely different but what relates to almost all of them is that you use your mouth, your ability to produce a varied amount of sounds gets better with every trick or skill you pick up from the last time.
Types of Transfer learning
Inductive TL
Here the source task differs from the target task, despite them being in a similar domain.
Usually in traditional TL, the main goal is set on the target domain, in contrast, multilearning or multi-tasks learning, much like someone with ADHD, the goal is to excel at every available task. Inductive TL is classified into two cases. multi task and self taught like most of us
Multi task learning
Here the target and source domain are the same. The labeled information in the domain source is accessible. Here multi task learning, tries to gain insight from the target and source simultaneously.
Self taught learning
Here it’s the opposite, the source and target differ but still remain related in a sense. No labels are accessible from the source domain. Here, the TL model “discovers” spaces in between the source and target domains that are related that can give us new insight on how or why they’re even related.
Transductive TL
Both source and target tasks are identical in this case, but the domains are different. No labeled data is provided in the target domain but it is provided in the source domain.
Unsupervised TL
Both the source and target tasks have little similarity and neither domains work with labeled data.
Reference
