Transfer Learning in Natural Language Processing: A Survey
Abstract
Transfer learning has transformed Natural Language Processing (NLP) by leveraging pre-trained models on large datasets. This paper surveys recent advances in transfer learning for NLP, including models like BERT, GPT, and T5. We analyze their architectures, training methodologies, and performance across diverse downstream tasks, while discussing challenges such as domain adaptation and knowledge distillation.
References
Baars, H., & Kemper, H.-G. (2008). Management support with structured and unstructured data—An integrated business intelligence framework. Information Systems Management, 25(2), 132–148. https://doi.org/10.1080/10580530801941058
Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1798–1828. https://doi.org/10.1109/TPAMI.2013.50
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. https://doi.org/10.48550/arXiv.2005.14165
Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785–794). https://doi.org/10.1145/2939672.2939785
Nadella, G. S., Satish, S., Meduri, K., & Meduri, S. S. (2023). A Systematic Literature Review of Advancements, Challenges and Future Directions of AI And ML in Healthcare. International Journal of Machine Learning for Sustainable Development, 5(3), 115-130.
Maturi, M. H., Satish, S., Gonaygunta, H., & Meduri, K. (2022). The Intersection of Artificial Intelligence and Neuroscience: Unlocking the Mysteries of the Brain. International Journal of Creative Research In Computer Technology and Design, 4(4), 1-21.
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (Vol. 25, pp. 1097–1105). https://doi.org/10.1145/3065386
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539
Li, H., & Liu, X. (2018). Federated learning: Challenges, methods, and future directions. IEEE Communications Magazine, 56(12), 34–40. https://doi.org/10.1109/MCOM.2018.1800258
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2016). SSD: Single shot multibox detector. In Proceedings of the European Conference on Computer Vision (pp. 21–37). https://doi.org/10.1007/978-3-319-46448-0_2
Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. OpenAI Research Paper. https://doi.org/10.48550/arXiv.1801.06146
Ruder, S. (2019). Transfer learning—Machine learning’s next frontier. Machine Learning Research (Vol. 1). https://doi.org/10.48550/arXiv.1810.04805
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., & Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484–489. https://doi.org/10.1038/nature16961
Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations. https://doi.org/10.48550/arXiv.1409.1556
Maturi, M. H., Gonaygunta, H., Nadella, G. S., & Meduri, K. (2023). Fault Diagnosis and Prognosis using IoT in Industry 5.0. International Numeric Journal of Machine Learning and Robots, 7(7), 1-21.
