Abstract
This case study investigates the task of job classification in a real-world setting, where the goal is to determine whether an English-language is appropriate for a graduate or entry-level position. We explore multiple approaches to text classification, including supervised approaches such as traditional models like Support Vector Machines (SVMs) and state-of-the-art deep learning methods such as DeBERTa. We compare them with Large Language Models (LLMs) used in both few-shot and zero-shot classification settings. To accomplish this task, we employ prompt engineering, a technique that involves designing prompts to guide the LLMs towards the desired output. Specifically, we evaluate the performance of two commercially available state-of-the-art GPT-3.5-based language models, text-davinci-003 and gpt-3.5-turbo. We also conduct a detailed analysis of the impact of different aspects of prompt engineering on the model’s performance.
Our results show that, with a well-designed prompt, a zero-shot gpt-3.5-turboclassifier outperforms all other models, achieving a 6% increase in Precision@95% Recall compared to the best supervised approach. Furthermore, we observe that the wording of the prompt is a critical factor in eliciting the appropriate “reasoning” in the model, and that seemingly minor aspects of the prompt significantly affect the model’s performance.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
These models are accessed through OpenAI’s API.
- 2.
Introduced by OpenAI in a blog post rather than a technical report: https://help.openai.com/en/articles/6779149-how-do-text-davinci-002-and-text-davinci-003-differ.
- 3.
This model is also accessed through OpenAI’s API.
- 4.
Due to the long token length of job postings, providing it with more than two examples required us to truncate the postings, which resulted in a degradation in performance.
- 5.
As demonstrated by the widely circulated prompt https://simonwillison.net/2023/Feb/15/bing/.
- 6.
As reported by OpenAI, a partnered developer found that positive reinforcement resulted in increased accuracy.
References
Anders, G.: Hiring’s new red line: why newcomers can’t land 35% of “entry-level” jobs. LinkedIn Economic Graph Research (2021)
Bommasani, R., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)
Boselli, R., et al.: WoLMIS: a labor market intelligence system for classifying web job vacancies. J. Intell. Inf. Syst. 51, 477–502 (2018)
Brown, T.B., et al.: Language models are few-shot learners. In: NeurIPS 2020 (2020)
Clavié, B., Alphonsus, M.: The unreasonable effectiveness of the baseline: discussing SVMs in legal text classification. In: JURIX 2021 (2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding, vol. 1 (2019)
Fujita, S., Ramey, G.: Job matching and propagation. J. Econ. Dyn. Control 31(11), 3671–3698 (2007)
Gao, L., et al.: PAL: program-aided language models. arXiv preprint arXiv:2211.10435 (2022)
Green, F., Henseke, G.: The changing graduate labour market: analysis using a new indicator of graduate jobs. IZA J. Labor Policy 5(1), 1–25 (2016). https://doi.org/10.1186/s40173-016-0070-0
He, P., Gao, J., Chen, W.: DeBERTaV3: improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543 (2021)
He, P., Liu, X., Gao, J., Chen, W.: DeBERTa: decoding-enhanced BERT with disentangled attention. arXiv:2006.03654 cs.CL (2020)
Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: Proceedings of ACL 2018 (2018)
Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 (2022)
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 (2021)
Loth, R., Battistelli, D., Chaumartin, F.R., De Mazancourt, H., Minel, J.L., Vinckx, A.: Linguistic information extraction for job ads (sire project). In: 9th International Conference on Adaptivity, Personalization and Fusion of Heterogeneous Information, pp. 300–303 (2010)
McGuinness, S., Pouliakas, K., Redmond, P.: Skills mismatch: concepts, measurement and policy approaches. J. Econ. Surv. 32(4), 985–1015 (2018)
Microsoft: Learn how to work with the ChatGPT and GPT-4 models (preview) (2023)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: NeurIPS 2013 (2013)
Miller, R.A.: Job matching and occupational choice. J. Polit. Econ. 92(6), 1086–1120 (1984)
Morgan, K.: Why inexperienced workers can’t get entry-level jobs. BBC (2021)
Office for National Statistics: Vacancies and jobs in the UK (2022)
OpenAI: OpenAI API: Chat completions (2023). https://platform.openai.com/docs/guides/chat
Ouyang, L., et al.: Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022)
Peters, M.E., et al.: Deep contextualized word representations (2018)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
Rae, J.W., et al.: Scaling language models: methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 (2021)
Roll, I., Russell, D.M., Gašević, D.: Learning at scale. Int. J. Artif. Intell. Educ. 28 (2018)
Sun, T.X., Liu, X.Y., Qiu, X.P., Huang, X.J.: Paradigm shift in natural language processing. Mach. Intell. Res. 19(3), 169–183 (2022)
Vaswani, A., et al.: Attention is all you need, vol. 2017-December (2017)
Veselinović, L., Mangafić, J., Turulja, L.: The effect of education-job mismatch on net income: evidence from a developing country. Econ. Res. 33(1) (2020)
Wei, J., et al.: Emergent abilities of large language models. Trans. Mach. Learn. Res. (2022)
Wei, J., et al.: Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022)
White, J., et al.: A prompt pattern catalog to enhance prompt engineering with ChatgPT. arXiv preprint arXiv:2302.11382 (2023)
Wolf, T., et al.: Transformers: state-of-the-art natural language processing. In: Proceedings of NeurIPS 2020: System Demonstrations (2020)
World Economic Forum, V: The future of jobs report 2020. WEF Reports (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Clavié, B., Ciceu, A., Naylor, F., Soulié, G., Brightwell, T. (2023). Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification. In: Métais, E., Meziane, F., Sugumaran, V., Manning, W., Reiff-Marganiec, S. (eds) Natural Language Processing and Information Systems. NLDB 2023. Lecture Notes in Computer Science, vol 13913. Springer, Cham. https://doi.org/10.1007/978-3-031-35320-8_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-35320-8_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-35319-2
Online ISBN: 978-3-031-35320-8
eBook Packages: Computer ScienceComputer Science (R0)