The Natural Language Expertise Crossroads

Ever since I was a kid I’ve been fascinated with language.

I remember being amazed as I watched my younger brother (a full 12 years my junior) turn almost overnight from a languageless toddler into a full-sentence-weilding mini-person, coming out with phrases and expressions that you just knew were his own spontaneous creations.  I was amazed at his ability to create sentences that he’d never heard before, or to translate never-before-heard strings of sounds into blocks of meaning.  He was a normal (if you’re reading, oh brother of mine, I mean “normal” only in the most basic sense, ok?) 2 year-old kid, but he had somehow managed to figure out the finer points of what seemed to me to be a pretty complex grammatical system.

Some 20 years later I’m watching my own children go through the same process, and – even after reaching a far deeper understanding of the language acquisition process – I still find it utterly amazing.

I later found out that actually studying the science of human language solves very few of its puzzles.  Quite how humans are able to learn a system so powerful it can create an infinite number of sentences from a (theoretically) finite set of building blocks isn’t all that clear.  Quite why we are able to do so also isn’t clear.  Chuck a load a sounds into the brain of an infant, and a couple of years later out comes this thing we call language, capable of effortless interpretation of other people’s sounds.  It is one of nature’s most colossal achievements.

I believe that language is at the heart of what makes humans different from other species.  I also believe that it is at the heart of what separates us from Artificial Intelligence.  Creativity, emotion, imagination, self-awareness – all the things that we do that computers can’t do – are all intertwined with the systems that give us language.

This is the background to why – 5 years ago – I set off on a mission to apply my professional skills (complex people sourcing) to the field of Natural Language technology.    At the time, this was seriously niche stuff – a space inhabited by relatively few companies, primarily involved in doing interesting things with speech technology – building systems that could process human sounds into and begin the process of enabling humans to communicate with computers.  There was a smattering (yes, great word, isn’t it?) of activity in the information retrieval market – people commercializing technology that enables machines to process large amounts of linguistic data and give the human audience actionable information about it.  Other work was happening in the Semantic technology space, and there was some interest in commercialising dialogue systems, but it was at first tough going.  It was a space dominated by big companies, who primarily hired people straight from the academia.  In Europe (where my work was focussed), there are a number of established Computer Science departments where language technology plays a major role and for a while the route from academia into the R&D department of one of the big companies was relatively smooth.

Something was stirring though.

5 years later, things have changed.  There have been huge steps forwards in the ways that linguistic theory is being applied to computer systems.  We are starting to see technology that writes for you, technology that talks to you, technology that translates for you.  There’s systems to summarize for you, there are applications that read, digest and process text for you.  The workings in linguistics labs in Universities around the world are making their way into the real world and people are starting to take notice.  And primarily, these changes have been characterised by a change in the demographic of the average company in the space – in particular by the emergence of a significant number of start-ups.

Now is a very interesting time to be in the Natural Language space.  Computers will never have the gift of language, but they are starting to be able to do some great things with it.

But the industry is at a crossroads, and at the heart of the problem – as always – is people.  Since 2006, we’ve seen a 350% increase in the demand for people with Natural Language Processing skills (Language processing is just one of many language-related skills that we see demand for, but it’s a nice representative sample).  Primarily, that extra demand – driven by the commercial sector – has been serviced by people moving out of academia.    The vast majority of people involved in language technology development hold postgraduate qualifications – they are trained by academia and employed by industry.  This arrangement works well on the industry side: supply of expertise (I am always careful not to use the word talent) – whilst always a problem – has been manageable.  But with increased activity comes expertise supply problems, and this is starting to become particularly acute.

Demand for Natural Language expertise is growing rapidly

This problem really comes into focus when you consider that the number of people studying Computer Science in any form (let alone the highly specialised subjects relating to language technology) has dropped off considerably, in the UK at least.

For companies involved in developing language systems, an increasing amount of their target candidate pool is now employed elsewhere.  The right expertise is now significantly harder to find and even harder to hire than ever before.   (Post)Graduates are faced with an increasing number of options; people with previous industrial experience even more so.

The supply of expertise that the rapidly-growing band of start-ups and established companies in this space needs is dwindling, and it is only going to get worse.  For now, the options for organisations are:- 1) Continue as we are and hope that academia continues to supply the numbers or 2) Start offering language-technology specific training to non language specialist Computer Science graduates.

Until now, there has been a reluctance to go down the latter path.  And so here we are, at the crossroads.  Turn one way and the computer’s relationship with language continues to be largely at the periphery of technology.  Turn the other, and language technology goes mainstream.

We have spent the last three years hunting down and talking to people involved in language technology development.  Quite a lot of the time, this has been a difficult and thankless task.  I’ve pinned my colours to the mast: I’m betting that other people find language as beautiful as I do, and that the industry can continue to hunt for ways to help machines share our gift of language…or at least part of that gift.

Of course, if you’re interested in any help or advice on hiring language technologists, please get in touch.

Tags: , , ,

No comments yet.

Leave a Reply