My top 5 language AI books
Language AI is one of the most challenging areas of artificial intelligence, one where mainstream AI is far from coming near human-level performance, because it needs world knowledge to be solved (AI complete).
The shortcomings of modern machine learning approaches can be explained by the low efficiency of artificial neural networks. Because natural evolution is mainly driven by efficiency, I developed a strong interest for biologically inspired natural language understanding, hence the following book recommendations.
By Jeff Hawkins and Sandra Blakeslee
This book published in 2004 describes the core concepts in Jeff Hawkins’ cortical theory. Hawkins defines intelligence as the capacity to anticipate the future based on past experiences. The brain acts as a memory system that stores patterns and makes predictions out of current experiences based on memorized patterns. In order to be truly intelligent, machines must recreate this memory-prediction system.
Why you should read it:
This book opens a fundamentally new perspective on AI, away from traditional statistical approaches, and looks at biology as source of inspiration. It is easy to read for anyone who is interested in the area, and definitely a must-read for all AI experts who experience the limits of current machine learning approaches. It creates a true paradigm shift by describing sparse distributed representations as the sole data format used by the neocortex.
The birth of the Semantic Folding approach was the intuition that the most important thing to look at is the representation, not the algorithm.
#2 Surfaces and Essences: Analogy as fuel and fire of thinking
By Douglas Hofstadter and Emmanuel Sander
The theory exposed by Pulitzer Prize winning author Hofstadter and French psychologist Sander can be summarized in one sentence: analogy is the core of all thinking. The brain makes sense of the world, including the new and unpredictable, by doing nothing but continuously creating analogical links to past experiences.
Why you should read it:
Hofstadter explains with many practical examples (by analogy!) how humans process any kind of information (image, sound, text…) using the same unified representation in an endless chain of analogies. New incoming patterns are compared with those already stored. If a large overlap is detected, we "feel" that we understand; with little overlap, we "feel" curiosity to learn in order to increase future overlaps.
The idea for Semantic Folding -- a similarity-grounded theory for processing natural language -- originated in the amalgamation of Hawkins’ "sparse distributed representations" and Hofstadter’s concept of "analogy making".
By Pentti Kanerva
The book, published at MIT Press, presents a mathematical model of human long-term memory, where the main attribute of memory is sensitivity to similarity. Kanerva introduces the concept of Sparse Distributed Representation (SDR), a robust and not deterministic way of encoding, storing and processing information.
Why you should read it:
This book formalizes the link between Hawkins’ cortical SDRs and Hofstadter’s analogical processing and provided me with the mathematics for an engineering approach. The simple insight that similar words have to look similar led to the creation of semantic fingerprints, replacing complex matrix calculus with highly efficient Boolean set theory.
Semantic fingerprints encode the meaning of words into the topology of a two-dimensional, sparse binary vector. The analogy making corresponds to the bit-wise overlap between these fingerprints.
By Steven Pinker
In this famous book written for a general audience, Steven Pinker, a Canadian cognitive psychologist and linguist, describes how language is learnt, how it evolves, and argues that language is an innate behavior of humans.
Why you should read it:
This book is a classic and absolute must-read for anyone interested in language. Pinker interconnects consistently neuroscience, psychology, social sciences and linguistics and makes understandable in simple words how language works. His formulations and many examples are easy to understand and make this book a perfect starting point for further exploration of the topic. By the way, Steven Pinker is not only an excellent writer, he is also a great speaker: I warmly recommend listening to his online talks.
The Semantic Folding theory makes use of Pinker’s explanations around "mentalese", the universal inner language common to all humans that Jerry Fodor (Language of Thought) and Noam Chomsky (Universal Grammar) elaborated. Within the Semantic Folding framework, the universal inner language corresponds to the specific way the neocortex generates and processes language along the lines of Jeff Hawkins generic cortical information processing (hierarchical temporal memory).
#5 Synergetics: Explorations in the Geometry of Thinking
By Buckminster Fuller
A master of abstract system thinking, Buckminster provides in this book a description of the universe inspired by geometric logic, while using helpful metaphors from familiar human experiences.
Why you should read it:
Fuller meticulously worked on concepts like "generic systems" and "special case experiences" and called for "experienceable epistemology". In his view, something like a mathematical circle does not exist in the real world, a "practical circle" rather corresponds to a very high order regular polygon. Interestingly, this makes the irrational number pi obsolete and simplifies the practical use of circles.
Among many other sources of inspiration, Fuller’s Synergetics helped me understand how to get from generic sensory information to meaning and how words (symbolic labels of meaning) are created using motor commands of phoneme production.
Conclusion
Initially, I considered language as a brain phenomenon that underlies evolution. But ultimately, I came to the conclusion that language constitutes the essence of neocortical thinking and drives the evolution of the neocortex. In my opinion, this must be taken into account when building language processing systems. The reward will be a highly efficient AI implementation in which brute force, floating point combinatorics are replaced by the elegant simplicity of Boolean set theory.
Image credit: Andrey_Kuzmin/depositphotos.com
Francisco Webber is co-founder and CEO of Cortical.io and inventor of the company’s proprietary Retina technology. This technology applies the principles of cerebral processing to machine learning and natural language understanding (NLU) to solve real-world use cases related to big text data. Cortical.io solutions are based on the actual meaning of text, rather than on statistical occurrences.