Adapting the Efficient Market Hypothesis: a review of Andrew Lo’s Adaptive Markets

Theories of efficient markets mislead because they seek to explain market behaviour according to the laws of physics rather than biology, says Adaptive Market Hypothesis proponent Andrew Lo

The finance books I’ve been reviewing here make it clear I’m a fan of the passive investment philosophy. It seems to me the evidence is clear that over time a diversified portfolio that simply tracks the market secures the most reliable returns for investors who do not have the resources, expertise or luck to pick market-beating stocks — which in practice means most investors.

The theories and maxims that underpin passive strategies — like the Capital Asset Pricing Model (CAPM), and the Efficient Market Hypothesis (EMH) that flows from it — have a pleasing simplicity and elegance.

Of course, they model reality rather than represent it. As even occasional traders know, the markets do not serenely glide between states of equilibrium: they are volatile, emotional places, often overwhelmed by sudden surges of sentiment, driven by greed and — more often — fear.

In his short, sharp defence of active investment, 10 1/2 Lessons from Experience, which I reviewed last autumn, hedge fund manager Paul Marshall described markets as ‘highly complex non-linear systems created by a myriad of half-informed or uninformed decisions made by fallible (human) agents with multiple cognitive biases.’

Marshall likes Georges Soros’s ‘General Theory of Reflexivity’, which suggests that markets are self-referential systems, changed by the act of observation. A shift in valuations can generate snowballing effects, as traders pile in or pile out, amplifying — and distorting — market corrections. Financial markets do not only anticipate, observe and react to economic developments, but drive them, locked into tight feedback loops. Marshall also references Hyman Minksy’s observations on the capacity of markets to destabilise themselves, long periods of calm prompting investors to take risks in search of alpha, generating a bust followed by a renewed era of chastened calm, until temptation reasserts itself and the cycle repeats

But though Soros and Minsky’s offer useful correctives to purist formulations of the EMH, theories of efficient markets say something important about why it is so difficult to beat the market over the long term: in a market in which stocks are under close scrutiny by so many informed, high motivated participants, it does not seem possible to consistently identify winning stocks, year after year. The facts prove that ordinary investors are indeed better off letting a diversified portfolio generate returns over time. The market is a turbulent sea, the passive investor’s passage continually beset by squalls and storms, and subject to long periods in the doldrums when there is little tailwind, but those who stay the course are — usually — rewarded.

The shark on the beach

Adaptive Markets: Financial Evolution at the Speed of Thought by MIT economist Andrew Lo recognises the EMH’s historic strengths, but seeks to integrate insights by Soros, Minsky and others into a new theory of how markets work. First published in 2017, and revised in 2019, it is not a new book: I was prompted to read it after reviewing Lo’s latest, In Pursuit of the Perfect Portfolio, a history of the development of passive investment theory written with Stephen R Foerster. But though well received, the book is still relatively little referenced in the mainstream financial and investor press.

Lo’s subtle argument unfolds over more than 500 closely argued pages. But his thesis flows from a simple insight: theories of efficient markets are incomplete because they are suffused with misleading imagery, seeking to explain market behaviour with reference to the laws of physics, when in fact biological laws offer a more accurate model. Lo wants to upgrade the EMH with his Adaptive Market Hypothesis (AMH), which contends that ‘investors and financial markets behave more like biology than physics, comprising a population of living organisms competing to survive, not a collection of inanimate objects subject to immutable laws of motion’.

Theories of efficient markets offer a highly stylised vision of markets as systems governed by rational behaviour, tending towards equilibrium through the interplay of clear headed participants acting in perfect accord with their self interest. Like Marshall and Soros et al, Lo recognises that investors don’t behave like rational Platonic spirits, calmly assessing all of the evidence before them before deciding on their next move. But it would be facile to say that their behaviour is simply irrational. Rather, he suggests, they continually adapt their behaviour to new environments.

When market conditions change, investor behaviour can certainly look irrational. Participants continue to do what worked in the past, until they have had time to process what has changed, and are able to adjust their behaviour accordingly. Lo cites the archetypes of the investor who, having only experienced bull markets, buys near the top of a bubble, or the rogue trader who continually doubles down on losing trades, attempting to recoup ever accumulating losses by resorting to decision making formulae that may have worked in the past, but which have rendered obsolete by new circumstances. Lo captures the idea through a neat image, suggesting such behaviour is rather like the desperate flapping of a shark washed up on land — not irrational, but ‘maladaptive’:

The difference between the irrational investor and the shark on the beach is the shorter length of time the investor has had to adapt to the financial environment, and the much faster speed with which that environment is changing.

As Lo acknowledges, such adaptive behaviour is close to the idea of ‘bounded rationality’ associated with the political scientist Herbert Simon, who observed that in practice we make decisions according to ‘heuristics’, rules-of-thumb that are good enough for day-to-day purposes. When conditions change, these rules evolve, through a process of trial and error. Lo suggests market bubbles and crashes are prime examples of adaptive behaviour. Traders pile into stocks when markets are rising for fear of missing out, and continue to do so even as prices plateau: when caught up in the cut and thrust of day-to-trading it is not immediately apparent to participants that a market may have reached the crest of a wave. When prices peak, and then begin to fall, there is an increasingly rapid sell-off. Again, it isn’t clear to those close to the market when prices will bottom out, so it makes sense to keep selling. This behaviour isn’t optimal, but it has a certain logic: it is adaptive.

Where the EMH images investors as self-interested calculating machines, the AMH, following evolutionary theory, argues their overriding objective is rather more elemental: survival. Market participants are rational, but only in so far as rationality works for survival. Primal emotions kick-in when survival is threatened. And if the market can be conceived as a giant organism, a market crisis shows the organism under conditions of stress. Lo describes market crashes in evolutionary terms, suggesting that the ‘collective rush to the market’s nucleus accumbens [overwhelms] the fear response generated by its amygdala, and [induces] its left hemispheres to come up with a justification’. Or: ‘the wisdom of crowds is sometimes overwhelmed by the madness of mobs’.

Economics as physics

The perfectly rational homo economicus imagined by efficient markets proponents has, of course long been subject to critique, and indeed ridicule. Lo, is more generous, offering an intriguing account as to how economists came to propose such an unlikely model for human behaviour, and why some continue to defend it.

Essentially, it was, and is, a matter of pride. Following the intellectual zeitgeist of the post-war years, economists wanted to elevate their subject to the status of a social science. For much of the previous 200 years it had been known as ‘political economy’, closer perhaps to moral philosophy than mathematics, its great figures — Adam Smith, Karl Marx, John Maynard Keynes — as much essayists as systematic theoreticians.

That open ended approach had its merits, allowing Smith and Marx et al to suffuse their texts with acute psychological insights that have proven their value. But the post-war founders of modern economics wanted to rebuild their subject’s foundations, to reground it in a set of techniques that would allow them to analyse economic problems mathematically. The brilliant Paul Samuelson showed the way with his 1947 manifesto Foundations of Economic Analysis, reconceiving economics as a science, capable of producing theories open to empirical verification. Lo suggests:

They wanted a theory of economics as powerful and abstract as the nuclear physics that had given the United States the atomic bomb. They distrusted the measurement of the subjective, and they distrusted psychology as a whole. They wanted a theory that looked like mathematics and physics, not like biology.

Samuelson and the economists of the Chicago School the new approach offered their profession a cool glass of water, an invitation to wash away all of the verbiage, improvisation and supposition that characterised the dark arts of political economy, and derive reliable laws from rigorous study of patterns in economic activity.

And the new economics has indeed yielded a host of useful theories. The movements of particles in Brownian motion inspired the Random Walk Hypothesis, which sheds light on why it isn’t possible to accurately forecast stock prices, the insight at the heart of passive investment theory. The heat equation in thermodynamics, which holds that heat is also the product of random motion, suggested the tried and tested Black-Scholes/Merton option pricing formula. And the assumption that patterns of economic activity can be discerned and defined has facilitated the development of game theory, general equilibrium theory, rational expectations theory, the CAPM, and more. But Lo suggests economists have become too confident, too ready to see read immutable laws into ambiguous data. In one of the book’s sharpest passages, he writes:

Physicists can explain 99 percent of all observable physical phenomena using Newton’s three laws of motion. Economists wish we had three laws capable of explaining 99 percent of all observable behaviour in our professional purview. Instead, we probably have ninety-nine laws that explain 3 percent of all economic behaviour, and it’s a source of terrible frustration for us. So we sometimes cloak our ideas in the trappings of physics. We make axioms from which we derive seemingly mathematically rigorous universal economic principles, carefully calibrated simulations, and the very occasional empirical test of those theories.’

For Lo, economies are simply too complex to be fully explicable within the exacting compass of mathematical laws. They more closely resemble endlessly complex, organic, biological systems, driven by evolutionary categories such as mutation, competition, and natural selection. Indeed, though efficient markets theorists have aspired to interpret economic activity through the lens of mathematics, economists have often turned to biology to offer sufficiently nuanced accounts of how markets operate. Thomas Malthus famously drew on the biological category of scarcity to illustrate his theory of population growth, and Joseph Schumpeter suggested that economies continually evolve through ‘creative destruction’. Competitive markets are associated in the popular imagination with Darwinian notions of the ’survival of the fittest’, in which ‘wolves’ and ‘vultures’ prevail. Lo wants to recognise the achievements of 20th century economics, but to encourage the profession to expand its terms of reference, to move beyond an insistence that everything must be viewed in terms of physics.

An adaptive index fund

In the latter part of the book he suggests how the AMH might be applied, offering ideas as to how investors might navigate markets that are rather more volatile by nature than theories of efficient markets allow. These is much rich material here that cannot be adequately summarised within the scope of a single review. But one is of immediate interest to day-to-day investors, a proposal for a ‘dynamic index fund’ designed to protect its holders from the severe market downturns that sit uneasily with the EMH, but which the AMH can more readily acknowledge.

Lo has in mind a diversified equity portfolio that would not simply track the market like a traditional index fund, but take advantage of modern trading technology to shield accumulated gains from periods of market volatility. Rather than exposing investors to the sharp and prolonged downturns that can really cut into wealth, Lo’s fund would adapt, automatically switching a portion of its equity holdings to cash when such conditions are encountered.

Though a passive investment strategy of buy-and-hold will likely yield a decent return for patient investors over the long term, not everyone can afford to hold a portfolio for the number of years it requires to come good, and riding out market downturns requires a psychological resilience that not all will possess. The long term can be very long indeed.

Lo produces a battery of statistics to show the extent to which periods of volatility, such as during the 1930s, the early 2000s, the 2008 crash, and the early months of the pandemic in 2020, have cut into patiently accumulated holdings. Markets — perhaps most notably Japan — can be sluggish for many years. Lo notes that a dollar invested in 1926 would, by 2014, be worth $11,141 if it had been converted back into cash during downturns, as against $4,162 had it been held in an unmanaged index.

Lo’s proposed dynamic fund seeks to blend the strengths of passive and active investment, introducing an element of hands-on risk management into an essentially passive product. Innovations such as algorithmic trading, derivatives, securities exchange design, telecommunications, and back-office infrastructure would make it possible to design a fund with preset rules that would automatically switch holdings to cash under certain market conditions, or alert investors to give authorisation to make such a change. Lo suggests it would be ‘as simple as setting your car’s cruise control. If the estimated volatility of the index at a given date exceeds a pre-specified threshold, it invests a portion of the fund in cash … if the volatility falls below that threshold, it invests more than 100 percent of the fund in the index: in other words, it leverages the fund.’

Many investors are already familiar with these concepts. Those using trading platforms can configure their system to send stop-loss alerts when the value of a certain security falls by a certain percentage. And life-cycle funds adjust holdings of debt and equity over time in accordance with an automatic formula, shifting to fixed income as they approach their target date.

The idea, of course, depends ultimately on the old question of whether it is possible to time the market. Passive advocates know perfectly well how damaging downturns can be: they just don’t think it is possible to gauge when they are going to happen, and how long they will last. Lo believes that an intelligent fund, able to assess volatility through analysis of market data, would offer a more reliable means of navigating turbulence than dependence on human judgement.

Lo’s bold book offers a new language for describing how markets work. The grammar of ‘adaptation’ feels right. We know we don’t act as impartial agents when making investment decisions — or indeed any other kind of decision. Our capacity to choose wisely is compromised by the knowledge, time and energy we have available. And at times of crisis the imperative of survival pushes itself into the foreground. But we don’t feel we are acting irrationally. We try to exercise some kind of judgement even in a crisis. Lo has found the right word: we adapt. We flounder while seeking to adjust to new conditions, refining our judgement in response to trial and error.

That said, I’m wary of formulating the insight into a hypothesis, of simply exchanging one model for economic activity for another, tempting us to another form of determinism. Overly zealous applications of the AMH might be inclined to set bounds to the possible range of human behaviour as rigid as those imagined by efficient markets purists. I want to hold on to a sense of the inscrutability of human behaviour, a sense that we are too complex to be explained in terms of a particular theory. In the end questions of human subjectivity are philosophical, perhaps even theological.

But Lo’s core insight is compelling: so much depends on our use of language, which generates imagery able to deceive as much as illuminate. By noting the many similarities of the market to organic systems he opens up a rich new vocabulary for casting new light on the enigmas of the marketplace.

Adaptive Markets: Financial Evolution at the Speed of Thought by Andrew Lo is published by Princeton University Press. The image above is a detail from the book’s cover.