For a variety of unrelated reasons, quite a number of people in my life or whom I follow online have recently started buying into a variety of overarching pessimistic narratives. The thing that’s confusing to me as an outside observer to this is that the reasons for these pessimisms are so varied, and everyone has their own distinct set! Unfortunately for my friends, the human race can only go extinct once1. That said, the some dominant strands I can identify are the following:

1. Climate - People concerned about climate change

   The climate crisis is just getting started and I cannot bear to stand around—tinkering with microcontrollers and Python programs—while my future burns away.

   I’m frustrated. […] I wish I could accept what everyone else seems to have already accepted—that our climate catastrophe is inevitable and we will just have to adapt. I’m tempted. However, these are things I cannot accept for accepting such things would be ignoring the truth that individuals can and do catalyze change.

2. Population decline - the concern that population will collapse

   

3. AI X-Risk - The possibility that rogue AI will destroy civilisation or make humanity subservient

4. Rise of populism - The much discussed “rise of authoritarians” in various countries (examples — just google the phrase)

5. Falling apart of global order - the idea that the era of Pax Americana ending will lead to multitudinous catastrophes

6. Vague worries about internet rotting people’s brains / turning them into degenerates etc. (often wrapped up in concerns about fertility).

This is far from a complete list, and perhaps my list is somewhat of a Rorschach test of what I’m interested in or thinking about. If I didn’t include your personal doom scenario, sorry!

So why don’t I think you should be worried about these things? Well, you probably should! At least to some extent. In each case the data is kind of there that these things are a Big Problem™! The question is how much you should worry about them.

Implicitly or explicitly, the thought process taken by people who worry excessively about problems similar to the above seems to be “X is a problem; I can’t see a solution; therefore society is over.”

This form of “absolute” argument leading to a general vibes based pessimism is an appealing line of thought. It forms a kind of attractor in the space of ideas — once you are fixated on a problem with no obvious solution (or no solution which you can directly affect), the world feels fundamentally broken in some way and the actions of others not directly in service of working on this problem feel at best inexplicable or at worst malign.

At the level of reality, however, this line of thinking is unhelpful and potentially malignant. The history of pessimism is littered with examples where an impending disaster turned out to be irrelevant. Moral panics come and moral panics go, and the world moves on. There’s even a Twitter account dedicated to the exhibition of past instances of moral panics focused around things which seem laughable today.

In extreme forms, pessimism can be counterproductive at the object level in several ways. The first is that its logical conclusion can spawn an almost self-pitying state of inaction. This is perhaps exemplified by the (slightly tongue in cheek) “death with dignity” strategy announced by one of the most prominent groups focused on AI existential risk.

A second trap inherent in doomerism is that it can actually lead to an almost religious reverence for the problem itself, in which advocates for the problem being a problem, leading people worrying about a problem to take a position which I can only characterise as their saying “a mere solution will not suffice to solve this problem!” Or, even more bafflingly, they mockingly call anything proposed “solutionism” or “techno-solutionism” implying that the proposer is hopelessly naïve and has impure motives. Maybe they are and they do, but we can just as easily turn these sorts of ad hominem arguments against the doomers themselves and, further, such a reverance for a self-selected problem is risible, to the point that it feels like the problem has become a twisted sort of religious idol, so embedded into people’s identities that it would be painful to see it solved (or not be an issue in the first place).

Furthermore, often followers of a particular doom cult will claim to completely reject the validity of other problems, or other aspects of life, in the face of their own chosen issue. One of the most interesting things to watch is when tow forms of doomerism interact. While both sides agree that Everything Is Terrible And Becoming Worse, they will fight each other to the death of the reasons why we are so doomed. My favourite example of this is the fight between people who are worried about AI for social reasons (of amplifying biases / etc. in society), and true AI doomers, who hate each other even more than their common enemy of techno-accelerationists!

Tellingly, many such extremists still care about other issues in their personal lives. And of course we cannot begrudge them for this— unless one is actually a terrorist on behalf of an issue, there is kind of an inevitability in this: by definition, doing anything that is not furthering the cause of an issue is implicitly caring about things outside of the value system constructed around a particular issue.

Furthermore, reality is huge and extravagantly complex, so the remedies to problems of the past are often not only not clear at the time, but not clear in hindsight! An interesting example of this is to do with the “population crisis” itself — which used to be taken to mean the impending overpopulation of the world. As a result of falling birth rates this completely reversed in the subsequent decades to the point where now people are worried about a declining population. Which is probably a bad thing, but to say civilisation is going to crumble soon also seems like a gross exaggeration.

Social dynamics amongst human networks and exponentials in technology are powerful and very hard to predict in advance. This is obviously a double edged sword — pandemics spread much faster than one would expect them to from a linear extrapolation, and, yes, the intelligence singularity may creep up unexpectedly. To balance these concerns, though, if you’re worried about rapid progress then these worries should be tempered by an equal and opposite optimism that solutions can quickly emerge, such as that of the vaccines in Covid, which were widely predicted to be far slower than they were, or, potentially, alignment in the context of AI risk, or batteries and solar in the context of climate.

I like Tyler Cowen’s approach to thinking about these issues related to the future, prescribing a dual mandate of radical uncertainty and empiricism. We don’t know what the future will look like, but it will probably look somewhat like the past combined with things we can extrapolate given the current environment, and we have some large degree of agency over it. So while it is important to think about the problems of the future, we should work towards alleviating and yes, solving, them, tackling them with the practical mindset of problem solving rather than being paralysed by vague bad vibes.

Recently, some of the building blocks for systems that will enable individuals to virtually transport themselves and perform physical tasks remotely via robots have been coming together. By making people able to travel at the speed of the internet, and by abstracting human labour from the constraints of the human embodiment, telerobotics systems have the potential to upend many areas: travel, construction, manufacturing, laboratories, and more.

What is telerobotics?

Telerobotics is the set of technologies that allows a person to feel as if they are present in the world in another location, while simultaneously enabling them to affect the world in that location.

Why should I care about telerobotics?

If telerobotics happens in a big way, it will make the physical world location-independent.

Economically, this would have huge implications by making labour able to travel across borders at the speed of the internet. Construction workers could work remotely from halfway across the world just as easily as computer programmers.

In terms of leisure, it would allow travel to anywhere in the blink of an eye. Travel may shift from being plane-based to internet-based.

It could expand the limits of human physical abilities by enabling superhuman strength, faster reflexes, or any feasible extension of the human form when imbued with robotic components. 

What is the difference between telepresence and teleoperation?

Teleoperation is the operation of a robot at a distance. This is used in a wide variety of industries right now - operating cranes, drones, and so on. 

Telepresence is the subset that enables the user to feel as if they have been virtually transported to a new location, while at the same time giving them the freedom to move around and interact with the world. It requires much more general technology to be built, as what is being done is defined by the user's desire to act rather than the specific context of use. 

In general, the domains that are likely to benefit from general-purpose telerobotics as opposed to telepresence are more unstructured and closer to human scale.

What are some important applications for telepresence?

There are a huge number of possibilities both economic and non-economic for the technology; I will attempt to elucidate a few here.

Travel - If telepresence robots get good enough, the need to travel could be eliminated or substituted. Why travel to Japan when you can walk around Tokyo from your living room in Toronto?

This one cuts both ways, by the way. If you want an Italian dinner as part of your telepresence tour of Rome? Have the chef make it in your house via telepresence.

Business travel could also benefit from this. Things like factory tours or office tours or site inspections will be able to be automated.

Construction is notoriously hard to automate due to the diversity in the way that buildings are assembled, and the difficulty of automating physical interaction in many of the situations they encounter. Telerobotics could imbue construction machines with the ability to operate such hard-to-model situations by bringing human judgement and dexterity as required. Construction telerobots would have numerous benefits over humans:

• When there are no humans on site, safety requirements during construction can be reduced, bringing down building costs.

• Workers would not have to travel to construction sites, saving time

• Workers could be pulled from a larger pool of labour around the world, driving costs in these sectors towards the median (hopefully accelerating progress in the physical world and helping to combat cost disease).

Maintenance and Manufacturing - to the extent that a) developed countries want to bring manufacturing back onshore, b) that manufacturing involves labour which is prohibitively expensive in their jurisdictions, and c) those processes are difficult to impossible to automate, telerobotics provides a solution by allowing workers from locations where it is economically viable to remote-in to onshore factories from offshore.

There is also a long tail of cases where maintenance is required and either something has to be shipped to a manufacturer or a maintenance engineer has to go on site to repair something. These situations could be resolved by having a telerobot locally operated by a manufacturer's technician remotely.

Scientific research - there is a tremendous amount of lab work that is still not automated and requires people to be there. However, labs are unevenly distributed around the world, expensive equipment is often not used efficiently, etc. Improving access and improving utilisation of such equipment could allow the use of all lab hardware in the same way raw compute is used today - seamlessly timeshared and available to all researchers for a fee.

Elderly care - While on the upper end of requirements for safety, costs of in-home care could be reduced hugely with the availability of fractional time on a robot with similar capabilities to a human. Contrarily, telerobotics could also enable previously infirm or physically impaired people to operate in the physical world at a normal level of capacity.

What previous efforts have there been? Why did they fail?

The basic idea of telerobotics is obvious, and there have been various efforts towards creating such systems going back to the mid 20th century. However, nobody has built a full-stack solution that gives a person controlling a robotic platform a sense of presence and the capability to act in spaces designed for humans.

Earlier efforts failed for two reasons: lack of technology, and tailoring the technology towards specific problems without generality in mind. The technology for enabling technologies such as long-distance high bandwidth video IO, legged robotics, or virtual reality has not existed since recently.

For example, some recent efforts include "iPads on sticks", pioneered by such companies as Double Robotics and Anybots:

These failed because of the specificity of the use case, and compared to using general-purpose tablets and smartphones the tech was expensive.

There have also been projects trying to build systems with off-the-shelf components or adapting existing systems, such as The Centauro Project or Sanctuary AI.

Ben Reinhardt raises interesting questions about whether telepresence can be developed in a commercial or academic context. In the former the urge to develop immediately commercially useful systems drives projects towards domain-specific solutions. While in academia, incentives often drive labs towards working on small pieces of the problem. In both cases, the proximal milestone is often a demo which, abstracted from a serious context of use, results in systems that seem good when using them for ten minutes but fail upon contact with actual problems.

Why do we need robotic telepresence for travel and meetings when we have Zoom?

Video conferencing technologies are limited in a variety of ways. They don't give the remote user any ability to move around in the world, creating a dependence on the local user. There is also the burden of continuous undivided attention staring at someone through a webcam.

Perhaps VR will solve this need for most business meetings. But people want to communicate in a whole manner of different settings beyond the (virtual) boardroom, whether it be going on a factory tour, site inspection, or during a walk in a physical space.

Why do we need robotic telepresence when we have robot autonomy?

Teleoperation has taken a back seat to full autonomy in robotics as it is often seen by roboticists as a less 'clean' solution. But in many cases, full physical automation is currently far away (construction), and in others it is a category error (you can't automate the experience of a site tour).

The solution is robotic telepresence, bringing many of the benefits of bits (travel at the speed of light, independent of location, equitable access) to the world of atoms (travel at the speed of a ship or the cost of a plane fair; location matters).

What about AGI?

If AGI (which for the purposes of the question I take to include physical intelligence) actually happens, then yes telerobotics as a way to augment labour becomes irrelevant, as presumably at that point it will be taken over by machines. Though it is still likely to be useful for leisure and personal connection purposes. 1

How does telerobotics relate to the metaverse and VR? 

In the short run, the race to develop metaverses will stimulate investment in enabling technologies. This is because of the large areas of technological overlap between telerobotics and VR-type technologies (animation, simulation, and computer vision are all used by robotics to some degree, and telerobotics itself will rely on great VR). Medium term, there may be a desire to allow the merging of the physical world and metaverse(s) in specific contexts, in which case there would be a direct case for developing robotics in the context of a broader metaverse effort.

What are some wider social and economic considerations around telepresence?

Mobility of labour - with telepresence robotics, in theory, the wages for physical labour will be pushed towards equalisation around the world. Some places may try to regulate this to protect local jobs, however this may be difficult to police, and there would be pressure from projects that could vastly benefit from the technology.

Wealthy countries where capital is cheap will be able to afford to buy telerobots and will leverage labour from jurisdictions where it is cheaper. So it may discourage the development of physical infrastructure in low-wage jurisdictions if workers can telecommute to high-wage areas.

Another lens through which to view the impact of telepresence is that of Stratechery's framework on the dichotomy between the physical and virtual worlds:

...in our world the benevolent monopolist is the reality of atoms. Sure, we can construct borders and private clubs, just as the Metaverse has private property, but interoperability and a shared economy are inescapable in the real world; physical constraints are community.

From one perspective, telerobotics helps the real world to acquire some of the advantages of the digital world, such as location-independence and access control. However, the current limitations of the real world act act as positives in the way that they prevent it from becoming distorted by economic incentives towards proprietary standards and gatekeepers. If you can simply turn off someone's robot access when they deviate from the norms in your telerobot city, your agency is at the mercy of the entity in charge of the telerobotic equipment, in the same way that in the digital economy users are beholden to platform owners today.

Telepresence doesn't bring the scale inherent in many digital systems - we'll have to wait for autonomous robots for that. However, it enables the reading and writing of atoms over the network. This is something truly novel and a huge capability unlock for the internet, which thus far has mostly been limited to the domain of information and relied on poor proxies to the physical world (low-wage workers in-person, or only being able to organise systems rather than control them).

What are the functional requirements for telepresence technology?

There are 3 key components: the robot side, the user side, and the connection. We can break down each of these components into core functional areas.

The robot requires

1. Some way to move around (locomotion)

2. The ability to affect its environment (manipulation)

3. Sensors to perceive the world for the human (perception)

The human interface requires

1. Some way of providing sensory information for the human (display)

2. The ability for the human to communicate their intents about what the robot should do (input)

The connection requires

1. High bandwidth & low latency

2. Portable to allow freedom of movement in areas the robot will operate in

What current technologies exist to fulfil the listed needs? What needs more R&D?

On the locomotion front, legged robotics has made huge strides in recent years, with 4 legged locomotion basically being a solved problem using learning methods. It can be produced on a deterministic schedule using off-the-shelf software. And systems for bipedal walking are improving quickly, with products such as Digit reducing the cost to the order of hundreds of thousands per unit rather than millions (and in the realm of quadrupedal robots, you can even buy a small robot dog from Xiaomi for US$1500). Unitree is planning to release a humanoid for $90k(!) soon, and presumably at some point, the Tesla, Figure, and 1X offerings will roll out. The reliability for such systems would need to be high to produce more than a quick demo; this is unknown, or known to be bad, on many current systems.

For manipulation, progress in controls will allow the usage of more complex hardware than traditional 2-finger robotic grippers. For example, learned task-space style control can allow low-level robot motion to be generated from only inputs of the trajectories of objects to be manipulated. This could help combat latency and may make high-fidelity haptic user feedback less necessary for complex manipulations. Real-time hand tracking could be leveraged in tandem for input; combining both explicit hand pose reconstruction and task space control could help improve speed (which isn’t great with grippers).

Hardware for anthropomorphic robotic hands remains an open question. It is advancing quickly with both efforts to make more robust versions of products like the Allegro hand, as well as other startups (eg Clone robotics) trying to make much more anthropomorphic hands that are purportedly much stronger.

With regards to sensing on the robot, from a controls perspective for both manipulation and locomotion it seems RGBD input will provide most of what you need. Tactile sensors on the hands would also be ideal for manipulating soft objects; fortunately these are currently coming down dramatically in cost.

On the human interface side, many challenges are shared with Virtual reality: display size and FoV, convenience of the hardware, and how to provide tactile feedback. But there are a couple of key differences. So the display tech will likely be able to piggyback on increasingly good AR and VR tech from, for example, Apple and Meta. In telerobotics you are controlling the robot remotely, so in contrast to VR's rendered worlds, you can't get near real-time feedback to the user. Thus it seems likely that some degree of prediction of images, or reconstructing the world in a simulator, predicting forward, and rendering it live, will be needed to combat this.

The fact that the telerobot is interacting with the physical environment means that tactile feedback is likely important. There are a few proven ways of doing this, for example, HaptX's system which relies on directly applying force on the user's hands, and Facebook's wrist-based system. But many of these technologies are in relatively early development (the former requires you to wear a backpack!), so it's unclear what the right set of tradeoffs is - do you scrap haptics entirely and do manipulation through more gesture type interfaces with no tactile feedback, trusting the robot's own control system? Or is high-fidelity feedback essential for communicating user intent?

For connection hardware, it seems piggybacking off either 5G or some of the newer LEO satellite platforms like Starlink if you needed the robots to go into areas with poor service would be the way to go. There is the question of figuring out what level of bandwidth and realism you need for your application. It seems the ceiling is quite high here based on what I’ve heard about Starline, the most realistic telepresence system to date.

With regards to what to work on out of these areas, I agree with Ben Reinhardt's take that a systems approach is needed, thus the impact of working on any individual area in isolation is limited. Most of the components exist but can only be built properly in their final form when accounting for the whole system.

The adjacent possible

Developing telerobotics technology will have several auxiliary benefits:

Bootstrapping autonomous robots - If telerobotics reaches scale it will dramatically reduce the cost and improve the hardware capabilities of robots. If scaled up enough, data on a large variety of manipulation tasks will be collected. These two factors combined are likely to allow gradual automation of many repetitive tasks in software; likely starting with "Tesla Autopilot" like assistive features before scaling up to full manipulation tasks.

A "killer app" for VR? - As telepresence requires a VR experience to be truly immersive, if it becomes widely adopted (for example as a proxy for travel) it will pull VR along with it. This is likely to cause a much larger shift towards VR technologies once individuals have access to and are used to these systems.

Any good resources for further reading?

Marvin Minsky’s Telepresence Manifesto (1980s)

Ben Reinhardt's "Bottlenecks in Telerobotics"

Venkatesh Rao articulates a good case for biomorphic robots here, which I agree is a key aspect of making telerobotics actually general-purpose

CWT with Balaj Srinivasan has a short segment on the topic

There was an XPRIZE for telerobotics in the recent past.

If you have suggestions for more items for me to include on this FAQ or are interested in discussing these topics further, get in touch.

This post takes stylistic inspiration from the various longevity and climate FAQs which others have written.

I spent the summer in Switzerland working at Nvidia and ETH Zürich (on robotic parkour). It was not my first time there - I had lived in Switzerland many years ago and have spent some time there since, but that was a) in Romandy (the French part), and b) before I was old enough to make interesting observations about my time there. So here are a few collected thoughts that may be of note about the country to those who haven’t visited or only been there for short periods of time. Even if you have no connection to the country, it is hopefully still of interest at least for the reason that things are inexplicably so much nicer there.

Work & School

Zürich has the feel of a tech hub in a way that I haven’t really experienced outside of North America. This is helped by the small size of the city, and also because they have many of the ingredients to make it a tech hub — high wages, loose immigration policies (especially from other European countries), a world-class CS research university (ETH), and the presence of large tech companies. I think the question for Zürich longer term is whether they can have successful startups of their own rather than relying on large foreign employers. To this end, there is a startup scene there, but their ability to create a domestic tech champion is yet to be proven, so the jury is still out on this one.1

With regards to the working culture, it feels as if remote-work is less common even post-Pandemic. Labour laws are also much stricter than in North America — good luck trying to find an open supermarket on Sunday or after 8pm or so.

I don’t have anything particularly new to say on the topic of schooling but it is interesting that in Switzerland overall a much lower proportion of people attend university. They have a two-track education system similar to Germany, and the streaming happens early, around ages 11 or 12, and entry to the tracks is merit-tested. The flip side of this is that entry of university is not competitive at all — people with high school diplomas from the university stream are automatically admitted. This means that “too many” people end up going to top universities like ETHz and EPFL, and consequently these places make the first year brutally difficult, basically to filter students, and consequently the graduation rates are only between 10-30% in STEM fields. I don’t know whether this is good or bad but it it is a very different system than is in place in many other Western and particularly Anglo countries. I think overall it probably helps prevent a march of credential inflation like we have in the Anglo world, but probably misses some late bloomers and also means that Switzerland has to import a lot of talent to fill positions that would otherwise be filled by domestic PhDs.

Language

One of the most interesting aspects of the experience as an international in Switzerland is the country’s relationship with language. Swiss German - the dialect or more accurately collection of dialects which are used there - is both primarily an oral language and isn’t used in formal contexts, though you will sometimes see it in ads. This means that while understanding/speaking German (very broken in my case :) can be helpful for filling out forms or getting by in shops, foreigners can’t really talk to the locals in their own language due to the difficulty of learning or even understanding it.2 This makes somewhere like Zürich feel distinct from many other international hubs, as there is more of a difference between true locals and first-gen immigrants due to the language barrier, despite most of the former speaking English.

Culture

Compared to what you’d find in a North American city, people in clubs etc. there are a much wider diversity of ages in Zürich. It is also a huge techno city so the energy in these places is distinct from more pop-y places I’m used to with a much more chill vibe. I think that this probably reflects a difference in attitudes to a) leisure and b) alcohol in Europe broadly vs NA culture. Smoking in clubs is also not uncommon.

I was lucky that this summer coincided with the first running of Züri Fäscht (Zurich Festival) and Street Parade, both events where millions flood the center of downtown Zürich over the course of a weekend. The scale of these events relative to the size of the city is really crazy, with more than the normal population of the city worth of people being downtown at once. Of course, being Switzerland, the downtown which gets covered in litter is completely spotless by Monday morning…

More people read books in Switzerland than in Canada, and there are consequently better stocked bookstores. Every moderately sized Bahnhof has an Orell Füssli (and possibly other smaller book sellers) which always has, aside from the overflowing shelves of Swiss Krimis, ample classics and philosophy sections. And Amazon won’t be of much help in Switzerland — they don’t operate there, and one can expect to wait for a week and pay exorbitant shipping and tax for books from Amazon.de.

Government and Politics

Switzerland is a weird mix between European social democracy and deep-seated small-c conservatism. On the one hand they have a bureaucracy where everything must be declared - you need to register in the exact area where you live, notify the government exactly which health insurance you participate in, and so on. But this exists side-by-side with a deep scepticism of change and government interference — every week there is some “scandal” in the papers about how solar panels are going to be mandated in some municipality or how the federal government will destroy the country by letting in too many immigrants.

Despite its immense wealth and privileged position as a talent magnet, Switzerland suffers from the same affliction of populism in the form of the SVP (Swiss People’s Party.) But interestingly, as has been the case with many places in Western Europe, despite being the biggest party they have struggled to enact many super reactionary changes. Even their signature achivement, (the revocation EU citizens’ right to movement to Switzerland) a compromise was found which effectively enabled the previous position to continue as before.

Transit

Swiss transit is interesting because it serves dual-purposes — being both a critical artery for local commuters but also as a gateway and huge source of revenue for Swiss tourism. The network's density, even in the many sparsely populated areas of the country, is convenient, and its reliability is famously amazing.3 And it extends to even relatively remote regions of the country which is wonderful for going hiking.

Perhaps part of the reason why the Swiss mid distance rail network is reliable is also that geographically even the denser part of the country is composed of relatively small cities without the population to support their own metro systems, and hence there are large relative numbers of inter-city commuters. Even Zürich, while having a few stations which are connected, doesn’t really have something like a subway system.

For tourists buying one-off tickets, transit is shockingly expensive, but living there as a local there are ways to pay much less (cheaper tickets for locals travelling at night, monthly and yearly passes, and so on.) The difference between these pricings is an interesting example of market segmentation.

Climate Control (of two kinds)

Probably the single worst functioning thing in Switzerland is climate control. Like much of Northern Europe, Switzerland eschews air conditioning, thinking their climate to be sufficiently frigid to can sweat it out in the summer without AC and switch on fans when it gets too hot. This may have been the case 20 years ago, however it is clearly at this point untenable and suboptimal for productivity, as the summers now are so hot that it is hard for many to work in swealtering office blocks designed for temperate alpine climes rather than summers which are now more akin to something you may have once seen in the Riviera.

Relatedly, one thing that struck me is how seriously my friends there, and people in general, take climate change compared to those in English-speaking countries. I do have some friends in Canada who care about climate change a lot as an issue, to the extent that they have changed their lifestyles for it (eg stopped or reduced flying, become vegetarian, etc.) but they are isolated cases. This is the opposite in Switzerland and Europe in general, and people over there generally both seem to be more pessimistic about the impacts of climate change and more willing to make personal sacrafices to combat it.

Moving

Having been back in Toronto for 2 months now, it surprises me how both how quickly I slid into a new routine once I moved to Switzerland, and the speed with which I re-adjusted to being back in Toronto. It feels like a weird form of amnesia as my life just over two months ago was thousands of miles away in a completely milieu and yet I fell very quickly back into old habits. This is both somewhat frightening and interesting to me as I consider what I’m going to do next…

For more reading like this post, but longer and more comprehensive, see Why Switzerland? which has a lot more info than this post but aligns with my experience or Europe Notes, which focuses on Germany not Switzerland but which I felt the desire to link to because it is so fascinating.

Georgia

On the way from Dublin to Tbilisi I learned the Georgian script using this online tutor. While no great achievement (the Georgian alphabet is only about 30 characters and phonetic), it was the first time I have learned to read in a script other than Latin and was a fun experience to be able to decode signs (though not necessary if you are there since most signs are surprisingly re-written in English). I obviously didn’t know the meaning of most Georgian words but a lot of words for newer objects or places are the same as in English with slightly altered endings, so it was actually somewhat useful.

Tbilisi in winter is not the most colourful but there is something appealing about the combination of the run-down post-Soviet aesthetic of the city with the backdrop of mountains. As a tourist, it is less of a city with “major attractions” and more a place with lots of small discoveries and observations to make about times past and history in the making.

The spectre of Georgia’s precarious political situation hangs over Tbilisi and is constantly in your face. In the old town (which doubles as a commercial district) on many streets you are more likely to hear Russian than Georgian spoken. The government has quite loose visa requirements so Russians dodging the draft or domestic inflation fill local upscale eateries and cafes, boosting the economy but pushing up house prices for locals. Russian has been the lingua franca since Soviet times, and old books in Cyrillic line many street markets and bookstores.

However the age of those books also suggests it is possible that this orientation is an atavism. The country drapes itself in the European flag; support for Ukraine is everywhere. Although the government is more apathetic or even sometimes conciliatory towards the great northern neighbour than the populace out of fear of retribution, ordinary people continue to seethe about the regions under occupation and feel solidarity. This causes tension, which I saw first-hand in a protest outside the presidential palace one day while walking by. English rather than Russian is the language taught in schools there today. Even since our visit, there have been major protests against the government’s Russia-appeasing policies.

We also went to the Caucasian mountains. My favourite single sight from the whole trip was the monument of “Friendship” between the Georgian and Russian people, a Soviet commissioned concrete circle, which contains a couple of discrete subversive jokes on the 360° mosaic on the inside. The road we took leads right up to the Russian border, and it is clear even post-invasion that this remains major corridor for the exchange of goods and people (interestingly it seemed that there were quite a few EU registered companies running the trucks).

There was some laughing on our tour to the Caucasian mountains when the tour guide asked if anyone was vegetarian. Georgian food is quite meat-heavy, with the highlight being beef-filled dumplings called Khinkali. There is a great debate over where to get the best Khinkali, and rules on how to achieve this ideal remain impenetrable to me, though the variance in quality between different varieties isn’t especially high and they are mostly great. Other highlights are the Sulguni cheese which they use to make Khachapuri (flat bread with cheese), which is amazing. Overall I think Georgian food represents the greatest aspects of Western cuisine - heavy meals rendered to resist a harsh cold and environment unapologetic in their usage of meats and breads.

We spent New Year’s Eve in Tbilisi watching from the hill overlooking the city with the TV tower. Georgia polices private fireworks very lightly; never before have I been scared when watching a fireworks display. Firework displays here are decentralised. Which means chaos. Rather than having one big display, private citizens launch their own fireworks, so the city becomes a blur of pulses of light and machine-gun like blast sounds as 00:00 nears. From where we watched on the hill, there were also people launching rockets over the balcony from the middle of the crowd. One of the fireworks misfired, and the coat of one of our fellow onlookers was tempoarily set ablaze (he was OK in the end, though only after sequentially ignoring 10 people who were trying to tell him). Not for the faint of heart but an unforgettable experience.

We visited to one of the contemporary galleries in the centre of Tbilisi. Being relatively ignorant of how art worked in the USSR, I was surprised by how much got done during the Soviet period, although disappointingly many of the artists too closely mimicked various earlier styles from other countries. The best of the exhibited artists for me was Vladimir Kandelaki, who seems to quite effectively fuse political subversion, a pleasing palette, and a sense of whimsy in his work.

Istanbul

The rumours are true, Istanbul is filled with cats, and more than one black cat cross my path. Even as someone generally suspicious of cats this was fun.

The attractions in the main Sultanakhmet area are undoubtedly overtouristed but worthwhile nonetheless. The most spectacular is the Hagia Sophia, which is externally underwhelming but incredible inside. Topkapi Palace was underwhelming to me but is steeped in history.

The Ottomans become huge Francophiles towards the end of their rule, so there is a lot amount of fusion between classically French and traditionally Eastern architectures, food, and art in late Ottoman artefacts. The pinnacle of this for me was the Dolmabahçe palace, an absurdly ornate 19th century construction filled with amazing architecture and orientalist paintings.

The food in Istanbul is consistently good but nothing blew my mind. Perhaps it is because the Middle Eastern cuisine in Toronto is amazing overall1 but fare in Istanbul did not elevate itself far above other Middle Eastern or even Turkish I have had. Eating in Sultanahmet will leave you overcharged and underwhelmed, at a minimum walking just across the river to Karakoy will give better selection at lower prices. Food on the Asian side is cheaper and in my (brief) time there more varied overall.

Signifiers of the stormy Turkish political situation are constantly present in the background. The Hagia Sophia was recently converted from a museum back to a mosque, with Erdoğan the secular government’s decision from the ‘30s. And unlike in Georgia, nobody in Turkey cares about Ukraine other than instrumentally (the one flag I did see was on the German embassy). Ataturk (the founder of Turkey in the ‘20s and ‘30s) is also highly revered - his image is everywhere.

The museums are where you feel the Ottoman legacy most strongly. A cornucopia of imperial artefacts fills the museums2. Byzantium3 gets short shrift in captions and descriptions of artefacts and buildings.

Istanbul in many ways feels very European and so to me familiar. The streets in the old town are a busier and more lively variant of what you could find in, say, Budapest, but the skyline filled with Minarets and multiple-times daily calls to prayer provide a reminder that you at the nexus between Europe and Asia.

Istanbul as a modern city of efficiencies copes surprisingly well with the Bosphorus as its surroundings. I was astonished by the efficiency of transit links across it - ~20 minutes by boat, and less than that by undersea metro system (which doesn’t drop 4G connection even as you pass through it).

While Georgia and Istanbul are hard to compare directly, the overall atmosphere in Tbilisi was more amicable to me, while my brother preferred Istanbul. If I were to visit Georgia again I would want to rent a car since there is not very centralised or developed outside the centre, and there is a lot which you need one to access. In Turkey I would want to visit somewhere on the Mediterranean and probably Izmir to see a different part of the country and more history.

Many (including Tyler Cowen himself) think that they will negatively impact the ability of authoritarian governments (such as that of China) to censor information, and even if it is possible they will require China to develop inferior or hobbled models in some way in order to provide censored answers. From MR -

Will Chinese LLMs be much worse? Presumably these are being built right now.  But which texts will they be trained upon?  Let’s say you can keep out any talk of T. Square.  What about broader Chinese history?  Do you allow English-language sources?  Japanese-language accounts of the war with Japan?  Do you allow economics blogs in English?  JStor?  Discussions of John Stuart Mill on free speech?

Just how good is the Chinese-language, censorship-passed body of training data?  Does China end up with a much worse set of LLMs?  Or do they in essence anglicize most of what they learn and in time know?

Pre-LLM news censorship was an easier problem, because you could let the stock sit in a library somewhere, mostly neglected, while regulating the flow.  But when the new flow is so directly derived from the stock, statistically speaking that is?  What then?

Source

and

LLMs will have many second-order effects on censorship.  For instance, Chinese may be more likely to use VPNs to access Western LLMs, if they need to.  The practical reason for going the VPN route just shot way up.  Of course, if Chinese citizens are allowed unfettered access (as I believe is currently the case?), they will get more and more of their information — and worldview — from Western LLMs.

In short, the West has just won a huge soft power and propaganda battle with China, and hardly anyone is talking about that.

Source

In a recent episode of the ChinaTalk podcast the guest (a director at CNAS) also claims it will be hard for China to tamp down on language models as any form of diverse training data contains views that are contrary to those of the ruling party.

I think that these arguments are wrong and fall prey to the same style of logical traps that Tyler accuses AI doomers of promulgating. Consider the following

1. LLMs make it far easier to explicitly ask whether a piece of content in textual format contains information that would be sensitive to a particular party

2. They can do this at the same scale as the amount of compute available which is available at the scale that fake content that can be produced

Given this, a platform or government with a desire to censor could do it using another LLM to "review" the output of the first model and modify it according to the desired guidelines. In this way, LLMs can self-censor in the style of Just-ask-for-X. A platform which provides a ChatGPT-style interface somewhere like China could just pass over the output of an unfiltered LLM with (possibly even the same) LLM that is explicitly asked to detect content sensitive to the censoring authority and not respond to certain questions or modify the output to conform to the stated ideology.

Such a system would not involve “hobbling” the abilities of the underlying model in any way by fiddling with the training data or even performing RLHF or similar on it - it can work out of the box using the same model which the user is interacting with, and would only impact the output or ability to debate when a sensitive prompt or series of prompts is posed. For example this flow already works with GPT-4 out of the box —

You could even go further and have RLHF on top to make the filter wise to the subtleties of getting around censorship on the Chinese internet in real time. In fact, it already seems as if Midjourney does something similar to censor topics sensitive in China — and nobody is even forcing them to do it!

I also disagree with people claiming that LLMs will be an effective tool to overwhelm censors by generating adversarial content. To the extent that the best models are openly available (in the sense of open source), China can use them unrestrictedly for censorship by incorporating them into their own systems as described above, filtering out even generated content which they don’t want. To the extent that they are not open, access to them will either be geo-blocked by the provider (ie OpenAI, Google, Anthropic, etc) or by the GFW.1

Of course one can also say that you could smuggle something like LLaMA into China, along with the ability to ask it uncensored questions. However, already if behind a firewall you can circumvent it by loading up Wikipedia on a hard drive, or going through a VPN to get access to articles about sensitive events — but people don’t do it enough to matter politically as it is hard. There are language barriers, and huge downloads of databases of websites are cumbersome. None of this is different for LLMs2. Convenience has always trumped all in consumer tech and I don’t see why this time should be different.

The same reasoning goes for using VPNs to access ChatGPT etc. unrestrictedly. If it was easy enough to go through VPNs to access such services, nobody would have felt the need to build (now banned) WeChat plugins to proxy it.

Furthermore, it is actually harder to pull out information about sensitive events from LLMs than it is for something like Wikipedia, since you have to know ask it the right questions instead of stumbling across it in other contexts.

The argument has also been made that the smaller amount of training data in Chinese could have an impact on quality in the Chinese language and thus cause them to “fall behind” in a classical arms race in the ability of their models. I think that this is less of a big deal than predicted — Chinese companies can train on the Western internet, and it seems that many of the capabilities in LLMs port across languages (for example models which are RLHFd in only English in my experience carry over the underlying modified behaviour to German if the base model is multi-lingual). Also Mandarin as a language is not that much smaller than English in terms of number of speakers, thus there is ample training data for it to become decent at speaking the language and hence I don’t think there is a data reason why models should be worse speaking Chinese than English.

I obviously hope I am wrong on all this as censorship is terrible, but to me the debate around how AI will circumvent it sounds strikingly similar to 90’s era claims about how the internet was going to bring down dictators which we now know were abjectly wrong.

Subscribe now

Niall Ferguson's Kissinger biography was structurally interesting in the way in which it doubled as a history of the early part of the Cold War, as well as an account of Kissinger's life. Which, whatever you think of his later actions (I have no opinion as volume 2 isn't out yet :) is astounding. Born a German Jew and having to flee the Nazis, he escaped to the US only to be drafted into the military and return to Germany as a liberator. The book, as many from times of crisis do, made me wonder about whether it people of similar mettle emerge in other times, or whether they can only be forged amidst social strife and upheaval. It might be telling that it was Kissinger, now 99, was the person giving the keynote speech less than 2 months ago at the World Economic Forum.

I am slowly trudging through Die Welt Von Gestern (The World of Yesterday) by Stefan Zweig1, which is his account of the changes in Europe and particularly Vienna through the end of the Bellé Epoque, through WWI and then the rise of facism. Like Kissinger, the book is really interesting when thinking about stability vs. rapidly changing times and the kinds of people they produce. His horror at what befell Europe during his lifetime (the early 20th century) and especially the way in which was not only a tragic event but also extremely surprising from a contemporary perspective was fascinating. As are his descriptions of how the pathologies of decadence in the Vienna of the old Austro Hungarian empire echo many of those today - the stultification of education, the over-professionalisation and regulation of youth, etc, which are yet almost compensated for by the benefits of such a time of plenty, as manifested for instance in the literary heights someone like Zweig was driven to in such a zealously competative atmosphere of overly studious peers.

I also recently read Anna Karenina. It was decent overall, my favourite parts being the drawn-out but beautiful descriptions of Levin on the farm as well as of Anna's angst, but overall it didn't strike me as a particular masterpiece: he scenes from society were relatively flat, and many of the characters were somewhat caracaturish and not well developed in terms of their motivations.

As an aside, I was also struck struck by how closely this passage describes what irks me with a lot of the argumentation in current rationality and EA style movements:

I just started reading Knausgård's My Struggle (Min Kamp) and, being halfway through the second volume I cannot provide a full evaluation, but I will say what a fantastic book thus far. Often compared to Proust2, but transposed into a 21st century context which is in many ways more interesting and painful as the narrator is forced to deal with and describe the picturesque mundanity of growing up in Norway, his alcoholic father, andsudden move to Sweden. How can a story be so crushingly bleak and also in many places hilarious.

What a stupid, bloody idiotic country this was. All the young women drank water in such vast quantities it was coming out of their ears, they thought it was ‘beneficial’ and ‘healthy’, but all it did was send the graph of incontinent young people soaring. Children ate wholemeal pasta and wholemeal bread and all sorts of weird coarse-grained rice which their stomachs could not digest properly, but that didn’t matter because it was ‘beneficial’, it was ‘healthy’, it was ‘wholesome’. Oh, they were confusing food with the mind, they thought they could eat their way to being better human beings without understanding that food is one thing and the notions food evokes another.

My Struggle, Vol 2.

Some other books I tried and failed to read recently (my anti-portfolio so as to speak) --

• Ulysses - the book I was supposed to like... I know many of the locations in Dublin and the vernacular (ah, you'll be grand). And yet something just didn't click for me. To be attempted again...

• Madame Bovary - sorry, I was just bored to death after the first hundred pages. Obviously “well written,” whatever that means, but not enough to hold my attention with not much else going on.

• The Idiot - I think I may be missing some of the theology motivating it but while I think I got what he was going for in the chunk I did read nothing really affected me strongly.

On to other stuff!

• I recently visited the Bay Area for the first time with some friends; we stayed for the first few days in the South Bay where we were honestly bored out of our minds. And we then went to San Francisco, which was one of the nicest places I've ever been. It felt like interesting things were happening and things to do and see and people to meet were in abundance. The homelessness felt like less of an issue than I heard from other sources but possibly we were there at the wrong time? Anyway well worth a visit.

• I will be in Zürich for the summer -- if you made it this far reading this and will be near there we should meet up!

• Now some other random links, mostly to ML stuff…:

I recently watched of Andrej Karpathy's videos on training neural networks. While I knew most of the topics covered quite well already, I still learned some subtle tricks and it was great to go back to reinforce the basics and hear it explained very clearly step by step -- recommended!

The furore about Lex Fridman's reading list from January continues to live rent free in my head.

And then there is the petition and that letter about slowing down AI. Sigh. At least we live in interesting times…

Section 1: World

1. Will Vladimir Putin be President of Russia? 95%: with no election in Russia this year and the relatively stable situation in Ukraine, I think that de Neufville's basic reasoning still holds.

2. Will Ukraine control the city of Sevastopol? 15%: Unlikely given that the city can be resupplied by sea; also I am sceptical of Ukraine's ability to take Crimea while maintaining international support.

3. Will Ukraine control the city of Luhansk? 20%: Ukraine has shown ability to take ground in east however Russian supply lines are short to Luhansk and they have held it against attack for 8 years already.

4. Will Ukraine control the city of Zaporizhzhia? 80%: Ukraine currently controls Zaporizhzhia but it is on the left bank of the river, meaning it is in comparison to eg. Kherson relatively exposed. Russian supply lines are quite stretched in this region so Russia would have to have quite a successful offensive to take it.

5. Will there be a lasting cease-fire in the Russia-Ukraine war? 30%: Right now both sides could not even agree to Christmas cease-fires, so I see this as unlikely.

6. Will the Kerch Bridge be destroyed, such that no vehicle can pass over it? 30%: Both rail and road lines have to be destroyed by the deadline. If the Ukrainians were able to knock both out, one must ask why they didn't do it the first time they tried.

7. Will an issue involving a nuclear power plant in Ukraine require evacuation of a populated area? 8%: Most reactors in Ukraine are either not currently near the front line, or are deactivated (Zaporizhzhia). Thus there would need to be a large shift in the distribution of fighting probably through Russian advances for this to happen, and on top this offensive would need to end up causing a nuclear incident requiring evacuation - fairly low likelihood of both happening in sequence. Alternatively there could be a long range Russian strike on a nuclear facility but this gets into the realm of nuclear escalation which, while worrying, is quite unlikely.

8. Will a nuclear weapon be detonated (including tests and accidents)? 40%: seems as if North Korea is likely to want to test another device, and they have a proven history of doing so.

9. Will a nuclear weapon be used in war (ie not a test or accident) and kill at least 10 people? 2%: There has been sabre rattling but historically there is a pretty low probability of going nuclear and it is not obvious what tactical objectives this would achieve on the Russian side, unless Ukraine advances into Russia itself which it has not shown a willingness to do.

10. Will China launch a full-scale invasion of Taiwan? 5%: A blockade and negotiations are more likely to precede this, as an invasion would be incredibly messy, and China at the moment is in a mood for detenté in the wake of Covid-Zero damaging the economy.

11. Will any new country join NATO? 80%: As of Jan 9th, Turkey is still rumbling about extracting concessions. I think these negotiations can still fall apart.

12. Will Ali Khameini cease to be Supreme Leader of Iran? 10%: He is relatively old so the actuarial tables are simply not good on this one, coupled with the nonzero coup probability.

13. Will any other war have more casualties than Russia-Ukraine? 40%: My starting point is that the major possible flashpoints are the Middle East, particularly any major war along the Iran-Saudi axis, Taiwan, ethnic conflicts in Africa (such as the situation with Tigray), and Syria (e.g. Turkish action to create a safe zone). 

14. Will there be more than 25 million confirmed COVID cases in China? 99%: unsure about the OWID tracker but officials are claiming much more than this.

Section 2: US/UK Politics

15. Will prediction markets say Joe Biden is the most likely Democratic nominee for President in 2024? 80%: If Biden runs, I give this a 60% chance, as the field will likely be cleared for him but the probability for R's probability will be split between at least Trump and DeSantis. I think there is a 75% chance of Biden running at this point - historically presidents run for re-election, but giving a slight discount for Biden being very old. If he doesn't run, obviously there is a 0% chance.

16. Will prediction markets say Gavin Newsom is the most likely Democratic nominee for President in 2024? 10%: You have to both believe that Biden doesn't run and that Newsom becomes the frontrunner.

17. Will prediction markets say Donald Trump is the most likely Republican nominee for President in 2024? 60%: DeSantis is riding high right now after the midterms, but I have read previously that if other candidates enter the field it is likely to dilute his support more than Trump's. Furthermore Trump is ahead in polls even at a high point for DeSantis.

18. Will prediction markets say Ron DeSantis is the most likely Republican nominee for President in 2024? 25%: Splitting difference on previous question and leaving some probability for another candidate to emerge.

19. Will the Supreme Court rule against affirmative action? 90%: All the current rumblings are that they will.

20. Will there be any change in the composition of the Supreme Court? 5%: All justices are quite young and there is no partisan incentive for anyone to retire

21. Will Donald Trump make at least one tweet? 60%: When Trump is explicitly running for president, he will want to maximise his chances.

22. Will Joe Biden have a positive (approval minus dispproval) rating? 20%: It does seem to be trending that way right now, but reversion to the mean and polarisation during a re-election campaign make this become less likely.

23. Will Donald Trump get indicted on criminal charges? 20%: Up to the DOJ but they likely don't want to look like they are pursuing a witch hunt against Trump.

24. Will a major US political figure be killed or wounded in an assassination attempt? 10%: Briefly looking into US assassinations (attempts for just wounds are not easy in my brief research to find good data for), this seems quite unlikely.

25. Will Rishi Sunak be Prime Minister of the UK? 75%: with no incentive to call a GE, and the conservatives unlikely to want to embroil themselves in another costly leadership spill, Sunak looks like a relatively secure if boring figure.

26. Will the UK hold a general election? 15%: Barring gross calamity, there is little incentive for Tories to call an election.

Section 3: Business And Economy

27. Will Elon Musk remain owner of Twitter? 80%: very hard to predict. The most likely reason for him to divest may be if they get into sufficient financial trouble regarding the debt position and Musk has a margin call (?) based on the fall in value of Tesla stock.

28. Will Twitter's net income be higher in 2023 than in 2022? 80%: They have cut many staff and seem to have stabilised the advertising situation.

29. Will Twitter's average monetizable daily users be higher in 2023 than in 2022? 95%: Based on Musk's claims about constantly hitting concurrent users are true and the fact that mDAU was growing already.

30. Will US CPI inflation for 2023 average above 4%? 30%: CPI is trending down currently and the 6-month average is below the level to achieve 4% inflation. We would have to see a considerable re-acceleration in price inflation. Given the Fed's hawkish tone, this is unlikely to come from monetary policy. A more likely scenario for price increases would be a geopolitical shock. Scenarios for this seem to be: Chinese re-acceleration in oil demand due to re-opening, Iran-Saudi conflict both of which could push up oil, while a Taiwan blockade or similar could create a supply chain crisis. None of these are inconceivable but one of them needs to happen to push CPI back up.

31. Will the S&P 500 index go up over 2023? 60%: The Fed is only expected to push up interest rates by an extra percentage point or so and in fact is expected to start easing before the end of the year, so interest rates don't seem to be the likeliest source of further declines. However, there is a looming US recession risk. So a real economic decline could push down stocks of "real-economy" companies. However I think psychological buoyancy over the easing of monetary conditions is a big factor here.

32. Will the S&P 500 index reach a new all-time high? 20%: While the bruising experiences of the last year would make this seem unlikely, as mentioned previously we shouldn't under-estimate the jubilation that could break out upon easing of monetary conditions.

33. Will the Shanghai index of Chinese stocks go up over 2023? 70%: Re-opening makes this much more likely but there are apparently been mini-bull markets already in some sectors and it sounds like there is real economic risk.

34. Will Bitcoin go up over 2023? 40%: There is a pretty high chance of blow ups, and monetary conditions are unlikely to end up being significantly looser by the end of the year.

35. Will Bitcoin end 2023 above $30,000? 20%: The route to this seems similar to the aforementioned potential stock-market machinations (a new bubble driven by lowering rates).

36. Will Tether de-peg? 30%: The story around Tether is weird, given that it does seem that they were making questionable bets with deposited USD assets but have thus far survived. Given that yet another market fall is less likely as inflation is coming down and rates are likely to peak, I think it is more likely than not that they don't blow up, but we will see...

37. Will the US unemployment rate (now 3.7%) be above 4% in November 2023? 60%: Likely recession globally and in the US but the US labour market is still quite tight, hard to predict but more likely than not it will creep above that figure.

38. Will any FAANG or Musk company accept crypto as a payment? 90%: Tesla already does this, so the question is only whether they will backtrack.

Section 4: Science And Tech

39. Will OpenAI release GPT-4? 80%: there are significant rumblings and rumours about this, I think the question becomes mostly about timing.

40. Will SpaceX's Starship reach orbit? 80%: It seems that they are likely to do it early in the year, so something would have to go quite wrong for this not to happen.

41. Will an image model win Scott Alexander’s bet on compositionality, to Edwin Chen’s satisfaction? 40%: Most of the ambiguity around this comes down to the resolution of the question. 

42. Will COVID kill at least 50% as many people in 2023 as it did in 2022? 60%: China's re-opening would appear to make this much more likely despite the drop in mortality worldwide post-omicron.

43. Will a new version of COVID be substantially able to escape Omicron vaccines? 20%: Based on the fact that the original strain from Hubei was used to produce the original vaccines, and the magnitude of escape described did not happen in the first two years of the pandemic, I would rate this as relatively unlikely.

44. Will Google, Meta, Amazon, or Apple  release an AR headset? 80%: Apple - announcement likely mid this year. Google - apparently not till '24. Amazon - doesn't seem like they have any product in an advanced stage of development. Meta - ambiguous but it seems like they already pushed out their initial attempt at AR-ish device. So any such release seems most likely to come from apple.

45. Will an ordinary person be able to take a self-driving taxi from Oakland → SF during rush hour? 30%: Hard to say but seems like they are being quite cautious with the rollout in my quick googling, and both expansion of the area and times would be needed.

46. Will a cultured meat product be available in at least one US store or restaurant for less than $30? 20%: Both regulatory approval will be needed and either reduction in the cost of production or selling below cost.

47. Will a successful deepfake attempt causing real damage make the front page of a major news source? 20%: It seems conceivable that as the next US election cycle ramps up and image and video get better that this could become a storyline.

48. Will WHO declare a new Global Health Emergency? 20%: Apparent base-rate.

49. Will AI win a programming competition? 25%: It seems that the best model right now is AlphaCode and it is at the median level of performance in programming competitions. It needs to improve and these results are new so it may take more time but is within reach with current techniques.

50. Will someone release "DALL-E, but for videos"? 70%: Imagen Video and a similar model from Meta already can do this, but not to a satisfactory level. The question then becomes about rate of progress and resolution of the question - hard to say but I am optimistic given the previous timeline going from latent diffusion models working to having Imagen Video at all.

• Use delta-based rewards instead of kernels. For example, if you are trying to get a manipulator to a certain point, have reward of the following form

  In my experience this works better than rewards of the form

  I suspect this is because the change in the relative magnitude of the reward for making progress towards the goal vs not making progress towards the goal over a single timestep is much bigger.

  • Also when designing stuff from scratch, look for what successful papers have used as reward functions, as they will likely have experimented a lot to find the right reward function and you don't want to repeat this work. For robot manipulation & locomotion, many of the Robotic Systems Lab papers have good references for this which are proven to transfer to the real world [1, p15] [2].

• Episode length - depending on how you calculate advantage, do updates, and how your environment resets, having too long of an episode length can slow down training. Alternatively, if the episode length is too short your agent may not be able to complete the task.

• Control frequency - if set too high, each completion of your task in the simulator will take many timesteps and it will take longer to train.

• The RL algorithm that will work best for you will depend on what kind of simulator you use. On-policy algorithms such as PPO work better when it is cheap to get environment samples (fast simulator). Off-policy algorithms such as SAC work better when your simulator is very slow.

  • Don't try to implement one of these from scratch yourself (unless you are doing it for educational purposes). (Hopefully) tested code in quasi-official repositories will almost certainly work better than whatever you can hack together. Denis Yarats' SAC algorithm is a well-documented extensible implementation. It's also fairly easy to hook this up to Stable Baselines' SubprocVecEnv to parallelise your environment. RLLib has a wide variety of algorithms implemented, which come in with built-in support for large-scale parallelisation (though they are somewhat annoying to modify).

• Action spaces can have a big impact on the ability to learn tasks, and this impact will vary by the nature of the task.

• If you are having trouble training an environment, especially with multiple shaped rewards, try stripping back to the minimal possible objective and seeing if it trains on that one, and add additional objectives progressively.

• Completion termination & bonus - if you want to improve the throughput of episodes as your RL algorithm gets better at achieving a goal, try terminating the episode when the agent gets close to the goal. But be careful to give it a bonus reward for doing this, especially if the value of your reward is positive. If you don't the optimal strategy to maximise the discounted sum of future rewards will be to *not* achieve the goal, since terminating the episode by reaching the goal obviously means that the future value will be limited to the reward at the terminating timestep.

• Use fast simulators like IsaacGym or RaiSim. This will both allow you to train on more complex tasks and improve the speed of your tweak env/algo -> train -> tweak env/algo loop.

• If you expect your algorithm to depend on velocity information or other such things which may not be available with the state in a single timestep, make sure it has a way to infer this. It's possible to do this by stacking observations (concatenate the raw observations from the last 3 timesteps), by directly passing velocity information, or by using a recurrent policy.

Real Robots / Sim2Real

I only have limited experience with transferring RL algorithms to real robots, however I have worked with Other Algorithms™ (yes, I know, they do exist) a fair bit on real robots, so some of these tips will generalise:

• First ensure you have the basics working - are the scaling on algorithm outputs correct, is your action space right, is the control frequency you are running your algorithm matching the simulator etc.

• Before trying 10001 tricks with Domain Randomisation, tuning simulator/algorithm parameters etc., ensure you can transfer your vanilla algorithm with reasonable performance.

• There are better methods than DR for getting sim2real transfer to work coming online which you can try, especially better calibration of simulators [1] [2]. In general, improving how close your simulator is to reality > adding ever more randomisation to your simulator (less work and it produces better results).

• In general, don't try to fix problems in software if there is an issue with your physical robot. Fix that first and you will have much less frustration.

Creating Understanding

Creating and maintaining understanding should be the goal of the kind of learning I'm attempting to describe with this list.

Transcription

One of the most useful techniques for picking up highly technical / mathematical fields I have used is re-writing the content. With a pen and paper, by hand. Not the prose word for word, but writing out equations and proofs helps you both check and build understanding, as it makes it more difficult for you to skip over crucial bits.

Transcription works best when you are actively recalling the information, if only for a short time (ie look at the next couple of lines, store in short term memory, and then try to reproduce). This kind of active recall also helps to force engagement.

I usually don't refer back to these types of notes later (my handwriting is far too poor for that), but without doing this I usually find the content goes... in one eye and out the other?

Spaced repetition systems [SRS]

Spaced Repetition is a technique for building long-term understanding by reviewing flashcards at increasing intervals. It leverages the mechanics of memory to give you exponential recall for linearly increasing effort. SRS is super useful for learning stuff you won't use every day but will need to recall at some point in the future (so if you're a programmer, you wouldn't ankify your favourite programming language but you may add that piece of maths you use once every 3 months). I still find the best software tool for doing SRS to be Anki.

Many others have written far better on this topic than I could hope to, so you should go read them to get the lowdown (and you really should, the method is a game changer):

• Michael Nielsen's essay, which serves an an excellent introduction to the topic.

• SRS is remarkably good at enabling fluid thought in highly technical disciplines..

• Andy Matuschak's essay using spaced repetition will help you write better prompts (how meta...).

• Be sure to check Alexey Guzey's life-changing tip on how to actually follow your own advice using Anki...

Writing

Often the best way to get to grips with fields which are more vague than hard tech/maths is to write about them. The reason writing is valuable isn’t that it is super ‘correct’ or original or anything like that, it simply needs to be written out so you can reflect - it is often difficult to assess ideas in your own head.

As an added bonus, if others like your writing, you can get leverage on your learning by sharing your insights on a blog (hey!)

I was using Roam Research which is all the rage at the moment, but for some reason I can't use it properly and always fail to backlink, and find it hard to navigate. I've really taken to the beautiful Bear app, though.

See also - Why You Should Start a Blog Right Now, and the million other things on the internet explaining why writing in public is valuable2.

Teaching

For much the same reasons as writing, teaching can really help clarify stuff in your head by forcing you to express it. If you are in a course and have others around you learning the same material, teach them. Doesn't work as well if you are learning on your own, unfortunately.

Sidenote - I often wonder if one of the useful functions teaching performs for graduate students (apart from funding...) is to force them to actually learn all that material they forgot in undergrad by continually revising it, in essence performing a 'poor-man's spaced-repetition' function.

Those exercises at the end of the chapter that you probably want to skip

Yup, you should prolly do them to improve your understanding, but with a few of caveats

• Too easy = don't do them, you aren't learning anything

• If you only need a high-level overview rather of a topic than a working knowledge, doing them probably isn't needed. In these cases a good strategy can to be picking out the key facts you need and putting them into spaced repetition.

• If you are going to be imminently using this knowledge in a project, you will probably learn equivalent skills by doing that and so doing the exercises may be a waste of time.

Practical experience

There is no substitute for practical experience. Having to apply knowledge forces you to internalise it on a level that not even spaced repetition will enable. Plus, in an environment with others working on the same tasks, you will be forced to learn from people who are better versed than you in the same topic and feel social pressure to improve yourself.

Information Ingestion

Notes on publishing formats...

Books

There are probably too many books. It depends what your goal is. If your goal is simply to learn something, so often, reading a blog post is better than reading a book. Even if the book is, of course, much longer. Books embody knowledge, they store knowledge, they certify knowledge. Those are important, I’m not anti-book. But as a means of communicating knowledge, once you’ve read a certain number of key, earthquake, worldview-shattering books, books are way overrated. They’re actually a pretty weak, impotent way of learning new things. --Tyler Cowen

Books in general are pretty overrated as a tool for learning. The amount you retain from most is pretty minimal (do you remember anything from that book you read last year? -- in many cases, the answer will be no...)

My advice? Don’t read books super linearly - skip bits you aren’t interested in. And super don’t feel guilty about this - reading 30% of a book in sections is far better than not reading it at all, especially if you don't have the time or need to go into everything on a topic in detail. Especially for many nontechnical topics (and even for some technical ones) only 30% of a book is worth reading anyways...

Engagement the material is what actually produces lasting understanding, whether that comes from doing exercises or reviewing Anki3. Ankification or doing exercises inherently slows your reading down dramatically, so be strategic in what read, engage with it deeply, and don’t fall for the trap of feeling like you need to read all the things just because PC has a long book list!

Blogs

Blogs are a fantastic source of info especially for non-technical subjects and advice (hey!) They provide a short introduction to a wide breadth of topics that cuts the fat present in many books (just imagine how long this blog would be if I had signed a contract with a publisher). However, blogs often do not have an explicit way for you to understand the material long-term4; for this revert to SRS. When you're just beginning learning highly-technical subjects books are probably better though as they can include neccacary foundational knowledge.

See also: My linked list with some blogs you may like to follow. Marginal Revolution is one great entry point into the blogosphere. Twitter is a great place to find blogs from interesting people too, although it can be expensive.

Lectures

I find watching lectures to absorb content basically completely useless. They are the worst of all worlds - they dont require any active effort from (even less than reading a book without doing exercises), go into less detail, and take up more of your time 5! The two scenario where I find that lectures can be pretty useful is

• When you are stuck understanding something and need it explained in a different way.

• If you want to get an overview of something, seminars are quite useful (as long as you take notes.)

Papers

...are an awful format for trying to learn stuff, because they have to follow relatively fixed conventions resulting in a ton of irrelevant boilerplate. Reading papers to stay up-to-date in your field is obviously necessary in academic disciplines, but for getting up to the frontier of knowledge, blogs and even lectures provide far less convoluted introductions...

Questions

How do I actually apply the advice in this list?

Make Spaced Repetition a habit, and follow Alexey Guzey's advice on instilling novel thought patterns with Anki to remind yourself to use these techniques. The "habits" deck really works for shaping my thought patterns and habits.

Some more good advice in this LessWrong post.

What should I learn?

IDK because I don't know what field you are in, but a couple of possible strategies:

• Try to learn stuff that is fundamental to the field of your core interests (kinda obvious).

• Learn things that are immediately practically useful.

• Optimize for long-term understanding - learn fundamentals to the greatest extent possible, but don’t get bored by areas of the stack you have no interest in and thus stop learning. * For example in computing, I wish I had learned a lot about lower-level areas of the stack (OS details, assembly, etc) earlier. Even though to do any particular task you don’t need to know this stuff, in the end it is pretty important to be able to incorporate new stuff into your mental model quickly and to ensure the mental model of what you are actually doing at high levels of the stack is correct.

• Make sure you have a clear teleology for what you are learning, otherwise at some point you will lose motivation to learn it and stop.

Appendix: Min-maxing marks

That’s all fine and good, but what if I have an upcoming test, I don’t care about the subject, and I just need to {pass, get 80%, whatever} expending as little time and energy as possible…?

How to get decent marks with minimal effort... any instructors reading this, prolly stop here 😌. It's pretty hard to describe this strategy as "learning" (you will not remember anything two weeks after the test) but it serves its purpose.

0. Wait until you actually need the knowledge…

One of the biggest mistakes students make is thinking that they need to ‘keep up’ with content through a semester. This is inefficient, as you will likely then have to go and re-learn the content when the time comes.

The time you need to cover content before a test is shorter than you might think. Milage will vary but for me for finals in undergrad, probably less (<=3 days) now that most things (at least for me) are open book with COVID (so I need not concern yourself with memorizing formulae). You almost certainly don’t need to go to lectures and tutorials, unless you are struggling to understand particular concepts in which case watching the recordings of these can be super helpful.

1. Find out what the minimal amount of knowledge you need is. Find the minimal spanning list of concepts to learn. Anything not on the syllabus need not apply, and if you are learning it you are wasting your time.

2. Read the textbook to load this minimal amount of knowledge in your working memory. Cross out items on the aforementioned list to learn until you're done. Refer to lectures when confused or stuck on a concept. You don’t need to understand every proof in-depth unless it will be on the exam - surface level details only.

3. Find as many past tests as you can and do as many as you are willing to within your time budget. The aim here should be coverage of question types. Start by doing questions at a variety of difficulty levels, but gradually increase the cutoff for minimal difficulty questions (practising easy ones over and over will make you feel good but you won’t learn much).

If there are particularly difficult questions on past tests, make a note with your solutions to these problems so in the test you can refer back to them during the test (assuming you're allowed to).

I find that this practice phase works best when you do it with maniacal focus the day or two before the test to the exclusion of basically all other activities. If you are doing this method you're probably pretty bored by the course anyway, so just try and compress the time into a single block to minimise the pain.

See also: Karpathy’s advice for undergrads.

People sometimes ask me for robotics resources. There aren’t many high quality internet-first lists available on this topic, so I made one I can link to. I'll continually update this list so feel free to give suggestions if things are missing or you have better links.

There's some more interesting comments on this topic in the Hacker News discussion.

General Advice

Robotics can be overwhelming from the sheer number of fields involved (everything from mathematics to physics to computer science to mechanical engineering enters in). Remember: you won’t start as an expert in anything; you can’t end up as an expert in everything. Also learning will not be a linear path; outside of courses, acquire knowledge as you need it in projects.

Try as much as possible to avoid this antipattern (I still struggle)… {{< tweet 1343701831210307586 >}}

Communities

Instead of attempting to pre-load your head with maximum theory, try to get as much practical experience as possible (see below), whether that be joining a club, lab, or if you’re lucky in an internship or job.

It is incredibly hard to do something on your own without some kind of environmental forcing function. If at all possible, embed yourself in a community of people working on problems of interest to you.

• FIRST robotics competition - Simply put, FRC was the awesomest thing ever for me. If you can find a team, beg them to let you participate…

• If you are a student at a university - try joining a lab!

Unfortunately outside of these robotics is currently not an easy field to break into. The sheer number of ‘moving parts’ to a (useful) robotic system - from hardware to drivers to algorithms to user interfaces - means it’s tough to hack together a demo in a few weeks, get users, and bootstrap / apply to some accelerator like in many software projects. Making the jump from “hacker” to “career”, even with all the knowledge in the world is hard. If you know solutions to this, let me know.

Prerequisites

Computer programming - this is essential if you want to do basically anything in robotics. Python and C++ are the most important languages to know - learn both. Familiarity with basic data-structures and algorithms concepts is helpful for problem solving and in some fields like motion planning but initially less vital than you may think - it’s not necessary to be a master leetcoder.

Maths

• Linear algebra - vital in most topics beyond basic basic control. If you want to become an expert, Axler’s book is good (though I didn’t make it through the whole thing 😳).

• Calculus - fundamental differential calculus and partial derivatives is required to understand optimisation algorithms which are everywhere in robo.

• The book A Programmer’s Introduction to Mathematics is actually a solid foundation if you want a quick overview of both of the above topics.

Important Concepts

Controllers

• Wiki's PID controller isn’t bad if you want to learn the basics (95% of controllers used in industry are PID). Ben Recht's blog post is also decent.

• These course slides are good for basics and more advanced concepts like MPC .

• Control challenges is a set of (very fun) challenges that will allow you to test your ability to write basic controllers.

• This talk from FIRST Robotics team 254 is great for going over the basics of robot control and showing how we can improve on basic PID using simple interpolation.

Mathematics and Mechanics of Robotics & Manipulation - This set of notes by Nathan Ratliff provides a great overview of many topics. These notes on robot dynamics from ETH are also great.

Filtering - Kalman and Bayesian Filters in Python is a great read with interactive notebook examples throughout. Good for building intuition on both Bayesian statistics as well as how filters for linear and nonlinear dynamical systems work. The wiki article on the Kalman Filter and this blog post is also well written if you want a shorter overview.

Machine Learning & Neural Networks - Focus less on the basics of machine learning than the application of neural networks as function approximators, which is their primary utility as perception and decision making modules in robotics. Michael Nielsen’s book is my favourite resource in this regard. The canonical deep learning book is good reference but you don’t need to read the whole thing - pick out specific topics as required.

Path and motion planning - Wiki has a decent overview and details some basic algorithms. I haven't read it but this book is apparently a good comprehensive guide, and a course based on it is available here.

Reinforcement Learning & more on controllers - Ben Recht’s “outsider’s tour of RL” provides a nice concise overview of many relevant concepts and ties them to classical control. Spinning up in Deep Reinforcement Learning is another great reference on the topic and exhibits many of the most important algorithms for research in the field.

Other Topics & Resources

Books

• For beginners looking for a single book, Elements of Robotics looks like an a great resource, and I'm only sorry it didn't exist when I was starting. Provides self-contained introductions to many topics listed, pitched at a level high-schoolers / beginning university students can understand.

• Robotics: Modelling, Planning and Control is apparently an all time classic covering many topics listed above..

• Convex Optimisation is used in everything from simulators to controllers and beyond. I intend to read the canonical book (free) at some point.

Courses

• The Berkley Advanced Robotics course provides relatively self contained expositions of many topics in lecture format (don't be scared off by the ‘advanced' moniker, it's not that incomprehensible 😅.)

• Russ Tedrake’s courses on Underactuated Robotics and Manipulation seem really high quality (I haven’t gotten around to doing them myself yet).

Software

The space of software is so vast I couldn’t fit it all here... this list provides a much more comprehensive overview in this regard. Some utilities I’ve found useful are:

• ROS is very widely used many projects, worth picking up the basics.

• RLLib has implementations of many reinforcement learning algorithms and support for distributed training.

Simulators

• Robosuite is great for robot learning and has many richly-designed built-in environments.

• Isaac Gym is recently released and provides extremely fast samples for reinforcement learning. It does this by running thousands of environments in parallel on the GPU (in almost all other sims physics is run on the CPU meaning you can only run one environment at once). You can train tasks in minutes that previously took hours.

Interesting Lines of Work

• Domain Randomisation is highly underrated, and will likely get more important as we get faster simulators which are able to generalize better. Learning Dexterity from OpenAI is a very interesting paper, and the line of work around improving simulator parameter show two different and interesting takes on this powerful approach.

• Hierarchy in reinforcement learning has shown promising results in getting behaviour that generalises, the DeepGait and UniCon paper being two interesting examples.

• DexPilot is a really cool low-cost system for teleoperation (video). Partially inspired my piece on teleoperation.

Hardware

Probably my weakest area… up for recommendations about specific great resources. OnShape is a great resource for designing things and is great even with minimal CAD experience (even I am able to use it!)

Open Dynamic Robot initiative has many robots that you can build with off-the-shelf components.

Others' recommendations -

In the HN thread, contingencies had some tips on how to get started yourself on the hardware side.

I'm told cadathons such as those organised by Solidworks are a good way to get started. They also have a series of lessons on CADding fundamentals.

GrabCAD has big libries of pre-existing models you can use.

Business & Higher-Level Thinking

Very interesting side of things, however there is somewhat of a paucity of great resources (I’m pretty sure many in the space refrain from publishing their theses.) Please send if you have others I have missed!

• Michael Dempsey has some great writing on the business of autonomy.

• Trucks VC has an interesting newsletter about autonomy.

• Jessy Lin wrote some essays on topic surrounding the economics and future of autonomy and HRI.

• Rodney Brooks makes predictions in the robotics space on a yearly basis and has some good rigorous thinking on there.

• EE Times has news and frequent opinion pieces in this space.

• Though inexperienced, I sometimes try my best to write on this topic :)

“You can always tell how serious a company is about unmanned by how seriously they talk about teleop” a vendor once told me. Nevertheless, we found an incredible amount of industry and investor resistance to our teleop-dependent approach. Source

The dream of full autonomy is a compelling one for us roboticists, but I wonder whether it is a practical one if you are looking for medium-term impact. Teleoperation solves many of the same economic and practical problems as full autonomy in many use cases of robotics while being easier to actually bring to users:

1 Teleoperation solves the safety problem

As discussed previously , safety is very hard in robotics. But if you have humans as providing the high level supervision or control input and monitoring systems in-the-loop, you don’t need to prove a priori that the agent will not have catastrophic failures to 1-in-a-billion accuracy. Yes there are concerns about hacking and the reliability of the hardware, but all of these same concerns apply to full autonomy.

2 We can still reap economic benefits

The founder of the now-defunct self-driving truck startup Starsky did a great writeup of why exactly it was that they failed. However, I found the far less popular follow up post covering the economics of their industry even more interesting:

Decoupling the driver from the truck is the easiest way to cut labor costs. American truck drivers are paid only for the miles they haul freight — the hours spent waiting to be loaded or unloaded and taking mandatory breaks are all unpaid. As a result, the $200/day drivers typically earn is really only for the 7 hours/day they move freight and not for the 14 total hours they’re on-duty. This delta is big — it means that the trucking company feels like they pay drivers $28/hr while drivers feel like they only get $14/hr (or $8/hr if you consider the 24 hours/day drivers spend in a truck).

Starsky's graphs are informative, showing asymptotic gains as you decrease the level of human supervision from human in-the-cabin to 1:1 human teleoperation and 1:many teleoperation, nearing full autonomy:

The lesson on the economics of labour may apply to many more industries than trucking. Decoupling physical labour from the need for physical presence allows for elimination of switching costs. This could increasing dynamism in sectors that require physical labour but have as of yet not been revolutionized by software, but without requiring an algorithm and interface as complex as the self-driving, and the safety and reliability verification to go along with it in every single industry. The internet didn’t remove the need for writers, but it did fundamentally the way information consumption works.

Basic economic efficiency will increase when you don’t need to physically move humans to the site of labour. This is already happening in mining, where semi-autonomous mining vehicles are continuously monitored from thousands of miles away.

3 You get human level reasoning today, instead of in “10 years”

State of the art language models operating in the domain of text generation, while very good, fail to show true understanding and instead generate what are essentially maximum likelihood estimates of what should be said next. Such problems of understanding are even harder in the relatively more complex environment of the physical world. We can teach very complex motor-skills, but there is arguably only a tenuous path to getting agents to generalize reliably in unstructured physical environments (consider the fact that researchers are still attempting to define some of the challenges in this space).

While humans are weak, get fatigued and struggle with fine motor control, they find actual supervision of tasks at a high level very easy. It is possible human supervision is the best way for us to specify what we want of robots. For example, coming up with the exact order of operations of moving material and equipment around on a construction site, and what happens under every scenario if something doesn’t go according to plan is a problem humans can handle through supervision, but no current heuristic or RL based system could handle.

With teleoperation, we can get the best of both worlds - domain adaptation and decision making from a human, and tireless muscles and perfect motor control from robotic embodiments.

4 The resulting data can bootstrap full autonomy

Efforts such as Scale AI and Roboturk are very promising for data collection for supervised learning and even training robotic systems. However, the amount of data able to be collected thorough financing collection upfront is likely going to be capped as the collection process itself is not economically productive. With teleoperation you can end up with a positive feedback loop - with doing a task leading to more data leading to doing the task better, all the while being paid to improve as you start off with a useful system. This could actually be a viable path to full autonomy in some domains, if the quantity of data eventually becomes large enough that you can provide safety and capability guarantees and take the human out of the equation entirely.

5 Teleop can be cool, too

A silly but I think important often overlooked final point. There is sometimes a resistance to embracing more practical but near term applications because of the perceived lack of aesthetic qualities. But teleop can definitely be awesome, too - just check out DexPilot:

Real-world deployment of learning based systems in non safety-critical environments should be leveraged to accelerate their rollout in safety critical applications. The current approach - starting by solving problems that simultaneously stretch current algorithms’ capabilities and need approaching 0% failure rates was almost guaranteed to be a huge challenge.

Self-driving is an instructive lens through which we can view the challenges of the rollout of autonomy. Billions have been poured into autonomous vehicle companies. However, we have yet to see the scaled deployment of this technology. What went wrong?

You should watch this excellent talk from Richard Murray, which gives a detailed sense of what it will take to ensure safety in autonomous systems.

In the US1, there are

• 7 deaths per billion miles of driving

• 0.1-0.4 deaths per billion miles on public transport

• 0.07 deaths per billion miles of flying

The first death from a prototype self-driving system was extremely widely publicised at the time, and continues to be discussed two years after. If the rollout accelerates, not every death will be publicised to this extent. At 1 fatality per billion miles (an order of magnitude better than humans), you would get at one death per day in California. Even if every single one is not national news, this would make for like quite a few law-suits indeed; is probably not sustainable.

Since autonomous vehicles must be perceived to be superior to human drivers, we can take this 1 fatality / billion miles figure as a high upper bound on acceptable safety rates.

Vision systems are unacceptably flaky when deployed on current level-2 systems. Errors are both far too common and unpredictable - more progress is needed on robustness in this area.

On the planning side, analysis of progress on the learning side paints a bleak picture. Heuristic + algorithm based approaches do not generalise very well; Reinforcement Learning methods currently have very poor sample efficiency. Neither has safety guarantees attached.

It is clear that more progress is needed on both the algorithms and data collection fronts for both perception and planning.

Scaling capability safely

There are two crucial questions to be asked when deploying an autonomous system - robustness and capability.

The capability question has largely been answered by massively scaling neural networks. Current methods may not be able to do everything, but it seems likely that with sufficient data, they will eventually be able to amply generalise.

Robustness remains a challenge because:

1. To produce highly capable and robust learning-based systems, you have to have trained them on a dense and representative distribution to ensure they have fully explored the state / action space.

2. To have systems that are safe, you need to first validate them on a representative sample (or else prove formally that they are safe - good luck doing that with a deep net!). One estimate puts this figure potentially as high as tens of billions of miles.

The raw capability of the learning algorithms has outstripped what can be verified, leading to a safety deficit in these cutting edge systems:

This is borne out if you look at the recent history of self-driving. Machine Learning combined with exponentially increasing compute enabled rapid improvement of the capability of these systems, but they have not yet achieved the robustness required to climb the tricky part of the sigmoid curve of performance.

The safety deficit results in a classic chicken and egg problem: you can’t gather enough data to train your models and ensure the system is robust, but to gather this data you need many systems in production.

Taking a broader view we can approximately plot the safety and data requirements for different systems:

What becomes apparent is how current successes lie close to one of the axes on either training + verification data requirements or safety requirements.

Autonomous cars push the boundaries on both axes: they both must have failure rates comparable to existing transportation systems due to their safety critical nature, and they require the most cutting edge algorithms ever built.

In this model, it becomes apparent why self-driving was the wrong problem to choose. The extremely high safety requirements mean that large deployment in the hands of real users has been impossible. The result has been sub-scale deployments funded up-front and slow progress.

We should seek application areas for learning algorithms in the/missing middle, shown in red in the chart above. These application areas push the boundaries on the performance of current-generation learning algorithms, but which allow for safe exploration of their action space.

Such agents can continually improve from the data collected while being safe doing so. In this model, capability is the bottleneck:

When you take this approach, by the time you get systems that are marginal on capability, they will also satisfy the safety requirements of the problem at hand. Once this threshold is met, you can start pushing the technology into application areas.

This will lead to a positive feedback loop. More deployment will lead to more data leading will lead to better systems and hence more deployment. The classic technology climbing of the sigmoid curve will be the result.

It makes sense to target application areas with a) lower safety verification requirements, and b) that can be trained with less data (either due to the nature of the problem, or through training in simulation).

The canonical example of this from the past is industrial robots. You can bound the design domain, and nobody is going to die because of a wrong controller input (in many cases you can remove human interaction from the equation entirely). Autonomous mining trucks are a more recent example of a truly autonomous system operating in an unstructured environment.

Robots pushing into domestic spaces are a promising opportunity. If mobile manipulators can be made to work they could reduce labour requirements in many sectors. As does low-speed transportation (for example in pedestrian spaces). In both of these cases, the operational domain is such that small errors do not lead to large safety risk.

Making the economics work in these scenarios is a vital piece of the puzzle. Drone delivery systems are in instructive failure in this space: the technology is relatively safe and the algorithms mature. Yet deployment outside of very specific settings has been impossible - drone delivery simply cannot compete on price with a fleet of vans.

Does this mean that the self-driving project is doomed? No. There are multiple companies chipping away at the issue. The most sensible approaches are by Waymo with huge scale simulation, and Tesla using data to bootstrap a level-4 autonomous system from a level-2 one.

These may or may not be successful. For practitioners looking for a faster route to the frontier, I think that there are smarter routes towards improvement of robotic systems operating in unstructured environments.

Thanks to Ben Agro and Shrey Jain for reading drafts of this.