Hybrid AI for safe, high-performance robotic planning

To address safety constraints or to program or define planning or reconfiguration tasks in robotics, today’s AI systems require a high level of expertise and are only effective within a given scope.

Because it leverages automated logical reasoning based on formalized knowledge, Symbolic AI offers the advantage of explainability. It can be used in robotic planning to determine a sequence of safe, predictable actions from an initial situation to the desired goal. Logically-rigorous symbolic AI is ideal for this kind of reasoning. However, it is not effective on imprecise data like text, images, and natural language.

Generative AI, on the other hand, is based on large language models (LLMs). It can encode very large quantities of information and is extremely flexible. The ability to interpret images, to follow instructions given in natural language, and to leverage implicit knowledge (common sense) makes for seamless interaction between humans and robots. The downside is that generative AI’s output is not guaranteed to be correct and lacks traceability. Plus, it needs significant amounts of training data to learn.

On their own, neither type of AI can solve robotics challenges like understanding unpredictable environments. Together, however, they are rigorous yet flexible and combine logical reasoning and powerful learning capabilities.

It is precisely this kind of hybrid AI solution that CEA-List researchers developed for the Magicoders project. This novel approach draws on CEA-List’s expertise in software engineering, AI, robotics, and high-performance computing.

The project, at the crossroads of requirements engineering, automated planning, generative AI, and robotics, is truly multidisciplinary. The hybrid solution developed enables several new capabilities—such as reasoning that can better adapt to complex or ambiguous contexts—that support explainable, integrated, and operational AI.

Our researchers experimented with the integration of symbolic AI and generative AI (LLMs) for robotic planning using a standard robotic planning language, PDDL (Planning Domain Description Language). Magicoders is based on the following workflow:

1. First, a scene is interpreted by a VLM-type (vision-language model) generative AI, which provides a semantic description.
2. Based on the initial scene, the user then states a goal in natural language. The goal is then interpreted by an LLM.
3. This information is combined with business rules (including safety rules) and then sent to a PDDL solver for the symbolic AI step.
4. If the solver finds an action, it is sent to the robot for completion. If, on the other hand, the solver finds that the goal cannot be achieved, the user is informed and can then review and correct the goal. This results in the continuous improvement of both control and understanding of the process.

Magicoders was tested on two use cases, a simulator and a robotic arm. In both cases, the expected goal was achieved within the constraints of the business rules used. These two test implementations demonstrated the feasibility of this hybrid AI approach to robotic planning.

This research is part of a modular, end-to-end approach designed to open new avenues for future architectures combining symbolic reasoning, LLMs, and autonomous systems.

The next project milestones—advanced understanding of the scene, automated guided review and correction of an action plan, a recovery strategy if the plan has failed, replanning, and integration of business knowledge to control and contextualize LLM responses—are ambitious.

CEA-List is pursuing these pathways to generative-AI-based advanced robotic systems to solve complex manipulation problems.