Mercedes-Benz unveils new AI-based virtual assistant for cars 

It uses advanced software and generative AI to create an even more natural and intuitive relationship with the car, with proactive support that makes life easier.

This development takes the ‘Hey Mercedes’ voice assistant into a whole new visual dimension made with Unity’s high-resolution game-engine graphics. Running on the Mercedes-Benz Operating System (MB.OS) developed in-house, its rollout starts with the vehicles on the forthcoming MMA platform (Mercedes-Benz Modular Architecture). The Concept CLA Class, which celebrates its North American premiere at CES 2024, is based on this platform and likewise provides a preview of MB.OS.

“Through advances in artificial intelligence, tomorrow’s Mercedes-Benz will know its driver like never before. It will enhance and complement our customers’ lives – in their cars and in other areas too. We have already made strong progress on this journey. The latest proof points on show at CES provide an exciting glimpse of what lies ahead," said Markus Schäfer, Member of the Board of Management at Mercedes-Benz Group AG, Chief Technology Officer

MBUX Virtual Assistant makes life easier with natural interaction and proactive intelligence

The class-defining MBUX Virtual Assistant running on MB.OS is the most human-like interface with a Mercedes-Benz yet. Combining the intelligent systems of MBUX into a single entity, it presents a new face to the customer with natural and empathetic interaction.

The MBUX Virtual Assistant is a further development of the system first showcased in the VISION EQXX. It uses generative AI and proactive intelligence to make life as easy, convenient and comfortable as possible.

For instance, it can offer helpful suggestions based on learned behaviour and situational context. Examples include playing the latest news in the morning or initiating a preferred massage programme at the end of the working day. And if the driver is running late for a meeting entered into the calendar, the system can offer to dial in directly from the car. If the driver wishes, these processes can also be automated.

The experience can be further augmented through incorporation of the vehicle’s ambient lighting and sound system. For example, warm light and soft music can accompany a relaxing massage.

“The Mercedes-Benz user experience of tomorrow will be hyper-personalised. With generative AI, our MBUX Virtual Assistant brings more trust and empathy to the relationship between car and driver. Thanks to our MB.OS chip-to-cloud architecture, our future vehicles will provide customers with exactly what they need when they need it," said Magnus Östberg, Chief Software Officer, Mercedes-Benz AG.

Four characteristics ensure the assistant is always in tune with customer needs and preferences

Underpinning the MBUX Virtual Assistant are four ‘personality traits’ – Natural, Predictive, Personal and Empathetic – blended to create a fully-rounded whole. Natural is always the underlying principle, offering seamless interaction that leverages the best of voice and graphic interfaces.

The customer can choose to speak with the assistant with or without using the keyword ‘Hey Mercedes’. Speech takes advantage of large language models (LLMs) for natural dialogue in response to queries and commands. The effect is more like a conversation with a friend.

The assistant answers questions based on general knowledge and can also ask intelligent questions to clarify intent and draw conclusions. By being predictive, the assistant can also offer proactive suggestions and routines based on situational context, learned behaviour and using generative AI.

At the same time, a more emotional neural voice that can express different speaking styles generates a sense of empathy. And to ensure it is always a very personal assistant, the driver can tailor it to their own specific preferences via a range of options in a central hub.

Visual communication made more intuitive and emotional through powerful 3D graphics

The holistic approach pairs more natural dialogue with visual feedback through advanced 3D graphics made with Unity’s game engine. In the new MBUX Virtual Assistant, a ‘living’ star avatar uses animations to express different moods and states of being. Emotions range from calm to excited and even sensitive. Further animations indicate whether the assistant is speaking, listening, thinking, making a suggestion or providing a warning. Movement, brightness, intensity and colour interact seamlessly to communicate intuitively with the driver.

MBUX Surround Navigation marries assistance with route guidance for a seamless experience

Integrated visual communication also enters a new dimension with MBUX Surround Navigation. Facilitated by the domain integration of MB.OS, this advanced feature pairs a real-world view of the car’s surroundings with its reactions to it in real time.

The driver benefits from significantly enhanced situational awareness by being able to ‘see what the car sees’. The seamless incorporation of information into the driver display using 3D graphics made with Unity’s game engine provides a highly intuitive overview. It offers everything the driver needs to know in a single glance. For instance, it shows the type of traffic, be it another car, van, truck or even a cyclist.

It shows pedestrians close to the roadside as well as potential hazards. And it superimposes route guidance into a realistic representation of the surroundings – particularly helpful in busy urban environments.

The customer can clearly see where exactly their next turn will take them. Buildings and other infrastructure are easy and quick to recognise.

The system also provides integrated oversight of the car’s status, through virtual-reality real-time visualisation.

This includes flashing indicators, high/low headlight beam and even acceleration or deceleration depicted by spinning wheels.

More productivity, entertainment and convenience with new 3rd-party apps and content

Over-the-air (OTA) updates ensure customers have access to the latest content and features tailored to their wishes and regional preferences.

The new E-Class, for example, is already delivering these benefits – its infotainment domain is a precursor of MB.OS technology. Starting in the first quarter of 2024, productivity in the E-Class will be further elevated through the addition of our new Meetings app where our customers can participate in Microsoft Teams conferences.

This, paired with existing offerings with Zoom and WebEx, covers all market leaders for video conferencing.

The extensive update also includes the addition of further apps for everyday convenience. These include Booking.com, getAbstract, Just Eat Takeaway.com and more. In entertainment, there are plans to introduce cloud-based retro gaming from Antstream Arcade and further immersive experiences.

Video streaming offering to be enhanced with Sony Pictures Entertainment’s RIDEVU

In an exciting development, the video streaming offering will be enhanced by an integration of Sony Pictures Entertainment's (SPE) cutting-edge RIDEVU service. RIDEVU promises to elevate the new vehicle entertainment experience for Mercedes-Benz customers, granting them access to a diverse array of video entertainment directly within their vehicles. RIDEVU provides users with advanced controls and features designed for the car. These enable consumers to conveniently rent and purchase Sony Pictures Entertainment’s content on demand, directly from their vehicles. It includes a curated library of films available for streaming, enriching the Mercedes-Benz entertainment bundle.

Hyper-personalisation through digital art with MBUX Collectibles and more

MBUX Collectibles is Mercedes-Benz’s first in-car app to showcase digital art and collectibles based on NFTs. It enables the creation of a private art gallery by connecting the customer's personal NFT wallet to the user interface. Users can also browse a digital art exhibition curated by Mercedes-Benz.

On show at CES are NFT collections from Mercedes-Benz NXT within the app. The collection Mercedes-Benz NXT Icons Era 1 ‘Luxury’ is the first in a series of seven exploring the rich Mercedes-Benz design heritage.

At CES, there is also a first sneak peak of Era 2 ‘Technology’, which is yet to be launched. And in homage to last year’s CES, visitors can enjoy a first glimpse of the Mercedes-Benz NXT Superdackel NFT. A limited edition of 1,000 Superdackel NFTs by Mercedes-Benz NXT will drop soon.

Concept CLA Class showcases MB.OS as the foundation for the hyper-personalised Mercedes-Benz

The first vehicles to run on MB.OS will be the family based on the forthcoming MMA platform. The Concept CLA Class, which celebrates its North American premiere at CES 2024, offers a close-to-production insight into the future range of four vehicles. To stress the central role played by MB.OS in shaping the customer experience, the interior of the show car offers a striking visualisation of it.

The complex multi-faceted architecture is represented by one of its many hardware components, the high-performance water-cooled chip from collaboration partner NVIDIA.

The advanced three-display MBUX Superscreen hints at the role of the Mercedes-Benz of the future as a hyper-personalised part of the customer’s digital life. MB.OS provides the foundation for achieving this.

It comprises four domains – Infotainment, Automated Driving, Body & Comfort and Driving & Charging – and the connectivity module linked to the Mercedes-Benz Intelligent Cloud. The chip-to-cloud architecture ensures deep integration into the vehicle and full control of all sensors and actuators. Furthermore, by decoupling the software and hardware innovation cycles, the company can ensure its vehicles are constantly up to date.

The use of the Digital Extras requires the permanent acceptance of the Terms of Use for Digital Extras and the Mercedes me ID Service Terms in their applicable versions, the permanent pairing of the vehicle with the Mercedes me user account and - if applicable - the activation of the Digital Extras.

Upon expiry of the term, the Digital Extras can be renewed for a fee in the Mercedes me Store provided they are still offered for the corresponding vehicle at that point. In addition, for the use of some Digital Extras there may be further usage requirements or limitations specifically separate customer-owned contracts with third parties (e. g., streaming, charging, conclusion of a data contract covering comfort data volume), selected third-party products (e.g. smartphone, smartwatch) or they are only available for specific customer groups.

Alternatively to comfort data volume, depending on the generation of your multimedia system, data volume must be available via Internet in the car or, where possible, your own data volume will need to be used via tethering. Information on personal data processed for the use of Digital Extras may be found in the Mercedes me connect privacy policy.