Embedded AI How Embedded AI Reduces Power Consumption While Promoting Efficiency

Related Vendor

Wuxi InfiMotion Technology Co., Ltd.

The immense energy demand of artificial intelligence (AI) is increasingly being criticized. No wonder, just the training of the GPT-3 model – that is, the publicly accessible version of ChatGPT since the end of 2022 – consumed as much electricity as a medium-sized nuclear power plant can produce in about an hour, namely 1,287 megawatt-hours.

Viacheslav Gromov is the CEO of the AI provider Aitad.

Nowadays, the energy consumption of the entire information and communication technology sector causes about two to four percent of global greenhouse gas emissions – as much as global air traffic. And the demand is growing: So scientists estimate that the energy consumption of Artificial Intelligence could increase to up to 134 terawatt hours by the year 2027. In addition, AI training also requires water in significant quantities – the training of the GPT model is said to have consumed around 700,000 liters (184,000 US liquid gallon) of cooling water.

Unprecedented demand for computing power

Training large AI models requires unprecedented amounts of computing power - for example, Nvidia plans to sell more than 2 million AI accelerators of the latest H100 'Hopper' type by the end of 2024. If all of these run at full load, they need 1.6 gigawatts of power, more than one of the largest nuclear reactors can supply.

In addition to training, the operation of AI systems also requires energy. Thus, the average energy consumption of a request to ChatGPT is estimated at 3 to 9 watt-hours. If all of the 9 billion daily search queries were answered by AI, energy consumption would be expected to increase thirtyfold. The integration of ChatGPT into Microsoft's Bing and Gemini into Google's search engine indicates that the number of AI-generated search responses will indeed significantly increase.

Even in medical technology and industry, such as in production, the use of AI is increasing. Although the use of AI in industry promises to make processes more efficient and prevent production failures, the increase in machine efficiency also leads to a significantly higher energy demand here.

Artificial intelligence needs to become more energy efficient

Against this backdrop, the development and use of AI must become more energy efficient, not only to save costs, but also to address energy shortages and the resource consumption required for energy generation. Last but not least, global warming and geopolitical events force us to use resources sparingly.

Of course, the solution is not to abandon AI, but to focus on increasing energy efficiency in its use. AI and smart sensors can not only transform the industry, but also make production more energy efficient. The central focus should already be on the development of neural networks and the question of where the desired information from the data collected by sensors is extracted.

Discriminative AI aids in data analysis

Significant savings opportunities exist in the field of discriminative AI, which is particularly important for industrial use. Unlike generative AI (such as ChatGPT, Midjourney, Gemini, etc.), this type of AI is not used to create content but to analyze and evaluate data. Discriminative AI therefore answers questions directed at devices and machines such as: "What is happening right now, and what will happen in the future?".

Bigger doesn't always mean more powerful

Ultimately, energy consumption largely depends on the size of the artificial neural network and consequently the underlying, usually parallelized, computing operations. This affects both the training phase and the use of the model. So what could be more obvious than keeping the models as small and efficient in processing as possible? However, small does not necessarily mean less performance. For a few years now, it has been possible to run AI on the smallest semiconductors – from the MCU to the FPGA – as an embedded system.

When comparing a Cortex-M microcontroller (possibly with NPU and DMA) with the same MAC parallel operations to a typical Intel-i PC processor, natural limitations make them 20 to 60 times more efficient. In detail, this means: due to the semiconductor size, technology, and price (natural limitations), both the computing performance (including clocking) and the instruction set are significantly limited in small embedded microcontrollers. And depending on the specific factors of model and data type, size and the necessary operations, as well as the dimension of the model transformation (i.e., shrinking to a smaller size), this is more or less possible, which is why a rough estimate of 20 to 60 applies to the factor.

In addition, there is the option of the NPU in the Cortex-M MCUs that processes the MAC operations in parallel and very efficiently (could be compared to Nvidia GPUs on a small scale). By the way, Arm speaks of a factor of 480 here (Ethos-U55 Machine Learning Processor – Arm), which we consider too much marketing because the NPU also has its own energy consumption. Taking all factors into account (including energy comparison: ST STM32L0: max. 88 µA/MHz / Intel i5-11Gen: max. 1.4 mA/MHz) we therefore come up with a real efficiency range factor of between 20 to 60 depending on the constellation.

Such embedded AI operates completely autonomously at the network edge - with the great advantage that it can evaluate all the data received from the sensor in real time and therefore requires hardly any connectivity. For the industry, embedded AI is becoming increasingly important in many areas of application and at the same time helps to massively curb energy consumption.

Energy-saving training on a standard PC is achievable with embedded AI

This starts with the training of AI models for embedded AI system components. While large models for training require an extensive server infrastructure (sometimes tens of thousands of GPU servers – costing more than 400,000 USD per server – need to be connected), AI development for an embedded system is often possible on a standard PC. This is of course also because embedded AI answers very specific questions, such as predictions about the health status of a drive, simple voice controls with recognition of more complex word structures, monitoring and testing of weld seams, or assessment of the health status of teeth based on the ultrasound noises recorded during brushing - and countless other use cases, e.g. for relieving service or predicting failures in industrial robots.

As embedded AI systems are always strongly resource-limited, the energy demand is very low, with only a few milliamperes required depending on the use case. This means that these systems can be operated with a battery, and in many cases, energy harvesting would even be possible.

Embedded AI saves electricity and can be used in a variety of ways

Even though embedded AI systems are not a universal solution – their use is fundamentally possible wherever sensors are used. In order to save energy and achieve more efficiency with a simultaneously better climate balance, they are ideal. Pleasant side effects are the much deeper possible analysis of the sensor data, real-time capability and protection of data privacy – after all, it's not the sensor data that is transmitted, but only the evaluation results. Last but not least, development and unit costs should be significantly lower than those of networked AI systems.

With the help of embedded AI, the use of AI can be decentralized, so that even in large networked production operations, work can be done with significantly less energy usage without compromising the advantages of AI. Therefore, in order to stay future-proof and conserve resources, embedded AI should be used wherever possible and necessary. (mk)