A neural network has been taught to suggest recipes from the food available in the fridge

A neural network has been taught to suggest recipes from the food in your fridge. How does it work and what’s next for us?

What is it about?

Chat-bot ChatGPT based on a new version of neural network GPT-4 is able to make recipes with step-by-step preparation based on a photo of the products you have.

To do this, you need to upload a photo with the contents of your refrigerator, after which the program will recognize the ingredients and suggest variants of dishes.

According to the creators of the chatbot, the technology can become an assistant for blind and visually impaired people. The neural network collaborates with the Be My Eyes project and tells users everything their smartphone sees.

How does it work?

Ilya Karpuk

Director of Printeria service center, neural network programmer

It’s not difficult to teach a neural network to recognize products and make a list of suitable recipes based on a photo from the refrigerator. You don’t even need ChatGPT with its huge resources, any student studying AI (artificial intelligence) can make such a neural network.

The neural network breaks down the task into recognizing objects and products, classifying them, then scours the Internet for matching recipes. And it looks like magic – you take a picture of the refrigerator, ask a question, for example: what should I cook for breakfast if I don’t want scrambled eggs. You get the result in the form of recipes.

During training, the neural network digests a huge amount of information: both pictures and text. At the same time, all the inputs are mixed together, as in a huge cauldron.

According to the input data, in this case it is a picture with products, the neural network understands as a query “recipe of eggs, milk, ham and cheese” (conditionally) and tries to “degenerate” it from its parameters. The neural network can also create something more interesting, for example, variants of dishes from pickle with milk.

Services are improved by combining algorithms. Chat-bot GPT has made a flexible model, which knows and can do more than its predecessors and is universal, not sharpened for any one action.

What’s in store for us next?

Pavel Korolev

ML-engineer of a social entertainment network

Recipe matching services based on product availability have existed for a very long time. Many popular recipe sites have such functionality. But now, for the first time, you can simply take a picture of available products, and neuron will determine what you can cook with them in a few seconds. This effect became available after the introduction of product recognition.

In general, if the trend continues, it will soon be possible to see smart refrigerators ordering products themselves and obsessively offering you recipes based on them. The main thing is that this smart gadget should not catch a virus and start ordering only cucumbers and bananas.

As for ChatGPT, it is a very interesting thing with almost unlimited resources. It already surprises and sometimes frightens with answers with its share of reasonableness, and in new versions, as it learns, it simply removes restrictions and gives more freedom.

Neural network is already changing the lives of designers and copywriters, becoming a convenient tool for them or making them think about changing jobs. I think we’re in for a very interesting future.