Foodie Programmer’s “Cashier Robot” is a Recipe for Success

The Alibaba Tech Team has developed a robot that uses visual recognition technology to serve at the staff canteen

Liu Yongkai(刘永凯) is a Senior Algorithm Engineer at Alibaba. Here is his story:

My name is Liu Yongkai. I love two things: food and programming.

One of my favorite dishes is yang xiezi (lamb spine hotpot). It’s a soupy dish cooked with the meat from the spine of a lamb. Once I was in the hospital for an examination when they showed me the scan of my spine. Instead of checking the scan to see what kind of shape my spine was in, I started daydreaming about lamb spine hotpot.

In September 2017, my love for food and programming inspired me to design a cashier robot with a group of friends.

I wanted to find a way to make service at the Alibaba canteen faster. Thanks to friends and programming skills, I was able to do just that. Now, our robot could end up changing how 60,000 Alibaba employees pay for their daily meals.

Inspiration strikes in the lunch queue

The day I came up with the cashier robot idea, I had a big appetite, and my plate was fully loaded. The cashier was new to her job and it took her nearly 5 minutes to add up the cost of my meal.

Standing there with my stomach rumbling, I started thinking about how we could make payment faster.

That was when the idea flashed across my mind: Why don’t we use visual recognition technology to calculate our meal prices?

At the time I was part of the Alibaba security team, which specializes in visual recognition technology. In simple terms, this technology shows robots massive amounts of pictures to teach them to recognize objects. The robots also learn to react to the pictures in certain ways according to instructions.

Visual recognition technology was first used at Alibaba for a different purpose: deleting pornography. When questionable photos appeared on a network, the robot could quickly identify and delete the pornographic images.

Unmanned supermarkets also use visual recognition technology. Without cashiers, robots can scan shopping baskets and tell which items are being purchased.

With the technology already in place, I thought, all we needed to do was change the code, and a machine could be taught to identify steamed lamb, braised duck, roasted chicken, and pork stomach. If the machine also knew the price of each dish, meal prices could be calculated just by scanning plates, and we could all sit down to eat much faster.

How the Cashier Robot was born

I love two things: food and programming. Here was a food problem that could be solved with programming. I was so excited that even a year’s supply of lamb spine hotpot couldn’t have made me much happier. My enthusiasm bubbled over and I had soon recruited a small team to work with.

The equipment we used was makeshift. I disassembled my home desk lamp, took its stand and attached a webcam. The webcam was connected to a cellphone equipped with a visual recognition program. We used a computer cooling pad as the base for the whole setup.

We had a person responsible for testing, for going to the canteen to collect food, for photographing food, writing code, and teaching the robot to identify what twice-cooked pork was, what shredded pork in salted fish sauce was …

We also had a person responsible for the algorithm that allows the robot to calculate meal prices faster.

Our colleagues in the admin department brought back countless dishes from the canteen to help the robot learn. A month into the project, each of us had gained more than ten pounds. That’s not surprising when you consider we shared an average of 20 dishes per meal.

Dishes are recognized during online processing

A product manager also helped us develop another feature.

With the aid of IoT technology, the food ingredients from the dishes we scan daily are stored in the cloud.

Our daily nutrient intake is calculated in the cloud. At night, if we open DingTalk and look at the robot’s report, we will know whether to snack on a protein-filled yogurt, or to hit the gym.

This idea made our whole team excited. Each dish we eat could be become useful data and could even become part of the big data that affects everyone’s health in the future. It was hard to imagine a more meaningful and interesting project.

A curious foodie can change the world

In the first Alibaba IoT geek contest, our cashier robot project won a gold medal, and our project is expected to go live soon at the Xixi Park canteen. (Even though I helped create it, telling the robot you know me won’t get you any discounts.)

Yongkai presenting the cashier robot project at the 1st Alibaba Geek Competition

Being a foodie doesn’t mean all you can do is eat. By combining my love for food with curiosity and imagination, I came up with an idea that could affect the canteen experience for thousands of Alibaba employees.

If Isaac Newton had just eaten the apples that fell from trees, without ever asking why they fell, he would have never discovered gravity.

It all just goes to show, just by asking the right questions, a curious foodie has the power to change the world.

Alibaba Tech

First-hand and in-depth information about Alibaba’s latest technology → Search on Facebook