XJTLU students designing smart robotic solution to organise your mess

If you wish your living areas were more orderly, software by a student team at Xi’an Jiaotong-Liverpool University (XJTLU)’s Entrepreneur College (Taicang) (XEC) may one day be able to help. The software design provides the “smarts” behind robots that can perform helpful tasks in the home that involve moving objects, intelligently predicting what humans want.

“The software will be able to help robots predict users’ needs through observation, and then perform customised tasks for users in various scenarios, such as tidying up the desk in a home environment,” says Year Four student Clarine Beryl, the leader of the Intelligent Future team. Beryl, from Indonesia, is studying BEng Intelligent Robotics Engineering with Contemporary Entrepreneurialism at XEC.

From left to right: Zhenhao Guo, Hanyang Xu, Bojun Long, and Clarine Beryl

The software is one of the control system solutions the team plans to develop for robotics companies, particularly service robotics manufacturers.

Teamwork makes dream work

To make it work, the software needs to be able to command robots to intelligently detect, pick up, and move items in line with a user’s patterns and preferences. The team has already met multiple major challenges to make this happen.

One was to enable the software to electronically “see” objects in the environment. To accomplish this, the team learned to use YOLOv5, a popular computer vision model designed to detect and recognise objects around the environment, says Zhenhao Guo, Chief Product Officer.

Another was to ensure the software could predict a user’s needs and preferences. To do this, they used algorithms similar to those used in apps that serve up content they predict you will like. These algorithms, called “collaborative filtering algorithms”, use data collected from many users to make intelligent predictions.

Zhenhao Guo (right) works on software development with the robotic arm at a lab. Credit: Clarine Beryl

However, during the software programming process, the team found that the predicted results did not always match a user's real preferences, says Bojun Long, Chief Technology Officer and a Year Four student at BEng Intelligent Robotics Engineering with Contemporary Entrepreneurialism.

To solve this problem, the team developed a continuous learning function. Now, the robot arm’s camera  can continuously observe the user's behaviour and learn their tidying habits. The software directs the robot arm to adjust its strategy for placing items according to those habits .

“If a user places their phone on the right side of the desk several times but only once on the left, the software will record this preference,” Long says. “When the phone is misplaced, the robot will move it to the preferred side.”

Dr Fan Zhu, Assistant Professor at XEC’s School of Robotics, provided guidance to the team.

She says the team’s use of tools such as YOLOv5 and collaborative filtering algorithms shows “they’re comfortable with industry-standard machine learning techniques, which adds a sophisticated layer to their project.”

 Support from X3 Co-Venture

X3 Co-Venture, a startup hub at XEC and a key component of XJTLU’s Syntegrative Education model, provided guidance and support to the team. The hub assists students with launching entrepreneurial projects, collaborating with industrial partners, and building new business partnerships.

X3 Co-Venture provided the team with the budget to purchase the equipment needed to carry out the research, Beryl says.

“We also received support from X3 Co-Venture in terms of having our office room, which allows us to discuss and work together in the room. This increases our efficiency as we do not need to book a separate room for meetings,” she adds.

Long says: “Our final product is planned to be a miniaturised home robotic arm that can help users organise their desktop or kitchen and so on, and will be developed in-house by our team in terms of both software and hardware.”

Dr Mikhail Zenchenkov, Director of X3 Co-Venture, believes the team’s project could both tap into existing markets and also create new opportunities in sectors that haven’t fully embraced automation.

“In consultation with the team, we plan to establish pilot opportunities with robotics manufacturers and potential end-users to refine the technology, gain insights into market needs, and build credibility,” he says.

By Clarine Beryl and Haolun Xu
Edited by Xinmin Han and Tamara Kaup
Photos courtesy of Clarine Beryl