Digital chefs and intelligent cooking systems based on multimodal large language model

LI Xinyuan; LI Bai; SUN Yueshuo; ZHANG Tantan; TIAN Yonglin; YIN Zhuyan; WANG Fei-Yue

doi:10.11959/j.issn.2096-6652.202448

您当前的位置：

首页 >

文章列表页 >

Digital chefs and intelligent cooking systems based on multimodal large language model

Papers and Reports | 更新时间：2025-01-24

- Digital chefs and intelligent cooking systems based on multimodal large language model
- Chinese Journal of Intelligent Science and Technology Vol. 6, Issue 4, Pages: 429-444(2024)
- 作者机构：
  
  1.湖南大学机械与运载工程学院，湖南长沙 410082
  2.湖南大学整车先进设计制造技术全国重点实验室，湖南长沙 410082
  3.中国科学院自动化研究所多模态人工智能系统全国重点实验室，北京 100190
  4.中国科学院自动化研究所复杂系统管理与控制国家重点实验室，北京 100190
  5.中国科学院大学人工智能学院，北京 100049
  6.澳门科技大学创新工程学院工程科学系，澳门 999078
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(62103139);Lotus Youth Talent Program of Hunan Province(2023RC3115)
- DOI：10.11959/j.issn.2096-6652.202448
  CLC： TP391.4
- Received：27 September 2024，
  
  Revised：2024-11-08，
  
  Published：15 December 2024
- 稿件说明：
移动端阅览
李鑫源,李柏,孙跃硕等.基于多模态大语言模型的数字厨师与智能烹饪系统[J].智能科学与技术学报,2024,06(04):429-444.

LI Xinyuan,LI Bai,SUN Yueshuo,et al.Digital chefs and intelligent cooking systems based on multimodal large language model[J].Chinese Journal of Intelligent Science and Technology,2024,06(04):429-444.
李鑫源,李柏,孙跃硕等.基于多模态大语言模型的数字厨师与智能烹饪系统[J].智能科学与技术学报,2024,06(04):429-444. DOI： 10.11959/j.issn.2096-6652.202448.

LI Xinyuan,LI Bai,SUN Yueshuo,et al.Digital chefs and intelligent cooking systems based on multimodal large language model[J].Chinese Journal of Intelligent Science and Technology,2024,06(04):429-444. DOI： 10.11959/j.issn.2096-6652.202448.

摘要

面向高质量和精准烹饪的需求，提出一种基于多模态大语言模型的数字厨师与智能烹饪方法。离线阶段利用视觉、声音、温度等多源传感器记录专业厨师的连续操作，将图像与多轮问答文本融合，建立烹饪专家知识库，并采用低秩适配方法对预训练多模态大语言模型进行微调，以构建能够理解烹饪意图的多模态大语言模型。在线阶段将实时感知的数据转换为图文输入微调后的大语言模型，经模型分析后生成烹饪指令，引导用户完成相应的烹饪动作。以煎牛排任务为例，搭建了智能烹饪软硬件系统并进行实验验证。实验结果表明，经过微调后的智能烹饪系统能有效控制牛排的熟度与品质，相较于微调前的模型，显著提升了烹饪指令的合理性和针对性。

Abstract

A digital chef and an intelligent cooking method were proposed to achieve high-quality

precise cooking results. In the offline phase

visual

auditory and thermal sensors record professional chefs' continuous cooking operations. The collected frame-by-frame images and multi-round Q&A texts form a culinary expert knowledge base. A low-rank adaptation method was applied to fine-tune a pretrained multimodal large language model

enabling it to understand cooking intentions. In the online phase

real-time sensory data were converted into image-text inputs for the fine-tuned model

which then generated cooking instructions to guide users through the cooking steps. A hardware-software cooking system was implemented and tested with a pan-frying steak task. Experimental results show that the fine-tuned system effectively controls the steak's doneness and quality

and significantly improves the accuracy and rationality of cooking instructions compared to the model before fine-tuning.

关键词

Keywords

references

陈涵, 姜慧. 我国社区餐饮发展现状、问题与对策[J]. 商业经济研究, 2023(11): 30-33.

CHEN H, JIANG H. Present situation, problems and countermeasures of community catering development in China[J]. Journal of Commercial Economics, 2023(11): 30-33.

杨铭铎, 陈健. 对本科烹饪专业建设几个问题的探讨[J]. 美食研究, 2022, 39(3): 45-50.

YANG M D, CHEN J. Discussion on several practical problems in the construction of undergraduate culinary major[J]. Journal of Researches on Dietetic Science and Culture, 2022, 39(3): 45-50.

左武举. 烹饪机器人与智能烹饪市场[J]. 现代食品, 2022, 28(9): 83-85.

ZUO W J. Cooking robots and smart cooking market[J]. Modern Food, 2022, 28(9): 83-85.

雷鸣, 管声启, 王杰, 等. 新型商用炒菜机器人搅拌系统结构设计与轨迹分析[J]. 西安工程大学学报, 2019, 33(6): 660-665.

LEI M, GUAN S Q, WANG J, et al. Structural design and trajectory analysis of a new commercial cooking robot mixing system[J]. Journal of Xi'an Polytechnic University, 2019, 33(6): 660-665.

邓鹏, 唐文涛, 罗静. 机器人大模型发展与挑战[J]. 电子测量与仪器学报, 已录用.

DENG P, TANG W T, LUO J. Robotic large model development and challenges[J]. Journal of Electronic Measurement and Instrumentation, accepted.

王婉娇. 中国传统饮食文化的发展及当代价值[J]. 食品与机械, 2024, 40(6): 241.

WANG W J. The development and contemporary value of traditional diet culture in China[J]. Food and Machinery, 2024, 40(6): 241.

青平, 王玉泽, 李剑, 等. 大食物观与国民营养健康[J]. 农业经济问题, 2023, 44(5): 61-73.

QING P, WANG Y Z, LI J, et al. The greater food approach and national nutritional health[J]. Issues in Agricultural Economy, 2023, 44(5): 61-73.

李程, 常静一, 朱婧. 食欲的评价方法[J]. 卫生研究, 2023, 52(5): 832-836.

LI C, CHANG J Y, ZHU J. Evaluation method of appetite[J]. Journal of Hygiene Research, 2023, 52(5): 832-836.

赵海, 王路, 滕立新, 等. 北京市大中学生抑郁流行现状及与饮食相关行为的关系[J]. 中国学校卫生, 2022, 43(5): 715-717.

ZHAO H, WANG L, TENG L X, et al. Prevalence of depression and the association between depressive symptoms and dietary behaviors among students in Beijing[J]. Chinese Journal of School Health, 2022, 43(5): 715-717.

薛羽, 张逸轩. 深层神经网络架构搜索综述[J]. 信息网络安全, 2023, 23(9): 58-74.

XUE Y, ZHANG Y X. Survey on Deep Neural Architecture Search[J]. Netinfo Security, 2023, 23(9): 58-74.

张淼, 贾洪锋, 张振宇, 等. 我国牛排品质识别技术研究现状[J]. 食品工业, 2023, 44(2): 175-180.

ZHANG M, JIA H F, ZHANG Z Y, et al. Research situation on recognition technology of steak in China[J]. The Food Industry, 2023, 44(2): 175-180.

姚力为, 邓静, 易宇文, 等. 烹饪熟度对牛排挥发性风味物质的影响研究[J]. 中国调味品, 2024, 49(9): 47-53.

YAO L W, DENG J, YI Y W, et al. Study on effect of degree of doneness on volatile flavor substances of steak[J]. China Condiment, 2024, 49(9): 47-53.

季鸿崑. 饮食美感和饮食风味[J]. 扬州大学烹饪学报, 2005, 22(1): 11-16.

JI H K. Dietetic aesthetic perception and dietetic flavor[J]. Cuisine Journal of Yangzhou University, 2005, 22(1): 11-16.

陈梦晓. 雕塑艺术在食品造型中的应用研究[J]. 食品研究与开发, 2020, 41(6): 2.

CHEN M X. Research on the application of sculpture in food modeling[J]. Food Research and Development, 2020, 41(6): 2.

MORQUECHO-CAMPOS P, GRAAF K D, BOESVELDT S. Smelling our appetite? The influence of food odors on congruent appetite, food preferences and intake[J]. Food Quality and Preference, 2020, 85: 103959.

侯兵. 新时代我国烹饪高等教育发展审视: 成就、挑战与展望: 基于扬州大学烹饪与营养教育专业办学40周年的思考[J]. 美食研究, 2023, 40(1): 1-10.

HOU B. Research on the development of Chinese higher culinary education in the new era: achievements, challenges and prospects: for the 40th anniversary of the founding of the school of tourism and cuisine of Yangzhou University[J]. Journal of Researches on Dietetic Science and Culture, 2023, 40(1): 1-10.

李楠, 王昆仑, 王起赫, 等. 餐饮食品营养成分影响因素分析[J]. 中国食品卫生杂志, 2023, 35(8): 1192-1198.

LI N, WANG K L, WANG Q H, et al. Factors influencing the nutritional composition of catering food[J]. Chinese Journal of Food Hygiene, 2023, 35(8): 1192-1198.

SUHAIROM N, MUSTA'AMAL A H, MOHD A N F, et al. Quality culinary workforce competencies for sustainable career development among culinary professionals[J]. International Journal of Hospitality Management, 2019, 81: 205-220.

吴晓蒙, 李想, 刘楠, 等. 智慧餐饮4.0: 融合人工智能的未来餐饮发展新方向[J]. 食品科学技术学报, 2024, 42(1): 10-19.

WU X M, LI X, LIU N, et al. Smart catering 4.0: new development direction of future catering integrated with artificial intelligence technology[J]. Journal of Food Science and Technology, 2024, 42(1): 10-19.

高莹婷, 胡懿昕, 路奇. 基于电子鼻的智能厨房烹饪方式检测[J]. 计算机辅助设计与图形学学报, 2023, 35(2): 185-194.

GAO Y T, HU Y X, LU Q. Cooking method detection based on electronic nose in smart kitchen[J]. Journal of Computer-Aided Design & Computer Graphics, 2023, 35(2): 185-194.

LIU J J, CHEN Y T, DONG Z P, et al. Robot cooking with stir-fry: bimanual non-prehensile manipulation of semi-fluid objects[J]. IEEE Robotics and Automation Letters, 2022, 7(2): 5159-5166.

张腾超, 田永林, 林飞, 等. 平行旅游: 基础智能驱动的智慧出游服务[J]. 智能科学与技术学报, 2024, 6(2): 164-178.

ZHANG T C, TIAN Y L, LIN F, et al. Parallel tourism: foundation intelligence driven smart trip services[J]. Chinese Journal of Intelligent Science and Technology, 2024, 6(2): 164-178.

田永林, 王兴霞, 王雨桐, 等. RAG-PHI: 检索增强生成驱动的平行人与平行智能[J]. 智能科学与技术学报, 2024, 6(1): 41-51.

TIAN Y L, WANG X X, WANG Y T, et al. RAG-PHI: searching for parallel people and parallel intelligence driven by enhanced generation[J]. Chinese Journal of Intelligent Science and Technology, 2024, 6(1): 41-51.

LIU H T, LI C Y, WU Q Y, et al. Visual instruction tuning[J]. Advances in Neural Information Processing Systems, 2024, 36: 1-25.

李耕, 王梓烁, 何相腾, 等. 从ChatGPT到多模态大模型: 现状与未来[J]. 中国科学基金, 2023, 37(5): 724-734.

LI G, WANG Z S, HE X T, et al. From ChatGPT to large multimodal model: present and future[J]. Bulletin of National Natural Science Foundation of China, 2023, 37(5): 724-734.

JIANG Y, YAN X Y, JI G P, et al. Effectiveness assessment of recent large vision-language models[J]. Visual Intelligence, 2024, 2(1): 1-17.

HU E J, SHEN Y, WALLIS P, et al. LoRA: Low-rank adaptation oflarge language models[EB]. arXiv preprint, 2021, arXiv: 2106.09685.

Views

340

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

On intelligence: the origin and the goal

LinYun: a domain-specific large language model for intelligent question answering and decision support in forestry

“Breathing sovereignty” in new AI era: global technological competition with no genesis traps

A review of the application of artificial intelligence in deep excavation engineering

RadiFlow: AI agents reconstructing radiology workflow

Related Author

Wang Fei-Yue

Zhao Weikang

Wang Haoyu

Hua Jing

Liu Yaobing

Wang Xiujuan

Kang Mengzhen

GE Jingwei

Related Institution

The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences

Faculty of Innovation Engineering, Macau University of Science and Technology, Macao

Jiangheng Digital Intelligence Ecological Technology Co., Ltd.

The School of Artificial Intelligence, University of Chinese Academy of Sciences

The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences

Postal code：100079
Tel：（010）53879206 Email：cjist@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：161982 Visits Today：145

⁰