OrthoGPT: multimodal generative pre-trained transformer models for precise diagnosis and treatment of orthopedics

ZHANG Zhicheng; WANG Jing; ZHANG Yang; TIAN Yonglin; ZHANG Mengmeng; LYU Yisheng; WANG Fei-Yue

doi:10.11959/j.issn.2096-6652.202433

您当前的位置：

首页 >

文章列表页 >

OrthoGPT: multimodal generative pre-trained transformer models for precise diagnosis and treatment of orthopedics

Papers and Reports | 更新时间：2024-11-04

- OrthoGPT: multimodal generative pre-trained transformer models for precise diagnosis and treatment of orthopedics
- Chinese Journal of Intelligent Science and Technology Vol. 6, Issue 3, Pages: 338-346(2024)
- 作者机构：
  
  1.中国人民解放军总医院第七医学中心，北京 100010
  2.中国科学院自动化研究所多模态人工智能系统全国重点实验室，北京 100190
  3.澳门科技大学创新工程学院，澳门 999078
  4.中国科学院大学，北京 100049
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(62176266);Beijing Natural Science Foundation(J200012);The Science and Technology Development Fund of Macao Special Administrative Region(0145/2023/RIA3;0093/2023/RIA2)
- DOI：10.11959/j.issn.2096-6652.202433
  CLC： TP399
- Received：06 April 2024，
  
  Revised：2024-07-12，
  
  Published：15 September 2024
- 稿件说明：
移动端阅览
张志成,王静,张阳等.OrthoGPT：面向精准诊疗的多模态骨科大模型[J].智能科学与技术学报,2024,06(03):338-346.

ZHANG Zhicheng,WANG Jing,ZHANG Yang,et al.OrthoGPT: multimodal generative pre-trained transformer models for precise diagnosis and treatment of orthopedics[J].Chinese Journal of Intelligent Science and Technology,2024,06(03):338-346.
张志成,王静,张阳等.OrthoGPT：面向精准诊疗的多模态骨科大模型[J].智能科学与技术学报,2024,06(03):338-346. DOI： 10.11959/j.issn.2096-6652.202433.

ZHANG Zhicheng,WANG Jing,ZHANG Yang,et al.OrthoGPT: multimodal generative pre-trained transformer models for precise diagnosis and treatment of orthopedics[J].Chinese Journal of Intelligent Science and Technology,2024,06(03):338-346. DOI： 10.11959/j.issn.2096-6652.202433.

摘要

针对骨科疾病诊疗面临的非医院场景下应急处理困难、手术规划复杂且风险高以及康复治疗个性化需求强等诸多挑战，提出了面向骨科精准诊疗的多模态骨科大模型——OrthoGPT。该模型充分利用多模态特征对齐与融合技术和多场景个性化诊疗方案生成技术，结合基于平行理论的骨科世界模型和多Agent智能交互方法，为非医院场景提供初步诊断和应急指导，模拟和预测个性化术后治疗效果，辅助医生进行手术规划，并提供个性化康复治疗建议。该模型旨在探讨OrthoGPT在骨科疾病诊疗中的应用潜力，以期提高诊疗效率和提升效果，改善患者体验，推动人工智能技术在骨科疾病诊疗的发展和应用。

Abstract

This paper proposes OrthoGPT

a multimodal generative pre-trained transformer model designed for precise diagnosis and treatment of orthopedic diseases. It addresses several challenges in orthopedic care

such as the difficulties in emergency management outside hospital settings

the complexity and high risks associated with surgical planning

and the strong demand for personalized rehabilitation therapy. The model leverages multimodal feature alignment and fusion technology

multi-scenario personalized diagnosis and treatment plan generation technology

combined with an orthopedic world model based on parallel theory and multi-agent methods. It aims to provide preliminary diagnosis and emergency guidance in non-hospital scenarios

simulate and predict personalized postoperative treatment outcomes

assist doctors in surgical planning

and offer personalized rehabilitation treatment suggestions. The purpose of this paper is to explore the potential applications of OrthoGPT in improving the efficiency and effectiveness of orthopedic diagnosis and treatment

enhancing the patient experience

and advancing the development and application of artificial intelligence in the field of orthopedics.

关键词

Keywords

references

ACHIAM J, ADLER S, AGARWAL S, et al. GPT-4 technical report[EB ] . arXiv preprint, 2023, arXiv: 2303.08774 .

LUO R Q, SUN L A, XIA Y C, et al. BioGPT: generative pre-trained transformer for biomedical text generation and mining[J]. Briefings in Bioinformatics, 2022, 23(6): 409.

SINGHAL K, AZIZI S, TU T, et al. Large language models encode clinical knowledge[J]. Nature, 2023, 620(7972): 172-180.

ALSENTZER E, MURPHY J R, BOAG W, et al. Publicly available clinical BERT embeddings[EB ] . arXiv preprint, 2019, arXiv: 1904.03323 .

CIEZA A, CAUSEY K, KAMENOV K, et al. Global estimates of the need for rehabilitation based on the Global Burden of Disease study 2019: a systematic analysis for the Global Burden of Disease Study 2019[J]. The Lancet, 2021, 396(10267): 2006-2017.

王飞跃. 平行系统方法与复杂系统的管理和控制[J]. 控制与决策, 2004, 19(5): 485-489, 514.

WANG F Y. Parallel system methods for management and control of complex systems[J]. Control and Decision, 2004, 19(5): 485-489, 514.

王飞跃. 平行医学: 从医学的温度到智慧的医学[J]. 智能科学与技术学报, 2021, 3(1): 1-9.

WANG F Y. Parallel medicine: from warmness of medicare to medicine of smartness[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 1-9.

王拥军, 王飞跃, 王戈, 等. 平行医院: 从医院信息管理系统到智慧医院操作系统[J]. 自动化学报, 2021, 47(11): 2585-2599.

WANG Y J, WANG F Y, WANG G, et al. Parallel hospitals: from hospital information system(HIS)to hospital smart operating system(HSOS)[J]. Acta Automatica Sinica, 2021, 47(11): 2585-2599.

王飞跃, 李长贵, 国元元, 等. 平行高特: 基于ACP的平行痛风诊疗系统框架[J]. 模式识别与人工智能, 2017, 30(12): 1057-1068.

WANG F Y, LI C G, GUO Y Y, et al. Parallel gout: an ACP-based system framework for gout diagnosis and treatment[J]. Pattern Recognition and Artificial Intelligence, 2017, 30(12): 1057-1068.

WANG J, WANG X, GUO Y Y, et al. A parallel medical diagnostic and treatment system for chronic diseases[C]//Proceedings of the 2020 Chinese Automation Congress (CAC). Piscataway: IEEE Press, 2020: 7412-7416.

王飞跃. 数字医生与平行医疗: 从医疗知识自动化到系统化智能医学[J]. 协和医学杂志, 2021, 12(6): 829-833.

WANG F Y. Digital doctors and parallel healthcare: from medical knowledge automation to intelligent metasystems medicine[J]. Medical Journal of Peking Union Medical College Hospital, 2021, 12(6): 829-833.

王飞跃, 张梅, 孟祥冰, 等. 平行手术: 基于ACP的智能手术计算方法[J]. 模式识别与人工智能, 2017, 30(11): 961-970.

WANG F Y, ZHANG M, MENG X B, et al. Parallel surgery: an ACP-based approach for intelligent operations[J]. Pattern Recognition and Artificial Intelligence, 2017, 30(11): 961-970.

赵毅飞, 申乐, 叶佩军, 等. 平行麻醉: 从麻醉自动化走向智慧型全周期麻醉平台[J]. 智能科学与技术学报, 2023, 5(2): 234-246.

ZHAO Y F, SHEN L, YE P J, et al. Parallel anesthesia: from anesthesia automation to intelligent full-cycle anesthesia platform[J]. Chinese Journal of Intelligent Science and Technology, 2023, 5(2): 234-246.

王惠珍, 张捷, 俞怡, 等. 平行手术室: 围术期护理流程与智慧手术平台管理的新模式[J]. 模式识别与人工智能, 2023, 36(10): 867-876.

WANG H Z, ZHANG J, YU Y, et al. Parallel operating rooms: a new model of perioperative nursing process and smart surgical platform management[J]. Pattern Recognition and Artificial Intelligence, 2023, 36(10): 867-876.

ZHAO Y F, WANG J, WANG F Y, et al. Paragraph vector based retrieval model for similar cases recommendation[C]//Proceedings of the 2016 12th World Congress on Intelligent Control and Automation (WCICA). Piscataway: IEEE Press, 2016: 2220-2225.

WANG J, MAN C T, ZHAO Y F, et al. An answer recommendation algorithm for medical community question answering systems[C]//Proceedings of the 2016 IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI). Piscataway: IEEE Press, 2016: 139-144.

TIAN Y L, WANG X, SHEN Y, et al. Parallel point clouds: hybrid point cloud generation and 3D model enhancement via virtual–real integration[J]. Remote Sensing, 2021, 13(15): 2868.

林飞, 王飞跃, 田永林, 等. 平行药物系统: 基于大语言模型和三类人的框架与方法[J]. 智能科学与技术学报, 2024, 6(1): 88-99.

LIN F, WANG F Y, TIAN Y L, et al. Parallel drug system: framework and method based on big language model and three kinds of people[J]. Chinese Journal of Intelligent Science and Technology, 2024, 6(1): 88-99.

LIN F, GAO T, SUN D, et al. Parallel medical devices and instruments: integrating edge and cloud intelligence for smart treatment and health systems[J]. IEEE/CAA Journal of Automatica Sinica, accepted, doi: 10.1109/JAS.2024.124614.

WANG F Y, LI J J, QIN R, et al. ChatGPT for computational social systems: from conversational applications to human-oriented operating systems[J]. IEEE Transactions on Computational Social Systems, 2023, 10(2): 414-425.

QIN R, WANG F Y, ZHENG X L, et al. Sora for computational social systems: from counterfactual experiments to artificiofactual experiments with parallel intelligence[J]. IEEE Transactions on Computational Social Systems, 2024, 11(2): 1531-1550.

TIAN Y L, LI X, ZHANG H, et al. VistaGPT: generative parallel transformers for vehicles with intelligent systems for transport automation[J]. IEEE Transactions on Intelligent Vehicles, 2023, 8(9): 4198-4207.

FAN L L, GUO C, TIAN Y L, et al. Sora for foundation robots with parallel intelligence: three world models, three robotic systems[J]. Frontiers of Information Technology & Electronic Engineering, 2024, 25(7): 917-923.

DAI X Y, GUO C, TANG Y, et al. VistaRAG: toward safe and trustworthy autonomous driving through retrieval-augmented generation[J]. IEEE Transactions on Intelligent Vehicles, 2024, 9(4): 4579-4582.

YANG J, WANG X, ZHAO Y, et al. RAG-based crowdsourcing task decomposition via masked contrastive learning with prompts[EB ] . arXiv preprint, 2024, arXiv: 2406.06577 .

LI X, LIU E L, SHEN T Y, et al. ChatGPT-based scenario engineer: a new framework on scenario generation for trajectory prediction[J]. IEEE Transactions on Intelligent Vehicles, 2024, 9(3): 4422-4431.

卢经纬, 郭超, 戴星原, 等. 问答ChatGPT之后: 超大预训练模型的机遇和挑战[J]. 自动化学报, 2023, 49(4): 705-717.

LU J W, GUO C, DAI X Y, et al. The ChatGPT after: opportunities and challenges of very large scale pre-trained models[J]. Acta Automatica Sinica, 2023, 49(4): 705-717.

缪青海, 王兴霞, 杨静, 等. 从基础智能到通用智能:基于大模型的GenAI和AGI之现状与展望[J]. 自动化学报, 2024, 50(4):674-687.

MIAO Q H, WANG X X, YANG J, et al. From foundation intelligence to general intelligence: the state-of-art and perspectives of GenAI and AGI based on foundation models[J]. Acta Automatica Sinica, 2024, 50(4): 674-687.

WANG F Y, MIAO Q, LI L, et al. When does Sora show: the beginning of TAO to imaginative intelligence and scenarios engineering[J]. IEEE/CAA Journal of Automatica Sinica, 2024, 11(4): 809-815.

田永林, 王雨桐, 王建功, 等. 视觉Transformer研究的关键问题: 现状及展望[J]. 自动化学报, 2022, 48(4): 957-979.

TIAN Y L, WANG Y T, WANG J G, et al. Key problems and progress of vision transformers: the state of the art and prospects[J]. Acta Automatica Sinica, 2022, 48(4): 957-979.

WANG F Y, MIAO Q H, LI X, et al. What does ChatGPT say: the DAO from algorithmic intelligence to linguistic intelligence[J]. IEEE/CAA Journal of Automatica Sinica, 2023, 10(3): 575-579.

MUMTAZ U, AHMED A, MUMTAZ S. LLMs-healthcare: current applications and challenges of large language models in various medical specialties[EB ] . arXiv preprint, 2023, arXiv: 2311.12882 .

WANG G Y, YANG G X, DU Z X, et al. ClinicalGPT: large language models finetuned with diverse medical data and comprehensive evaluation[EB ] . arXiv preprint, 2023, arXiv: 2306.09968 .

GIORGINO R, ALESSANDRI-BONETTI M, LUCA A, et al. ChatGPT in orthopedics: a narrative review exploring the potential of artificial intelligence in orthopedic practice[J]. Frontiers in Surgery, 2023, 10: 1284015.

GENGATHARAN D, SAGGI S S, BIN ABD RAZAK H R. Pre-operative planning of high tibial osteotomy with ChatGPT: are we there yet?[J]. Cureus, 2024, 16(2): e54858.

NAKAJIMA N, FUJIMORI T, FURUYA M, et al. A comparison between GPT-3.5, GPT-4, and GPT-4V: can the large language model (ChatGPT) pass the Japanese board of orthopaedic surgery examination?[J]. Cureus, 2024, 16(3): e56402.

HAKAM H T, PRILL R, KORTE L, et al. Human-written vs AI-generated texts in orthopedic academic literature: comparative qualitative analysis[J]. JMIR Formative Research, 2024, 8: e52164.

BUTLER J J, HARRINGTON M C, TONG Y X, et al. From jargon to clarity: Improving the readability of foot and ankle radiology reports with an artificial intelligence large language model[J]. Foot and Ankle Surgery, 2024, 30(4): 331-337.

ZAIDAT B, LAHOTI Y S, YU A, et al. Artificially intelligent billing in spine surgery: an analysis of a large language model[J]. Global Spine Journal, accepted, doi: 10.1177/21925682231224753

KILLEEN B D, CHAUDHARY S, OSGOOD G, et al. Take a shot! Natural language control of intelligent robotic X-ray systems in surgery[J]. International Journal of Computer Assisted Radiology and Surgery, 2024, 19(6): 1165-1173.

KOTZUR T, SINGH A, PARKER J, et al. Evaluation of a large language model's ability to assist in an orthopedic hand clinic[J]. Hand, accepted, doi: 10.1177/15589447241257643

SHRESTHA N, SHEN Z K, ZAIDAT B, et al. Performance of ChatGPT on NASS clinical guidelines for the diagnosis and treatment of low back pain: a comparison study[J]. Spine, 2024, 49(9): 640-651.

LUBITZ M, LATARIO L. Performance of two artificial intelligence generative language models on the orthopaedic in-training examination[J]. Orthopedics, 2024, 47(3): 146-150.

CATERSON J, AMBLER O, CERECEDA-MONTEOLIVA N, et al. Application of generative language models to orthopaedic practice[J]. BMJ Open, 2024, 14(3): e076484.

田永林, 王兴霞, 王雨桐, 等. RAG-PHI: 检索增强生成驱动的平行人与平行智能[J]. 智能科学与技术学报, 2024, 6(1): 41-51.

TIAN Y L, WANG X X, WANG Y T, et al. RAG-PHI: searching for parallel people and parallel intelligence driven by enhanced generation[J]. Chinese Journal of Intelligent Science and Technology, 2024, 6(1): 41-51.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Cerberus: a cross-site social bot detection system based on deep learning

Related Author

No data

Related Institution

No data

Postal code：100079
Tel：（010）53879206 Email：cjist@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：161982 Visits Today：145

⁰