\u201cIt\u2019s entertaining, and perhaps mildly useful as a creative help,\u201d LeCun wrote. \u201cBut trying to build intelligent machines by scaling up language models is like building high-altitude airplanes to go to the moon. You might beat altitude records, but going to the moon will require a completely different approach.\u201d<\/p>\n
To support his claims, LeCun pointed to a new study of the model\u2019s<\/a> performance in healthcare scenarios by Nabla, a medtech firm cofounded by two of his former colleagues at Facebook.<\/span><\/p>\n [Read: What audience intelligence data tells us about the 2020 US presidential election<\/a>]<\/span><\/em><\/p>\n The researchers note that Open AI\u2019s GPT-3 guidelines put healthcare \u201cin the high stakes category because people rely on accurate medical information for life-or-death decisions, and mistakes here could result in serious harm.\u201d In addition, diagnosing medical or psychiatric conditions are unsupported uses of the model.<\/p>\n Nonetheless, Nabla tried it out on a range of healthcare use cases.<\/span><\/p>\n The researchers found that GPT-3 seemed helpful in finding information in long documents and in basic admin tasks such as appointment booking. But it lacked the memory, logic, and understanding of time for many more specific questions.<\/p>\n Nabla also found that GPT-3 was an unreliable Q&A support tool for doctors, dangerously oversimplified medical documentation analysis, and struggled to associate causes with consequences.<\/p>\n The model also made some basic errors in diagnosis and provided some reckless mental health advice.<\/p>\n The researchers do see some potential for using language models in medical settings. But they conclude that GPT-3 is \u201cnowhere near\u201d ready to provide significant help in the sector:<\/p>\n Because of the way it was trained, it lacks the scientific and medical expertise that would make it useful for medical documentation, diagnosis support, treatment recommendation or any medical Q&A. Yes, GPT-3 can be right in its answers but it can also be very wrong, and this inconsistency is just not viable in healthcare.<\/span><\/p>\n<\/blockquote>\n Their findings won\u2019t shock OpenAI, given the firm\u2019s warnings against using GPT-3 in healthcare. But they do show that many expectations for the model are wildly unrealistic.<\/p>\n Published October 28, 2020 \u2014 15:07 UTC <\/p>\nHow did GPT-3 perform?<\/h2>\n
![](\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2020\/10\/Screenshot-2020-10-28-at-13.16.15.png\")
![](\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2020\/10\/Screenshot-2020-10-28-at-13.20.22.png\")
\n