Chatting (More) About ChatGPT and Generative AI
As a follow-up to the ISMPP U webinar, The Rise of ChatGPT and Generative AI: Implications for Medical Writing, Authorship, Content Development and QA Systems, this episode of InformEd aims to address unanswered audience questions and discuss new developments in this quickly changing field. Our focus is on understanding ChatGPT and other large language models, exploring how they work – including strengths and limitations, and discussing ways to get the most out of ChatGPT now – plus new skills we may need to learn. Our guests will also look to the future and consider what that landscape may look like.
Guest host Jen Webster, Senior Director, Precision Medicine RWE Lead at Pfizer has spent the last 12 years building and using real world data systems to generate evidence and insights. She is joined by Jenny Ghith, Senior Director, Omnichannel Strategy and Innovations Lead, Global Scientific Communications, at Pfizer Oncology who works to advance the understanding of science by bridging AI technologies from data scientists to the healthcare community; and James Turnbull, Founder of Camino Communications where he uses his background in computer science and medcomms to bring the value of digital to healthcare communications.
Where to Listen
Find us in your favorite podcast app.
Jen Webster (00:00):
Hello, and welcome to InformED, a podcast series where you will hear industry experts share their thought-provoking insights and lessons in the field of medical communications. This series is brought to you by ISMPP and is generously sponsored by MedThink SciCom.
Jen Webster (00:15):
The views expressed in this recording are those of the individuals and not necessarily of ISMPP or of the individuals' companies with which they are affiliated. The presentation is for informational purposes only and is not intended as legal or regulatory advice.
Jen Webster (00:30):
Today, we will be chatting more about ChatGPT and generative AI. This podcast is picking up where we left off from a recently held webinar on this same subject where we received a lot of engagement and a lot of questions from the attendees.
Jen Webster (00:49):
What we hope to do today is to pick up where we left off and based on those questions, go into depth in a few more areas. I'm going to start off with some introductions. I'd like to introduce you to our esteemed panel. I will hand it off first to James Turnbull.
James Turnbull (01:06):
Hi, my name's James Turnbull, I'm the founder at a company called Camino, which is a newly launched agency. I've worked in digital healthcare for about 15 years now. So, today, I'm bringing my experience of having a degree in computer science and working in technology, but also working in the Medcomms industry and bringing those two things together.
Jen Webster (01:28):
And Jennifer Ghith.
Jennifer Ghith (01:31):
Hi, Jen Webster and James, it's so good to have you both here and to join you. I'm the omnichannel strategy and innovations lead for global scientific communications at Pfizer, and I work in oncology.
Jennifer Ghith (01:44):
My role is to develop big, new innovative ideas and scale them up in our oncology franchise as well as ensure they're executed with impact in the context of our omnichannel strategies. I work in social media, closed platforms, and I have a special interest in artificial intelligence and large language models as well as patient centricity initiatives.
Jennifer Ghith (02:11):
I've been in the industry for almost five years, and in Medcomms for over 15 years before that. And that's all I'll admit to before you start to date me too much. But I'm, really passionate about what we do. I think this is a very interesting time and I'm very much looking forward to our discussion today.
Jen Webster (02:31):
Great. And I am Jen Webster, I will be your MC for today. I am a real-world evidence scientist at Pfizer, along with Jenny Ghith. In addition to the more statistically-driven real world evidence work that I do, I have a deep interest in AI and how we can use AI to extract insights both from things like real world data, but also from the literature, so I'm thrilled to be here today.
Jen Webster (02:53):
Let's start it off. So, just as a reminder for our audience, James, can you tell us what is ChatGPT?
James Turnbull (03:04):
So, let's start at the beginning. It's a currently free website chats.openai.com. Where what you do is you interact, you chat with what's called a large language model. To the company, Open AI have created this language model, essentially, which is the latest in a series of developments in their GPT series. That's generative pre-trained transformer. Scientific term for, they've basically scraped billions of bits of text off the internet put that all together and modeled that in a computer program and an algorithm.
James Turnbull (03:42):
So, what that means is that by feeding it with such a huge amount of text is able to not understand, but be able to draw correlations between the pieces of text and therefore, what a logical order of how they go together is. And that understanding of text leads it to have the ability to also generate text. And that's the interesting bit, is that it is able to produce human-sounding language from this massive amount of data script off the internet.
James Turnbull (04:14):
There's two layers that go on top of that. There's what they call the reinforcement learning with human feedback, which is essentially the model will output a bunch of potential ways of talking, and then real humans have scored those as this sounds more human as opposed to this thing.
James Turnbull (04:32):
And that's why it is so good at sounding human, because it's being trained by humans and that sounds weird, that sounds real. And then the next layer on top of that is they have a safety layer, which stops it from outputting all the awful, biased things that you would get from scraping all the content of the internet. So, all the dark recesses of Reddit are kept to bay by that safety layer that they put on top.
James Turnbull (04:56):
But ultimately, it's a really good auto complete. It's not intelligent, it's just a great way of auto completing text. So, if you give it "Once upon ..." it will look at its huge database of text and go, statistically, the next word is, "A" and then statistically the next word is "Time," and so on and so on.
James Turnbull (05:14):
The transformer bit is what the clever element is because it's able to look at the whole sentence and therefore, generate the next bit of text based on everything it's already said. And that's why it's so good at telling the whole story because it's able to understand what it said before and therefore, what to say next. It has that wide context.
James Turnbull (05:32):
But I think crucially, it's not a database of information. It's not a knowledge model, it's a language model. So, the background is its ability to generate text, which is based on text that's read from the internet. That doesn't mean it really outputs truth.
James Turnbull (05:53):
Open AI themselves, say ChatGPT sometimes writes plausible sounding, but incorrect or nonsensical answers. And obviously, there's been many examples we've seen on the internet of some crazy and wild outlandish things that it's generated.
Jen Webster (06:08):
Indeed, I think that point you made about it sounding human is so important. I know that's probably the part that impacted me the most when I first saw it. On that note, Jenny, do you want to talk a little bit about why this has become such a hot topic? Why is this an inflection point for these large language models?
Jennifer Ghith (06:27):
Oh, gosh, Jen, so great question. It had over a million users in less than five days when it launched at the end of November of last year. That's faster, way faster in fact than Instagram, Facebook, Netflix, some of the other platforms that are used abroad by the broader public.
Jennifer Ghith (06:52):
But it's really been very interesting. You watch the news outlets and you see these captivated reporters also using it and it's been reported in the New York Times and CNN, et cetera, as well.
Jennifer Ghith (07:08):
But to answer your question, why, I think it's been percolating for a while. There are a number of large language models that have been worked on for a very long time — quite a few in fact. And so this isn't new information, and we've also seen the image generators and DALL.E 2 caused a lot of excitement, and you see those avatars online as well.
Jennifer Ghith (07:35):
But I think we're having a moment because there's a lot of interest amongst the public in science right now. There's a change in how we're consuming information through social media as well that's making information more accessible.
Jennifer Ghith (07:53):
So, all of this has come to a head, if you will, because the timing has been right. And even since ChatGPT, it's also triggered a lot of competition. We could call it an arms race, but the combative things are a little difficult. But it's struck a code with Google, and search engine work, and they have their own language model, and they launched Bard via YouTube right, a little while ago.
Jennifer Ghith (08:27):
So, everyone's also trying to capitalize on this time, and I think as folks who are working in science, we have an opportunity to provide a clear point of view on what we think about ChatGPT, and to build from our learnings through database with the pandemic, but also even through real-world evidence, Jen, because there are a lot of parallels in what we're seeing in terms of potential for evidence generation there.
Jen Webster (09:00):
I totally agree and I think you framing this as kind of us as scientists is so important in this time when everybody's so concerned with what is the science really saying, and how do we really know what's true?
Jen Webster (09:11):
Jenny, I'm going to hand it back to you and ask if you could (and this is starting to get now into some of the questions that we heard over and over in our webinar) talk a little bit in the area of disclosure, use and trust. Should we be disclosing our use of ChatGPT as we're working as scientists? When should we disclose it? Should it be an author?
Jen Webster (09:34):
There's all of these questions right now, but what do you do when you use ChatGPT to generate your work? And could you talk a little bit about how do we really maintain our trust in each other and the public's trust in us as scientists in our communication as we start working with this new tool set?
Jennifer Ghith (09:51):
So, great questions because we're also part of ISMPP, and we're working on scientific publications, which are in many ways, the currency of how we communicate our science too. So, I think that what may be reassuring to people is that basic principles still apply.
Jennifer Ghith (10:13):
So, we have ICMJE criteria for authorship, and that requires substantial contributions to the conception of design of the work, to participating in drafting of the work, to final approval of the version that's published as well as agreement to be accountable.
Jennifer Ghith (10:35):
And even if you ask ChatGPT itself (which I encourage you guys to try out) it will tell you that it is a source for information, but it is not necessarily appropriate for it to be listed as an author.
Jennifer Ghith (10:54):
Now, all that being said, there are (to my knowledge) four current publications that have actually listed ChatGPT as an author, and all this hit right at about the time the model was capturing the public's attention. And I think that it would be interesting to hear a bit more about (from the authors themselves) how that occurred and the process and how the editors felt comfortable with that.
Jennifer Ghith (11:24):
Because I think journals and societies are really trying to move with the environment and update their policies, and also do so in a transparent way for the public so that the contributions are acknowledged and understood if we do use ChatGPT.
Jennifer Ghith (11:42):
So, that's a lot of information, but again, basic rules apply and I would also just add too, that Open AI and other groups are developing classifiers, and what's a classifier in this instance? This is a way for us to tag or be aware that text is generated by AI.
Jennifer Ghith (12:07):
So, right now, there's still a lot of work to do in that field, and the classifiers, they're working to improve their accuracy. But I think that that is promising in terms of transparency and for the public, and for us all to be aware of what is being generated by AI versus what's not.
Jen Webster (12:28):
That's great.
James Turnbull (12:29):
Just to add to that, Jenny, I completely agree that I think authorship isn't going to really make sense in terms of assigning authorship to ChatGPT. I don't think that will ever make sense until we get to some very distant future where we do have general artificial intelligence, which this isn't.
James Turnbull (12:47):
But I think acknowledgement is actually really important. So, yes, I acknowledge that I used a model such as this. But I actually read a paper just today about a study where people were using ChatGPT to write an apology, and then they did an experiment to see how that apology was received.
James Turnbull (13:06):
And unsurprisingly, an apology delivered with, "I used AI to help me write this" was not taken so well, it was seen as, "Well, I don't really believe you're apologizing then."
Jennifer Ghith (13:17):
I've had job applications already where people put at the bottom of the job application, "I did not use ChatGPT to write this."
James Turnbull (13:26):
So, I think there's also the opposite effect now we may find of people acknowledging that they haven't used it to infer some sort of human quality on, "I produced this, I didn't use someone else to write this, some tools to write this."
Jen Webster (13:40):
It's very interesting. You gave us a nice overview and this is a bit of a philosophical question for you, James. But you gave us a nice overview on how ChatGPT actually works. I think one of the big controversies about these large language models generally was when a Google engineer came out and claimed that their model was sentient.
Jen Webster (13:58):
Can you talk a little bit about what can these models do? They can pass certain exams, they're sort of scored at the level of a nine-year-old? How do you put all of that into context?
James Turnbull (14:12):
That person was ultimately fired by Google for what they said, and the AI they were using was not the sentient, but it was very good at appearance form. We have the Turing test, which is the kind of classical is AI sentient test.
James Turnbull (14:33):
Some people say ChatGPT is past that because it can pass off as human, some people will be forgiven for thinking they're talking to a human. But in the spirit of that test, it is not intelligent, it's just very good at printing words.
Jen Webster (14:49):
That's great. Now, I'm going to go from the philosophical to the very pragmatic. One of the biggest questions that we got asked, Jenny, I'm going to pose to you: should we just blindly trust and input data into these models? Are they storing our data? What happens if we use these models to help us do our work? Is that kept confidential or do we need to worry about that?
Jennifer Ghith (15:13):
It's a very important philosophical and ethical question. Again, though, basic rules apply. So, we all work with confidential data on a regular basis in many of our jobs in the day-to-day. And the same rules apply in terms of sharing of that data that apply in other instances.
Jennifer Ghith (15:43):
The other thing to be aware of is that ChatGPT only uses data through 2021, and if you're using the publicly accessible part of ChatGPT, your queries are actually being used to train it further, which is a good thing and that the AI is improving, but it also does mean that your queries are in the system.
Jennifer Ghith (16:09):
So, I think that it's important to be conscious as with any tools that we use when we're inputting our data, of the technology and the privacy considerations that come into play. And if you're in doubt, I encourage you to speak with your governance and your legal colleagues, and really just phone a friend and make sure you truly understand what you're working with.
Jennifer Ghith (16:37):
And I think that's really a part of upskilling and understanding these new systems. And look, it's a daunting task, there's a lot going on out there. We all are very busy in our day-to-day, but this is something that is worth understanding and learning about, and again, talking with your cross-functional colleagues to really understand what you need to know in order to use these systems.
Jen Webster (17:03):
That's great. One more really pragmatic question this time, James, for you — could you talk us through some tips and trick tricks for writing prompts? What new skill sets are we all going to have to develop as we start using these models?
James Turnbull (17:16):
And it has a name, prompt engineering is a new skill set and I reckon it's going to be a new job title in not a very short space of time. But certainly, working with ChatGPT, there's already examples and nuances about how to get the best out of it.
James Turnbull (17:34):
I think the first thing is how you give it instruction. You can either say, "Give me something, write me a recipe for beans and toast." Or you can ask it to complete something, that example, like I said, "Once upon a ..." I find it quite useful when I'm writing an email, I go, "What's the next words?" Stick it in there, it'll finish the sentence for you. So, that kind of completion.
James Turnbull (17:57):
And then the third one is the demonstration or what's called few short learning, which is essentially, where you say, given this structure for example, question, answer, question, answer, question, answer — I'll give you a question and then you can continue the answers. So, you're giving it a little bit of extra learning by demonstrating the kind of output that you want.
James Turnbull (18:17):
So, those are the three ways in which you can make your request. But then once you get into that, quite often it'll just stop mid-sentence either because it hits a buffer because it confuses itself and says ... I've had it say kind of there was two people thingy, thingy and then it gets confused because that sentence no longer works.
James Turnbull (18:39):
So, when it gets at that stopped point, you can just say, "Please continue, go on," and it'll resume. You can try and enforce a little bit of truthfulness by just saying, "If you don't know, say, I don't know" which doesn't work all the time. It will still produce some quite nonsensical words, but it limits it a bit more by just enforcing that on top of it.
James Turnbull (19:04):
But the way a lot of people are using it, is in a more actor sort of approach. So, you say, "I want you to act as this sort of person, I will give you my job application and you will then respond to me as if you were the recruiter at a large pharma company."
James Turnbull (19:25):
And by giving it that context, it gives you much better responses because you've created an environment in which it will respond with the appropriate words. So, I will, you will works very well.
James Turnbull (19:38):
And then, lastly, it's weird because as a computer program, you'd expect it to be very good at maths, it's actually terrible at maths. If you are trying to do step by step math equations — so the classic things like there's 14 cars in the car park and half of them are BMWs and half of them are blue, how many were green? That sort of stuff. ChatGPT is terrible at answering those sorts of maths questions if you give it the whole question.
James Turnbull (20:06):
But a paper found that if you just say, "Let's think step by step and then do the question," it will give you a much more accurate answer. Because by saying, "Let's think step by step," it will step through the process and go, "Well, there's this many cars and there's that many cars, and therefore, the answer is x."
James Turnbull (20:23):
Apparently, it goes up from 18% accuracy rate at maths, which is rubbish to 79%, which is probably better than me . So, few keywords, I'm sure more will come out over the next few weeks as people use and use and use these models and find these nuances.
Jen Webster (20:44):
I think that's great. We are nearing the end of our time here, so I'm going to set us up to close here by having us all go around and talk about what's the thing you're most excited about using these models for now. And then I will close us out. So, James, I'm going to hand it back to you and then, Jenny, and then I'll finish this up.
James Turnbull (21:07):
So, I saw something on Twitter, a guy called Kevin Cannon posted this on Twitter, and it's a scenario where somebody writes "I want a job, here's my resume, Brian." And they stick that into ChatGPT and it outputs "Salutations and greetings most esteemed, I'm filled with exuberance ... et cetera, et cetera."
James Turnbull (21:25):
And then the recruiter takes that message and puts it back into ChatGPT and gets out, "Brian wants a job, here is his PDF." If you take that and extrapolate to the future as we've all got this integrated into our email and nobody's seeing the email in the middle process, do we bother writing that to take it to our world in terms of communicating science?
James Turnbull (21:48):
And I don't know if this will happen, but in a future where everybody's using these sorts of tools, all our audiences, your specialist oncologists, your patients, everybody in between, if they're all using large language models to pars the science that we're generating, do we end up writing for the large language models? Do we then change the way we communicate science because we're just communicating with this model to then communicate to the person on the other side? I don't know.
Jennifer Ghith (22:20):
I think it's an interesting thought, James, and Jen, I'll jump right in. I think there's just a diversification of knowledge and of the science, and I think that that's wonderful. And having patients be able to ask questions of these systems is really important for them. So, I look forward to seeing more of that and seeing it done in a responsible and ethical way.
Jennifer Ghith (22:49):
But I also look forward to the days of the other models and seeing the improvements that are going to happen and very fast. So, Bing has launched its own integration of GPT-3 into Microsoft, and they've got a wait list for having access to it, but what it spits out to you is referenced. So, I think we're going to see improvements that will be very helpful.
Jennifer Ghith (23:20):
I look forward to understanding the abilities for more personalized searches because the literature is expanding so exponentially, and being able to find those rare signals in the literature that we all look for and try and monitor for.
Jennifer Ghith (23:34):
I look forward to living guidelines, living systematic literature review documents, using this technology using it to help us generate hypotheses. But really truly, what I think is the most fun right now is that it's bringing the community together, folks like us, bridging conversation between the AI experts, publications teams, Medcomms professionals, med affairs, industry.
Jennifer Ghith (24:04):
I think that is very exciting and we're going to learn a lot about how to talk to each other and how to learn from each other in the coming months as well.
Jen Webster (24:12):
That's great, I'm going to agree with a version of that. ChatGPT is already a very good coder for certain things in my corner of the world. So, there's a real-world evidence data standard that we use called OMOP and you can ask ChatGPT to write you OMOP code to answer specific evidence generation questions.
Jen Webster (24:37):
And so, I think there's going to be a ton of changes about the way that we can transparently and quickly generate evidence using real-world data. I have this dream of a day when we can actually just put a whole protocol into one of these models and it spits out a perfectly executed study.
Jen Webster (24:54):
We are obviously very far from it, but I think that might be one of the keys to ever really getting to true precision medicine. Drug rate, patient rate time — is being able to do evidence generation on that scale.
Jennifer Ghith (25:07):
So, with that, I'm going to thank you both so much for being here, and close this up for today.
James Turnbull (25:16):
Thanks, Jenny.
Jennifer Ghith (25:18):
Thanks guys, be well.
Jen Webster (25:19):
Thanks for your time, everyone, and we encourage you to also access and view the recordings of the original ChatGPT webinar and also, the Plain Language GPT webinar at ismpp.org under the education tab.
Jen Webster (25:31):
Thank you for listening to InformED for Medical Communication Professionals. Please take a minute to subscribe to the show on your favorite podcast app, inform your colleagues, and rate our show highly, if you liked what you heard today.
Jen Webster (25:42):
We hope you will also join us at an upcoming ISMPP U webinar or even consider becoming a member of our association. Just go to ismpp.org ...
Hide TranscriptRecent Episodes
View AllBringing the Patient Voice into Company-Sponsored Publications: The Med Comms Perspective
InformEDBringing the Patient Voice into Company-Sponsored Publications: The Pharma Perspective
InformEDDigital Features: Are They Worth the Effort? Questions Answered
InformEDAccessible Conference Presentations: Results and Insights From a Study
InformEDHear More From Us!
Subscribe Today and get the newest Evergreen content delivered straight to your inbox!