ChatGPT (GPT-4, API version “2023-03-15-preview”) was used on these 765 notes to extract all instances of the cognitive tests—MMSE and CDR—along with the dates at which the tests were mentioned to have been administered. Two examples of our task are provided in the
supplementary section S2. Inference was successful for 742 notes. The complete API call, along with the exact prompt, the temperature, and other hyper-parameters are included in
Supplementary Table S3. The prompt included a request to return these results in a JSON format. ChatGPT’s response (full), as well as the JSON formatted dialogue response were recorded in one session on June 9th 2023. The notes sent to ChatGPT were text-only, stripped of the rich-text formatting (RTF) native to our
EHR system (Epic Systems, Verona, WI). This reduced token count by approximately ten-fold, enabling notes to fit into the GPT4–8K input window and removing a substantial source of confusion for the LLM in prompt tuning. The date that the encounter was recorded in Epic was appended at the beginning of the note, proceeding with a column (“:”) then the note text. See
Supplementary Table S3 for the API request, including the prompt.
Zhang H., Jethani N., Jones S., Genes N., Major V.J., Jaffe I.S., Cardillo A.B., Heilenbach N., Ali N.F., Bonanni L.J., Clayburn A.J., Khera Z., Sadler E.C., Prasad J., Schlacter J., Liu K., Silva B., Montgomery S., Kim E.J., Lester J., Hill T.M., Avoricani A., Chervonski E., Davydov J., Small W., Chakravartty E., Grover H., Dodson J.A., Brody A.A., Aphinyanaphongs Y., Masurkar A, & Razavian N. (2024). Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores. medRxiv.