/ docs-website / versioned_docs / version-2.21 / pipeline-components / generators / openaichatgenerator.mdx
openaichatgenerator.mdx
1 --- 2 title: "OpenAIChatGenerator" 3 id: openaichatgenerator 4 slug: "/openaichatgenerator" 5 description: "`OpenAIChatGenerator` enables chat completion using OpenAI’s large language models (LLMs)." 6 --- 7 8 # OpenAIChatGenerator 9 10 `OpenAIChatGenerator` enables chat completion using OpenAI's large language models (LLMs). 11 12 <div className="key-value-table"> 13 14 | | | 15 | --- | --- | 16 | **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | 17 | **Mandatory init variables** | `api_key`: An OpenAI API key. Can be set with `OPENAI_API_KEY` env var. | 18 | **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat | 19 | **Output variables** | `replies`: A list of alternative replies of the LLM to the input chat | 20 | **API reference** | [Generators](/reference/generators-api) | 21 | **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/openai.py | 22 23 </div> 24 25 ## Overview 26 27 `OpenAIChatGenerator` supports OpenAI models starting from gpt-3.5-turbo and later (gpt-4, gpt-4-turbo, and so on). 28 29 `OpenAIChatGenerator` needs an OpenAI key to work. It uses an ` OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: 30 31 ```python 32 generator = OpenAIChatGenerator(model="gpt-4o-mini") 33 ``` 34 35 Then, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. See the [usage](#usage) section for an example. 36 37 You can pass any chat completion parameters valid for the `openai.ChatCompletion.create` method directly to `OpenAIChatGenerator` using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the parameters supported by the OpenAI API, refer to the [OpenAI documentation](https://platform.openai.com/docs/api-reference/chat). 38 39 `OpenAIChatGenerator` can support custom deployments of your OpenAI models through the `api_base_url` init parameter. 40 41 ### Structured Output 42 43 `OpenAIChatGenerator` supports structured output generation, allowing you to receive responses in a predictable format. You can use Pydantic models or JSON schemas to define the structure of the output through the `response_format` parameter in `generation_kwargs`. 44 45 This is useful when you need to extract structured data from text or generate responses that match a specific format. 46 47 ```python 48 from pydantic import BaseModel 49 from haystack.components.generators.chat import OpenAIChatGenerator 50 from haystack.dataclasses import ChatMessage 51 52 class NobelPrizeInfo(BaseModel): 53 recipient_name: str 54 award_year: int 55 category: str 56 achievement_description: str 57 nationality: str 58 59 client = OpenAIChatGenerator( 60 model="gpt-4o-2024-08-06", 61 generation_kwargs={"response_format": NobelPrizeInfo} 62 ) 63 64 response = client.run(messages=[ 65 ChatMessage.from_user( 66 "In 2021, American scientist David Julius received the Nobel Prize in" 67 " Physiology or Medicine for his groundbreaking discoveries on how the human body" 68 " senses temperature and touch." 69 ) 70 ]) 71 print(response["replies"][0].text) 72 73 >> {"recipient_name":"David Julius","award_year":2021,"category":"Physiology or Medicine", 74 >> "achievement_description":"David Julius was awarded for his transformative findings 75 >> regarding the molecular mechanisms underlying the human body's sense of temperature 76 >> and touch. Through innovative experiments, he identified specific receptors responsible 77 >> for detecting heat and mechanical stimuli, ranging from gentle touch to pain-inducing 78 >> pressure.","nationality":"American"} 79 ``` 80 81 :::info[Model Compatibility and Limitations] 82 83 - Pydantic models and JSON schemas are supported for latest models starting from `gpt-4o-2024-08-06`. 84 - Older models only support basic JSON mode through `{"type": "json_object"}`. For details, see [OpenAI JSON mode documentation](https://platform.openai.com/docs/guides/structured-outputs#json-mode). 85 - Streaming limitation: When using streaming with structured outputs, you must provide a JSON schema instead of a Pydantic model for `response_format`. 86 - For complete information, check the [OpenAI Structured Outputs documentation](https://platform.openai.com/docs/guides/structured-outputs). 87 ::: 88 89 ### Streaming 90 91 You can stream output as it’s generated. Pass a callback to `streaming_callback`. Use the built-in `print_streaming_chunk` to print text tokens and tool events (tool calls and tool results). 92 93 ```python 94 from haystack.components.generators.utils import print_streaming_chunk 95 96 ## Configure any `Generator` or `ChatGenerator` with a streaming callback 97 component = SomeGeneratorOrChatGenerator(streaming_callback=print_streaming_chunk) 98 99 ## If this is a `ChatGenerator`, pass a list of messages: 100 ## from haystack.dataclasses import ChatMessage 101 ## component.run([ChatMessage.from_user("Your question here")]) 102 103 ## If this is a (non-chat) `Generator`, pass a prompt: 104 ## component.run({"prompt": "Your prompt here"}) 105 ``` 106 107 :::info 108 Streaming works only with a single response. If a provider supports multiple candidates, set `n=1`. 109 ::: 110 111 See our [Streaming Support](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) docs to learn more how `StreamingChunk` works and how to write a custom callback. 112 113 Give preference to `print_streaming_chunk` by default. Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting. 114 115 ## Usage 116 117 ### On its own 118 119 Basic usage: 120 121 ```python 122 from haystack.dataclasses import ChatMessage 123 from haystack.components.generators.chat import OpenAIChatGenerator 124 125 client = OpenAIChatGenerator() 126 response = client.run( 127 [ChatMessage.from_user("What's Natural Language Processing? Be brief.")] 128 ) 129 print(response) 130 131 >> {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content= 132 >> [TextContent(text='Natural Language Processing (NLP) is a field of artificial 133 >> intelligence that focuses on the interaction between computers and humans through 134 >> natural language. It involves enabling machines to understand, interpret, and 135 >> generate human language in a meaningful way, facilitating tasks such as 136 >> language translation, sentiment analysis, and text summarization.')], 137 >> _name=None, _meta={'model': 'gpt-4o-mini-2024-07-18', 'index': 0, 138 >> 'finish_reason': 'stop', 'usage': {'completion_tokens': 59, 'prompt_tokens': 15, 139 >> 'total_tokens': 74, 'completion_tokens_details': {'accepted_prediction_tokens': 140 >> 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 141 >> 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}})]} 142 ``` 143 144 With streaming: 145 146 ```python 147 from haystack.dataclasses import ChatMessage 148 from haystack.components.generators.chat import OpenAIChatGenerator 149 150 client = OpenAIChatGenerator(streaming_callback=lambda chunk: print(chunk.content, end="", flush=True)) 151 response = client.run( 152 [ChatMessage.from_user("What's Natural Language Processing? Be brief.")] 153 ) 154 print(response) 155 156 >> Natural Language Processing (NLP) is a field of artificial intelligence that 157 >> focuses on the interaction between computers and humans through natural language. 158 >> It involves enabling machines to understand, interpret, and generate human 159 >> language in a way that is both meaningful and useful. NLP encompasses various 160 >> tasks, including speech recognition, language translation, sentiment analysis, 161 >> and text summarization.{'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 162 >> 'assistant'>, _content=[TextContent(text='Natural Language Processing (NLP) is a 163 >> field of artificial intelligence that focuses on the interaction between computers 164 >> and humans through natural language. It involves enabling machines to understand, 165 >> interpret, and generate human language in a way that is both meaningful and 166 >> useful. NLP encompasses various tasks, including speech recognition, language 167 >> translation, sentiment analysis, and text summarization.')], _name=None, _meta={' 168 >> model': 'gpt-4o-mini-2024-07-18', 'index': 0, 'finish_reason': 'stop', 169 >> 'completion_start_time': '2025-05-15T13:32:16.572912', 'usage': None})]} 170 ``` 171 172 With multimodal inputs: 173 174 ```python 175 from haystack.dataclasses import ChatMessage, ImageContent 176 from haystack.components.generators.chat import OpenAIChatGenerator 177 178 llm = OpenAIChatGenerator(model="gpt-4o-mini") 179 180 image = ImageContent.from_file_path("apple.jpg", detail="low") 181 user_message = ChatMessage.from_user(content_parts=[ 182 "What does the image show? Max 5 words.", 183 image 184 ]) 185 186 response = llm.run([user_message])["replies"][0].text 187 print(response) 188 189 >>> Red apple on straw. 190 ``` 191 192 ### In a Pipeline 193 194 ```python 195 from haystack.components.builders import ChatPromptBuilder 196 from haystack.components.generators.chat import OpenAIChatGenerator 197 from haystack.dataclasses import ChatMessage 198 from haystack import Pipeline 199 from haystack.utils import Secret 200 201 ## no parameter init, we don't use any runtime template variables 202 prompt_builder = ChatPromptBuilder() 203 llm = OpenAIChatGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY"), model="gpt-4o-mini") 204 205 pipe = Pipeline() 206 pipe.add_component("prompt_builder", prompt_builder) 207 pipe.add_component("llm", llm) 208 pipe.connect("prompt_builder.prompt", "llm.messages") 209 location = "Berlin" 210 messages = [ChatMessage.from_system("Always respond in German even if some input data is in other languages."), 211 ChatMessage.from_user("Tell me about {{location}}")] 212 pipe.run(data={"prompt_builder": {"template_variables":{"location": location}, "template": messages}}) 213 214 >> {'llm': {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, 215 >> _content=[TextContent(text='Berlin ist die Hauptstadt Deutschlands und eine der 216 >> bedeutendsten Städte Europas. Es ist bekannt für ihre reiche Geschichte, 217 >> kulturelle Vielfalt und kreative Scene. \n\nDie Stadt hat eine bewegte 218 >> Vergangenheit, die stark von der Teilung zwischen Ost- und Westberlin während 219 >> des Kalten Krieges geprägt war. Die Berliner Mauer, die von 1961 bis 1989 die 220 >> Stadt teilte, ist heute ein Symbol für die Wiedervereinigung und die Freiheit. 221 >> \n\nBerlin bietet eine Fülle von Sehenswürdigkeiten, darunter das Brandenburger 222 >> Tor, den Reichstag, die Museumsinsel und den Alexanderplatz. Die Stadt ist auch 223 >> für ihre lebendige Kunst- und Musikszene bekannt, mit zahlreichen Galerien, 224 >> Theatern und Clubs. ')], _name=None, _meta={'model': 'gpt-4o-mini-2024-07-18', 225 >> 'index': 0, 'finish_reason': 'stop', 'usage': {'completion_tokens': 260, 226 >> 'prompt_tokens': 29, 'total_tokens': 289, 'completion_tokens_details': 227 >> {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 228 >> 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 229 >> 'cached_tokens': 0}}})]}} 230 ``` 231 232 ## Additional References 233 234 :notebook: Tutorial: [Building a Chat Application with Function Calling](https://haystack.deepset.ai/tutorials/40_building_chat_application_with_function_calling) 235 236 🧑🍳 Cookbook: [Function Calling with OpenAIChatGenerator](https://haystack.deepset.ai/cookbook/function_calling_with_openaichatgenerator)