metallamachatgenerator.mdx
1 --- 2 title: "MetaLlamaChatGenerator" 3 id: metallamachatgenerator 4 slug: "/metallamachatgenerator" 5 description: "This component enables chat completion with any model hosted available with Meta Llama API." 6 --- 7 8 # MetaLlamaChatGenerator 9 10 This component enables chat completion with any model hosted available with Meta Llama API. 11 12 <div className="key-value-table"> 13 14 | | | 15 | --- | --- | 16 | **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | 17 | **Mandatory init variables** | `api_key`: A Meta Llama API key. Can be set with `LLAMA_API_KEY` env variable or passed to `init()` method. | 18 | **Mandatory run variables** | `messages`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects | 19 | **Output variables** | `replies`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects | 20 | **API reference** | [Meta Llama API](/reference/integrations-meta-llama) | 21 | **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/meta_llama | 22 23 </div> 24 25 ## Overview 26 27 The `MetaLlamaChatGenerator` enables you to use multiple Meta Llama models by making chat completion calls to the Meta [Llama API](https://llama.developer.meta.com/?utm_source=partner-haystack&utm_medium=website). The default model is `Llama-4-Scout-17B-16E-Instruct-FP8`. 28 29 Currently available models are: 30 31 <div className="key-value-table"> 32 33 | | | | | | 34 | --- | --- | --- | --- | --- | 35 | Model ID | Input context length | Output context length | Input Modalities | Output Modalities | 36 | `Llama-4-Scout-17B-16E-Instruct-FP8` | 128k | 4028 | Text, Image | Text | 37 | `Llama-4-Maverick-17B-128E-Instruct-FP8` | 128k | 4028 | Text, Image | Text | 38 | `Llama-3.3-70B-Instruct` | 128k | 4028 | Text | Text | 39 | `Llama-3.3-8B-Instruct` | 128k | 4028 | Text | Text | 40 41 </div> 42 This component uses the same `ChatMessage` format as other Haystack Chat Generators for structured input and output. For more information, see the [ChatMessage documentation](../../concepts/data-classes/chatmessage.mdx). 43 44 ### Tool Support 45 46 `MetaLlamaChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations: 47 48 - **A list of Tool objects**: Pass individual tools as a list 49 - **A single Toolset**: Pass an entire Toolset directly 50 - **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list 51 52 This allows you to organize related tools into logical groups while also including standalone tools as needed. 53 54 ```python 55 from haystack.tools import Tool, Toolset 56 from haystack_integrations.components.generators.meta_llama import MetaLlamaChatGenerator 57 58 # Create individual tools 59 weather_tool = Tool(name="weather", description="Get weather info", ...) 60 news_tool = Tool(name="news", description="Get latest news", ...) 61 62 # Group related tools into a toolset 63 math_toolset = Toolset([add_tool, subtract_tool, multiply_tool]) 64 65 # Pass mixed tools and toolsets to the generator 66 generator = MetaLlamaChatGenerator( 67 tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects 68 ) 69 ``` 70 71 For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation. 72 73 ### Initialization 74 75 To use this integration, you must have a Meta Llama API key. You can provide it with the `LLAMA_API_KEY` environment variable or by using a [Secret](../../concepts/secret-management.mdx). 76 77 Then, install the `meta-llama-haystack` integration: 78 79 ```shell 80 pip install meta-llama-haystack 81 ``` 82 83 ### Streaming 84 85 `MetaLlamaChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization. 86 87 ## Usage 88 89 ### On its own 90 91 ```python 92 from haystack.dataclasses import ChatMessage 93 from haystack_integrations.components.generators.meta_llama import ( 94 MetaLlamaChatGenerator, 95 ) 96 97 llm = MetaLlamaChatGenerator() 98 response = llm.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")]) 99 print(response["replies"][0].text) 100 ``` 101 102 With streaming and model routing: 103 104 ```python 105 from haystack.dataclasses import ChatMessage 106 from haystack_integrations.components.generators.meta_llama import ( 107 MetaLlamaChatGenerator, 108 ) 109 110 llm = MetaLlamaChatGenerator( 111 model="Llama-3.3-8B-Instruct", 112 streaming_callback=lambda chunk: print(chunk.content, end="", flush=True), 113 ) 114 115 response = llm.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")]) 116 117 ## check the model used for the response 118 print("\n\n Model used: ", response["replies"][0].meta["model"]) 119 ``` 120 121 With multimodal inputs: 122 123 ```python 124 from haystack.dataclasses import ChatMessage, ImageContent 125 from haystack_integrations.components.generators.meta_llama import ( 126 MetaLlamaChatGenerator, 127 ) 128 129 llm = MetaLlamaChatGenerator(model="Llama-4-Scout-17B-16E-Instruct-FP8") 130 131 image = ImageContent.from_file_path("apple.jpg") 132 user_message = ChatMessage.from_user( 133 content_parts=["What does the image show? Max 5 words.", image], 134 ) 135 136 response = llm.run([user_message])["replies"][0].text 137 print(response) 138 139 # Red apple on straw. 140 ``` 141 142 ### In a pipeline 143 144 ```python 145 ## To run this example, you will need to set a `LLAMA_API_KEY` environment variable. 146 147 from haystack import Document, Pipeline 148 from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder 149 from haystack.components.generators.utils import print_streaming_chunk 150 from haystack.components.retrievers.in_memory import InMemoryBM25Retriever 151 from haystack.dataclasses import ChatMessage 152 from haystack.document_stores.in_memory import InMemoryDocumentStore 153 from haystack.utils import Secret 154 155 from haystack_integrations.components.generators.meta_llama import ( 156 MetaLlamaChatGenerator, 157 ) 158 159 ## Write documents to InMemoryDocumentStore 160 document_store = InMemoryDocumentStore() 161 document_store.write_documents( 162 [ 163 Document(content="My name is Jean and I live in Paris."), 164 Document(content="My name is Mark and I live in Berlin."), 165 Document(content="My name is Giorgio and I live in Rome."), 166 ], 167 ) 168 169 ## Build a RAG pipeline 170 prompt_template = [ 171 ChatMessage.from_user( 172 "Given these documents, answer the question.\n" 173 "Documents:\n{% for doc in documents %}{{ doc.content }}{% endfor %}\n" 174 "Question: {{question}}\n" 175 "Answer:", 176 ), 177 ] 178 179 ## Define required variables explicitly 180 prompt_builder = ChatPromptBuilder( 181 template=prompt_template, 182 required_variables={"question", "documents"}, 183 ) 184 185 retriever = InMemoryBM25Retriever(document_store=document_store) 186 llm = MetaLlamaChatGenerator( 187 api_key=Secret.from_env_var("LLAMA_API_KEY"), 188 streaming_callback=print_streaming_chunk, 189 ) 190 191 rag_pipeline = Pipeline() 192 rag_pipeline.add_component("retriever", retriever) 193 rag_pipeline.add_component("prompt_builder", prompt_builder) 194 rag_pipeline.add_component("llm", llm) 195 rag_pipeline.connect("retriever", "prompt_builder.documents") 196 rag_pipeline.connect("prompt_builder", "llm.messages") 197 198 ## Ask a question 199 question = "Who lives in Paris?" 200 rag_pipeline.run( 201 { 202 "retriever": {"query": question}, 203 "prompt_builder": {"question": question}, 204 }, 205 ) 206 ```