1  ---
  2  title: "SagemakerGenerator"
  3  id: sagemakergenerator
  4  slug: "/sagemakergenerator"
  5  description: "This component enables text generation using LLMs deployed on Amazon Sagemaker."
  6  ---
  7  
  8  # SagemakerGenerator
  9  
 10  This component enables text generation using LLMs deployed on Amazon Sagemaker.
 11  
 12  <div className="key-value-table">
 13  
 14  |  |  |
 15  | --- | --- |
 16  | **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
 17  | **Mandatory init variables** | `model`: The model to use  <br /> <br />`aws_access_key_id`: AWS access key ID. Can be set with `AWS_ACCESS_KEY_ID` env var.  <br /> <br />`aws_secret_access_key`: AWS secret access key. Can be set with `AWS_SECRET_ACCESS_KEY` env var. |
 18  | **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
 19  | **Output variables** | `replies`: A list of strings with all the replies generated by the LLM  <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
 20  | **API reference** | [Amazon Sagemaker](/reference/integrations-amazon-sagemaker) |
 21  | **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/amazon_sagemaker |
 22  
 23  </div>
 24  
 25  `SagemakerGenerator` allows you to make use of models deployed on [AWS SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/whatis.html).
 26  
 27  ## Parameters Overview
 28  
 29  `SagemakerGenerator` needs AWS credentials to work.  Set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.
 30  
 31  You also need to specify your Sagemaker endpoint at initialization time for the component to work. Pass the endpoint name to the `model` parameter like this:
 32  
 33  ```python
 34  generator = SagemakerGenerator(model="jumpstart-dft-hf-llm-falcon-7b-instruct-bf16")
 35  ```
 36  
 37  Additionally, you can pass any text generation parameters valid for your specific model directly to `SagemakerGenerator` using the `generation_kwargs` parameter, both at initialization and to `run()` method.
 38  
 39  If your model also needs custom attributes, pass those as a dictionary at initialization time by setting the `aws_custom_attributes` parameter.
 40  
 41  One notable family of models that needs these custom parameters is Llama2, which needs to be initialized with `{"accept_eula": True}` :
 42  
 43  ```python
 44  generator = SagemakerGenerator(
 45      model="jumpstart-dft-meta-textgenerationneuron-llama-2-7b",
 46      aws_custom_attributes={"accept_eula": True},
 47  )
 48  ```
 49  
 50  ## Usage
 51  
 52  You need to install `amazon-sagemaker-haystack` package to use the  `SagemakerGenerator`:
 53  
 54  ```shell
 55  pip install amazon-sagemaker-haystack
 56  ```
 57  
 58  ### On its own
 59  
 60  Basic usage:
 61  
 62  ```python
 63  from haystack_integrations.components.generators.amazon_sagemaker import SagemakerGenerator
 64  
 65  client = SagemakerGenerator(model="jumpstart-dft-hf-llm-falcon-7b-instruct-bf16")
 66  client.warm_up()
 67  response = client.run("Briefly explain what NLP is in one sentence.")
 68  print(response)
 69  
 70  >>> {'replies': ["Natural Language Processing (NLP) is a subfield of artificial intelligence and computational linguistics that focuses on the interaction between computers and human languages..."],
 71   'metadata': [{}]}
 72  ```
 73  
 74  ### In a pipeline
 75  
 76  In a RAG pipeline:
 77  
 78  ```python
 79  from haystack_integrations.components.generators.amazon_sagemaker import (
 80      SagemakerGenerator,
 81  )
 82  from haystack import Pipeline
 83  from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
 84  from haystack.components.builders import PromptBuilder
 85  
 86  template = """
 87  Given the following information, answer the question.
 88  
 89  Context:
 90  {% for document in documents %}
 91      {{ document.content }}
 92  {% endfor %}
 93  
 94  Question: What's the official language of {{ country }}?
 95  """
 96  pipe = Pipeline()
 97  
 98  pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
 99  pipe.add_component("prompt_builder", PromptBuilder(template=template))
100  pipe.add_component(
101      "llm",
102      SagemakerGenerator(model="jumpstart-dft-hf-llm-falcon-7b-instruct-bf16"),
103  )
104  pipe.connect("retriever", "prompt_builder.documents")
105  pipe.connect("prompt_builder", "llm")
106  
107  pipe.run({"prompt_builder": {"country": "France"}})
108  ```