/ source-basic-api-fetcher / README.md
README.md
1 # Bank Balance Fetcher Source 2 3 This is the repository for the Bank Balance Fetcher source connector, written in Python. 4 For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/basic-api-fetcher). 5 6 ## Local development 7 8 ### Prerequisites 9 **To iterate on this connector, make sure to complete this prerequisites section.** 10 11 #### Minimum Python version required `= 3.9.0` 12 13 #### Activate Virtual Environment and install dependencies 14 From this connector directory, create a virtual environment: 15 ``` 16 python -m venv .venv 17 ``` 18 19 This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your 20 development environment of choice. To activate it from the terminal, run: 21 ``` 22 source .venv/bin/activate 23 pip install -r requirements.txt 24 pip install '.[tests]' 25 ``` 26 If you are in an IDE, follow your IDE's instructions to activate the virtualenv. 27 28 Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is 29 used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`. 30 If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything 31 should work as you expect. 32 33 #### Create credentials 34 **If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/basic-api-fetcher) 35 to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_basic_api_fetcher/spec.yaml` file. 36 Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information. 37 See `integration_tests/sample_config.json` for a sample config file. 38 39 **If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source basic-api-fetcher test creds` 40 and place them into `secrets/config.json`. 41 42 ### Locally running the connector 43 ``` 44 python main.py spec 45 python main.py check --config secrets/config.json 46 python main.py discover --config secrets/config.json 47 python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json 48 ``` 49 50 ### Locally running the connector docker image 51 52 #### Use `airbyte-ci` to build your connector 53 The Airbyte way of building this connector is to use our `airbyte-ci` tool. 54 You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1). 55 Then running the following command will build your connector: 56 57 ```bash 58 airbyte-ci connectors --name source-basic-api-fetcher build 59 ``` 60 Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-basic-api-fetcher:dev`. 61 62 ##### Customizing our build process 63 When contributing on our connector you might need to customize the build process to add a system dependency or set an env var. 64 You can customize our build process by adding a `build_customization.py` module to your connector. 65 This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively. 66 It will be imported at runtime by our build process and the functions will be called if they exist. 67 68 Here is an example of a `build_customization.py` module: 69 ```python 70 from __future__ import annotations 71 72 from typing import TYPE_CHECKING 73 74 if TYPE_CHECKING: 75 # Feel free to check the dagger documentation for more information on the Container object and its methods. 76 # https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/ 77 from dagger import Container 78 79 80 async def pre_connector_install(base_image_container: Container) -> Container: 81 return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value") 82 83 async def post_connector_install(connector_container: Container) -> Container: 84 return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value") 85 ``` 86 87 #### Build your own connector image 88 This connector is built using our dynamic built process in `airbyte-ci`. 89 The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`. 90 The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py). 91 It does not rely on a Dockerfile. 92 93 If you would like to patch our connector and build your own a simple approach would be to: 94 95 1. Create your own Dockerfile based on the latest version of the connector image. 96 ```Dockerfile 97 FROM airbyte/source-basic-api-fetcher:latest 98 99 COPY . ./airbyte/integration_code 100 RUN pip install ./airbyte/integration_code 101 102 # The entrypoint and default env vars are already set in the base image 103 # ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py" 104 # ENTRYPOINT ["python", "/airbyte/integration_code/main.py"] 105 ``` 106 Please use this as an example. This is not optimized. 107 108 2. Build your image: 109 ```bash 110 docker build -t airbyte/source-basic-api-fetcher:dev . 111 # Running the spec command against your patched connector 112 docker run airbyte/source-basic-api-fetcher:dev spec 113 ```` 114 115 #### Run 116 Then run any of the connector commands as follows: 117 ``` 118 docker run --rm airbyte/source-basic-api-fetcher:dev spec 119 docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-basic-api-fetcher:dev check --config /secrets/config.json 120 docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-basic-api-fetcher:dev discover --config /secrets/config.json 121 docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-basic-api-fetcher:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json 122 ``` 123 ## Testing 124 Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named. 125 First install test dependencies into your virtual environment: 126 ``` 127 pip install .[tests] 128 ``` 129 ### Unit Tests 130 To run unit tests locally, from the connector directory run: 131 ``` 132 python -m pytest unit_tests 133 ``` 134 135 ### Integration Tests 136 There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector). 137 #### Custom Integration tests 138 Place custom tests inside `integration_tests/` folder, then, from the connector root, run 139 ``` 140 python -m pytest integration_tests 141 ``` 142 143 ### Acceptance Tests 144 Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information. 145 If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py. 146 Please run acceptance tests via [airbyte-ci](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#connectors-test-command): 147 ```bash 148 airbyte-ci connectors --name source-basic-api-fetcher test 149 ``` 150 151 ## Dependency Management 152 All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development. 153 We split dependencies between two groups, dependencies that are: 154 * required for your connector to work need to go to `MAIN_REQUIREMENTS` list. 155 * required for the testing need to go to `TEST_REQUIREMENTS` list 156 157 ### Publishing a new version of the connector 158 You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what? 159 1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-basic-api-fetcher test` 160 2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors). 161 3. Make sure the `metadata.yaml` content is up to date. 162 4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/basic-api-fetcher.md`). 163 5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention). 164 6. Pat yourself on the back for being an awesome contributor. 165 7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master. 166