How to add standard tests to an integration

When creating either a custom class for yourself or to publish in a LangChain integration, it is important to add standard tests to ensure it works as expected. This guide will show you how to add standard tests to each integration type.

Setup

If you're coming from the previous guide, you have already installed these dependencies, and you can skip this section.

First, let's install 2 dependencies:

• langchain-core will define the interfaces we want to import to define our custom tool.
• langchain-tests will provide the standard tests we want to use. Recommended to pin to the latest version:

note

Because added tests in new versions of langchain-tests can break your CI/CD pipelines, we recommend pinning the version of langchain-tests to avoid unexpected changes.

Poetry

If you followed the previous guide, you should already have these dependencies installed!

poetry add langchain-core
poetry add --group test pytest pytest-socket pytest-asyncio langchain-tests==<latest_version>
poetry install --with test

Pip

pip install -U langchain-core pytest pytest-socket pytest-asyncio langchain-tests

# install current package in editable mode
pip install --editable .

Let's say we're publishing a package, langchain_parrot_link, that exposes the chat model from the guide on implementing the package. We can add the standard tests to the package by following the steps below.

And we'll assume you've structured your package the same way as the main LangChain packages:

langchain-parrot-link/
├── langchain_parrot_link/
│   ├── __init__.py
│   └── chat_models.py
├── tests/
│   ├── __init__.py
│   ├── unit_tests/
│   │   ├── __init__.py
│   │   └── test_chat_models.py
│   └── integration_tests/
│       ├── __init__.py
│       └── test_chat_models.py
├── pyproject.toml
└── README.md

Add and configure standard tests

There are 2 namespaces in the langchain-tests package:

• unit tests (langchain_tests.unit_tests): designed to be used to test the component in isolation and without access to external services
• integration tests (langchain_tests.integration_tests): designed to be used to test the component with access to external services (in particular, the external service that the component is designed to interact with).

Both types of tests are implemented as pytest class-based test suites.

By subclassing the base classes for each type of standard test (see below), you get all of the standard tests for that type, and you can override the properties that the test suite uses to configure the tests.

Implementing standard tests

In the following tabs, we show how to implement the standard tests for each component type:

Chat models

Here's how you would configure the standard unit tests for the custom chat model:

Chat model standard tests test a range of behaviors, from the most basic requirements (generating a response to a query) to optional capabilities like multi-modal support and tool-calling. For a test run to be successful:

1. If a feature is intended to be supported by the model, it should pass;
2. If a feature is not intended to be supported by the model, it should be skipped.

Tests for "optional" capabilities are controlled via a set of properties that can be overridden on the test model subclass.

You can see the entire list of properties in the API references for unit tests and integration tests.

For example, to enable integration tests for image inputs, we can implement

@property
def supports_image_inputs(self) -> bool:
    return True

on the integration test class.

note

Details on what tests are run, how each test can be skipped, and troubleshooting tips for each test can be found in the API references. See details:

• Unit tests API reference
• Integration tests API reference

Unit test example:

tests/unit_tests/test_chat_models.py

"""Test chat model integration."""

from typing import Type

from langchain_parrot_link.chat_models import ChatParrotLink
from langchain_tests.unit_tests import ChatModelUnitTests


class TestChatParrotLinkUnit(ChatModelUnitTests):
    @property
    def chat_model_class(self) -> Type[ChatParrotLink]:
        return ChatParrotLink

    @property
    def chat_model_params(self) -> dict:
        # These should be parameters used to initialize your integration for testing
        return {
            "model": "bird-brain-001",
            "temperature": 0,
            "parrot_buffer_length": 50,
        }

Integration test example:

tests/integration_tests/test_chat_models.py

"""Test ChatParrotLink chat model."""

from typing import Type

from langchain_parrot_link.chat_models import ChatParrotLink
from langchain_tests.integration_tests import ChatModelIntegrationTests


class TestChatParrotLinkIntegration(ChatModelIntegrationTests):
    @property
    def chat_model_class(self) -> Type[ChatParrotLink]:
        return ChatParrotLink

    @property
    def chat_model_params(self) -> dict:
        # These should be parameters used to initialize your integration for testing
        return {
            "model": "bird-brain-001",
            "temperature": 0,
            "parrot_buffer_length": 50,
        }

Vector stores

Here's how you would configure the standard tests for a typical vector store (using ParrotVectorStore as a placeholder):

Note that unlike chat models, vector stores do not have optional capabilities that can be enabled or disabled at this time.

tests/integration_tests/test_vectorstores.py

from typing import Generator

import pytest
from langchain_parrot_link.vectorstores import ParrotVectorStore
from langchain_core.vectorstores import VectorStore
from langchain_tests.integration_tests import VectorStoreIntegrationTests


class TestParrotVectorStore(VectorStoreIntegrationTests):
    @pytest.fixture()
    def vectorstore(self) -> Generator[VectorStore, None, None]:  # type: ignore
        """Get an empty vectorstore for unit tests."""
        store = ParrotVectorStore(self.get_embeddings())
        # note: store should be EMPTY at this point
        # if you need to delete data, you may do so here
        try:
            yield store
        finally:
            # cleanup operations, or deleting data
            pass

Configuring the tests consists of implementing pytest fixtures for setting up an empty vector store and tearing down the vector store after the test run ends.

For example, below is the VectorStoreIntegrationTests class for the Chroma integration:

from typing import Generator

import pytest
from langchain_core.vectorstores import VectorStore
from langchain_tests.integration_tests.vectorstores import VectorStoreIntegrationTests

from langchain_chroma import Chroma


class TestChromaStandard(VectorStoreIntegrationTests):
    @pytest.fixture()
    def vectorstore(self) -> Generator[VectorStore, None, None]:  # type: ignore
        """Get an empty vectorstore for unit tests."""
        store = Chroma(embedding_function=self.get_embeddings())
        try:
            yield store
        finally:
            store.delete_collection()
            pass

API Reference:VectorStore

Note that before the initial yield, we instantiate the vector store with an embeddings object. This is a pre-defined "fake" embeddings model that will generate short, arbitrary vectors for documents. You can use a different embeddings object if desired.

In the finally block, we call whatever integration-specific logic is needed to bring the vector store to a clean state. This logic is executed in between each test (e.g., even if tests fail).

note

Details on what tests are run and troubleshooting tips for each test can be found in the API reference.

Embeddings

To configure standard tests for an embeddings model, we subclass EmbeddingsUnitTests and EmbeddingsIntegrationTests. On each subclass, we implement the embeddings_class property to specify the embeddings model to be tested. We can also override the embedding_model_params property to specify initialization parameters. See examples below.

note

Details on what tests are run, how each test can be skipped, and troubleshooting tips for each test can be found in the API references. See details:

• Unit tests API reference
• Integration tests API reference

Unit test example:

tests/unit_tests/test_embeddings.py

"""Test embedding model integration."""

from typing import Type

from langchain_parrot_link.embeddings import ParrotLinkEmbeddings
from langchain_tests.unit_tests import EmbeddingsUnitTests


class TestParrotLinkEmbeddingsUnit(EmbeddingsUnitTests):
    @property
    def embeddings_class(self) -> Type[ParrotLinkEmbeddings]:
        return ParrotLinkEmbeddings

    @property
    def embedding_model_params(self) -> dict:
        return {"model": "nest-embed-001"}

Integration test example:

tests/integration_tests/test_embeddings.py

from typing import Type

from langchain_parrot_link.embeddings import ParrotLinkEmbeddings
from langchain_tests.integration_tests import EmbeddingsIntegrationTests


class TestParrotLinkEmbeddingsIntegration(EmbeddingsIntegrationTests):
    @property
    def embeddings_class(self) -> Type[ParrotLinkEmbeddings]:
        return ParrotLinkEmbeddings

    @property
    def embedding_model_params(self) -> dict:
        return {"model": "nest-embed-001"}

tests/integration_tests/test_embeddings.py

"""Test ParrotLink embeddings."""

from typing import Type

from langchain_parrot_link.embeddings import ParrotLinkEmbeddings
from langchain_tests.integration_tests import EmbeddingsIntegrationTests


class TestParrotLinkEmbeddingsIntegration(EmbeddingsIntegrationTests):
    @property
    def embeddings_class(self) -> Type[ParrotLinkEmbeddings]:
        return ParrotLinkEmbeddings

    @property
    def embedding_model_params(self) -> dict:
        return {"model": "nest-embed-001"}

Tools

To configure standard tests for a tool, we subclass ToolsUnitTests and ToolsIntegrationTests. On each subclass, we override the following @property methods to specify the tool to be tested and the tool's configuration:

Property	Description
tool_constructor	The constructor for the tool to be tested, or an instantiated tool.
tool_constructor_params	The parameters to pass to the tool (optional).
tool_invoke_params_example	An example of the parameters to pass to the tool's invoke method.

If you are testing a tool class and pass a class like MyTool to tool_constructor, you can pass the parameters to the constructor in tool_constructor_params.

If you are testing an instantiated tool, you can pass the instantiated tool to tool_constructor and do not override tool_constructor_params.

note

Details on what tests are run, how each test can be skipped, and troubleshooting tips for each test can be found in the API references. See details:

• Unit tests API reference
• Integration tests API reference

tests/unit_tests/test_tools.py

from typing import Type

from langchain_parrot_link.tools import ParrotTool
from langchain_tests.unit_tests import ToolsUnitTests


class TestParrotMultiplyToolUnit(ToolsUnitTests):
    @property
    def tool_constructor(self) -> Type[ParrotTool]:
        return ParrotTool

    @property
    def tool_constructor_params(self) -> dict:
        # if your tool constructor instead required initialization arguments like
        # `def __init__(self, some_arg: int):`, you would return those here
        # as a dictionary, e.g.: `return {'some_arg': 42}`
        return {}

    @property
    def tool_invoke_params_example(self) -> dict:
        """
        Returns a dictionary representing the "args" of an example tool call.

        This should NOT be a ToolCall dict - i.e. it should not
        have {"name", "id", "args"} keys.
        """
        return {"a": 2, "b": 3}

tests/integration_tests/test_tools.py

from typing import Type

from langchain_parrot_link.tools import ParrotTool
from langchain_tests.integration_tests import ToolsIntegrationTests


class TestParrotMultiplyToolIntegration(ToolsIntegrationTests):
    @property
    def tool_constructor(self) -> Type[ParrotTool]:
        return ParrotTool

    @property
    def tool_constructor_params(self) -> dict:
        # if your tool constructor instead required initialization arguments like
        # `def __init__(self, some_arg: int):`, you would return those here
        # as a dictionary, e.g.: `return {'some_arg': 42}`
        return {}

    @property
    def tool_invoke_params_example(self) -> dict:
        """
        Returns a dictionary representing the "args" of an example tool call.

        This should NOT be a ToolCall dict - i.e. it should not
        have {"name", "id", "args"} keys.
        """
        return {"a": 2, "b": 3}

Retrievers

To configure standard tests for a retriever, we subclass RetrieversUnitTests and RetrieversIntegrationTests. On each subclass, we override the following @property methods

Property	Description
retriever_constructor	The class for the retriever to be tested
retriever_constructor_params	The parameters to pass to the retriever's constructor
retriever_query_example	An example of the query to pass to the retriever's invoke method

note

Details on what tests are run and troubleshooting tips for each test can be found in the API reference.

tests/integration_tests/test_retrievers.py

from typing import Type

from langchain_parrot_link.retrievers import ParrotRetriever
from langchain_tests.integration_tests import (
    RetrieversIntegrationTests,
)


class TestParrotRetriever(RetrieversIntegrationTests):
    @property
    def retriever_constructor(self) -> Type[ParrotRetriever]:
        """Get an empty vectorstore for unit tests."""
        return ParrotRetriever

    @property
    def retriever_constructor_params(self) -> dict:
        return {"k": 2}

    @property
    def retriever_query_example(self) -> str:
        """
        Returns a str representing the "query" of an example retriever call.
        """
        return "example query"

---

Running the tests

You can run these with the following commands from your project root

Poetry

# run unit tests without network access
poetry run pytest --disable-socket --allow-unix-socket --asyncio-mode=auto tests/unit_tests

# run integration tests
poetry run pytest --asyncio-mode=auto tests/integration_tests

Pip

# run unit tests without network access
pytest --disable-socket --allow-unix-socket --asyncio-mode=auto tests/unit_tests

# run integration tests
pytest --asyncio-mode=auto tests/integration_tests

Test suite information and troubleshooting

For a full list of the standard test suites that are available, as well as information on which tests are included and how to troubleshoot common issues, see the Standard Tests API Reference.

You can see troubleshooting guides under the individual test suites listed in that API Reference. For example, here is the guide for ChatModelIntegrationTests.test_usage_metadata.