LangChain Study¶

In LLM application there are a lot of steps to do, trying different prompting, integrating different LLMs, implementing conversation history, at the end there is a lot of glue code to implement.

LangChain is a open-source framework for developing applications powered by large language models, connecting them to external data sources, and manage conversation with human.

Value propositions¶

Develop apps with context awareness, and that can reason using LLMs. It includes Python and Typescript packages, and a Java one under construction.

It focuses on composition and modularity. The components defined by the framework can be combined to address specific use cases, and developers can add new components.

• LangChain: Python and Javascript libraries
• LangServe: a library for deploying LangChain chains as a REST API.
• LangSmith: a platform that lets developers debug, test, evaluate, and monitor chains
• Predefined prompt template from langChain Hub.

They are adding new products to their portfolio quickly like LangSmith (get visibility on LLMs execution), and LangServe (server API for LangChain apps).

Sources¶

The content comes from different sources:

• Excellent product documentation, should be the go to place.
• deeplearning.ai LangChain introduction by Harisson Chase and Andrew Ng
• LLM Powered Autonomous Agents
• Retrieval and RAG blog.

LangChain libraries¶

The core building block of LangChain applications is the LLMChain:

• A LLM
• Prompt templates
• Output parsers

PromptTemplate helps to structure the prompt and facilitate reuse by creating model agnostic templates. The library includes output parsers to get content extracted from the keyword defined in the prompt. Example is the chain of thought keywords of Thought, Action, Observation.

Getting started¶

The LangChain documentation is excellent so no need to write more. All my study codes with LangChain and LLM are in different folders of this repo:

Each code needs to define only the needed LangChain modules to keep the executable size low.

Main Concepts¶

Model I/O¶

Model I/O are building blocks to interface with any language model. It facilitates the interface of model input (prompts) with the LLM model to produce the model output.

• LangChain supports two types of language: LLM (for pure text completion models) or ChatModel (conversation on top of LLM using constructs of AIMessage, HumanMessage)

• LangChain uses Prompt templates to control LLM behavior.

  • Two common prompt templates: string prompt templates and chat prompt templates.

  from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
  prompt = ChatPromptTemplate.from_messages([
      ("system", "Answer the user's questions based on the below context:\n\n{context}"),
      MessagesPlaceholder(variable_name="chat_history"),
      ("user", "{input}"),
  ])
  

  * We can build custom prompt by extending existing default templates. An example is a 'few-shot-examples' in a chat prompt using FewShotChatMessagePromptTemplate. * LangChain offers a prompt hub to get predefined prompts easily loadable:

  from langchain import hub
  prompt = hub.pull("hwchase17/openai-functions-agent")
  

• Chains allow developers to combine multiple components together (or to combine other chains) to create a single, coherent application.

• OutputParsers convert the raw output of a language model into a format that can be used downstream

Feature stores, like Feast, can be a great way to keep information about the user conversation or query, and LangChain provides an easy way to combine data from Feast with LLMs.

Chain¶

Chains are runnable, observable and composable. The LangChain framework uses the Runnable class to encapsulate operations that can be run synchronously or asynchronously.

• LLMChain class is the basic chain to integrate with any LLM.

  # Basic chain
  chain = LLMChain(llm=model, prompt = prompt)
  chain.invoke("a query")
  

• Sequential chain combines chains in sequence with single input and output (SimpleSequentialChain)

  overall_simple_chain = SimpleSequentialChain(chains=[chain_one, chain_two],
                                              verbose=True
                                              )
  

  or multiple inputs and outputs with prompt using the different environment variables (see this code).

• LLMRouterChain is a chain that outputs the name of a destination chain and the inputs to it.

• LangChain Expression Language is a declarative way to define chains. It looks similar to Unix shell pipe: input for one runnable comes from the output of predecessor (This is why prompt below is a runnable).

  # a chain definition using Langchain expression language
  chain = prompt | model | output_parser
  

• Chain can be executed asynchronously in its own Thread using the ainvoke method.

Runnable¶

Runnable interface is a protocol to define custom chains and invoke them. Each Runnable exposes methods to get input, output and config schemas. Each implements synchronous and async invoke methods and batch. Runnable can run in parallel or in sequence.

To pass data to a Runnable there is the RunnablePassthrough class. This is used in conjunction with RunnableParallel to assign data to key in a map.

from langchain.schema.runnable import RunnableParallel, RunnablePassthrough

runnable = RunnableParallel(
   passed = RunnablePassthrough(),
   extra= RunnablePassthrough.assign(mult= lambda x:x["num"] * 3),
   modified=lambda x:x["num"] +1   
)

print(runnable.invoke({"num": 6}))
{'passed': {'num': 6}, 'extra': {'num': 6, 'mult': 18}, 'modified': 7}

• RunnableLambda is a type of Runnable that wraps a callable function.

sequence = RunnableLambda(lambda x: x + 1) | {
    'mul_2': RunnableLambda(lambda x: x * 2),
    'mul_5': RunnableLambda(lambda x: x * 5)
}
sequence.invoke(1)

The RunnablePassthrough.assign method is used to create a Runnable that passes the input through while adding some keys to the output.

We can use Runnable.bind() to pass arguments as constants accessible within a runnable sequence (a chain) where argument is not part of the output of preceding runnables in the sequence.

See some code RunnableExamples

Memory¶

Large Language Models are stateless and do not remember anything. Chatbot seems to have memory, because conversation is kept in the context.

With a simple conversation like the following code, the conversation is added as string into the context:

llm = ChatOpenAI(temperature = 0)
memory = ConversationBufferMemory()
conversation = ConversationChain(
    llm= llm,
    memory=memory,
    verbose=True   # trace the chain
)

The memory is just a container in which we can save {"input:""} and {"output": ""} content.

But as the conversation goes, the size of the context grows, and so the cost of operating this chatbot, as API are charged by the size of the token. Using ConversationBufferWindowMemory(k=1) with a k necessary to keep enough context, we can limit cost. Same with ConversationTokenBufferMemory to limit the token in memory.

ConversationChain is a predefined chain to have a conversation and load context from memory.

As part of memory component there is the ConversationSummaryMemory to get the conversation summary so far.

The other important memory is Vector Data memory and entity memory or knowledgeGraph

See related code conversation_with_memory.py

Retrieval Augmented Generation¶

The goal for Retrieval Augmented Generation (RAG) is to add custom dataset not already part of a trained model and use the dataset as input sent to the LLM. RAG is illustrated in figure below:

Embed is the vector representation of a chunk of text. Different embedding can be used.

from langchain_openai import OpenAIEmbeddings
vectorstore =  Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings(),
                                     persist_directory=DOMAIN_VS_PATH)
# With HuggingFace
from sentence_transformers import SentenceTransformer
def build_embedding(docs):
    model = SentenceTransformer("all-MiniLM-L6-v2")
    return model.encode(docs)
# With AWS embedding
from langchain.embeddings import BedrockEmbeddings

Different code that implement RAG

Creating chunks is necessary because language models generally have a limit to the amount of token they can deal with. It also improve the similarity search based on vector.

# ...
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import FAISS
from langchain.indexes.vectorstore import VectorStoreIndexWrapper

loader = PyPDFDirectoryLoader("./data/")
documents = loader.load()
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size = 1000,
    chunk_overlap  = 100,
)
docs = text_splitter.split_documents(documents)

vectorstore_faiss = FAISS.from_documents(
    docs,
    embeddings,
)
vectorstore_faiss.save_local("faiss_index")

embeddings = OpenAIEmbeddings(model="text-embedding-3-large", dimensions=1024)
query = """Is it possible that ...?"""
query_embedding = embeddings.embed_query(query)
relevant_documents = vectorstore_faiss.similarity_search_by_vector(query_embedding)

During the interaction with the end-user, the system (a chain in LangChain) retrieves the most relevant data to the question asked, and passes it to LLM in the generation step.

• Embeddings capture the semantic meaning of the text to help do similarity search
• Persist the embeddings into a Vector store. Faiss and ChromaDB are common vector stores to use, but OpenSearch, Postgresql can also being used.
• Retriever includes semantic search and efficient algorithm to prepare the prompt. To improve on vector similarity search we can generate variants of the input question.

See Q&A with FAISS store qa-faiss-store.py.

• Another example of LLM Chain with AWS Bedrock llm and Feast as feature store

• Web scraping for LLM based web research. It uses the same process: document/page loading, transformation with tool like BeautifulSoup, to HTML2Text.

Q&A app¶

For Q&A the pipeline will most likely integrate with existing documents as illustrated in the figure below:

Embeddings capture the semantic meaning of the text, which helps to do similarity search. Vector store supports storage and searching of these embeddings. Retrievers use different algorithms for the semantic search to load vectors.

from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

prompt_template = """Human: Use the following pieces of context to provide a concise answer to the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

{context}

Question: {question}
Assistant:"""

PROMPT = PromptTemplate(
    template=prompt_template, input_variables=["context", "question"]
)

qa = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore_faiss.as_retriever(
        search_type="similarity", search_kwargs={"k": 3}
    ),
    return_source_documents=True,
    chain_type_kwargs={"prompt": PROMPT}
)
query = "Is it possible that I get sentenced to jail due to failure in filings?"
result = qa({"query": query})
print_ww(result['result'])

ChatBot¶

Chatbots is the most common app for LLM: Aside from basic prompting and LLMs call, chatbots have memory and retrievers:

Text Generation Examples¶

• Basic query with unknown content to generate hallucination: 1st_openAI_lc.py
• Simple test to call Bedrock with Langchain using on zero_shot generation.
• Response to an email of unhappy customer using Claude 2 and PromptTemplate. PromptTemplates allow us to create generic shells which can be populated with information and get model outputs based on different scenarios. See the text_generation/ResponseToUnhappyCustomer.py code.

Summarization chain¶

Always assess the size of the content to send, as the approach can be different: for big document, we need to split the doc in chunks.

• Small text summary with OpenAI.
• Small text to summarize, with bedrock client and the invoke_model on the client see the code in llm-langchain/summarization/SmallTextSummarization.py
• For big document, langchain provides the load_summarize_chain to summarize by chunks and get the summary of the summaries. See code with 'manual' extraction of the summaries as insights and then creating a summary of insights in summarization/long-text-summarization.py or using a LangChain summarization with map-reduce in summarization/long-text-summarization-mr.py.

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.llms.bedrock import Bedrock
from langchain.chains.summarize import load_summarize_chain

llm = Bedrock(
    model_id=modelId,
    model_kwargs={
        "max_tokens_to_sample": 1000,
    },
    client=boto3_bedrock,
) 

text_splitter = RecursiveCharacterTextSplitter(
    separators=["\n\n", "\n"], chunk_size=4000, chunk_overlap=100
)
docs = text_splitter.create_documents([letter])

summary_chain = load_summarize_chain(llm=llm, chain_type="map_reduce", verbose=True)
output = summary_chain.run(docs)

Evaluating results¶

The evaluation of a chain, we need to define the data points to be measured. Building Questions and accepted answers is a classical approach.

We can use LLM and a special chain (QAGenerateChain) to build Q&A from a document.

Agent¶

Agent is an orchestrator pattern where the LLM decides what actions to take from the current query and context. With chain, developer code the sequence of tasks, with agent the LLM decides. LangGraph is an extension of LangChain specifically aimed at creating highly controllable and customizable agents.

Chains let create a pre-defined sequence of tool usage(s), while Agents let the model uses tools in a loop, so that it can decide how many times to use its defined tools.

from langchain.agents import create_tool_calling_agent
from langchain.agents import AgentExecutor
from langchain.tools.retriever import create_retriever_tool
from langchain_community.tools.tavily_search import TavilySearchResults

...
tools = [retriever_tool, search, llm_math, wikipedia]

agent = create_tool_calling_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

Tool Calling¶

With Tool Calling we can define function or tool to be referenced as part of the LLM response, and LLM will prepare the arguments for the function. It is used to generate tool invocations, not to execute it.

Tool calling allows a model to detect when one or more tools should be called and responds with the inputs that should be passed to those tools. The inputs match a defined schema. Below is an example structured answer from OpenAI LLM: "tool_calls" is the key to get the list of function names and arguments the orchestrator needs to call.

    "tool_calls": [
            {
            "name": "tavily_search_results_json",
            "args": {
                "query": "weather in San Francisco"
            },
            "id": "call_Vg6JRaaz8d06OXbG5Gv7Ea5J"
            }

Prompt defines placeholders to get tools parameters. The following langchain prompt for OpenAI uses agent_scratchpad variable, which is a MessagesPlaceholder. Intermediate agent actions and tool output messages, will be passed in here.

LangChain has a lot of predefined tool definitions to be reused.

We can use tool calling in chain (to use tools in sequence) or in agent (to use tools in loop).

LangChain offers an API to the LLM called bind_tools to pass the definition of the tool, as part of each call to the model, so that the application can invoke the tool when appropriate.

See also the load tools api with a list of predefined tools.

Below is the classical application flow using tool calling. The exposed function wraps a remote microservice.

When developing a solution based on agent, consider the tools, the services, the agent needs to access. See a code example openAI_agent.py.

Many LLM providers support for tool calling, including Anthropic, Cohere, Google, Mistral, OpenAI, see the existing LangChain tools.

Interesting tools¶

Search recent news¶

A common tool integrated in agent, is the Tavily search API, used to get the last trusted News, so the most recent information created after the cutoff date of the LLM.

retriever_tool = create_retriever_tool(
    retriever,
    "langsmith_search",
    "Search for information about LangSmith. For any questions about LangSmith, you must use this tool!",
)
search = TavilySearchResults()
tools = [retriever_tool, search]

Python REPLtool¶

PythonREPLTool is a tool for running python code in REPL (look like a jupiter notebook).

A base model¶

It is possible to bind a BaseModel class as below, where a LLM is used to create prompt, so the prompt instruction entity is a json used as tool. (Tool definition are structured system prompt for the LLMs as they just understand text)

class PromptInstructions(BaseModel):
    """Instructions on how to prompt the LLM."""
    objective: str
    variables: List[str]
    constraints: List[str]
    requirements: List[str]

llm_with_tool = llm.bind_tools([PromptInstructions])

See the LangGraph sample: prompt_builder_graph.py.

Our own tools¶

Define custom tool using the @tool annotation on a function to expose it as a tool. It uses the function name as the tool name and the function’s docstring as the tool’s description.

A second approach is to subclass the langchain.pydantic_v1.BaseModel class.

Finally the last possible approach is to use StructuredTool dataclass.

When doing agent we need to manage exception and implement handle_tool_error.

To map the tools to OpenAI function call there is a module called: from langchain_core.utils.function_calling import convert_to_openai_function.

It may be interesting to use embeddings to do tool selection before calling LLM. See this code agent_wt_tool_retrieval.py The approach is to dynamically select the N tools we want at run time, without having to pass all the tool definitions within the context window. It uses a vector store to create embeddings for each tool description.

How Tos¶

import langchain
langchain.debug = True

# x is the response from LLM 
agent = (
        {
            "input": lambda x: x["input"],
            "agent_scratchpad": lambda x: format_to_openai_tool_messages(
                x["intermediate_steps"]
            ),
             "chat_history": lambda x: x["chat_history"],
        }
        | prompt
        | llm_with_tools
        | OpenAIToolsAgentOutputParser()
    )

LangChain Expression Language (LCEL)¶

LCEL supports streaming the LLM results, use async communication, run in parallel, retries and fallbacks, access intermediate results.