We will then select which embeddings we want to use.
from langchain.embeddings import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()
We now create the vectorstore to use as the index.
from langchain.vectorstores import Chroma
db = Chroma.from_documents(texts, embeddings)
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.
So that’s creating the index. Then, we expose this index in a retriever interface.
retriever = db.as_retriever()
Then, as before, we create a chain and use it to answer questions!
qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=retriever)
query = "What did the president say about Ketanji Brown Jackson"
qa.run(query)
" The President said that Judge Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He said she is a consensus builder and has received a broad range of support from organizations such as the Fraternal Order of Police and former judges appointed by Democrats and Republicans."
VectorstoreIndexCreator is just a wrapper around all this logic. It is configurable in the text splitter it uses, the embeddings it uses, and the vectorstore it uses. For example, you can configure it as below:
index_creator = VectorstoreIndexCreator(
    vectorstore_cls=Chroma,
    embedding=OpenAIEmbeddings(),
    text_splitter=CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
)


)
Hopefully this highlights what is going on under the hood of VectorstoreIndexCreator. While we think it’s important to have a simple way to create indexes, we also think it’s important to understand what’s going on under the hood.
previous
Indexes
next
Document Loaders
 Contents

One Line Index Creation
Walkthrough
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Getting Started
Getting Started#
The default recommended text splitter is the RecursiveCharacterTextSplitter. This text splitter takes a list of characters. It tries to create chunks based on splitting on the first character, but if any chunks are too large it then moves onto the next character, and so forth. By default the characters it tries to split on are ["\n\n", "\n", " ", ""]
In addition to controlling which characters you can split on, you can also control a few other things:
length_function: how the length of chunks is calculated. Defaults to just counting number of characters, but it’s pretty common to pass a token counter here.
chunk_size: the maximum size of your chunks (as measured by the length function).
chunk_overlap: the maximum overlap between chunks. It can be nice to have some overlap to maintain some continuity between chunks (eg do a sliding window).
# This is a long document we can split up.
with open('../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()
from langchain.text_splitter import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
    # Set a really small chunk size, just to show.
    chunk_size = 100,
    chunk_overlap  = 20,
    length_function = len,
)
texts = text_splitter.create_documents([state_of_the_union])
print(texts[0])
print(texts[1])
page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and' lookup_str='' metadata={} lookup_index=0
page_content='of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.' lookup_str='' metadata={} lookup_index=0
previous
Text Splitters
next
Character Text Splitter
By Harrison Chase


previous
Text Splitters
next
Character Text Splitter
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Latex Text Splitter
Latex Text Splitter#
LatexTextSplitter splits text along Latex headings, headlines, enumerations and more. It’s implemented as a simple subclass of RecursiveCharacterSplitter with Latex-specific separators. See the source code to see the Latex syntax expected by default.
How the text is split: by list of latex specific tags
How the chunk size is measured: by length function passed in (defaults to number of characters)
from langchain.text_splitter import LatexTextSplitter
latex_text = """
\documentclass{article}
\begin{document}
\maketitle
\section{Introduction}
Large language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing tasks, including language translation, text generation, and sentiment analysis.
\subsection{History of LLMs}
The earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made it possible to train LLMs on massive datasets, leading to significant improvements in performance.
\subsection{Applications of LLMs}
LLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.
\end{document}
"""
latex_splitter = LatexTextSplitter(chunk_size=400, chunk_overlap=0)
docs = latex_splitter.create_documents([latex_text])
docs


docs = latex_splitter.create_documents([latex_text])
docs
[Document(page_content='\\documentclass{article}\n\n\x08egin{document}\n\n\\maketitle', lookup_str='', metadata={}, lookup_index=0),
 Document(page_content='Introduction}\nLarge language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing tasks, including language translation, text generation, and sentiment analysis.', lookup_str='', metadata={}, lookup_index=0),
 Document(page_content='History of LLMs}\nThe earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made it possible to train LLMs on massive datasets, leading to significant improvements in performance.', lookup_str='', metadata={}, lookup_index=0),
 Document(page_content='Applications of LLMs}\nLLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.\n\n\\end{document}', lookup_str='', metadata={}, lookup_index=0)]
previous
Hugging Face Length Function
next
Markdown Text Splitter
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
RecursiveCharacterTextSplitter
RecursiveCharacterTextSplitter#
This text splitter is the recommended one for generic text. It is parameterized by a list of characters. It tries to split on them in order until the chunks are small enough. The default list is ["\n\n", "\n", " ", ""]. This has the effect of trying to keep all paragraphs (and then sentences, and then words) together as long as possible, as those would generically seem to be the strongest semantically related pieces of text.
How the text is split: by list of characters
How the chunk size is measured: by length function passed in (defaults to number of characters)
# This is a long document we can split up.
with open('../../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()
from langchain.text_splitter import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
    # Set a really small chunk size, just to show.
    chunk_size = 100,
    chunk_overlap  = 20,
    length_function = len,
)
texts = text_splitter.create_documents([state_of_the_union])
print(texts[0])
print(texts[1])
page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and' lookup_str='' metadata={} lookup_index=0
page_content='of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.' lookup_str='' metadata={} lookup_index=0
previous
Python Code Text Splitter
next
Spacy Text Splitter
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Python Code Text Splitter
Python Code Text Splitter#
PythonCodeTextSplitter splits text along python class and method definitions. It’s implemented as a simple subclass of RecursiveCharacterSplitter with Python-specific separators. See the source code to see the Python syntax expected by default.
How the text is split: by list of python specific characters
How the chunk size is measured: by length function passed in (defaults to number of characters)
from langchain.text_splitter import PythonCodeTextSplitter
python_text = """
class Foo:
    def bar():

def foo():
def testing_func():
def bar():
"""
python_splitter = PythonCodeTextSplitter(chunk_size=30, chunk_overlap=0)
docs = python_splitter.create_documents([python_text])
docs
[Document(page_content='Foo:\n\n    def bar():', lookup_str='', metadata={}, lookup_index=0),
 Document(page_content='foo():\n\ndef testing_func():', lookup_str='', metadata={}, lookup_index=0),
 Document(page_content='bar():', lookup_str='', metadata={}, lookup_index=0)]
previous
NLTK Text Splitter
next
RecursiveCharacterTextSplitter
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
NLTK Text Splitter
NLTK Text Splitter#
Rather than just splitting on “\n\n”, we can use NLTK to split based on tokenizers.
How the text is split: by NLTK
How the chunk size is measured: by length function passed in (defaults to number of characters)
# This is a long document we can split up.
with open('../../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()
from langchain.text_splitter import NLTKTextSplitter
text_splitter = NLTKTextSplitter(chunk_size=1000)
texts = text_splitter.split_text(state_of_the_union)
print(texts[0])
Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.
Members of Congress and the Cabinet.
Justices of the Supreme Court.
My fellow Americans.
Last year COVID-19 kept us apart.
This year we are finally together again.
Tonight, we meet as Democrats Republicans and Independents.
But most importantly as Americans.
With a duty to one another to the American people to the Constitution.
And with an unwavering resolve that freedom will always triumph over tyranny.
Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.
But he badly miscalculated.
He thought he could roll into Ukraine and the world would roll over.
Instead he met a wall of strength he never imagined.
He met the Ukrainian people.
From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.
Groups of citizens blocking tanks with their bodies.
previous
Markdown Text Splitter
next
Python Code Text Splitter
By Harrison Chase


previous
Markdown Text Splitter
next
Python Code Text Splitter
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Markdown Text Splitter
Markdown Text Splitter#
MarkdownTextSplitter splits text along Markdown headings, code blocks, or horizontal rules. It’s implemented as a simple subclass of RecursiveCharacterSplitter with Markdown-specific separators. See the source code to see the Markdown syntax expected by default.
How the text is split: by list of markdown specific characters
How the chunk size is measured: by length function passed in (defaults to number of characters)
from langchain.text_splitter import MarkdownTextSplitter
markdown_text = """
# 🦜🔗 LangChain
⚡ Building applications with LLMs through composability ⚡
## Quick Install
```bash
# Hopefully this code block isn't split
pip install langchain
```
As an open source project in a rapidly developing field, we are extremely open to contributions.
"""
markdown_splitter = MarkdownTextSplitter(chunk_size=100, chunk_overlap=0)
docs = markdown_splitter.create_documents([markdown_text])
docs
[Document(page_content='# 🦜🔗 LangChain\n\n⚡ Building applications with LLMs through composability ⚡', lookup_str='', metadata={}, lookup_index=0),
 Document(page_content="Quick Install\n\n```bash\n# Hopefully this code block isn't split\npip install langchain", lookup_str='', metadata={}, lookup_index=0),
 Document(page_content='As an open source project in a rapidly developing field, we are extremely open to contributions.', lookup_str='', metadata={}, lookup_index=0)]
previous
Latex Text Splitter
next
NLTK Text Splitter
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
tiktoken (OpenAI) Length Function
tiktoken (OpenAI) Length Function#
You can also use tiktoken, an open source tokenizer package from OpenAI to estimate tokens used. Will probably be more accurate for their models.
How the text is split: by character passed in
How the chunk size is measured: by tiktoken tokenizer
# This is a long document we can split up.
with open('../../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()
from langchain.text_splitter import CharacterTextSplitter
text_splitter = CharacterTextSplitter.from_tiktoken_encoder(chunk_size=100, chunk_overlap=0)
texts = text_splitter.split_text(state_of_the_union)
print(texts[0])
Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.
Last year COVID-19 kept us apart. This year we are finally together again.
Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans.
With a duty to one another to the American people to the Constitution.
previous
Spacy Text Splitter
next
TiktokenText Splitter
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
TiktokenText Splitter
TiktokenText Splitter#
How the text is split: by tiktoken tokens
How the chunk size is measured: by tiktoken tokens
# This is a long document we can split up.
with open('../../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()
from langchain.text_splitter import TokenTextSplitter
text_splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=0)
texts = text_splitter.split_text(state_of_the_union)
print(texts[0])
Madam Speaker, Madam Vice President, our
previous
tiktoken (OpenAI) Length Function
next
Vectorstores
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Spacy Text Splitter
Spacy Text Splitter#
Another alternative to NLTK is to use Spacy.
How the text is split: by Spacy
How the chunk size is measured: by length function passed in (defaults to number of characters)
# This is a long document we can split up.
with open('../../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()
from langchain.text_splitter import SpacyTextSplitter
text_splitter = SpacyTextSplitter(chunk_size=1000)
texts = text_splitter.split_text(state_of_the_union)
print(texts[0])
Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.
Members of Congress and the Cabinet.
Justices of the Supreme Court.
My fellow Americans.
Last year COVID-19 kept us apart.
This year we are finally together again.
Tonight, we meet as Democrats Republicans and Independents.
But most importantly as Americans.
With a duty to one another to the American people to the Constitution.
And with an unwavering resolve that freedom will always triumph over tyranny.
Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.
But he badly miscalculated.
He thought he could roll into Ukraine and the world would roll over.
Instead he met a wall of strength he never imagined.
He met the Ukrainian people.
From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.
previous
RecursiveCharacterTextSplitter
next
tiktoken (OpenAI) Length Function
By Harrison Chase

      © Copyright 2023, Harrison Chase.


By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Character Text Splitter
Character Text Splitter#
This is a more simple method. This splits based on characters (by default “\n\n”) and measure chunk length by number of characters.
How the text is split: by single character
How the chunk size is measured: by length function passed in (defaults to number of characters)
# This is a long document we can split up.
with open('../../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()
from langchain.text_splitter import CharacterTextSplitter
text_splitter = CharacterTextSplitter(
    separator = "\n\n",
    chunk_size = 1000,
    chunk_overlap  = 200,
    length_function = len,
)
texts = text_splitter.create_documents([state_of_the_union])
print(texts[0])


texts = text_splitter.create_documents([state_of_the_union])
print(texts[0])
page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n\nLast year COVID-19 kept us apart. This year we are finally together again. \n\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n\nWith a duty to one another to the American people to the Constitution. \n\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \n\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n\nHe met the Ukrainian people. \n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.' lookup_str='' metadata={} lookup_index=0
Here’s an example of passing metadata along with the documents, notice that it is split along with the documents.
metadatas = [{"document": 1}, {"document": 2}]
documents = text_splitter.create_documents([state_of_the_union, state_of_the_union], metadatas=metadatas)
print(documents[0])


print(documents[0])
page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n\nLast year COVID-19 kept us apart. This year we are finally together again. \n\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n\nWith a duty to one another to the American people to the Constitution. \n\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \n\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n\nHe met the Ukrainian people. \n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.' lookup_str='' metadata={'document': 1} lookup_index=0
previous
Getting Started
next
Hugging Face Length Function
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Hugging Face Length Function
Hugging Face Length Function#
Most LLMs are constrained by the number of tokens that you can pass in, which is not the same as the number of characters. In order to get a more accurate estimate, we can use Hugging Face tokenizers to count the text length.
How the text is split: by character passed in
How the chunk size is measured: by Hugging Face tokenizer
from transformers import GPT2TokenizerFast
tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
# This is a long document we can split up.
with open('../../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()
from langchain.text_splitter import CharacterTextSplitter
text_splitter = CharacterTextSplitter.from_huggingface_tokenizer(tokenizer, chunk_size=100, chunk_overlap=0)
texts = text_splitter.split_text(state_of_the_union)
print(texts[0])
Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.
Last year COVID-19 kept us apart. This year we are finally together again.
Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans.
With a duty to one another to the American people to the Constitution.
previous
Character Text Splitter
next
Latex Text Splitter
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
HTML
 Contents
Loading HTML with BeautifulSoup4
HTML#
This covers how to load HTML documents into a document format that we can use downstream.
from langchain.document_loaders import UnstructuredHTMLLoader
loader = UnstructuredHTMLLoader("example_data/fake-content.html")
data = loader.load()
data
[Document(page_content='My First Heading\n\nMy first paragraph.', lookup_str='', metadata={'source': 'example_data/fake-content.html'}, lookup_index=0)]
Loading HTML with BeautifulSoup4#
We can also use BeautifulSoup4 to load HTML documents using the BSHTMLLoader.  This will extract the text from the html into page_content, and the page title as title into metadata.
from langchain.document_loaders import BSHTMLLoader
loader = BSHTMLLoader("example_data/fake-content.html")
data = loader.load()
data
[Document(page_content='\n\nTest Title\n\n\nMy First Heading\nMy first paragraph.\n\n\n', lookup_str='', metadata={'source': 'example_data/fake-content.html', 'title': 'Test Title'}, lookup_index=0)]
previous
Hacker News
next
HuggingFace dataset loader
 Contents

Loading HTML with BeautifulSoup4
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Copy Paste
 Contents
Metadata
Copy Paste#
This notebook covers how to load a document object from something you just want to copy and paste. In this case, you don’t even need to use a DocumentLoader, but rather can just construct the Document directly.
from langchain.docstore.document import Document
text = "..... put the text you copy pasted here......"
doc = Document(page_content=text)
Metadata#
If you want to add metadata about the where you got this piece of text, you easily can with the metadata key.
metadata = {"source": "internet", "date": "Friday"}
doc = Document(page_content=text, metadata=metadata)
previous
Confluence
next
CSV Loader
 Contents

Metadata
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Spreedly
Spreedly#
This notebook covers how to load data from the Spreedly REST API into a format that can be ingested into LangChain, along with example usage for vectorization.
Note: this notebook assumes the following packages are installed: openai, chromadb, and tiktoken.
import os
from langchain.document_loaders import SpreedlyLoader
from langchain.indexes import VectorstoreIndexCreator
Spreedly API requires an access token, which can be found inside the Spreedly Admin Console.
This document loader does not currently support pagination, nor access to more complex objects which require additional parameters. It also requires a resource option which defines what objects you want to load.
Following resources are available:
gateways_options: Documentation
gateways: Documentation
receivers_options: Documentation
receivers: Documentation
payment_methods: Documentation
certificates: Documentation
transactions: Documentation
environments: Documentation
spreedly_loader = SpreedlyLoader(os.environ["SPREEDLY_ACCESS_TOKEN"], "gateways_options")
# Create a vectorstore retriver from the loader
# see https://python.langchain.com/en/latest/modules/indexes/getting_started.html for more details
index = VectorstoreIndexCreator().from_loaders([spreedly_loader])
spreedly_doc_retriever = index.vectorstore.as_retriever()
Using embedded DuckDB without persistence: data will be transient
# Test the retriever
spreedly_doc_retriever.get_relevant_documents("CRC")


I\nVN\nVU\nWF\nWS\nYE\nYT\nZA\nZM\nsupported_cardtypes:


visa\nmaster\namerican_express\ndiscover\njcb\nmaestro\nelo\nnaranja\ncabal\nunionpay\nregions: asia_pacific\neurope\nmiddle_east\nnorth_america\nhomepage: http://worldpay.com\ndisplay_api_url: https://secure.worldpay.com/jsp/merchant/xml/paymentService.jsp\ncompany_name: WorldPay', metadata={'source': 'https://core.spreedly.com/v1/gateways_options.json'}),


https://api.cybersource.com\ncompany_name: CyberSource REST', metadata={'source': 'https://core.spreedly.com/v1/gateways_options.json'})]


previous
Slack (Local Exported Zipfile)
next
Subtitle Files
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
AZLyrics
AZLyrics#
This covers how to load AZLyrics webpages into a document format that we can use downstream.
from langchain.document_loaders import AZLyricsLoader
loader = AZLyricsLoader("https://www.azlyrics.com/lyrics/mileycyrus/flowers.html")
data = loader.load()
data


[Document(page_content="Miley Cyrus - Flowers Lyrics | AZLyrics.com\n\r\nWe were good, we were gold\nKinda dream that can't be sold\nWe were right till we weren't\nBuilt a home and watched it burn\n\nI didn't wanna leave you\nI didn't wanna lie\nStarted to cry but then remembered I\n\nI can buy myself flowers\nWrite my name in the sand\nTalk to myself for hours\nSay things you don't understand\nI can take myself dancing\nAnd I can hold my own hand\nYeah, I can love me better than you can\n\nCan love me better\nI can love me better, baby\nCan love me better\nI can love me better, baby\n\nPaint my nails, cherry red\nMatch the roses that you left\nNo remorse, no regret\nI forgive every word you said\n\nI didn't wanna leave you, baby\nI didn't wanna fight\nStarted to cry but then remembered I\n\nI can buy myself flowers\nWrite my name in the sand\nTalk to myself for hours, yeah\nSay things you don't


to myself for hours, yeah\nSay things you don't understand\nI can take myself dancing\nAnd I can hold my own hand\nYeah, I can love me better than you can\n\nCan love me better\nI can love me better, baby\nCan love me better\nI can love me better, baby\nCan love me better\nI can love me better, baby\nCan love me better\nI\n\nI didn't wanna wanna leave you\nI didn't wanna fight\nStarted to cry but then remembered I\n\nI can buy myself flowers\nWrite my name in the sand\nTalk to myself for hours (Yeah)\nSay things you don't understand\nI can take myself dancing\nAnd I can hold my own hand\nYeah, I can love me better than\nYeah, I can love me better than you can, uh\n\nCan love me better\nI can love me better, baby\nCan love me better\nI can love me better, baby (Than you can)\nCan love me better\nI can love me better, baby\nCan love me better\nI\n", lookup_str='', metadata={'source':


love me better\nI\n", lookup_str='', metadata={'source': 'https://www.azlyrics.com/lyrics/mileycyrus/flowers.html'}, lookup_index=0)]


previous
Arxiv
next
Azure Blob Storage Container
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Airbyte JSON
Airbyte JSON#
This covers how to load any source from Airbyte into a local JSON file that can be read in as a document
Prereqs:
Have docker desktop installed
Steps:
Clone Airbyte from GitHub - git clone https://github.com/airbytehq/airbyte.git
Switch into Airbyte directory - cd airbyte
Start Airbyte - docker compose up
In your browser, just visit http://localhost:8000. You will be asked for a username and password. By default, that’s username airbyte and password password.
Setup any source you wish.
Set destination as Local JSON, with specified destination path - lets say /json_data. Set up manual sync.
Run the connection!
To see what files are create, you can navigate to: file:///tmp/airbyte_local
Find your data and copy path. That path should be saved in the file variable below. It should start with /tmp/airbyte_local
from langchain.document_loaders import AirbyteJSONLoader
!ls /tmp/airbyte_local/json_data/
_airbyte_raw_pokemon.jsonl
loader = AirbyteJSONLoader('/tmp/airbyte_local/json_data/_airbyte_raw_pokemon.jsonl')
data = loader.load()
print(data[0].page_content[:500])
abilities:
ability:
name: blaze
url: https://pokeapi.co/api/v2/ability/66/
is_hidden: False
slot: 1
ability:
name: solar-power
url: https://pokeapi.co/api/v2/ability/94/
is_hidden: True
slot: 3
base_experience: 267
forms:
name: charizard
url: https://pokeapi.co/api/v2/pokemon-form/6/
game_indices:


game_indices:
game_index: 180
version:
name: red
url: https://pokeapi.co/api/v2/version/1/
game_index: 180
version:
name: blue
url: https://pokeapi.co/api/v2/version/2/
game_index: 180
version:
n
previous
CoNLL-U
next
Apify Dataset
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Azure Blob Storage Container
 Contents
Specifying a prefix
Azure Blob Storage Container#
This covers how to load document objects from a container on Azure Blob Storage.
from langchain.document_loaders import AzureBlobStorageContainerLoader
#!pip install azure-storage-blob
loader = AzureBlobStorageContainerLoader(conn_str="<conn_str>", container="<container>")
loader.load()
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpaa9xl6ch/fake.docx'}, lookup_index=0)]
Specifying a prefix#
You can also specify a prefix for more finegrained control over what files to load.
loader = AzureBlobStorageContainerLoader(conn_str="<conn_str>", container="<container>", prefix="<prefix>")
loader.load()
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpujbkzf_l/fake.docx'}, lookup_index=0)]
previous
AZLyrics
next
Azure Blob Storage File
 Contents

Specifying a prefix
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Arxiv
 Contents
Installation
Examples
Arxiv#
arXiv is an open-access archive for 2 million scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.
This notebook shows how to load scientific articles from Arxiv.org into a document format that we can use downstream.
Installation#
First, you need to install arxiv python package.
!pip install arxiv
Second, you need to install PyMuPDF python package which transform PDF files from the arxiv.org site into the text format.
!pip install pymupdf
Examples#
ArxivLoader has these arguments:
query: free text which used to find documents in the Arxiv
optional load_max_docs: default=100. Use it to limit number of downloaded documents. It takes time to download all 100 documents, so use a small number for experiments.
optional load_all_available_meta: default=False. By default only the most important fields downloaded: Published (date when document was published/last updated), Title, Authors, Summary. If True, other fields also downloaded.
from langchain.document_loaders.base import Document
from langchain.document_loaders import ArxivLoader
docs = ArxivLoader(query="1605.08386", load_max_docs=2).load()
len(docs)
docs[0].metadata  # meta-information of the Document
{'Published': '2016-05-26',
 'Title': 'Heat-bath random walks with Markov bases',
 'Authors': 'Caprice Stanley, Tobias Windisch',


'Authors': 'Caprice Stanley, Tobias Windisch',
 'Summary': 'Graphs on lattice points are studied whose edges come from a finite set of\nallowed moves of arbitrary length. We show that the diameter of these graphs on\nfibers of a fixed integer matrix can be bounded from above by a constant. We\nthen study the mixing behaviour of heat-bath random walks on these graphs. We\nalso state explicit conditions on the set of moves so that the heat-bath random\nwalk, a generalization of the Glauber dynamics, is an expander in fixed\ndimension.'}
docs[0].page_content[:400]  # all pages of the Document content
'arXiv:1605.08386v1  [math.CO]  26 May 2016\nHEAT-BATH RANDOM WALKS WITH MARKOV BASES\nCAPRICE STANLEY AND TOBIAS WINDISCH\nAbstract. Graphs on lattice points are studied whose edges come from a ﬁnite set of\nallowed moves of arbitrary length. We show that the diameter of these graphs on ﬁbers of a\nﬁxed integer matrix can be bounded from above by a constant. We then study the mixing\nbehaviour of heat-b'
previous
Apify Dataset
next
AZLyrics
 Contents

Installation
Examples
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Confluence
Confluence#
A loader for Confluence pages.
This currently supports both username/api_key and Oauth2 login.
Specify a list page_ids and/or space_key to load in the corresponding pages into Document objects, if both are specified the union of both sets will be returned.
You can also specify a boolean include_attachments to include attachments, this is set to False by default, if set to True all attachments will be downloaded and ConfluenceReader will extract the text from the attachments and add it to the Document object. Currently supported attachment types are: PDF, PNG, JPEG/JPG, SVG, Word and Excel.
Hint: space_key and page_id can both be found in the URL of a page in Confluence - https://yoursite.atlassian.com/wiki/spaces/<space_key>/pages/<page_id>
from langchain.document_loaders import ConfluenceLoader
loader = ConfluenceLoader(
    url="https://yoursite.atlassian.com/wiki",
    username="me",
    api_key="12345"
)
documents = loader.load(space_key="SPACE", include_attachments=True, limit=50)
previous
College Confidential
next
Copy Paste
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
Apify Dataset
 Contents
Prerequisites
An example with question answering
Apify Dataset#
This notebook shows how to load Apify datasets to LangChain.
Apify Dataset is a scaleable append-only storage with sequential access built for storing structured web scraping results, such as a list of products or Google SERPs, and then export them to various formats like JSON, CSV, or Excel. Datasets are mainly used to save results of Apify Actors—serverless cloud programs for varius web scraping, crawling, and data extraction use cases.
Prerequisites#
You need to have an existing dataset on the Apify platform. If you don’t have one, please first check out this notebook on how to use Apify to extract content from documentation, knowledge bases, help centers, or blogs.
First, import ApifyDatasetLoader into your source code:
from langchain.document_loaders import ApifyDatasetLoader
from langchain.document_loaders.base import Document
Then provide a function that maps Apify dataset record fields to LangChain Document format.
For example, if your dataset items are structured like this:
{
    "url": "https://apify.com",
    "text": "Apify is the best web scraping and automation platform."
}
The mapping function in the code below will convert them to LangChain Document format, so that you can use them further with any LLM model (e.g. for question answering).
loader = ApifyDatasetLoader(
    dataset_id="your-dataset-id",
    dataset_mapping_function=lambda dataset_item: Document(
        page_content=dataset_item["text"], metadata={"source": dataset_item["url"]}
    ),
)
data = loader.load()
An example with question answering#
In this example, we use data from a dataset to answer a question.
from langchain.docstore.document import Document


from langchain.docstore.document import Document
from langchain.document_loaders import ApifyDatasetLoader
from langchain.indexes import VectorstoreIndexCreator
loader = ApifyDatasetLoader(
    dataset_id="your-dataset-id",
    dataset_mapping_function=lambda item: Document(
        page_content=item["text"] or "", metadata={"source": item["url"]}
    ),
)
index = VectorstoreIndexCreator().from_loaders([loader])
query = "What is Apify?"
result = index.query_with_sources(query)
print(result["answer"])
print(result["sources"])
 Apify is a platform for developing, running, and sharing serverless cloud programs. It enables users to create web scraping and automation tools and publish them on the Apify platform.
https://docs.apify.com/platform/actors, https://docs.apify.com/platform/actors/running/actors-in-store, https://docs.apify.com/platform/security, https://docs.apify.com/platform/actors/examples
previous
Airbyte JSON
next
Arxiv
 Contents

Prerequisites
An example with question answering
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
HuggingFace dataset loader
 Contents
Example
HuggingFace dataset loader#
This notebook shows how to load Hugging Face Hub datasets to LangChain.
The Hugging Face Hub hosts a large number of community-curated datasets for a diverse range of tasks such as translation, automatic speech recognition, and image classification.
from langchain.document_loaders import HuggingFaceDatasetLoader
dataset_name="imdb"
page_content_column="text"
loader=HuggingFaceDatasetLoader(dataset_name,page_content_column)
data = loader.load()
data[:15]


data = loader.load()
data[:15]
[Document(page_content='I rented I AM CURIOUS-YELLOW from my video store because of all the controversy that surrounded it when it was first released in 1967. I also heard that at first it was seized by U.S. customs if it ever tried to enter this country, therefore being a fan of films considered "controversial" I really had to see this for myself.

The plot is centered around a young Swedish drama student named Lena who wants to learn everything she can about life. In particular she wants to focus her attentions to making some sort of documentary on what the average Swede thought about certain political issues such as the Vietnam War and race issues in the United States. In between asking politicians and ordinary denizens of Stockholm about their opinions on politics, she has sex with her drama teacher, classmates, and married men.

What kills me about I AM CURIOUS-YELLOW is that 40 years ago, this was considered pornographic. Really, the sex and nudity scenes are few and far between, even then it\'s not shot like some cheaply made porno. While my countrymen mind find it shocking, in reality sex and nudity are a major staple in Swedish cinema. Even Ingmar Bergman, arguably their answer to good old boy John Ford, had sex scenes in his films.

I do commend the filmmakers for the fact that any sex shown in the film is shown for artistic purposes rather than just to shock people and make money to be shown in pornographic theaters in America. I AM CURIOUS-YELLOW is a good film for anyone wanting to study the meat and potatoes (no pun intended) of Swedish cinema. But really, this film doesn\'t have much of a plot.', metadata={'label': 0}),


Document(page_content='"I Am Curious: Yellow" is a risible and pretentious steaming pile. It doesn\'t matter what one\'s political views are because this film can hardly be taken seriously on any level. As for the claim that frontal male nudity is an automatic NC-17, that isn\'t true. I\'ve seen R-rated films with male nudity. Granted, they only offer some fleeting views, but where are the R-rated films with gaping vulvas and flapping labia? Nowhere, because they don\'t exist. The same goes for those crappy cable shows: schlongs swinging in the breeze but not a clitoris in sight. And those pretentious indie movies like The Brown Bunny, in which we\'re treated to the site of Vincent Gallo\'s throbbing johnson, but not a trace of pink visible on Chloe Sevigny. Before crying (or implying) "double-standard" in matters of nudity, the mentally obtuse should take into account one unavoidably obvious anatomical difference between men and women: there are no genitals on display when actresses appears nude, and the same cannot be said for a man. In fact, you generally won\'t see female genitals in an American film in anything short of porn or explicit erotica. This alleged double-standard is less a double standard than an admittedly depressing ability to come to terms culturally with the insides of women\'s bodies.', metadata={'label': 0}),


Document(page_content="If only to avoid making this type of film in the future. This film is interesting as an experiment but tells no cogent story.

One might feel virtuous for sitting thru it because it touches on so many IMPORTANT issues but it does so without any discernable motive. The viewer comes away with no new perspectives (unless one comes up with one while one's mind wanders, as it will invariably do during this pointless film).

One might better spend one's time staring out a window at a tree growing.

", metadata={'label': 0}),
 Document(page_content="This film was probably inspired by Godard's Masculin, féminin and I urge you to see that film instead.

The film has two strong elements and those are, (1) the realistic acting (2) the impressive, undeservedly good, photo. Apart from that, what strikes me most is the endless stream of silliness. Lena Nyman has to be most annoying actress in the world. She acts so stupid and with all the nudity in this film,...it's unattractive. Comparing to Godard's film, intellectuality has been replaced with stupidity. Without going too far on this subject, I would say that follows from the difference in ideals between the French and the Swedish society.

A movie of its time, and place. 2/10.", metadata={'label': 0}),


Document(page_content='Oh, brother...after hearing about this ridiculous film for umpteen years all I can think of is that old Peggy Lee song..

"Is that all there is??" ...I was just an early teen when this smoked fish hit the U.S. I was too young to get in the theater (although I did manage to sneak into "Goodbye Columbus"). Then a screening at a local film museum beckoned - Finally I could see this film, except now I was as old as my parents were when they schlepped to see it!!

The ONLY reason this film was not condemned to the anonymous sands of time was because of the obscenity case sparked by its U.S. release. MILLIONS of people flocked to this stinker, thinking they were going to see a sex film...Instead, they got lots of closeups of gnarly, repulsive Swedes, on-street interviews in bland shopping malls, asinie political pretension...and feeble who-cares simulated sex scenes with


pretension...and feeble who-cares simulated sex scenes with saggy, pale actors.

Cultural icon, holy grail, historic artifact..whatever this thing was, shred it, burn it, then stuff the ashes in a lead box!

Elite esthetes still scrape to find value in its boring pseudo revolutionary political spewings..But if it weren\'t for the censorship scandal, it would have been ignored, then forgotten.

Instead, the "I Am Blank, Blank" rhythymed title was repeated endlessly for years as a titilation for porno films (I am Curious, Lavender - for gay films, I Am Curious, Black - for blaxploitation films, etc..) and every ten years or so the thing rises from the dead, to be viewed by a new generation of suckers who want to see that "naughty sex film" that "revolutionized the film industry"...

Yeesh, avoid like the plague..Or if you MUST see it - rent the video and fast forward to the


it - rent the video and fast forward to the "dirty" parts, just to get it over with.

', metadata={'label': 0}),


Document(page_content="I would put this at the top of my list of films in the category of unwatchable trash! There are films that are bad, but the worst kind are the ones that are unwatchable but you are suppose to like them because they are supposed to be good for you! The sex sequences, so shocking in its day, couldn't even arouse a rabbit. The so called controversial politics is strictly high school sophomore amateur night Marxism. The film is self-consciously arty in the worst sense of the term. The photography is in a harsh grainy black and white. Some scenes are out of focus or taken from the wrong angle. Even the sound is bad! And some people call this art?

", metadata={'label': 0}),
 Document(page_content="Whoever wrote the screenplay for this movie obviously never consulted any books about Lucille Ball, especially her autobiography. I've never seen so many mistakes in a biopic, ranging from her early years in Celoron and Jamestown to her later years with Desi. I could write a whole list of factual errors, but it would go on for pages. In all, I believe that Lucille Ball is one of those inimitable people who simply cannot be portrayed by anyone other than themselves. If I were Lucie Arnaz and Desi, Jr., I would be irate at how many mistakes were made in this film. The filmmakers tried hard, but the movie seems awfully sloppy to me.", metadata={'label': 0}),


Document(page_content='When I first saw a glimpse of this movie, I quickly noticed the actress who was playing the role of Lucille Ball. Rachel York\'s portrayal of Lucy is absolutely awful. Lucille Ball was an astounding comedian with incredible talent. To think about a legend like Lucille Ball being portrayed the way she was in the movie is horrendous. I cannot believe out of all the actresses in the world who could play a much better Lucy, the producers decided to get Rachel York. She might be a good actress in other roles but to play the role of Lucille Ball is tough. It is pretty hard to find someone who could resemble Lucille Ball, but they could at least find someone a bit similar in looks and talent. If you noticed York\'s portrayal of Lucy in episodes of I Love Lucy like the chocolate factory or vitavetavegamin, nothing is similar in any way-her expression, voice, or movement.

To top it all off, Danny Pino playing Desi Arnaz is horrible. Pino does not qualify to play as Ricky. He\'s small and skinny, his accent is unreal, and once again, his acting is unbelievable. Although Fred and Ethel were not similar either, they were not as bad as the characters of Lucy and Ricky.

Overall, extremely horrible casting and the story is badly told. If people want to understand the real life situation of Lucille Ball, I suggest watching A&E Biography of Lucy and Desi, read the book from Lucille Ball herself, or PBS\' American Masters: Finding Lucy. If you want to see a docudrama, "Before the Laughter" would be a better choice. The casting of Lucille Ball and Desi Arnaz in "Before the Laughter" is much better compared to this. At least, a similar aspect is shown rather than nothing.', metadata={'label': 0}),


Document(page_content='Who are these "They"- the actors? the filmmakers? Certainly couldn\'t be the audience- this is among the most air-puffed productions in existence. It\'s the kind of movie that looks like it was a lot of fun to shoot\x97 TOO much fun, nobody is getting any actual work done, and that almost always makes for a movie that\'s no fun to watch.

Ritter dons glasses so as to hammer home his character\'s status as a sort of doppleganger of the bespectacled Bogdanovich; the scenes with the breezy Ms. Stratten are sweet, but have an embarrassing, look-guys-I\'m-dating-the-prom-queen feel to them. Ben Gazzara sports his usual cat\'s-got-canary grin in a futile attempt to elevate the meager plot, which requires him to pursue Audrey Hepburn with all the interest of a narcoleptic at an insomnia clinic. In the meantime, the budding couple\'s respective children (nepotism alert: Bogdanovich\'s


respective children (nepotism alert: Bogdanovich\'s daughters) spew cute and pick up some fairly disturbing pointers on \'love\' while observing their parents. (Ms. Hepburn, drawing on her dignity, manages to rise above the proceedings- but she has the monumental challenge of playing herself, ostensibly.) Everybody looks great, but so what? It\'s a movie and we can expect that much, if that\'s what you\'re looking for you\'d be better off picking up a copy of Vogue.

Oh- and it has to be mentioned that Colleen Camp thoroughly annoys, even apart from her singing, which, while competent, is wholly unconvincing... the country and western numbers are woefully mismatched with the standards on the soundtrack. Surely this is NOT what Gershwin (who wrote the song from which the movie\'s title is derived) had in mind; his stage musicals of the 20\'s may have been slight, but at least they were long on charm. "They All


but at least they were long on charm. "They All Laughed" tries to coast on its good intentions, but nobody- least of all Peter Bogdanovich - has the good sense to put on the brakes.

Due in no small part to the tragic death of Dorothy Stratten, this movie has a special place in the heart of Mr. Bogdanovich- he even bought it back from its producers, then distributed it on his own and went bankrupt when it didn\'t prove popular. His rise and fall is among the more sympathetic and tragic of Hollywood stories, so there\'s no joy in criticizing the film... there _is_ real emotional investment in Ms. Stratten\'s scenes. But "Laughed" is a faint echo of "The Last Picture Show", "Paper Moon" or "What\'s Up, Doc"- following "Daisy Miller" and "At Long Last Love", it was a thundering confirmation of the phase from which P.B. has never emerged.

All in all, though, the movie is harmless,


in all, though, the movie is harmless, only a waste of rental. I want to watch people having a good time, I\'ll go to the park on a sunny day. For filmic expressions of joy and love, I\'ll stick to Ernest Lubitsch and Jaques Demy...', metadata={'label': 0}),


Document(page_content="This is said to be a personal film for Peter Bogdonavitch. He based it on his life but changed things around to fit the characters, who are detectives. These detectives date beautiful models and have no problem getting them. Sounds more like a millionaire playboy filmmaker than a detective, doesn't it? This entire movie was written by Peter, and it shows how out of touch with real people he was. You're supposed to write what you know, and he did that, indeed. And leaves the audience bored and confused, and jealous, for that matter. This is a curio for people who want to see Dorothy Stratten, who was murdered right after filming. But Patti Hanson, who would, in real life, marry Keith Richards, was also a model, like Stratten, but is a lot better and has a more ample part. In fact, Stratten's part seemed forced; added. She doesn't have a lot to do with the story, which is pretty convoluted to begin with. All in all, every character in this film is somebody that very few people can relate with, unless you're millionaire from Manhattan with beautiful supermodels at your beckon call. For the rest of us, it's an irritating snore fest. That's what happens when you're out of touch. You entertain your few friends with inside jokes, and bore all the rest.", metadata={'label': 0}),


Document(page_content='It was great to see some of my favorite stars of 30 years ago including John Ritter, Ben Gazarra and Audrey Hepburn. They looked quite wonderful. But that was it. They were not given any characters or good lines to work with. I neither understood or cared what the characters were doing.

Some of the smaller female roles were fine, Patty Henson and Colleen Camp were quite competent and confident in their small sidekick parts. They showed some talent and it is sad they didn\'t go on to star in more and better films. Sadly, I didn\'t think Dorothy Stratten got a chance to act in this her only important film role.

The film appears to have some fans, and I was very open-minded when I started watching it. I am a big Peter Bogdanovich fan and I enjoyed his last movie, "Cat\'s Meow" and all his early ones from "Targets" to "Nickleodeon". So, it really surprised me that I was barely able to keep awake watching this one.

It is ironic that this movie is about a detective agency where the detectives and clients get romantically involved with each other. Five years later, Bogdanovich\'s ex-girlfriend, Cybil Shepherd had a hit television series called "Moonlighting" stealing the story idea from Bogdanovich. Of course, there was a great difference in that the series relied on tons of witty dialogue, while this tries to make do with slapstick and a few screwball lines.

Bottom line: It ain\'t no "Paper Moon" and only a very pale version of "What\'s Up, Doc".', metadata={'label': 0}),


Document(page_content="I can't believe that those praising this movie herein aren't thinking of some other film. I was prepared for the possibility that this would be awful, but the script (or lack thereof) makes for a film that's also pointless. On the plus side, the general level of craft on the part of the actors and technical crew is quite competent, but when you've got a sow's ear to work with you can't make a silk purse. Ben G fans should stick with just about any other movie he's been in. Dorothy S fans should stick to Galaxina. Peter B fans should stick to Last Picture Show and Target. Fans of cheap laughs at the expense of those who seem to be asking for it should stick to Peter B's amazingly awful book, Killing of the Unicorn.", metadata={'label': 0}),
 Document(page_content='Never cast models and Playboy bunnies in your films! Bob Fosse\'s "Star 80" about Dorothy Stratten, of whom Bogdanovich was obsessed enough to have married her SISTER after her murder at the hands of her low-life husband, is a zillion times more interesting than Dorothy herself on the silver screen. Patty Hansen is no actress either..I expected to see some sort of lost masterpiece a la Orson Welles but instead got Audrey Hepburn cavorting in jeans and a god-awful "poodlesque" hair-do....Very disappointing...."Paper Moon" and "The Last Picture Show" I could watch again and again. This clunker I could barely sit through once. This movie was reputedly not released because of the brouhaha surrounding Ms. Stratten\'s tawdry death; I think the real reason was because it was so bad!', metadata={'label': 0}),


Document(page_content="Its not the cast. A finer group of actors, you could not find. Its not the setting. The director is in love with New York City, and by the end of the film, so are we all! Woody Allen could not improve upon what Bogdonovich has done here. If you are going to fall in love, or find love, Manhattan is the place to go. No, the problem with the movie is the script. There is none. The actors fall in love at first sight, words are unnecessary. In the director's own experience in Hollywood that is what happens when they go to work on the set. It is reality to him, and his peers, but it is a fantasy to most of us in the real world. So, in the end, the movie is hollow, and shallow, and message-less.", metadata={'label': 0}),


Document(page_content='Today I found "They All Laughed" on VHS on sale in a rental. It was a really old and very used VHS, I had no information about this movie, but I liked the references listed on its cover: the names of Peter Bogdanovich, Audrey Hepburn, John Ritter and specially Dorothy Stratten attracted me, the price was very low and I decided to risk and buy it. I searched IMDb, and the User Rating of 6.0 was an excellent reference. I looked in "Mick Martin & Marsha Porter Video & DVD Guide 2003" and \x96 wow \x96 four stars! So, I decided that I could not waste more time and immediately see it. Indeed, I have just finished watching "They All Laughed" and I found it a very boring overrated movie. The characters are badly developed, and I spent lots of minutes to understand their roles in the story. The plot is supposed to be funny (private eyes who fall in love for the women they are


eyes who fall in love for the women they are chasing), but I have not laughed along the whole story. The coincidences, in a huge city like New York, are ridiculous. Ben Gazarra as an attractive and very seductive man, with the women falling for him as if her were a Brad Pitt, Antonio Banderas or George Clooney, is quite ridiculous. In the end, the greater attractions certainly are the presence of the Playboy centerfold and playmate of the year Dorothy Stratten, murdered by her husband pretty after the release of this movie, and whose life was showed in "Star 80" and "Death of a Centerfold: The Dorothy Stratten Story"; the amazing beauty of the sexy Patti Hansen, the future Mrs. Keith Richards; the always wonderful, even being fifty-two years old, Audrey Hepburn; and the song "Amigo", from Roberto Carlos. Although I do not like him, Roberto Carlos has been the most popular Brazilian singer since the end of the


most popular Brazilian singer since the end of the 60\'s and is called by his fans as "The King". I will keep this movie in my collection only because of these attractions (manly Dorothy Stratten). My vote is four.

Title (Brazil): "Muito Riso e Muita Alegria" ("Many Laughs and Lots of Happiness")', metadata={'label': 0})]


Example#
In this example, we use data from a dataset to answer a question
from langchain.indexes import VectorstoreIndexCreator
from langchain.document_loaders.hugging_face_dataset import HuggingFaceDatasetLoader
dataset_name="tweet_eval"
page_content_column="text"
name="stance_climate"
loader=HuggingFaceDatasetLoader(dataset_name,page_content_column,name)
index = VectorstoreIndexCreator().from_loaders([loader])
Found cached dataset tweet_eval
Using embedded DuckDB without persistence: data will be transient
query = "What are the most used hashtag?"
result = index.query(query)
result
' The most used hashtags in this context are #UKClimate2015, #Sustainability, #TakeDownTheFlag, #LoveWins, #CSOTA, #ClimateSummitoftheAmericas, #SM, and #SocialMedia.'
previous
HTML
next
iFixit
 Contents

Example
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
DataFrame Loader
DataFrame Loader#
This notebook goes over how to load data from a pandas dataframe
import pandas as pd
df = pd.read_csv('example_data/mlb_teams_2012.csv')
df.head()
Team
"Payroll (millions)"
"Wins"
0
Nationals
81.34
98
1
Reds
82.20
97
2
Yankees
197.96
95
3
Giants
117.62
94
4
Braves
83.31
94
from langchain.document_loaders import DataFrameLoader
loader = DataFrameLoader(df, page_content_column="Team")
loader.load()
[Document(page_content='Nationals', metadata={' "Payroll (millions)"': 81.34, ' "Wins"': 98}),
 Document(page_content='Reds', metadata={' "Payroll (millions)"': 82.2, ' "Wins"': 97}),
 Document(page_content='Yankees', metadata={' "Payroll (millions)"': 197.96, ' "Wins"': 95}),
 Document(page_content='Giants', metadata={' "Payroll (millions)"': 117.62, ' "Wins"': 94}),
 Document(page_content='Braves', metadata={' "Payroll (millions)"': 83.31, ' "Wins"': 94}),
 Document(page_content='Athletics', metadata={' "Payroll (millions)"': 55.37, ' "Wins"': 94}),
 Document(page_content='Rangers', metadata={' "Payroll (millions)"': 120.51, ' "Wins"': 93}),
 Document(page_content='Orioles', metadata={' "Payroll (millions)"': 81.43, ' "Wins"': 93}),


Document(page_content='Rays', metadata={' "Payroll (millions)"': 64.17, ' "Wins"': 90}),
 Document(page_content='Angels', metadata={' "Payroll (millions)"': 154.49, ' "Wins"': 89}),
 Document(page_content='Tigers', metadata={' "Payroll (millions)"': 132.3, ' "Wins"': 88}),
 Document(page_content='Cardinals', metadata={' "Payroll (millions)"': 110.3, ' "Wins"': 88}),
 Document(page_content='Dodgers', metadata={' "Payroll (millions)"': 95.14, ' "Wins"': 86}),
 Document(page_content='White Sox', metadata={' "Payroll (millions)"': 96.92, ' "Wins"': 85}),
 Document(page_content='Brewers', metadata={' "Payroll (millions)"': 97.65, ' "Wins"': 83}),
 Document(page_content='Phillies', metadata={' "Payroll (millions)"': 174.54, ' "Wins"': 81}),
 Document(page_content='Diamondbacks', metadata={' "Payroll (millions)"': 74.28, ' "Wins"': 81}),
 Document(page_content='Pirates', metadata={' "Payroll (millions)"': 63.43, ' "Wins"': 79}),
 Document(page_content='Padres', metadata={' "Payroll (millions)"': 55.24, ' "Wins"': 76}),
 Document(page_content='Mariners', metadata={' "Payroll (millions)"': 81.97, ' "Wins"': 75}),


Document(page_content='Mets', metadata={' "Payroll (millions)"': 93.35, ' "Wins"': 74}),
 Document(page_content='Blue Jays', metadata={' "Payroll (millions)"': 75.48, ' "Wins"': 73}),
 Document(page_content='Royals', metadata={' "Payroll (millions)"': 60.91, ' "Wins"': 72}),
 Document(page_content='Marlins', metadata={' "Payroll (millions)"': 118.07, ' "Wins"': 69}),
 Document(page_content='Red Sox', metadata={' "Payroll (millions)"': 173.18, ' "Wins"': 69}),
 Document(page_content='Indians', metadata={' "Payroll (millions)"': 78.43, ' "Wins"': 68}),
 Document(page_content='Twins', metadata={' "Payroll (millions)"': 94.08, ' "Wins"': 66}),
 Document(page_content='Rockies', metadata={' "Payroll (millions)"': 78.06, ' "Wins"': 64}),
 Document(page_content='Cubs', metadata={' "Payroll (millions)"': 88.19, ' "Wins"': 61}),
 Document(page_content='Astros', metadata={' "Payroll (millions)"': 60.65, ' "Wins"': 55})]
previous
CSV Loader
next
Diffbot
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
CoNLL-U
CoNLL-U#
This is an example of how to load a file in CoNLL-U format. The whole file is treated as one document. The example data (conllu.conllu) is based on one of the standard UD/CoNLL-U examples.
from langchain.document_loaders import CoNLLULoader
loader = CoNLLULoader("example_data/conllu.conllu")
document = loader.load()
document
previous
Document Loaders
next
Airbyte JSON
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on May 02, 2023.


.ipynb
.pdf
IMSDb
IMSDb#
This covers how to load IMSDb webpages into a document format that we can use downstream.
from langchain.document_loaders import IMSDbLoader
loader = IMSDbLoader("https://imsdb.com/scripts/BlacKkKlansman.html")
data = loader.load()
data


[Document(page_content='\n\r\n\r\n\r\n\r\n                                    BLACKKKLANSMAN\r\n                         \r\n                         \r\n                         \r\n                         \r\n                                      Written by\r\n\r\n                          Charlie Wachtel & David Rabinowitz\r\n\r\n                                         and\r\n\r\n                              Kevin Willmott & Spike Lee\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n                         FADE IN:\r\n


FADE IN:\r\n                         \r\n          SCENE FROM "GONE WITH THE WIND"\r\n                         \r\n          Scarlett O\'Hara, played by Vivian Leigh, walks through the\r\n          Thousands of injured Confederate Soldiers pulling back to\r\n          reveal the Famous Shot of the tattered Confederate Flag in\r\n          "Gone with the Wind" as The Max Stein Music Score swells from\r\n          Dixie to Taps.\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR (O.S.)\r\n                       They say they may have lost the\r\n


Battle but they didn\'t lose The War.\r\n                       Yes, Friends, We are under attack.\r\n                         \r\n                                                                          CUT TO:\r\n                         \r\n          A 1960\'S EDUCATIONAL STYLE FILM\r\n                         \r\n          Shot on Grainy COLOR 16MM EKTACHROME Film, The NARRATOR\r\n          BEAUREGARD, a Middle Aged but handsome, White Male, sits at a\r\n          desk, a Confederate Flag on a stand beside him. Very\r\n          Official. He


him. Very\r\n          Official. He is not a Southerner and speaks with articulation\r\n          and intelligence.\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR\r\n                       You\'ve read about it in your Local\r\n                       Newspapers or seen it on The Evening\r\n                       News. That\'s right. We\'re living in\r\n                       an Era marked by the spread of\r\n                       Integration and Miscegenation.\r\n                         \r\n


CUT TO:\r\n                         \r\n          FOOTAGE OF THE LITTLE ROCK NINE\r\n                         \r\n          being escorted into CENTRAL HIGH SCHOOL, Little Rock,\r\n          Arkansas by The National Guard.\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR\r\n                                   (V.O.)(CONT\'D)\r\n                       The Brown Decision forced upon us by\r\n                       The Jewish controlled Puppets on the\r\n


Jewish controlled Puppets on the\r\n                       U.S. Supreme Court compelling White\r\n                       children to go to School with an\r\n                       Inferior Race is The Final Nail in a\r\n                       Black Coffin towards America becoming\r\n                       a Mongrel Nation.\r\n                         \r\n          A QUICK SERIES OF IMAGES\r\n                         \r\n          Segregation Signs. Antebellum Photos. Happy Slaves in Old\r\n          Movies. Masters inspecting their Cotton and Tobacco with\r\n          their Slaves in The Fields. Blacks shining Shoes and working\r\n          as


Shoes and working\r\n          as Butlers, Porters and Maids.\r\n                                   BEAUREGARD- KLAN NARRATOR (V.O.)\r\n                                                  (CONT\'D)\r\n                       We had a great way of Life before The\r\n                       Martin Luther Coon\'s of The World...\r\n                         \r\n                                                                          CUT TO:\r\n                         \r\n          The Billboard of Dr. Martin Luther King Jr. sitting in


of Dr. Martin Luther King Jr. sitting in the\r\n          front row of a Classroom it reads: Martin Luther King in a\r\n          Communist Training School.\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR (CONT\'D)\r\n                       ...and their Army of Commies started\r\n                       their Civil Rights Assault on our\r\n                       Holy White Protestant Values.\r\n                         \r\n          CLOSE - BOUREGARD - KLAN NARRATOR\r\n                         \r\n                                   BEAUREGARD- KLAN


BEAUREGARD- KLAN NARRATOR (CONT\'D)\r\n                       Do you really want your precious\r\n                       White Child going to School with\r\n                       Negroes?\r\n                         \r\n          Footage of Black and White Children playing together,\r\n          innocent.\r\n                         \r\n          Beauregard now stands by a Large Screen and points at The\r\n          Screen.\r\n                         \r\n                                   BEAUREGARD-KLAN NARRATOR (CONT\'D)\r\n                       They are Lying, Dirty


They are Lying, Dirty Monkeys...\r\n                         \r\n          FOOTAGE and STILLS of Stereotype Blacks Coons, Bucks and\r\n          shining Black Mammies. Black Soldiers in D. W. Griffith\'s\r\n          "Birth of a Nation" pushing Whites around on the Street.\r\n                         \r\n          CLOSE - BEAUREGARD\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR (CONT\'D)\r\n                       ...Stopping at nothing to gain\r\n                       Equality with The White Man.\r\n                         \r\n


\r\n          Images and Scientific charts of Blacks compared to Apes and\r\n          Monkeys.\r\n                         \r\n          CLOSE - BEAUREGARD - KLAN NARRATOR\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR (CONT\'D)\r\n                       ...Rapists, Murderers...Craving The\r\n                       Virgin, Pure Flesh of White Women.\r\n                       They are Super Predators...\r\n                                                                             CUT TO:\r\n


CUT TO:\r\n                         \r\n          LYNCH, The MULATTO, lusting after our LILLIAN GISH in "Birth\r\n          of a Nation." Other Lusting Images of Craving Black\r\n          Beasts!!! SEXUAL PREDATORS!!!\r\n                         \r\n                                                                             CUT TO:\r\n                         \r\n          KING KONG on Empire State Building with Fay Wray in his hand.\r\n          GUS in "Birth of a Nation" chasing a White Woman he wants to\r\n          Rape.\r\n


\r\n                                                                             CUT TO:\r\n                         \r\n          CLOSE - BEAUREGARD - KLAN NARRATOR\r\n                         \r\n          A Stereotype illustration of Jews controlling Negroes.\r\n                         \r\n                                   BEAUREGARD- KLAN NARRATOR (CONT\'D)\r\n                       ...and the Negro\'s insidious tactics\r\n                       under the tutelage of High Ranking\r\n


of High Ranking\r\n                       Blood Sucking Jews! Using an Army of\r\n                       outside...\r\n                         \r\n          Beauregard continues.\r\n                         \r\n                                                                             CUT TO:\r\n                         \r\n                                   BEAUREGARD-KLAN NARRATOR(CONT\'D)\r\n                       ...Northern Black Beast Agitators...\r\n                         \r\n          Footage of The March on


Footage of The March on Washington.\r\n                         \r\n                                                                             CUT TO:\r\n                         \r\n          CLOSE - BOUREGARD - KLAN NARRATOR.\r\n                         \r\n                                   BOUREGARD- KLAN NARRATOR (CONT\'D)\r\n                       ...determined to overthrow The God\r\n                       Commanded and Biblically inspired\r\n                       Rule of The White Race.\r\n


\r\n                                                                             CUT TO:\r\n                         \r\n          An image of an All-American White Nuclear Family.\r\n                         \r\n                                                                             CUT TO:\r\n                         \r\n          Bouregard gives his Final Words.\r\n                         \r\n                                   BOUREGARD-KLAN


BOUREGARD-KLAN NARRATOR (CONT\'D)\r\n                       It\'s an International... Jewish...\r\n                       Conspiracy.\r\n          WE HEAR and end with the Corny Stinger of Music that goes\r\n          with these Education and Propaganda Films!\r\n                         \r\n                                                                             CUT TO:\r\n                         \r\n          EXT. COLORADO SPRINGS AREA - DAY\r\n                         \r\n                         DRONE SHOT\r\n


SHOT\r\n                         \r\n          Superimposed: Early 70s\r\n                         \r\n          An amazing contrast. The beautiful landscape of Colorado\r\n          Springs, the City sits nestled within the rugged Mountain\r\n          terrain. The majestic Pikes Peak, the jagged beauty of The\r\n          Garden of the Gods, The plush Broadmoor Resort, The Will\r\n          Rodgers Shrine of The Sun.\r\n                         \r\n                         \r\n          EXT. COLORADO SPRINGS STREET - DAY\r\n                         \r\n          RON STALLWORTH, Black, 21, Handsome, Intelligent, sporting a\r\n          good


sporting a\r\n          good sized Afro, rebellious but straight laced by most 1970\'s\r\n          standards.\r\n                         \r\n          Ron stares at an Ad attached to a bulletin board.\r\n                         \r\n          CLOSE - THE AD READS:\r\n                         \r\n          JOIN THE COLORADO SPRINGS POLICE FORCE, MINORITIES ENCOURAGED\r\n          TO APPLY! Ron rips the Ad from the board.\r\n                         \r\n          EXT. COLORADO SPRINGS POLICE DEPT BUILDING. - DAY\r\n                         \r\n          INT. OFFICE OF CHIEF BRIDGES - COLORADO SPRINGS POLICE DEPT -\r\n


- COLORADO SPRINGS POLICE DEPT -\r\n                         DAY\r\n                         \r\n          A drab, white-walled office. Ron sits across the table from\r\n          The Assistant City Personnel Manager, MR. TURRENTINE, Black,\r\n          40\'s, business like but progressive and CHIEF BRIDGES, White,\r\n          smart, 50\'s, in a Police Uniform, a Man ready for change.\r\n                         \r\n                                    MR. TURRENTINE\r\n                       Why weren\'t you drafted into the\r\n                       Vietnam War?\r\n                         \r\n


\r\n                                                  RON STALLWORTH\r\n                       I went to College.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       How do you feel about Vietnam?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       I have mixed feelings.\r\n                                     CHIEF BRIDGES\r\n          Would you call yourself a Womanizer?\r\n


Would you call yourself a Womanizer?\r\n                                                  RON STALLWORTH\r\n                       No Sir, I would not.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       Do you frequent Night Clubs?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       No Sir.\r\n                         \r\n


CHIEF BRIDGES\r\n                       Do you drink?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       On Special occasions, Sir.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       Have you ever done any Drugs?\r\n                         \r\n                                                  RON STALLWORTH\r\n


Only those prescribed by My Doctor,\r\n                       Sir.\r\n                         \r\n          Turrentine looks at Chief Bridges.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       That\'s kind of rare these days for a\r\n                       young Hip Soul Brother like you.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       I know but my Father was in The\r\n


but my Father was in The\r\n                       Military and I was raised up the\r\n                       Right way, Sir.\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       How are you with people, generally?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Sir, they treat me right, I treat\r\n                       them right, like I already said I was\r\n


raised...\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       ...Have you ever had any negative...\r\n                         \r\n          Mr. Turrentine jumps in, impatient.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       ...What would you do if another Cop\r\n                       called you a Nigger?\r\n                         \r\n


RON STALLWORTH\r\n                       Would that happen...\r\n                         \r\n                                   MR. TURRENTINE\r\n                       ...Sheeeeeeettt!!!\r\n          Bridges looks at him. Turrentine waits, Ron doesn\'t know how\r\n          to respond, finally. Turrentine leans forward.\r\n                         \r\n                                   MR. TURRENTINE (CONT\'D)\r\n                       There\'s never been a Black Cop in\r\n                       this City. If we


this City. If we make you an Officer,\r\n                       you would, in effect, be the Jackie\r\n                       Robinson of the Colorado Springs\r\n                       Police force.\r\n                         \r\n          Mr. Turrentine lets this sink in.\r\n                         \r\n                                   MR. TURRENTINE (CONT\'D)\r\n                       And if you know anything about Jackie\r\n                       Robinson you know he had to take a\r\n                       lot of... guff... from his fellow\r\n


his fellow\r\n                       Teammates, from Fans, other Teams,\r\n                       and The Press.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       I know Jackie\'s Story, Sir.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       Good. So, knowing that, when someone\r\n                       calls you Nigger will you be able to\r\n                       turn the other Cheek?\r\n


\r\n          Ron evaluates the hard reality of the question. Decides.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       If I need to, yes, Sir.\r\n                         \r\n                                   MR. TURRENTINE\r\n                       Son, The Mayor and I think you might\r\n                       be The Man to open things up here.\r\n                         \r\n          Ron looks at Chief Bridges.\r\n


Bridges.\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       I\'ll have your back but I can only do\r\n                       so much. The Weight of this is on\r\n                       You...and You alone.\r\n                         \r\n          Ron weighs The Journey ahead.\r\n                         \r\n                         OMITTED\r\n                         \r\n                         OMITTED\r\n


\r\n          INT. RECORDS ROOM - CSPD - DAY\r\n                         \r\n          Ron sorts a file cabinet of records as OFFICER CLAY MULANEY,\r\n          60\'s, White, sits on a stool, reading a Magazine clearly\r\n          looking at a Photo of something good.\r\n          Ron looks at the Photo of the Actress Cybill Shepherd.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Cybill Shepherd. She was great in The\r\n                       Last Picture Show.\r\n


\r\n                                                  OFFICER MULANEY\r\n                       Never saw it but what you think?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       She\'s a very good Actress.\r\n                         \r\n                                                  OFFICER MULANEY\r\n                       Y\'know you want some of that.\r\n


\r\n          Ron ignores it.\r\n                         \r\n                                   OFFICER MULANEY (CONT\'D)\r\n                       Truth be told when I see one of your\r\n                       kind with a White Woman it turns my\r\n                       Stomach.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Yeah. Why\'s that?\r\n                         \r\n


OFFICER MULANEY\r\n                       He could only want one thing.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       What would that be?\r\n                         \r\n                                                  OFFICER MULANEY\r\n                       You like acting dumb, Y\'know.\r\n                         \r\n


RON STALLWORTH\r\n                       No, I just like my questions to be\r\n                       answered.\r\n                         \r\n          A VOICE of UNIFORMED COP WHEATON calls from the other side of\r\n          the Counter.\r\n                         \r\n                                   WHEATON (O.S.)\r\n                       Hey! Anybody in there? Looking for a\r\n                       Toad here.\r\n                         \r\n          Ron walks to the Counter to see The White and


Ron walks to the Counter to see The White and sleep-deprived\r\n          Cop impatiently leaning on his elbows.\r\n                         \r\n                                                  WHEATON (CONT\'D)\r\n                       Get me the record for this Toad named\r\n                       Tippy Birdsong.\r\n                         \r\n          Ron pulls up the File for Tippy Birdsong. The Photo shows a\r\n          Black Man in his twenties.\r\n                                                  WHEATON (CONT\'D)\r\n                       While you\'re at it, why


While you\'re at it, why don\'t you\r\n                       grab another Toad... Steven Wilson.\r\n                         \r\n          Ron pulls the File... another young Black Male, ANOTHER\r\n          SEXUAL PREDATOR!\r\n                         \r\n          INT. CSPD HALLWAY - DAY\r\n                         \r\n          Chief Bridges strides down the hall with SGT. TRAPP a soft-\r\n          spoken White Man in his 40\'s, they are discussing a File. Ron\r\n          suddenly appears walking with them.\r\n                         \r\n


RON STALLWORTH\r\n                       While I\'ve got you both here. Sirs,\r\n                       I\'d like to be an Undercover\r\n                       Detective.\r\n                         \r\n          Chief Bridges and Sgt. Trapp both stop.\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       What Narcotics?\r\n                         \r\n                                                  RON


RON STALLWORTH\r\n                       Whatever Department works, Sir.\r\n                         \r\n                                   SGT. TRAPP\r\n                       You just joined The Force, Rookie.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       I know, Sir but I think I could do\r\n                       some good there.\r\n                         \r\n                                   SGT. TRAPP\r\n


SGT. TRAPP\r\n                       Is that right?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Well, I\'m young. I think there\'s a\r\n                       niche for me. Get In where I can Fit\r\n                       In.\r\n                         \r\n                                   SGT. TRAPP\r\n                       What do you think, Chief?\r\n                         \r\n          Sgt. Trapp sees


Sgt. Trapp sees the logic, looks to Chief Bridges, who stops,\r\n          considering.\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       Think a lot of yourself, don\'t cha?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Just trying to be of help, Chief.\r\n                       Plus, I hate working in The Records\r\n                       room.\r\n          Sgt. Trapp reacts knowing Ron


Sgt. Trapp reacts knowing Ron shouldn\'t have said that about\r\n          the Records Room. CHIEF BRIDGES looks at Ron, matter of fact.\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       Well, I think Records is a good place\r\n                       for you to start, Rookie.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Chief, want me clean shaven?\r\n                         \r\n


\r\n                                                  CHIEF BRIDGES\r\n                       Keep it. I like the look.\r\n                         \r\n          Chief Bridges walks off without another word. SGT. TRAPP\r\n          gives a knowing look to Ron, who watches them walk away.\r\n                         \r\n          INT. RECORDS ROOM - CSPD - DAY\r\n                         \r\n          Ron behind the Counter. MASTER PATROLMAN ANDY LANDERS, White,\r\n          Mid-30\'s, a regular guy but there is something dangerous\r\n          there, steps up.\r\n


\r\n                                                  LANDERS\r\n                       Need a File on a Toad.\r\n                         \r\n          Ron doesn\'t respond.\r\n                         \r\n                                                  LANDERS (CONT\'D)\r\n                       You Deaf? I said I need info on a\r\n                       Toad.\r\n                         \r\n


RON STALLWORTH\r\n                       No Toads here.\r\n                         \r\n                                                   LANDERS\r\n                       Excuse me?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       I said, I don\'t have any Toads. I do\r\n                       have Human Beings and if you give me\r\n                       their names I can pull the Files.\r\n


\r\n          Landers scowls. Ron stares back at him, Eye to Eye.\r\n                         \r\n                                                  LANDERS\r\n                       Heard you think you Hot Shit but you\r\n                       ain\'t nuthin\' but a Cold Fart. Name\'s\r\n                       Maurice, Maurice Smalls...That\r\n                       respectful enough for you, Officer\r\n                       Toad.\r\n                         \r\n          Ron pulls The File, throws it down on the Counter as


The File, throws it down on the Counter as Landers\r\n          snatches The File and storms off.\r\n          INT. RON\'S APARTMENT - BEDROOM - MORNING\r\n                         \r\n          As Ron sleeps, a phone rings. Ron snaps awake and grabs at\r\n          the phone on the night table.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Hello.\r\n                                    CHIEF BRIDGES (O.S.)\r\n                        It\'s Bridges. You sleeping?\r\n


\r\n                                                    RON STALLWORTH\r\n                        Yes, Chief, I was. Just worked a\r\n                        Night Shift.\r\n                         \r\n                                    CHIEF BRIDGES (O.S.)\r\n                        I changed my mind, you\'re gonna come\r\n                        in a little earlier today. We\'ve got\r\n                        an assignment for you. 12 Noon.\r\n                        Sharp. Narcotics Division. Wear\r\n


Sharp. Narcotics Division. Wear\r\n                        Street clothes.\r\n                         \r\n                                                   RON STALLWORTH\r\n                        Yes Sir, see you then. Thank You.\r\n                        Thank You.\r\n                         \r\n          Ron sits up in Bed, excited, thinking about the challenge\r\n          ahead.\r\n                         \r\n          INT. CSPD - NARCOTICS DIVISION - DAY\r\n                         \r\n          Ron, dressed in Bell-Bottoms and a Hip Italian Knit


dressed in Bell-Bottoms and a Hip Italian Knit Shirt,\r\n          Marshmallow Shoes steps inside the Narcotics office, which is\r\n          literally The Basement of The Station. He looks around at The\r\n          Area Buzzing with Activity and sees\r\n                         \r\n          ANGLE - UNDERCOVER COPS\r\n                         \r\n          at their desks. Looking less like Cops and more like unkempt\r\n          Hippies or Rock N\' Rollers.\r\n                         \r\n          CLOSE - RON\r\n                         \r\n          just stands there looking at all the activity.\r\n                         \r\n          CLOSE -


\r\n          CLOSE - CHIEF BRIDGES\r\n                         \r\n          waves Ron back to the rear of The Room for privacy.\r\n                         \r\n          CLOSE - FLIP ZIMMERMAN\r\n                         \r\n                                                   FLIP\r\n                        Rookie, you\'re late.\r\n                         \r\n                                                   RON STALLWORTH\r\n                        Sorry, it won\'t happen again.\r\n


again.\r\n                         \r\n          Flip, late 30\'s, long hair, looks like anything but a Cop, he\r\n          however is somewhat of a closed-off guy, all business, Ron\r\n          sits across from him. Chief Bridges steps before them.\r\n                                   CHIEF BRIDGES (CONT\'D)\r\n                       We\'ve got limited time so I\'ll be\r\n                       quick. That Black Radical Stokely\r\n                       Carmichael is giving a Speech Tonight\r\n                       at Bell\'s Nightingale.\r\n                         \r\n          Ron is surprised at this.\r\n


Ron is surprised at this.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       The Nightclub?\r\n                         \r\n                                                  CHIEF BRIDGES\r\n                       No, Emmanuel Missionary Baptist\r\n                       Church!!!\r\n                         \r\n          Flip just listens.\r\n                         \r\n                                   CHIEF BRIDGES


CHIEF BRIDGES (CONT\'D)\r\n                       Carmichael is a former High Muckity-\r\n                       Muck with The Black Panthers and as\r\n                       far as I\'m concerned, FBI Director J.\r\n                       Edgar Hoover was dead right when he\r\n                       said The Black Panthers are The\r\n                       Greatest Internal Threat to The\r\n                       Security of these United States. This\r\n                       Carmichael Joker, former Panther or\r\n                       not, they say he\'s a Damn Good\r\n


Good\r\n                       Speaker and we don\'t want this\r\n                       Carmichael getting into The Minds of\r\n                       the Black People here in Colorado\r\n                       Springs and stirring them up.\r\n                         \r\n          Ron\'s face cringes at Chief Bridges\'s words. He steps to Ron.\r\n                         \r\n                                   CHIEF BRIDGES (CONT\'D)\r\n                       Ron, your assignment is to go to this\r\n                       Speech tonight and infiltrate these\r\n


Bunch of Subversives and monitor The\r\n                       Audience reaction to Carmichael. You\r\n                       ready?\r\n                         \r\n          Flip and Chief Bridges stare at Ron.\r\n                         \r\n                                                     RON STALLWORTH\r\n                       Born Ready.\r\n                         \r\n          INT. NARCOTICS DIVISION - CSPD - NIGHT\r\n                         \r\n          Ron stands, his shirt off, as Flip wires a Wireless\r\n


off, as Flip wires a Wireless\r\n          Transmitter and Microphone to his body. Another Narcotics\r\n          Cop, JIMMY CREEK, 30\'s, observes the installation.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Any chance this thing Fucks Up?\r\n                                                    FLIP\r\n                       Fuck yeah.\r\n                         \r\n                                                    RON STALLWORTH\r\n


Then what?\r\n                         \r\n                                                  JIMMY\r\n                       Just stick to The Game Plan.\r\n                         \r\n                                                    RON STALLWORTH\r\n                       Which is?\r\n                         \r\n                                                  FLIP\r\n                       Improvise. Like Jazz. This isn\'t some\r\n


some\r\n                       Big Bust. We just want some Intel,\r\n                       that\'s it.\r\n                         \r\n                                                  JIMMY\r\n                       What happens if someone offers you a\r\n                       Marijuana Cigarette?\r\n                         \r\n                                                  RON STALLWORTH\r\n                       You mean a Joint?\r\n                         \r\n


JIMMY\r\n                       Yeah.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       "Soul Brother, I\'m already High on\r\n                       Life. Can you Dig It?"\r\n                         \r\n                                                  FLIP\r\n                       And if someone pulls a Gun on you?\r\n                         \r\n          Ron


\r\n          Ron is caught off guard.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       You expecting that?\r\n                         \r\n          Flip pulls his Gun.\r\n                         \r\n                                                  FLIP\r\n                       Barrel of a 45\'s in your face, Finger\r\n                       on the Trigger, now what?\r\n                         \r\n


\r\n                                                  RON STALLWORTH\r\n                       Blood, get that Gun out my face.\r\n                       Peace Love and Soul.\r\n                         \r\n                                                  FLIP\r\n                       Gun is still in your face.\r\n                         \r\n          Ron gives Jimmy a wary look speaking to Flip.\r\n                                                  RON STALLWORTH\r\n                       I de-escalate.


I de-escalate. Talk calmly, firmly.\r\n                       Find a way out of there, A-Sap.\r\n                         \r\n          Jimmy nods, satisfied. Flip is finished with The Wiring. Ron\r\n          takes a deep breath.\r\n                         \r\n                                                  FLIP\r\n                       Relax, we\'ll be outside, listening\r\n                       in.\r\n                         \r\n                                                  RON STALLWORTH\r\n


RON STALLWORTH\r\n                       Can I order a Drink at The Bar?\r\n                         \r\n          Flip steps away, no comment.\r\n                         \r\n                                                  JIMMY\r\n                       That\'s fine, just don\'t get Shit\r\n                       Faced.\r\n                         \r\n                                                  FLIP\r\n                       Got it?\r\n                         \r\n


\r\n                                                  RON STALLWORTH\r\n                       I got it. I\'m gone.\r\n                         \r\n          Jimmy laughs, Slaps Ron on the back.\r\n                         \r\n          EXT. CITY STREET - OUTSKIRTS OF DOWNTOWN - NIGHT\r\n                         \r\n          Ron pulls an unmarked Sedan to the curb. He gets out and\r\n          looks around.\r\n                         \r\n          A Crowded sidewalk overflows into The Street, filling a line\r\n          that Bottlenecks into The Club with


that Bottlenecks into The Club with the Sign: \r\n                         \r\n          CLOSE SIGN - BELL\'S NIGHTINGALE\r\n                         \r\n          ANGLE - TONIGHT: KWAME TURE SPEAKS\r\n                         \r\n          Ron walks to the back of the line. He becomes an Every\r\n          Brother slowly moving forward as People enter. As he moves\r\n          forward he notices a striking Woman at the Front Door.\r\n                         \r\n          ANGLE - PATRICE DUMAS\r\n                         \r\n          Mid 20\'s, an Angela Davis Afro, she wears a Hip array of\r\n


a Hip array of\r\n          Militant wear, Black Leather Jacket, Love Beads but on her it\r\n          looks fantastic. Ron is taken by her Beauty, he watches as\r\n          she monitors the door, clearly in charge.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       How are you doing, my Soul Sista?\r\n                         \r\n          Patrice gives Ron a good look summing him up.\r\n                                                  PATRICE\r\n                       I\'m doing fine, my Brother. This is\r\n


I\'m doing fine, my Brother. This is\r\n                       going to be an Amazing Night.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Indeed it is.\r\n                         \r\n                                                  PATRICE\r\n                       Have you heard Brother Kwame speak\r\n                       before?\r\n                         \r\n


RON STALLWORTH\r\n                       Who?\r\n                         \r\n                                                    PATRICE\r\n                       Kwame Ture.\r\n                         \r\n                                                  RON STALLWORTH\r\n                       Actually, I haven\'t, I didn\'t know he\r\n                       changed his name.\r\n                         \r\n