公司做网站需要服务器吗,辰景青岛网站建设,解释自己做的网站,网红网站建设LlamaIndex 是一个数据框架#xff0c;供 LLM 应用程序摄取、构建和访问私有或特定领域的数据。 LlamaIndex 是开源的#xff0c;可用于构建各种应用程序。 在 GitHub 上查看该项目。 安装 在 Docker 上设置 Elasticsearch 使用以下 docker 命令启动单节点 Elasticsearch 实…LlamaIndex 是一个数据框架供 LLM 应用程序摄取、构建和访问私有或特定领域的数据。 LlamaIndex 是开源的可用于构建各种应用程序。 在 GitHub 上查看该项目。 安装 在 Docker 上设置 Elasticsearch 使用以下 docker 命令启动单节点 Elasticsearch 实例。我们可以参考之前的文章 “Elasticsearch如何在 Docker 上运行 Elasticsearch 8.x 进行本地开发”。我选择不使用安全配置。直接使用 docker compose 来启动 Elasticsearch 及 Kibana .env $ pwd /Users/liuxg/data/docker8 $ ls -al total 16 drwxr-xr-x 4 liuxg staff 128 Jan 16 13:00 . drwxr-xr-x 193 liuxg staff 6176 Jan 12 08:31 .. -rw-r--r-- 1 liuxg staff 21 Jan 16 13:00 .env -rw-r--r-- 1 liuxg staff 733 Mar 14 2023 docker-compose.yml $ cat .env STACK_VERSION8.11.3 docker-compose.yml version: 3.9 services:elasticsearch:image: elasticsearch:${STACK_VERSION}container_name: elasticsearchenvironment:- discovery.typesingle-node- ES_JAVA_OPTS-Xms1g -Xmx1g- xpack.security.enabledfalsevolumes:- type: volumesource: es_datatarget: /usr/share/elasticsearch/dataports:- target: 9200published: 9200networks:- elastickibana:image: kibana:${STACK_VERSION}container_name: kibanaports:- target: 5601published: 5601depends_on:- elasticsearchnetworks:- elastic volumes:es_data:driver: localnetworks:elastic:name: elasticdriver: bridge 我们使用如下的命令来启动 docker-compose up ​ 这样我们就完成了 Elasticsearch 及 Kibana 的安装了。我们的 Elasticsearch 及 Kibana 都没有安全的设置。这个在生产环境中不被推荐使用。 应用设计 -  组装管道 我们将使用 Jupyter notebook 来进行设计。我们在命令行中打入: jupyter notebook 安装依赖 我们使用如下的命令来安装 Python 的依赖包 pip3 install llama-index openai elasticsearch load_dotenv 我们接下来在当前的工作目录中创建一个叫做 .env 的文件 .env OPENAI_API_KEYYourOpenAIKey 请在 .env 中创建如上所示的变量。你需要把自己的 openai key 写入到上面的文件里。 加载模块及读取环境变量 import logging import sys import os from dotenv import load_dotenvload_dotenv()logging.basicConfig(streamsys.stdout, levellogging.INFO) logging.getLogger().addHandler(logging.StreamHandler(streamsys.stdout))os.environ[OPENAI_API_KEY] os.getenv(OPENAI_API_KEY)import openai 连接到 Elasticsearch ElasticsearchStore 类用于连接到 Elasticsearch 实例。 它需要以下参数 index_nameElasticsearch 索引的名称。 必需的。es_client可选。 预先存在的 Elasticsearch 客户端。es_url可选。Elasticsearch 网址。es_cloud_id可选。 Elasticsearch 云 ID。es_api_key可选。 Elasticsearch API 密钥。es_user可选。 Elasticsearch 用户名。es_password可选。 弹性搜索密码。text_field可选。 存储文本的 Elasticsearch 字段的名称。vector_field可选。 存储 Elasticsearch 字段的名称嵌入。batch_size可选。 批量索引的批量大小。 默认为 200。distance_strategy可选。 用于相似性搜索的距离策略。默认为 “COSINE”。 针对我们的情况我们可以使用如下的示例方法来进行本地连接 from llama_index.vector_stores import ElasticsearchStorees ElasticsearchStore(index_namemy_index,es_urlhttp://localhost:9200, ) 我们将在下面的代码中使用上述的方法来连接 Elasticsearch。 加载文档使用 Elasticsearch 构建 VectorStoreIndex from llama_index import VectorStoreIndex, SimpleDirectoryReader from llama_index.vector_stores import ElasticsearchStore!mkdir -p data/paul_graham/ !wget https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt -O data/paul_graham/paul_graham_essay.txt 运行完上面的命令后我们可以在当前的目录下查看 $ pwd /Users/liuxg/python/elser $ ls data/paul_graham/ paul_graham_essay.txt 我们可以看到一个叫做 pau_graham_essay.txt 的文件。它的内容如下 What I Worked OnFebruary 2021Before college the two main things I worked on, outside of school, were writing and programming. I didnt write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.The first programs I tried writing were on the IBM 1401 that our school district used for what was then called data processing. This was in 9th grade, so I was 13 or 14. The school districts 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villains lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights....The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.I was puzzled by the 1401. I couldnt figure out what to do with it. And in retrospect theres not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didnt have any data stored on punched cards. The only other option was to do things that didnt rely on any input, like calculate approximations of pi, but I didnt know enough math to do anything interesting of that type. So Im not surprised I cant remember any programs I wrote, because they cant have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didnt. On a machine without time-sharing, this was a social as well as a technical error, as the data center managers expression made clear. ...Toward the end of the summer I got a big surprise: a letter from the Accademia, which had been delayed because theyd sent it to Cambridge England instead of Cambridge Massachusetts, inviting me to take the entrance exam in Florence that fall. This was now only weeks away. My nice landlady let me leave my stuff in her attic. I had some money saved from consulting work Id done in grad school; there was probably enough to last a year if I lived cheaply. Now all I had to do was learn Italian. ... documents SimpleDirectoryReader(./data/paul_graham/).load_data()from llama_index.storage.storage_context import StorageContextvector_store ElasticsearchStore(es_urlhttp://localhost:9200,# Or with Elastic Cloud# es_cloud_idmy_cloud_id,# es_userelastic,# es_passwordmy_password,index_namepaul_graham, )storage_context StorageContext.from_defaults(vector_storevector_store) index VectorStoreIndex.from_documents(documents, storage_contextstorage_context ) 基本查询 我们将向查询引擎询问有关我们刚刚索引的数据的问题。 # set Logging to DEBUG for more detailed outputs query_engine index.as_query_engine() response query_engine.query(what were his investments in Y Combinator?) print(response) 元数据过滤器 在这里我们将使用元数据索引一些文档以便我们可以将过滤器应用于查询引擎。 from llama_index.schema import TextNodenodes [TextNode(textThe Shawshank Redemption,metadata{author: Stephen King,theme: Friendship,},),TextNode(textThe Godfather,metadata{director: Francis Ford Coppola,theme: Mafia,},),TextNode(textBeautiful weather,metadata{director: Mark shuttle,theme: Mafia,},), TextNode(textInception,metadata{director: Christopher Nolan,},), ]# initialize the vector store vector_store_metadata_example ElasticsearchStore(index_namemovies_metadata_example,es_urlhttp://localhost:9200, ) storage_context StorageContext.from_defaults(vector_storevector_store_metadata_example ) index1 VectorStoreIndex(nodes, storage_contextstorage_context)# Metadata filter from llama_index.vector_stores.types import ExactMatchFilter, MetadataFiltersfilters MetadataFilters(filters[ExactMatchFilter(keytheme, valueMafia)] )retriever index1.as_retriever(filtersfilters)retriever.retrieve(weather is so beautiful) 在上面我们搜索的是 “weather is so beautiful”从而在两个 theme 为 Mafia 的 Texnode 里Mark shuttle 位列第一。这个是因为 “weather is so beautiful” 更和 “Beautiful weather” 更为贴近。 如果我们使用如下的查询 retriever.retrieve(The godfather is a nice person) 很显然这次我们的搜索结果的排序颠倒过来了。 自定义过滤器和覆盖查询 llama-index 目前仅支持 ExactMatchFilters。 Elasticsearch 支持多种过滤器包括范围过滤器、地理过滤器等。 要使用这些过滤器你可以将它们作为字典列表传递给 es_filter 参数。 def custom_query(query, query_str):print(custom query, query)return queryquery_engine index.as_query_engine(vector_store_kwargs{es_filter: [{match: {content: growing up}}],custom_query: custom_query,} )response query_engine.query(what were his investments in Y Combinator?) print(response) 为了方便大家学习我把所有的源码放到 githubhttps://github.com/liu-xiao-guo/semantic_search_es。其中相关的文件是 https://github.com/liu-xiao-guo/semantic_search_es/blob/main/Elasticsearch%20integration%20-%20LIamaIndex%20.ipynb 更多阅读使用 Elasticsearch 和 LlamaIndex 进行高级文本检索句子窗口检索