o ȳg@sbddlZddlmZmZmZmZmZmZddlm Z ddl m Z m Z e eZGddde ZdS)N)AnyCallableDictIteratorListOptional)Document) BaseModelmodel_validatorc@seZdZUdZdZeed<dZeed<dZe ed<dZ e e e gefed <eed <ed d ed e defddZdeefddZdeefddZdS)TensorflowDatasetsavAccess to the TensorFlow Datasets. The Current implementation can work only with datasets that fit in a memory. `TensorFlow Datasets` is a collection of datasets ready to use, with TensorFlow or other Python ML frameworks, such as Jax. All datasets are exposed as `tf.data.Datasets`. To get started see the Guide: https://www.tensorflow.org/datasets/overview and the list of datasets: https://www.tensorflow.org/datasets/catalog/ overview#all_datasets You have to provide the sample_to_document_function: a function that a sample from the dataset-specific format to the Document. Attributes: dataset_name: the name of the dataset to load split_name: the name of the split to load. Defaults to "train". load_max_docs: a limit to the number of loaded documents. Defaults to 100. sample_to_document_function: a function that converts a dataset sample to a Document Example: .. code-block:: python from langchain_community.utilities import TensorflowDatasets def mlqaen_example_to_document(example: dict) -> Document: return Document( page_content=decode_to_str(example["context"]), metadata={ "id": decode_to_str(example["id"]), "title": decode_to_str(example["title"]), "question": decode_to_str(example["question"]), "answer": decode_to_str(example["answers"]["text"][0]), }, ) tsds_client = TensorflowDatasets( dataset_name="mlqa/en", split_name="train", load_max_docs=MAX_DOCS, sample_to_document_function=mlqaen_example_to_document, ) dataset_nametrain split_named load_max_docsNsample_to_document_functiondatasetbefore)modevaluesreturncCszzddl}Wn tytdwzddl}Wn ty#tdw|ddur.td|j|d|dd |d <|S) z7Validate that the python package exists in environment.rNz\Could not import tensorflow python package. Please install it with `pip install tensorflow`.znCould not import tensorflow_datasets python package. Please install it with `pip install tensorflow-datasets`.rzmsample_to_document_function is None. Please provide a function that converts a dataset sample to a Document.r r)splitr) tensorflow ImportErrortensorflow_datasets ValueErrorload)clsrrrrm/var/www/html/chatdoc2/venv/lib/python3.10/site-packages/langchain_community/utilities/tensorflow_datasets.pyvalidate_environment?s,       z'TensorflowDatasets.validate_environmentcsfddjjDS)zYDownload a selected dataset lazily. Returns: an iterator of Documents. c3s$|] }jdur|VqdS)N)r).0sselfrr cs z/TensorflowDatasets.lazy_load..)rtakerr$rr$r lazy_load]s  zTensorflowDatasets.lazy_loadcCs t|S)zMDownload a selected dataset. Returns: a list of Documents. )listr(r$rrr ris zTensorflowDatasets.load)__name__ __module__ __qualname____doc__r str__annotations__rrintrrrrrrr classmethodr!rr(rrrrrr r s  .   r )loggingtypingrrrrrrlangchain_core.documentsrpydanticr r getLoggerr*loggerr rrrr s