foreign_storage::ForeignDataWrapper Class Reference

#include <ForeignDataWrapper.h>

Inheritance diagram for foreign_storage::ForeignDataWrapper:

Public Types
enum	ParallelismLevel { NONE, INTRA_FRAGMENT, INTER_FRAGMENT }

Public Member Functions
	ForeignDataWrapper ()=default
virtual	~ForeignDataWrapper ()=default
virtual void	populateChunkMetadata (ChunkMetadataVector &chunk_metadata_vector)=0
virtual void	populateChunkBuffers (const ChunkToBufferMap &required_buffers, const ChunkToBufferMap &optional_buffers, AbstractBuffer *delete_buffer=nullptr)=0
virtual std::string	getSerializedDataWrapper () const =0
virtual void	restoreDataWrapperInternals (const std::string &file_path, const ChunkMetadataVector &chunk_metadata)=0
virtual bool	isRestored () const =0
virtual void	validateServerOptions (const ForeignServer *foreign_server) const =0
virtual void	validateTableOptions (const ForeignTable *foreign_table) const =0
virtual const std::set < std::string_view > &	getSupportedTableOptions () const =0
virtual const std::set < std::string >	getAlterableTableOptions () const
virtual void	validateUserMappingOptions (const UserMapping *user_mapping, const ForeignServer *foreign_server) const =0
virtual const std::set < std::string_view > &	getSupportedUserMappingOptions () const =0
virtual void	validateSchema (const std::list< ColumnDescriptor > &columns) const
virtual ParallelismLevel	getCachedParallelismLevel () const
virtual ParallelismLevel	getNonCachedParallelismLevel () const
virtual bool	isLazyFragmentFetchingEnabled () const

Detailed Description

Definition at line 33 of file ForeignDataWrapper.h.

Member Enumeration Documentation

enum foreign_storage::ForeignDataWrapper::ParallelismLevel

ParallelismLevel describes the desired level of parallelism of the data wrapper. This level controls which optional_buffers are passed to populateChunkBuffers with the following behaviour:

NONE - no additional optional buffers are passed in

INTRA_FRAGMENT - additional optional buffers which are in the same fragment as the required buffers

INTER_FRAGMENT - additional optional buffers which may be in different fragments than those of the required buffers

Note, the optional buffers are passed in with the intention of allowing the data wrapper to employ parallelism in retrieving them. Each subsequent level allows for a greater degree of parallelism but does not have to be supported.

Enumerator
NONE
INTRA_FRAGMENT
INTER_FRAGMENT

Definition at line 136 of file ForeignDataWrapper.h.

{ NONE, INTRA_FRAGMENT, INTER_FRAGMENT };

Constructor & Destructor Documentation

foreign_storage::ForeignDataWrapper::ForeignDataWrapper ( )

default

virtual foreign_storage::ForeignDataWrapper::~ForeignDataWrapper ( )

virtualdefault

Member Function Documentation

virtual const std::set<std::string> foreign_storage::ForeignDataWrapper::getAlterableTableOptions ( ) const

inlinevirtual

Gets the subset of table options that can be altered for the data wrapper.

Reimplemented in foreign_storage::AbstractFileStorageDataWrapper.

Definition at line 99 of file ForeignDataWrapper.h.

{ return {}; };

virtual ParallelismLevel foreign_storage::ForeignDataWrapper::getCachedParallelismLevel ( ) const

inlinevirtual

Gets the desired level of parallelism for the data wrapper when a cache is in use. This affects the optional buffers that the data wrapper is made aware of during data requests.

Reimplemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, and foreign_storage::ParquetImporter.

Definition at line 143 of file ForeignDataWrapper.h.

References NONE.

{ return NONE; }

virtual ParallelismLevel foreign_storage::ForeignDataWrapper::getNonCachedParallelismLevel ( ) const

inlinevirtual

Gets the desired level of parallelism for the data wrapper when no cache is in use. This affects the optional buffers that the data wrapper is made aware of during data requests.

Reimplemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, and foreign_storage::ParquetImporter.

Definition at line 150 of file ForeignDataWrapper.h.

References NONE.

{ return NONE; }

virtual std::string foreign_storage::ForeignDataWrapper::getSerializedDataWrapper ( ) const

pure virtual

Serialize internal state of wrapper into file at given path if implemented

Implemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, foreign_storage::InternalSystemDataWrapper, and foreign_storage::ParquetImporter.

virtual const std::set<std::string_view>& foreign_storage::ForeignDataWrapper::getSupportedTableOptions ( ) const

pure virtual

Gets the set of supported table options for the data wrapper.

Implemented in foreign_storage::InternalSystemDataWrapper, foreign_storage::CsvDataWrapper, foreign_storage::RegexParserDataWrapper, and foreign_storage::AbstractFileStorageDataWrapper.

virtual const std::set<std::string_view>& foreign_storage::ForeignDataWrapper::getSupportedUserMappingOptions ( ) const

pure virtual

Gets the set of supported user mapping options for the data wrapper.

Implemented in foreign_storage::InternalSystemDataWrapper, and foreign_storage::AbstractFileStorageDataWrapper.

virtual bool foreign_storage::ForeignDataWrapper::isLazyFragmentFetchingEnabled ( ) const

inlinevirtual

If true data wrapper implements a lazy fragment fetching mode. This mode allows requests for fragments to be issued to populateChunks without the prerequisite that populateChunkMetadata has successfully finished execution. This is an optimization that has some specific use-cases and is not required.

NOTE: this mode is not guaranteed to work as expected when combined with certain types of refresh modes such as append. This is subject to change in the future, but has no impact on the intended use-cases of this mode.

Reimplemented in foreign_storage::AbstractTextFileDataWrapper.

Definition at line 163 of file ForeignDataWrapper.h.

{ return false; }

virtual bool foreign_storage::ForeignDataWrapper::isRestored ( ) const

pure virtual

Implemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, foreign_storage::InternalSystemDataWrapper, and foreign_storage::ParquetImporter.

virtual void foreign_storage::ForeignDataWrapper::populateChunkBuffers ( const ChunkToBufferMap &  required_buffers, const ChunkToBufferMap &  optional_buffers, AbstractBuffer *  delete_buffer = nullptr  )

pure virtual

Populates given chunk buffers identified by chunk keys. All provided chunk buffers are expected to be for the same fragment.

Parameters

required_buffers	- chunk buffers that must always be populated
optional_buffers	- chunk buffers that can be optionally populated, if the data wrapper has to scan through chunk data anyways (typically for row wise data formats)
delete_buffer	- chunk buffer for fragment's delete column, if non-null data wrapper is expected to mark deleted rows in buffer and continue processing

Implemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, foreign_storage::ParquetImporter, and foreign_storage::InternalSystemDataWrapper.

Referenced by anonymous_namespace{ForeignDataImporter.cpp}::import_foreign_data(), and foreign_storage::CachingForeignStorageMgr::populateChunkBuffersSafely().

Here is the caller graph for this function:

virtual void foreign_storage::ForeignDataWrapper::populateChunkMetadata ( ChunkMetadataVector &  chunk_metadata_vector )

pure virtual

Populates given chunk metadata vector with metadata for all chunks in related foreign table.

Parameters

chunk_metadata_vector

- vector that will be populated with chunk metadata

Implemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, foreign_storage::ParquetImporter, and foreign_storage::InternalSystemDataWrapper.

Referenced by anonymous_namespace{ForeignDataImporter.cpp}::metadata_scan().

Here is the caller graph for this function:

virtual void foreign_storage::ForeignDataWrapper::restoreDataWrapperInternals ( const std::string &  file_path, const ChunkMetadataVector &  chunk_metadata  )

pure virtual

Restore internal state of datawrapper

Parameters

file_path	- location of file created by serializeMetadata
chunk_metadata_vector	- vector of chunk metadata recovered from disk

Implemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, foreign_storage::InternalSystemDataWrapper, and foreign_storage::ParquetImporter.

virtual void foreign_storage::ForeignDataWrapper::validateSchema ( const std::list< ColumnDescriptor > &  columns ) const

inlinevirtual

Verifies the schema is supported by this foreign table

Parameters

columns

- column descriptors for this table

Definition at line 117 of file ForeignDataWrapper.h.

{};

virtual void foreign_storage::ForeignDataWrapper::validateServerOptions ( const ForeignServer *  foreign_server ) const

pure virtual

Checks that the options for the given foreign server object are valid.

Parameters

foreign_server

- foreign server object containing options to be validated

Implemented in foreign_storage::InternalSystemDataWrapper, and foreign_storage::AbstractFileStorageDataWrapper.

Referenced by foreign_storage::ForeignServer::validateStorageParameters().

Here is the caller graph for this function:

virtual void foreign_storage::ForeignDataWrapper::validateTableOptions ( const ForeignTable *  foreign_table ) const

pure virtual

Checks that the options for the given foreign table object are valid.

Parameters

foreign_table

- foreign table object containing options to be validated

Implemented in foreign_storage::InternalSystemDataWrapper, foreign_storage::CsvDataWrapper, foreign_storage::RegexParserDataWrapper, and foreign_storage::AbstractFileStorageDataWrapper.

virtual void foreign_storage::ForeignDataWrapper::validateUserMappingOptions ( const UserMapping *  user_mapping, const ForeignServer *  foreign_server  ) const

pure virtual

Checks that the options for the given user mapping object are valid.

Parameters

user_mapping

- user mapping object containing options to be validated

Implemented in foreign_storage::InternalSystemDataWrapper, and foreign_storage::AbstractFileStorageDataWrapper.

---

The documentation for this class was generated from the following file:

• /home/jenkins-slave/workspace/core-os-doxygen/DataMgr/ForeignStorage/ForeignDataWrapper.h