OmniSciDB  a5dc49c757
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
foreign_storage::ForeignDataWrapper Class Referenceabstract

#include <ForeignDataWrapper.h>

+ Inheritance diagram for foreign_storage::ForeignDataWrapper:

Public Types

enum  ParallelismLevel { NONE, INTRA_FRAGMENT, INTER_FRAGMENT }
 

Public Member Functions

 ForeignDataWrapper ()=default
 
virtual ~ForeignDataWrapper ()=default
 
virtual void populateChunkMetadata (ChunkMetadataVector &chunk_metadata_vector)=0
 
virtual void populateChunkBuffers (const ChunkToBufferMap &required_buffers, const ChunkToBufferMap &optional_buffers, AbstractBuffer *delete_buffer=nullptr)=0
 
virtual std::string getSerializedDataWrapper () const =0
 
virtual void restoreDataWrapperInternals (const std::string &file_path, const ChunkMetadataVector &chunk_metadata)=0
 
virtual bool isRestored () const =0
 
virtual void validateServerOptions (const ForeignServer *foreign_server) const =0
 
virtual void validateTableOptions (const ForeignTable *foreign_table) const =0
 
virtual const std::set
< std::string_view > & 
getSupportedTableOptions () const =0
 
virtual const std::set
< std::string > 
getAlterableTableOptions () const
 
virtual void validateUserMappingOptions (const UserMapping *user_mapping, const ForeignServer *foreign_server) const =0
 
virtual const std::set
< std::string_view > & 
getSupportedUserMappingOptions () const =0
 
virtual void validateSchema (const std::list< ColumnDescriptor > &columns) const
 
virtual ParallelismLevel getCachedParallelismLevel () const
 
virtual ParallelismLevel getNonCachedParallelismLevel () const
 
virtual bool isLazyFragmentFetchingEnabled () const
 

Detailed Description

Definition at line 33 of file ForeignDataWrapper.h.

Member Enumeration Documentation

ParallelismLevel describes the desired level of parallelism of the data wrapper. This level controls which optional_buffers are passed to populateChunkBuffers with the following behaviour:

NONE - no additional optional buffers are passed in

INTRA_FRAGMENT - additional optional buffers which are in the same fragment as the required buffers

INTER_FRAGMENT - additional optional buffers which may be in different fragments than those of the required buffers

Note, the optional buffers are passed in with the intention of allowing the data wrapper to employ parallelism in retrieving them. Each subsequent level allows for a greater degree of parallelism but does not have to be supported.

Enumerator
NONE 
INTRA_FRAGMENT 
INTER_FRAGMENT 

Definition at line 136 of file ForeignDataWrapper.h.

Constructor & Destructor Documentation

foreign_storage::ForeignDataWrapper::ForeignDataWrapper ( )
default
virtual foreign_storage::ForeignDataWrapper::~ForeignDataWrapper ( )
virtualdefault

Member Function Documentation

virtual const std::set<std::string> foreign_storage::ForeignDataWrapper::getAlterableTableOptions ( ) const
inlinevirtual

Gets the subset of table options that can be altered for the data wrapper.

Reimplemented in foreign_storage::AbstractFileStorageDataWrapper.

Definition at line 99 of file ForeignDataWrapper.h.

99 { return {}; };
virtual ParallelismLevel foreign_storage::ForeignDataWrapper::getCachedParallelismLevel ( ) const
inlinevirtual

Gets the desired level of parallelism for the data wrapper when a cache is in use. This affects the optional buffers that the data wrapper is made aware of during data requests.

Reimplemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, and foreign_storage::ParquetImporter.

Definition at line 143 of file ForeignDataWrapper.h.

References NONE.

virtual ParallelismLevel foreign_storage::ForeignDataWrapper::getNonCachedParallelismLevel ( ) const
inlinevirtual

Gets the desired level of parallelism for the data wrapper when no cache is in use. This affects the optional buffers that the data wrapper is made aware of during data requests.

Reimplemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, and foreign_storage::ParquetImporter.

Definition at line 150 of file ForeignDataWrapper.h.

References NONE.

virtual std::string foreign_storage::ForeignDataWrapper::getSerializedDataWrapper ( ) const
pure virtual

Serialize internal state of wrapper into file at given path if implemented

Implemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, foreign_storage::InternalSystemDataWrapper, and foreign_storage::ParquetImporter.

virtual const std::set<std::string_view>& foreign_storage::ForeignDataWrapper::getSupportedTableOptions ( ) const
pure virtual
virtual const std::set<std::string_view>& foreign_storage::ForeignDataWrapper::getSupportedUserMappingOptions ( ) const
pure virtual

Gets the set of supported user mapping options for the data wrapper.

Implemented in foreign_storage::InternalSystemDataWrapper, and foreign_storage::AbstractFileStorageDataWrapper.

virtual bool foreign_storage::ForeignDataWrapper::isLazyFragmentFetchingEnabled ( ) const
inlinevirtual

If true data wrapper implements a lazy fragment fetching mode. This mode allows requests for fragments to be issued to populateChunks without the prerequisite that populateChunkMetadata has successfully finished execution. This is an optimization that has some specific use-cases and is not required.

NOTE: this mode is not guaranteed to work as expected when combined with certain types of refresh modes such as append. This is subject to change in the future, but has no impact on the intended use-cases of this mode.

Reimplemented in foreign_storage::AbstractTextFileDataWrapper.

Definition at line 163 of file ForeignDataWrapper.h.

163 { return false; }
virtual bool foreign_storage::ForeignDataWrapper::isRestored ( ) const
pure virtual
virtual void foreign_storage::ForeignDataWrapper::populateChunkBuffers ( const ChunkToBufferMap required_buffers,
const ChunkToBufferMap optional_buffers,
AbstractBuffer delete_buffer = nullptr 
)
pure virtual

Populates given chunk buffers identified by chunk keys. All provided chunk buffers are expected to be for the same fragment.

Parameters
required_buffers- chunk buffers that must always be populated
optional_buffers- chunk buffers that can be optionally populated, if the data wrapper has to scan through chunk data anyways (typically for row wise data formats)
delete_buffer- chunk buffer for fragment's delete column, if non-null data wrapper is expected to mark deleted rows in buffer and continue processing

Implemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, foreign_storage::ParquetImporter, and foreign_storage::InternalSystemDataWrapper.

Referenced by anonymous_namespace{ForeignDataImporter.cpp}::import_foreign_data(), and foreign_storage::CachingForeignStorageMgr::populateChunkBuffersSafely().

+ Here is the caller graph for this function:

virtual void foreign_storage::ForeignDataWrapper::populateChunkMetadata ( ChunkMetadataVector chunk_metadata_vector)
pure virtual

Populates given chunk metadata vector with metadata for all chunks in related foreign table.

Parameters
chunk_metadata_vector- vector that will be populated with chunk metadata

Implemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, foreign_storage::ParquetImporter, and foreign_storage::InternalSystemDataWrapper.

Referenced by anonymous_namespace{ForeignDataImporter.cpp}::metadata_scan().

+ Here is the caller graph for this function:

virtual void foreign_storage::ForeignDataWrapper::restoreDataWrapperInternals ( const std::string &  file_path,
const ChunkMetadataVector chunk_metadata 
)
pure virtual

Restore internal state of datawrapper

Parameters
file_path- location of file created by serializeMetadata
chunk_metadata_vector- vector of chunk metadata recovered from disk

Implemented in foreign_storage::AbstractTextFileDataWrapper, foreign_storage::ParquetDataWrapper, foreign_storage::InternalSystemDataWrapper, and foreign_storage::ParquetImporter.

virtual void foreign_storage::ForeignDataWrapper::validateSchema ( const std::list< ColumnDescriptor > &  columns) const
inlinevirtual

Verifies the schema is supported by this foreign table

Parameters
columns- column descriptors for this table

Definition at line 117 of file ForeignDataWrapper.h.

117 {};
virtual void foreign_storage::ForeignDataWrapper::validateServerOptions ( const ForeignServer foreign_server) const
pure virtual

Checks that the options for the given foreign server object are valid.

Parameters
foreign_server- foreign server object containing options to be validated

Implemented in foreign_storage::InternalSystemDataWrapper, and foreign_storage::AbstractFileStorageDataWrapper.

Referenced by foreign_storage::ForeignServer::validateStorageParameters().

+ Here is the caller graph for this function:

virtual void foreign_storage::ForeignDataWrapper::validateTableOptions ( const ForeignTable foreign_table) const
pure virtual

Checks that the options for the given foreign table object are valid.

Parameters
foreign_table- foreign table object containing options to be validated

Implemented in foreign_storage::InternalSystemDataWrapper, foreign_storage::CsvDataWrapper, foreign_storage::RegexParserDataWrapper, and foreign_storage::AbstractFileStorageDataWrapper.

virtual void foreign_storage::ForeignDataWrapper::validateUserMappingOptions ( const UserMapping user_mapping,
const ForeignServer foreign_server 
) const
pure virtual

Checks that the options for the given user mapping object are valid.

Parameters
user_mapping- user mapping object containing options to be validated

Implemented in foreign_storage::InternalSystemDataWrapper, and foreign_storage::AbstractFileStorageDataWrapper.


The documentation for this class was generated from the following file: