foreign_storage::ParquetEncoder Class Reference

#include <ParquetEncoder.h>

Inheritance diagram for foreign_storage::ParquetEncoder:

Collaboration diagram for foreign_storage::ParquetEncoder:

Public Member Functions
	ParquetEncoder (Data_Namespace::AbstractBuffer *buffer)
virtual	~ParquetEncoder ()=default
virtual void	appendData (const int16_t *def_levels, const int16_t *rep_levels, const int64_t values_read, const int64_t levels_read, const bool is_last_batch, int8_t *values)=0
virtual std::shared_ptr < ChunkMetadata >	getRowGroupMetadata (const parquet::RowGroupMetaData *group_metadata, const int parquet_column_index, const SQLTypeInfo &column_type)

Static Protected Member Functions
static std::shared_ptr < ChunkMetadata >	createMetadata (const SQLTypeInfo &column_type)
static void	throwNotNullViolation (const std::string &parquet_column_name)
static void	validateNullCount (const std::string &parquet_column_name, int64_t null_count, const SQLTypeInfo &column_type)

Protected Attributes
Data_Namespace::AbstractBuffer *	buffer_

Detailed Description

Definition at line 28 of file ParquetEncoder.h.

Constructor & Destructor Documentation

foreign_storage::ParquetEncoder::ParquetEncoder ( Data_Namespace::AbstractBuffer *  buffer )

inline

Definition at line 30 of file ParquetEncoder.h.

: buffer_(buffer) {}

virtual foreign_storage::ParquetEncoder::~ParquetEncoder ( )

virtualdefault

Member Function Documentation

virtual void foreign_storage::ParquetEncoder::appendData ( const int16_t *  def_levels, const int16_t *  rep_levels, const int64_t  values_read, const int64_t  levels_read, const bool  is_last_batch, int8_t *  values  )

pure virtual

Implemented in foreign_storage::TypedParquetInPlaceEncoder< V, T >, foreign_storage::TypedParquetInPlaceEncoder< V, V >, foreign_storage::TypedParquetInPlaceEncoder< int64_t, int32_t >, foreign_storage::ParquetGeospatialEncoder, foreign_storage::ParquetInPlaceEncoder, foreign_storage::ParquetStringEncoder< V >, foreign_storage::ParquetArrayEncoder, foreign_storage::ParquetStringNoneEncoder, and foreign_storage::ParquetVariableLengthArrayEncoder.

static std::shared_ptr<ChunkMetadata> foreign_storage::ParquetEncoder::createMetadata ( const SQLTypeInfo &  column_type )

inlinestaticprotected

Definition at line 63 of file ParquetEncoder.h.

References SQLTypeInfo::get_elem_type(), Data_Namespace::AbstractBuffer::initEncoder(), and SQLTypeInfo::is_array().

Referenced by getRowGroupMetadata(), and foreign_storage::TypedParquetInPlaceEncoder< int64_t, int32_t >::getRowGroupMetadata().

                                                                                     {
    auto metadata = std::make_shared<ChunkMetadata>();
    ForeignStorageBuffer buffer;
    buffer.initEncoder(column_type.is_array() ? column_type.get_elem_type()
                                              : column_type);
    auto encoder = buffer.getEncoder();
    encoder->getMetadata(metadata);
    metadata->sqlType = column_type;
    return metadata;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

virtual std::shared_ptr<ChunkMetadata> foreign_storage::ParquetEncoder::getRowGroupMetadata ( const parquet::RowGroupMetaData *  group_metadata, const int  parquet_column_index, const SQLTypeInfo &  column_type  )

inlinevirtual

Reimplemented in foreign_storage::TypedParquetInPlaceEncoder< V, T >, foreign_storage::TypedParquetInPlaceEncoder< V, V >, foreign_storage::TypedParquetInPlaceEncoder< int64_t, int32_t >, foreign_storage::ParquetStringEncoder< V >, foreign_storage::ParquetArrayEncoder, and foreign_storage::ParquetFixedLengthArrayEncoder.

Definition at line 40 of file ParquetEncoder.h.

References createMetadata(), foreign_storage::validate_and_get_column_metadata_statistics(), and validateNullCount().

Referenced by foreign_storage::ParquetStringEncoder< V >::getRowGroupMetadata().

                                      {
    // update statistics
    auto column_metadata = group_metadata->ColumnChunk(parquet_column_index);
    auto stats = validate_and_get_column_metadata_statistics(column_metadata.get());
    auto metadata = createMetadata(column_type);

    auto null_count = stats->null_count();
    validateNullCount(group_metadata->schema()->Column(parquet_column_index)->name(),
                      null_count,
                      column_type);
    metadata->chunkStats.has_nulls = null_count > 0;

    // update sizing
    metadata->numElements = group_metadata->num_rows();
    return metadata;
  }

Here is the call graph for this function:

Here is the caller graph for this function:

static void foreign_storage::ParquetEncoder::throwNotNullViolation ( const std::string &  parquet_column_name )

inlinestaticprotected

Definition at line 74 of file ParquetEncoder.h.

Referenced by validateNullCount().

                                                                          {
    std::stringstream error_message;
    error_message << "A null value was detected in Parquet column '"
                  << parquet_column_name << "' but OmniSci column is set to not null";
    throw std::runtime_error(error_message.str());
  }

Here is the caller graph for this function:

static void foreign_storage::ParquetEncoder::validateNullCount ( const std::string &  parquet_column_name, int64_t  null_count, const SQLTypeInfo &  column_type  )

inlinestaticprotected

Definition at line 81 of file ParquetEncoder.h.

References SQLTypeInfo::get_notnull(), and throwNotNullViolation().

Referenced by getRowGroupMetadata(), and foreign_storage::TypedParquetInPlaceEncoder< int64_t, int32_t >::getRowGroupMetadata().

                                                                {
    bool has_nulls = null_count > 0;
    if (has_nulls && column_type.get_notnull()) {
      throwNotNullViolation(parquet_column_name);
    }
  }

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

Data_Namespace::AbstractBuffer* foreign_storage::ParquetEncoder::buffer_

protected

Definition at line 61 of file ParquetEncoder.h.

Referenced by foreign_storage::ParquetArrayEncoder::appendArraysToBuffer(), foreign_storage::ParquetStringNoneEncoder::appendData(), foreign_storage::ParquetInPlaceEncoder::appendData(), foreign_storage::TypedParquetInPlaceEncoder< int64_t, int32_t >::appendData(), foreign_storage::ParquetVariableLengthArrayEncoder::appendLastArrayOffset(), and foreign_storage::ParquetVariableLengthArrayEncoder::setFirstOffsetForBuffer().

---

The documentation for this class was generated from the following file:

• /home/jenkins-slave/workspace/core-os-doxygen/DataMgr/ForeignStorage/ParquetEncoder.h