OmniSciDB  085a039ca4
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
FixedLengthArrayNoneEncoder Class Reference

#include <FixedLengthArrayNoneEncoder.h>

+ Inheritance diagram for FixedLengthArrayNoneEncoder:
+ Collaboration diagram for FixedLengthArrayNoneEncoder:

Public Member Functions

 FixedLengthArrayNoneEncoder (AbstractBuffer *buffer, size_t as)
 
size_t getNumElemsForBytesEncodedDataAtIndices (const int8_t *index_data, const std::vector< size_t > &selected_idx, const size_t byte_limit) override
 
size_t getNumElemsForBytesInsertData (const std::vector< ArrayDatum > *srcData, const int start_idx, const size_t numAppendElems, const size_t byteLimit, const bool replicating=false)
 
std::shared_ptr< ChunkMetadataappendEncodedDataAtIndices (const int8_t *index_data, int8_t *data, const std::vector< size_t > &selected_idx) override
 
std::shared_ptr< ChunkMetadataappendEncodedData (const int8_t *index_data, int8_t *data, const size_t start_idx, const size_t num_elements) override
 
std::shared_ptr< ChunkMetadataappendData (int8_t *&src_data, const size_t num_elems_to_append, const SQLTypeInfo &ti, const bool replicating=false, const int64_t offset=-1) override
 
std::shared_ptr< ChunkMetadataappendData (const std::vector< ArrayDatum > *srcData, const int start_idx, const size_t numAppendElems, const bool replicating=false)
 
void getMetadata (const std::shared_ptr< ChunkMetadata > &chunkMetadata) override
 
std::shared_ptr< ChunkMetadatagetMetadata (const SQLTypeInfo &ti) override
 
void updateStats (const int64_t, const bool) override
 
void updateStats (const double, const bool) override
 
void reduceStats (const Encoder &) override
 
void updateStats (const int8_t *const src_data, const size_t num_elements) override
 
void updateStats (const std::vector< std::string > *const src_data, const size_t start_idx, const size_t num_elements) override
 
void updateStats (const std::vector< ArrayDatum > *const src_data, const size_t start_idx, const size_t num_elements) override
 
void writeMetadata (FILE *f) override
 
void readMetadata (FILE *f) override
 
void copyMetadata (const Encoder *copyFromEncoder) override
 
void updateMetadata (int8_t *array)
 
bool resetChunkStats (const ChunkStats &stats) override
 : Reset chunk level stats (min, max, nulls) using new values from the argument. More...
 
void resetChunkStats () override
 
- Public Member Functions inherited from Encoder
 Encoder (Data_Namespace::AbstractBuffer *buffer)
 
virtual ~Encoder ()
 
virtual void updateStatsEncoded (const int8_t *const dst_data, const size_t num_elements)
 
size_t getNumElems () const
 
void setNumElems (const size_t num_elems)
 

Static Public Member Functions

static bool is_null_ignore_not_null (const SQLTypeInfo &type, int8_t *array)
 
static bool is_null (const SQLTypeInfo &type, int8_t *array)
 
- Static Public Member Functions inherited from Encoder
static EncoderCreate (Data_Namespace::AbstractBuffer *buffer, const SQLTypeInfo sqlType)
 

Public Attributes

Datum elem_min
 
Datum elem_max
 
bool has_nulls
 
bool initialized
 

Private Member Functions

bool is_null (int8_t *array)
 
bool is_null_ignore_not_null (int8_t *array)
 
void update_elem_stats (const ArrayDatum &array)
 

Private Attributes

std::mutex EncoderMutex_
 
std::mutex print_mutex_
 
size_t array_size
 

Additional Inherited Members

- Protected Attributes inherited from Encoder
size_t num_elems_
 
Data_Namespace::AbstractBufferbuffer_
 
DecimalOverflowValidator decimal_overflow_validator_
 
DateDaysOverflowValidator date_days_overflow_validator_
 

Detailed Description

Definition at line 41 of file FixedLengthArrayNoneEncoder.h.

Constructor & Destructor Documentation

FixedLengthArrayNoneEncoder::FixedLengthArrayNoneEncoder ( AbstractBuffer buffer,
size_t  as 
)
inline

Definition at line 43 of file FixedLengthArrayNoneEncoder.h.

44  : Encoder(buffer), has_nulls(false), initialized(false), array_size(as) {}
Encoder(Data_Namespace::AbstractBuffer *buffer)
Definition: Encoder.cpp:221

Member Function Documentation

std::shared_ptr<ChunkMetadata> FixedLengthArrayNoneEncoder::appendData ( int8_t *&  src_data,
const size_t  num_elems_to_append,
const SQLTypeInfo ti,
const bool  replicating = false,
const int64_t  offset = -1 
)
inlineoverridevirtual

Append data to the chunk buffer backing this encoder.

Parameters
src_dataSource data for the append
num_elems_to_appendNumber of elements to append
tiSQL Type Info for the column TODO(adb): used?
replicatingPass one value and fill the chunk with it
offsetWrite data starting at a given offset. Default is -1 which indicates an append, an offset of 0 rewrites the chunk up to num_elems_to_append.

Implements Encoder.

Definition at line 98 of file FixedLengthArrayNoneEncoder.h.

References UNREACHABLE.

Referenced by Chunk_NS::Chunk::appendData(), appendEncodedData(), and appendEncodedDataAtIndices().

102  {
103  UNREACHABLE(); // should never be called for arrays
104  return nullptr;
105  }
#define UNREACHABLE()
Definition: Logger.h:267

+ Here is the caller graph for this function:

std::shared_ptr<ChunkMetadata> FixedLengthArrayNoneEncoder::appendData ( const std::vector< ArrayDatum > *  srcData,
const int  start_idx,
const size_t  numAppendElems,
const bool  replicating = false 
)
inline

Definition at line 107 of file FixedLengthArrayNoneEncoder.h.

References Data_Namespace::AbstractBuffer::append(), array_size, Encoder::buffer_, CHECK_EQ, getMetadata(), Data_Namespace::AbstractBuffer::isDirty(), Encoder::num_elems_, Data_Namespace::AbstractBuffer::reserve(), Data_Namespace::AbstractBuffer::setDirty(), and updateStats().

110  {
111  // Todo: The reserve call was changed to take into account the existing data size,
112  // but in other encoders (like ArrayNoneEncoder) we only reserve the append size,
113  // which will be a no-op likely after the first append on a chunk. This probably
114  // won't matter for disk writes as we just have static (default 2MB) page sizes, but
115  // could be an issue for temporary in-memory tables, as buffers for multi-column
116  // imports will likely need to be repeatedly migrated to grow them if they are
117  // "landlocked" amidst other buffers. We should follow-up with work to call reserve
118  // properly, accounting for both the new append size and existing size, for that
119  // reason and just for overall semantic correctness.
120 
121  const size_t existing_data_size = num_elems_ * array_size;
122  const size_t append_data_size = array_size * numAppendElems;
123  buffer_->reserve(existing_data_size + append_data_size);
124  std::vector<int8_t> append_buffer(append_data_size);
125  int8_t* append_ptr = append_buffer.data();
126 
127  // There was some worry about the change implemented to write the append data to an
128  // intermediate buffer, but testing on import and ctas of 20M points, we never append
129  // more than 1.6MB and 1MB of data at a time, respectively, so at least for fixed
130  // length types this should not be an issue (varlen types, which can be massive even
131  // for a single field/row, are a different story however)
132 
133  if (replicating) {
134  const size_t len = (*srcData)[0].length;
135  CHECK_EQ(len, array_size);
136  const int8_t* replicated_ptr = (*srcData)[0].pointer;
137  for (size_t i = 0; i < numAppendElems; ++i) {
138  std::memcpy(append_ptr + i * array_size, replicated_ptr, array_size);
139  }
140  } else {
141  for (size_t i = 0; i < numAppendElems; ++i) {
142  // Length of the appended array should be equal to the fixed length,
143  // all others should have been discarded, assert if something slips through
144  const size_t source_idx = start_idx + i;
145  const size_t len = (*srcData)[source_idx].length;
146  CHECK_EQ(len, array_size);
147  // NULL arrays have been filled with subtype's NULL sentinels,
148  // should be appended as regular data, same size
149  std::memcpy(
150  append_ptr + i * array_size, (*srcData)[source_idx].pointer, array_size);
151  }
152  }
153 
154  buffer_->append(append_ptr, append_data_size);
155 
156  if (replicating) {
157  updateStats(srcData, 0, 1);
158  } else {
159  updateStats(srcData, start_idx, numAppendElems);
160  }
161 
162  // make sure buffer_ is flushed even if no new data is appended to it
163  // (e.g. empty strings) because the metadata needs to be flushed.
164  if (!buffer_->isDirty()) {
165  buffer_->setDirty();
166  }
167 
168  num_elems_ += numAppendElems;
169  auto chunk_metadata = std::make_shared<ChunkMetadata>();
170  getMetadata(chunk_metadata);
171  return chunk_metadata;
172  }
#define CHECK_EQ(x, y)
Definition: Logger.h:231
size_t num_elems_
Definition: Encoder.h:288
void getMetadata(const std::shared_ptr< ChunkMetadata > &chunkMetadata) override
Data_Namespace::AbstractBuffer * buffer_
Definition: Encoder.h:290
virtual void append(int8_t *src, const size_t num_bytes, const MemoryLevel src_buffer_type=CPU_LEVEL, const int device_id=-1)=0
void updateStats(const int64_t, const bool) override
virtual void reserve(size_t num_bytes)=0

+ Here is the call graph for this function:

std::shared_ptr<ChunkMetadata> FixedLengthArrayNoneEncoder::appendEncodedData ( const int8_t *  index_data,
int8_t *  data,
const size_t  start_idx,
const size_t  num_elements 
)
inlineoverridevirtual

Append encoded data to the chunk buffer backing this encoder.

Parameters
index_data- (optional) the index data of data to append
data- the data to append
start_idx- the position to start encoding from in the data array
num_elements- the number of elements to encode from the data array
Returns
updated chunk metadata for the chunk buffer backing this encoder

NOTE: index_data must be non-null for varlen encoder types.

Implements Encoder.

Definition at line 84 of file FixedLengthArrayNoneEncoder.h.

References appendData(), and array_size.

87  {
88  std::vector<ArrayDatum> data_subset;
89  data_subset.reserve(num_elements);
90  for (size_t count = 0; count < num_elements; ++count) {
91  auto current_data = data + array_size * (start_idx + count);
92  data_subset.emplace_back(
93  ArrayDatum(array_size, current_data, false, DoNothingDeleter{}));
94  }
95  return appendData(&data_subset, 0, num_elements, false);
96  }
std::conditional_t< is_cuda_compiler(), DeviceArrayDatum, HostArrayDatum > ArrayDatum
Definition: sqltypes.h:208
std::shared_ptr< ChunkMetadata > appendData(int8_t *&src_data, const size_t num_elems_to_append, const SQLTypeInfo &ti, const bool replicating=false, const int64_t offset=-1) override

+ Here is the call graph for this function:

std::shared_ptr<ChunkMetadata> FixedLengthArrayNoneEncoder::appendEncodedDataAtIndices ( const int8_t *  index_data,
int8_t *  data,
const std::vector< size_t > &  selected_idx 
)
inlineoverridevirtual

Append selected encoded data to the chunk buffer backing this encoder.

Parameters
index_data- (optional) the index data of data to append
data- the data to append
selected_idx- which indices in the encoded data to append
Returns
updated chunk metadata for the chunk buffer backing this encoder

NOTE: index_data must be non-null for varlen encoder types.

Implements Encoder.

Definition at line 68 of file FixedLengthArrayNoneEncoder.h.

References appendData(), array_size, and is_null_ignore_not_null().

71  {
72  std::vector<ArrayDatum> data_subset;
73  data_subset.reserve(selected_idx.size());
74  for (const auto& index : selected_idx) {
75  auto current_data = data + array_size * (index);
76  data_subset.emplace_back(ArrayDatum(array_size,
77  current_data,
78  is_null_ignore_not_null(current_data),
79  DoNothingDeleter{}));
80  }
81  return appendData(&data_subset, 0, selected_idx.size(), false);
82  }
std::conditional_t< is_cuda_compiler(), DeviceArrayDatum, HostArrayDatum > ArrayDatum
Definition: sqltypes.h:208
std::shared_ptr< ChunkMetadata > appendData(int8_t *&src_data, const size_t num_elems_to_append, const SQLTypeInfo &ti, const bool replicating=false, const int64_t offset=-1) override
static bool is_null_ignore_not_null(const SQLTypeInfo &type, int8_t *array)

+ Here is the call graph for this function:

void FixedLengthArrayNoneEncoder::copyMetadata ( const Encoder copyFromEncoder)
inlineoverridevirtual

Implements Encoder.

Definition at line 228 of file FixedLengthArrayNoneEncoder.h.

References elem_max, elem_min, Encoder::getNumElems(), has_nulls, initialized, and Encoder::num_elems_.

228  {
229  num_elems_ = copyFromEncoder->getNumElems();
230  auto array_encoder =
231  dynamic_cast<const FixedLengthArrayNoneEncoder*>(copyFromEncoder);
232  elem_min = array_encoder->elem_min;
233  elem_max = array_encoder->elem_max;
234  has_nulls = array_encoder->has_nulls;
235  initialized = array_encoder->initialized;
236  }
size_t num_elems_
Definition: Encoder.h:288
size_t getNumElems() const
Definition: Encoder.h:284

+ Here is the call graph for this function:

void FixedLengthArrayNoneEncoder::getMetadata ( const std::shared_ptr< ChunkMetadata > &  chunkMetadata)
inlineoverridevirtual

Reimplemented from Encoder.

Definition at line 174 of file FixedLengthArrayNoneEncoder.h.

References elem_max, elem_min, Encoder::getMetadata(), and has_nulls.

Referenced by appendData().

174  {
175  Encoder::getMetadata(chunkMetadata); // call on parent class
176  chunkMetadata->fillChunkStats(elem_min, elem_max, has_nulls);
177  }
virtual void getMetadata(const std::shared_ptr< ChunkMetadata > &chunkMetadata)
Definition: Encoder.cpp:227

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::shared_ptr<ChunkMetadata> FixedLengthArrayNoneEncoder::getMetadata ( const SQLTypeInfo ti)
inlineoverridevirtual

Implements Encoder.

Definition at line 180 of file FixedLengthArrayNoneEncoder.h.

References elem_max, elem_min, and has_nulls.

180  {
181  auto chunk_metadata = std::make_shared<ChunkMetadata>(
182  ti, 0, 0, ChunkStats{elem_min, elem_max, has_nulls});
183  return chunk_metadata;
184  }
size_t FixedLengthArrayNoneEncoder::getNumElemsForBytesEncodedDataAtIndices ( const int8_t *  index_data,
const std::vector< size_t > &  selected_idx,
const size_t  byte_limit 
)
inlineoverridevirtual

Compute the maximum number of variable length encoded elements given a byte limit

Parameters
index_data- (optional) index data for the encoded type
selected_idx- which indices in the encoded data to consider
byte_limit- byte limit that must be respected
Returns
the number of elements

NOTE: optional parameters above may be ignored by the implementation, but may or may not be required depending on the encoder type backing the implementation.

Implements Encoder.

Definition at line 46 of file FixedLengthArrayNoneEncoder.h.

References array_size.

48  {
49  size_t data_size = selected_idx.size() * array_size;
50  if (data_size > byte_limit) {
51  data_size = byte_limit;
52  }
53  return data_size / array_size;
54  }
size_t FixedLengthArrayNoneEncoder::getNumElemsForBytesInsertData ( const std::vector< ArrayDatum > *  srcData,
const int  start_idx,
const size_t  numAppendElems,
const size_t  byteLimit,
const bool  replicating = false 
)
inline

Definition at line 56 of file FixedLengthArrayNoneEncoder.h.

References array_size.

Referenced by Chunk_NS::Chunk::getNumElemsForBytesInsertData().

60  {
61  size_t dataSize = numAppendElems * array_size;
62  if (dataSize > byteLimit) {
63  dataSize = byteLimit;
64  }
65  return dataSize / array_size;
66  }

+ Here is the caller graph for this function:

static bool FixedLengthArrayNoneEncoder::is_null ( const SQLTypeInfo type,
int8_t *  array 
)
inlinestatic

Definition at line 292 of file FixedLengthArrayNoneEncoder.h.

References SQLTypeInfo::get_notnull(), and is_null_ignore_not_null().

Referenced by Fragmenter_Namespace::FixedLenArrayChunkConverter::convertToColumnarFormat(), and updateMetadata().

292  {
293  if (type.get_notnull()) {
294  return false;
295  }
296  return is_null_ignore_not_null(type, array);
297  }
HOST DEVICE bool get_notnull() const
Definition: sqltypes.h:336
static bool is_null_ignore_not_null(const SQLTypeInfo &type, int8_t *array)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

bool FixedLengthArrayNoneEncoder::is_null ( int8_t *  array)
inlineprivate

Definition at line 326 of file FixedLengthArrayNoneEncoder.h.

References Encoder::buffer_, Data_Namespace::AbstractBuffer::getSqlType(), and is_null().

Referenced by is_null().

326 { return is_null(buffer_->getSqlType(), array); }
Data_Namespace::AbstractBuffer * buffer_
Definition: Encoder.h:290
SQLTypeInfo getSqlType() const
static bool is_null(const SQLTypeInfo &type, int8_t *array)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

static bool FixedLengthArrayNoneEncoder::is_null_ignore_not_null ( const SQLTypeInfo type,
int8_t *  array 
)
inlinestatic

Definition at line 242 of file FixedLengthArrayNoneEncoder.h.

References CHECK_EQ, SQLTypeInfo::get_compression(), SQLTypeInfo::get_subtype(), kBIGINT, kBOOLEAN, kCHAR, kDATE, kDECIMAL, kDOUBLE, kENCODING_DICT, kFLOAT, kINT, kNUMERIC, kSMALLINT, kTEXT, kTIME, kTIMESTAMP, kTINYINT, kVARCHAR, NULL_ARRAY_BIGINT, NULL_ARRAY_BOOLEAN, NULL_ARRAY_DOUBLE, NULL_ARRAY_FLOAT, NULL_ARRAY_INT, NULL_ARRAY_SMALLINT, NULL_ARRAY_TINYINT, and UNREACHABLE.

Referenced by appendEncodedDataAtIndices(), is_null(), and is_null_ignore_not_null().

242  {
243  switch (type.get_subtype()) {
244  case kBOOLEAN: {
245  return (array[0] == NULL_ARRAY_BOOLEAN);
246  }
247  case kINT: {
248  const int32_t* int_array = (int32_t*)array;
249  return (int_array[0] == NULL_ARRAY_INT);
250  }
251  case kSMALLINT: {
252  const int16_t* smallint_array = (int16_t*)array;
253  return (smallint_array[0] == NULL_ARRAY_SMALLINT);
254  }
255  case kTINYINT: {
256  const int8_t* tinyint_array = (int8_t*)array;
257  return (tinyint_array[0] == NULL_ARRAY_TINYINT);
258  }
259  case kBIGINT:
260  case kNUMERIC:
261  case kDECIMAL: {
262  const int64_t* bigint_array = (int64_t*)array;
263  return (bigint_array[0] == NULL_ARRAY_BIGINT);
264  }
265  case kFLOAT: {
266  const float* flt_array = (float*)array;
267  return (flt_array[0] == NULL_ARRAY_FLOAT);
268  }
269  case kDOUBLE: {
270  const double* dbl_array = (double*)array;
271  return (dbl_array[0] == NULL_ARRAY_DOUBLE);
272  }
273  case kTIME:
274  case kTIMESTAMP:
275  case kDATE: {
276  const int64_t* tm_array = reinterpret_cast<int64_t*>(array);
277  return (tm_array[0] == NULL_ARRAY_BIGINT);
278  }
279  case kCHAR:
280  case kVARCHAR:
281  case kTEXT: {
283  const int32_t* int_array = (int32_t*)array;
284  return (int_array[0] == NULL_ARRAY_INT);
285  }
286  default:
287  UNREACHABLE();
288  }
289  return false;
290  }
HOST DEVICE SQLTypes get_subtype() const
Definition: sqltypes.h:330
#define CHECK_EQ(x, y)
Definition: Logger.h:231
Definition: sqltypes.h:49
#define NULL_ARRAY_INT
#define NULL_ARRAY_SMALLINT
#define UNREACHABLE()
Definition: Logger.h:267
#define NULL_ARRAY_TINYINT
#define NULL_ARRAY_FLOAT
Definition: sqltypes.h:52
Definition: sqltypes.h:53
HOST DEVICE EncodingType get_compression() const
Definition: sqltypes.h:337
Definition: sqltypes.h:41
#define NULL_ARRAY_DOUBLE
#define NULL_ARRAY_BIGINT
Definition: sqltypes.h:45
#define NULL_ARRAY_BOOLEAN

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

bool FixedLengthArrayNoneEncoder::is_null_ignore_not_null ( int8_t *  array)
inlineprivate

Definition at line 328 of file FixedLengthArrayNoneEncoder.h.

References Encoder::buffer_, Data_Namespace::AbstractBuffer::getSqlType(), and is_null_ignore_not_null().

328  {
329  return is_null_ignore_not_null(buffer_->getSqlType(), array);
330  }
Data_Namespace::AbstractBuffer * buffer_
Definition: Encoder.h:290
SQLTypeInfo getSqlType() const
static bool is_null_ignore_not_null(const SQLTypeInfo &type, int8_t *array)

+ Here is the call graph for this function:

void FixedLengthArrayNoneEncoder::readMetadata ( FILE *  f)
inlineoverridevirtual

Implements Encoder.

Definition at line 219 of file FixedLengthArrayNoneEncoder.h.

References elem_max, elem_min, has_nulls, initialized, and Encoder::num_elems_.

219  {
220  // assumes pointer is already in right place
221  fread((int8_t*)&num_elems_, sizeof(size_t), 1, f);
222  fread((int8_t*)&elem_min, sizeof(Datum), 1, f);
223  fread((int8_t*)&elem_max, sizeof(Datum), 1, f);
224  fread((int8_t*)&has_nulls, sizeof(bool), 1, f);
225  fread((int8_t*)&initialized, sizeof(bool), 1, f);
226  }
size_t num_elems_
Definition: Encoder.h:288
constexpr double f
Definition: Utm.h:31
void FixedLengthArrayNoneEncoder::reduceStats ( const Encoder )
inlineoverridevirtual

Implements Encoder.

Definition at line 190 of file FixedLengthArrayNoneEncoder.h.

References CHECK.

190 { CHECK(false); }
#define CHECK(condition)
Definition: Logger.h:223
bool FixedLengthArrayNoneEncoder::resetChunkStats ( const ChunkStats )
inlineoverridevirtual

: Reset chunk level stats (min, max, nulls) using new values from the argument.

Returns
: True if an update occurred and the chunk needs to be flushed. False otherwise. Default false if metadata update is unsupported. Only reset chunk stats if the incoming stats differ from the current stats.

Reimplemented from Encoder.

Definition at line 299 of file FixedLengthArrayNoneEncoder.h.

References Encoder::buffer_, DatumEqual(), elem_max, elem_min, SQLTypeInfo::get_elem_type(), Data_Namespace::AbstractBuffer::getSqlType(), ChunkStats::has_nulls, has_nulls, initialized, ChunkStats::max, and ChunkStats::min.

299  {
300  auto elem_type = buffer_->getSqlType().get_elem_type();
301  if (initialized && DatumEqual(elem_min, stats.min, elem_type) &&
302  DatumEqual(elem_max, stats.max, elem_type) && has_nulls == stats.has_nulls) {
303  return false;
304  }
305  elem_min = stats.min;
306  elem_max = stats.max;
307  has_nulls = stats.has_nulls;
308  return true;
309  }
Data_Namespace::AbstractBuffer * buffer_
Definition: Encoder.h:290
bool DatumEqual(const Datum a, const Datum b, const SQLTypeInfo &ti)
Definition: Datum.cpp:342
SQLTypeInfo getSqlType() const
SQLTypeInfo get_elem_type() const
Definition: sqltypes.h:865

+ Here is the call graph for this function:

void FixedLengthArrayNoneEncoder::resetChunkStats ( )
inlineoverridevirtual

Resets chunk metadata stats to their default values.

Implements Encoder.

Definition at line 311 of file FixedLengthArrayNoneEncoder.h.

References has_nulls, and initialized.

void FixedLengthArrayNoneEncoder::update_elem_stats ( const ArrayDatum array)
inlineprivate

Definition at line 332 of file FixedLengthArrayNoneEncoder.h.

References Datum::bigintval, Datum::boolval, Encoder::buffer_, CHECK_EQ, Encoder::decimal_overflow_validator_, Datum::doubleval, elem_max, elem_min, Datum::floatval, SQLTypeInfo::get_compression(), SQLTypeInfo::get_subtype(), Data_Namespace::AbstractBuffer::getSqlType(), has_nulls, initialized, Datum::intval, kBIGINT, kBOOLEAN, kCHAR, kDATE, kDECIMAL, kDOUBLE, kENCODING_DICT, kFLOAT, kINT, kNUMERIC, kSMALLINT, kTEXT, kTIME, kTIMESTAMP, kTINYINT, kVARCHAR, NULL_BIGINT, NULL_BOOLEAN, NULL_DOUBLE, NULL_FLOAT, NULL_INT, NULL_SMALLINT, NULL_TINYINT, Datum::smallintval, Datum::tinyintval, UNREACHABLE, and DecimalOverflowValidator::validate().

Referenced by updateMetadata(), and updateStats().

332  {
333  if (array.is_null) {
334  has_nulls = true;
335  }
336  switch (buffer_->getSqlType().get_subtype()) {
337  case kBOOLEAN: {
338  if (!initialized) {
339  elem_min.boolval = true;
340  elem_max.boolval = false;
341  }
342  if (array.is_null) {
343  break;
344  }
345  const int8_t* bool_array = array.pointer;
346  for (size_t i = 0; i < array.length / sizeof(bool); i++) {
347  if (bool_array[i] == NULL_BOOLEAN) {
348  has_nulls = true;
349  } else if (initialized) {
350  elem_min.boolval = std::min(elem_min.boolval, bool_array[i]);
351  elem_max.boolval = std::max(elem_max.boolval, bool_array[i]);
352  } else {
353  elem_min.boolval = bool_array[i];
354  elem_max.boolval = bool_array[i];
355  initialized = true;
356  }
357  }
358  break;
359  }
360  case kINT: {
361  if (!initialized) {
362  elem_min.intval = 1;
363  elem_max.intval = 0;
364  }
365  if (array.is_null) {
366  break;
367  }
368  const int32_t* int_array = (int32_t*)array.pointer;
369  for (size_t i = 0; i < array.length / sizeof(int32_t); i++) {
370  if (int_array[i] == NULL_INT) {
371  has_nulls = true;
372  } else if (initialized) {
373  elem_min.intval = std::min(elem_min.intval, int_array[i]);
374  elem_max.intval = std::max(elem_max.intval, int_array[i]);
375  } else {
376  elem_min.intval = int_array[i];
377  elem_max.intval = int_array[i];
378  initialized = true;
379  }
380  }
381  break;
382  }
383  case kSMALLINT: {
384  if (!initialized) {
385  elem_min.smallintval = 1;
386  elem_max.smallintval = 0;
387  }
388  if (array.is_null) {
389  break;
390  }
391  const int16_t* smallint_array = (int16_t*)array.pointer;
392  for (size_t i = 0; i < array.length / sizeof(int16_t); i++) {
393  if (smallint_array[i] == NULL_SMALLINT) {
394  has_nulls = true;
395  } else if (initialized) {
396  elem_min.smallintval = std::min(elem_min.smallintval, smallint_array[i]);
397  elem_max.smallintval = std::max(elem_max.smallintval, smallint_array[i]);
398  } else {
399  elem_min.smallintval = smallint_array[i];
400  elem_max.smallintval = smallint_array[i];
401  initialized = true;
402  }
403  }
404  break;
405  }
406  case kTINYINT: {
407  if (!initialized) {
408  elem_min.tinyintval = 1;
409  elem_max.tinyintval = 0;
410  }
411  if (array.is_null) {
412  break;
413  }
414  const int8_t* tinyint_array = (int8_t*)array.pointer;
415  for (size_t i = 0; i < array.length / sizeof(int8_t); i++) {
416  if (tinyint_array[i] == NULL_TINYINT) {
417  has_nulls = true;
418  } else if (initialized) {
419  elem_min.tinyintval = std::min(elem_min.tinyintval, tinyint_array[i]);
420  elem_max.tinyintval = std::max(elem_max.tinyintval, tinyint_array[i]);
421  } else {
422  elem_min.tinyintval = tinyint_array[i];
423  elem_max.tinyintval = tinyint_array[i];
424  initialized = true;
425  }
426  }
427  break;
428  }
429  case kBIGINT:
430  case kNUMERIC:
431  case kDECIMAL: {
432  if (!initialized) {
433  elem_min.bigintval = 1;
434  elem_max.bigintval = 0;
435  }
436  if (array.is_null) {
437  break;
438  }
439  const int64_t* bigint_array = (int64_t*)array.pointer;
440  for (size_t i = 0; i < array.length / sizeof(int64_t); i++) {
441  if (bigint_array[i] == NULL_BIGINT) {
442  has_nulls = true;
443  } else if (initialized) {
444  decimal_overflow_validator_.validate(bigint_array[i]);
445  elem_min.bigintval = std::min(elem_min.bigintval, bigint_array[i]);
446  elem_max.bigintval = std::max(elem_max.bigintval, bigint_array[i]);
447  } else {
448  decimal_overflow_validator_.validate(bigint_array[i]);
449  elem_min.bigintval = bigint_array[i];
450  elem_max.bigintval = bigint_array[i];
451  initialized = true;
452  }
453  }
454  break;
455  }
456  case kFLOAT: {
457  if (!initialized) {
458  elem_min.floatval = 1.0;
459  elem_max.floatval = 0.0;
460  }
461  if (array.is_null) {
462  break;
463  }
464  const float* flt_array = (float*)array.pointer;
465  for (size_t i = 0; i < array.length / sizeof(float); i++) {
466  if (flt_array[i] == NULL_FLOAT) {
467  has_nulls = true;
468  } else if (initialized) {
469  elem_min.floatval = std::min(elem_min.floatval, flt_array[i]);
470  elem_max.floatval = std::max(elem_max.floatval, flt_array[i]);
471  } else {
472  elem_min.floatval = flt_array[i];
473  elem_max.floatval = flt_array[i];
474  initialized = true;
475  }
476  }
477  break;
478  }
479  case kDOUBLE: {
480  if (!initialized) {
481  elem_min.doubleval = 1.0;
482  elem_max.doubleval = 0.0;
483  }
484  if (array.is_null) {
485  break;
486  }
487  const double* dbl_array = (double*)array.pointer;
488  for (size_t i = 0; i < array.length / sizeof(double); i++) {
489  if (dbl_array[i] == NULL_DOUBLE) {
490  has_nulls = true;
491  } else if (initialized) {
492  elem_min.doubleval = std::min(elem_min.doubleval, dbl_array[i]);
493  elem_max.doubleval = std::max(elem_max.doubleval, dbl_array[i]);
494  } else {
495  elem_min.doubleval = dbl_array[i];
496  elem_max.doubleval = dbl_array[i];
497  initialized = true;
498  }
499  }
500  break;
501  }
502  case kTIME:
503  case kTIMESTAMP:
504  case kDATE: {
505  if (!initialized) {
506  elem_min.bigintval = 1;
507  elem_max.bigintval = 0;
508  }
509  if (array.is_null) {
510  break;
511  }
512  const int64_t* tm_array = reinterpret_cast<int64_t*>(array.pointer);
513  for (size_t i = 0; i < array.length / sizeof(int64_t); i++) {
514  if (tm_array[i] == NULL_BIGINT) {
515  has_nulls = true;
516  } else if (initialized) {
517  elem_min.bigintval = std::min(elem_min.bigintval, tm_array[i]);
518  elem_max.bigintval = std::max(elem_max.bigintval, tm_array[i]);
519  } else {
520  elem_min.bigintval = tm_array[i];
521  elem_max.bigintval = tm_array[i];
522  initialized = true;
523  }
524  }
525  break;
526  }
527  case kCHAR:
528  case kVARCHAR:
529  case kTEXT: {
531  if (!initialized) {
532  elem_min.intval = 1;
533  elem_max.intval = 0;
534  }
535  if (array.is_null) {
536  break;
537  }
538  const int32_t* int_array = (int32_t*)array.pointer;
539  for (size_t i = 0; i < array.length / sizeof(int32_t); i++) {
540  if (int_array[i] == NULL_INT) {
541  has_nulls = true;
542  } else if (initialized) {
543  elem_min.intval = std::min(elem_min.intval, int_array[i]);
544  elem_max.intval = std::max(elem_max.intval, int_array[i]);
545  } else {
546  elem_min.intval = int_array[i];
547  elem_max.intval = int_array[i];
548  initialized = true;
549  }
550  }
551  break;
552  }
553  default:
554  UNREACHABLE();
555  }
556  };
int8_t tinyintval
Definition: sqltypes.h:212
HOST DEVICE SQLTypes get_subtype() const
Definition: sqltypes.h:330
#define CHECK_EQ(x, y)
Definition: Logger.h:231
#define NULL_DOUBLE
Definition: sqltypes.h:49
#define NULL_FLOAT
DecimalOverflowValidator decimal_overflow_validator_
Definition: Encoder.h:292
#define NULL_BIGINT
int8_t boolval
Definition: sqltypes.h:211
#define UNREACHABLE()
Definition: Logger.h:267
int32_t intval
Definition: sqltypes.h:214
#define NULL_INT
float floatval
Definition: sqltypes.h:216
Data_Namespace::AbstractBuffer * buffer_
Definition: Encoder.h:290
int64_t bigintval
Definition: sqltypes.h:215
int16_t smallintval
Definition: sqltypes.h:213
#define NULL_BOOLEAN
Definition: sqltypes.h:52
Definition: sqltypes.h:53
HOST DEVICE EncodingType get_compression() const
Definition: sqltypes.h:337
Definition: sqltypes.h:41
SQLTypeInfo getSqlType() const
#define NULL_TINYINT
#define NULL_SMALLINT
Definition: sqltypes.h:45
void validate(T value) const
Definition: Encoder.h:54
double doubleval
Definition: sqltypes.h:217

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void FixedLengthArrayNoneEncoder::updateMetadata ( int8_t *  array)
inline

Definition at line 238 of file FixedLengthArrayNoneEncoder.h.

References array_size, is_null(), and update_elem_stats().

238  {
240  }
std::conditional_t< is_cuda_compiler(), DeviceArrayDatum, HostArrayDatum > ArrayDatum
Definition: sqltypes.h:208
void update_elem_stats(const ArrayDatum &array)
static bool is_null(const SQLTypeInfo &type, int8_t *array)

+ Here is the call graph for this function:

void FixedLengthArrayNoneEncoder::updateStats ( const int64_t  ,
const bool   
)
inlineoverridevirtual

Implements Encoder.

Definition at line 186 of file FixedLengthArrayNoneEncoder.h.

References CHECK.

Referenced by appendData().

186 { CHECK(false); }
#define CHECK(condition)
Definition: Logger.h:223

+ Here is the caller graph for this function:

void FixedLengthArrayNoneEncoder::updateStats ( const double  ,
const bool   
)
inlineoverridevirtual

Implements Encoder.

Definition at line 188 of file FixedLengthArrayNoneEncoder.h.

References CHECK.

188 { CHECK(false); }
#define CHECK(condition)
Definition: Logger.h:223
void FixedLengthArrayNoneEncoder::updateStats ( const int8_t *const  src_data,
const size_t  num_elements 
)
inlineoverridevirtual

Update statistics for data without appending.

Parameters
src_data- the data with which to update statistics
num_elements- the number of elements to scan in the data

Implements Encoder.

Definition at line 192 of file FixedLengthArrayNoneEncoder.h.

References UNREACHABLE.

192  {
193  UNREACHABLE();
194  }
#define UNREACHABLE()
Definition: Logger.h:267
void FixedLengthArrayNoneEncoder::updateStats ( const std::vector< std::string > *const  src_data,
const size_t  start_idx,
const size_t  num_elements 
)
inlineoverridevirtual

Update statistics for string data without appending.

Parameters
src_data- the string data with which to update statistics
start_idx- the offset into src_data to start the update
num_elements- the number of elements to scan in the string data

Implements Encoder.

Definition at line 196 of file FixedLengthArrayNoneEncoder.h.

References UNREACHABLE.

198  {
199  UNREACHABLE();
200  }
#define UNREACHABLE()
Definition: Logger.h:267
void FixedLengthArrayNoneEncoder::updateStats ( const std::vector< ArrayDatum > *const  src_data,
const size_t  start_idx,
const size_t  num_elements 
)
inlineoverridevirtual

Update statistics for array data without appending.

Parameters
src_data- the array data with which to update statistics
start_idx- the offset into src_data to start the update
num_elements- the number of elements to scan in the array data

Implements Encoder.

Definition at line 202 of file FixedLengthArrayNoneEncoder.h.

References anonymous_namespace{Utm.h}::n, and update_elem_stats().

204  {
205  for (size_t n = start_idx; n < start_idx + num_elements; n++) {
206  update_elem_stats((*src_data)[n]);
207  }
208  }
void update_elem_stats(const ArrayDatum &array)
constexpr double n
Definition: Utm.h:38

+ Here is the call graph for this function:

void FixedLengthArrayNoneEncoder::writeMetadata ( FILE *  f)
inlineoverridevirtual

Implements Encoder.

Definition at line 210 of file FixedLengthArrayNoneEncoder.h.

References elem_max, elem_min, has_nulls, initialized, and Encoder::num_elems_.

210  {
211  // assumes pointer is already in right place
212  fwrite((int8_t*)&num_elems_, sizeof(size_t), 1, f);
213  fwrite((int8_t*)&elem_min, sizeof(Datum), 1, f);
214  fwrite((int8_t*)&elem_max, sizeof(Datum), 1, f);
215  fwrite((int8_t*)&has_nulls, sizeof(bool), 1, f);
216  fwrite((int8_t*)&initialized, sizeof(bool), 1, f);
217  }
size_t num_elems_
Definition: Encoder.h:288
constexpr double f
Definition: Utm.h:31

Member Data Documentation

Datum FixedLengthArrayNoneEncoder::elem_max
Datum FixedLengthArrayNoneEncoder::elem_min
std::mutex FixedLengthArrayNoneEncoder::EncoderMutex_
private

Definition at line 322 of file FixedLengthArrayNoneEncoder.h.

bool FixedLengthArrayNoneEncoder::initialized
std::mutex FixedLengthArrayNoneEncoder::print_mutex_
private

Definition at line 323 of file FixedLengthArrayNoneEncoder.h.


The documentation for this class was generated from the following file: