OmniSciDB  72c90bc290
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
FixedLengthEncoder< T, V > Class Template Reference

#include <FixedLengthEncoder.h>

+ Inheritance diagram for FixedLengthEncoder< T, V >:
+ Collaboration diagram for FixedLengthEncoder< T, V >:

Public Member Functions

 FixedLengthEncoder (Data_Namespace::AbstractBuffer *buffer)
 
size_t getNumElemsForBytesEncodedDataAtIndices (const int8_t *index_data, const std::vector< size_t > &selected_idx, const size_t byte_limit) override
 
std::shared_ptr< ChunkMetadataappendEncodedDataAtIndices (const int8_t *, int8_t *data, const std::vector< size_t > &selected_idx) override
 
std::shared_ptr< ChunkMetadataappendEncodedData (const int8_t *, int8_t *data, const size_t start_idx, const size_t num_elements) override
 
std::shared_ptr< ChunkMetadataappendData (int8_t *&src_data, const size_t num_elems_to_append, const SQLTypeInfo &ti, const bool replicating=false, const int64_t offset=-1) override
 
void getMetadata (const std::shared_ptr< ChunkMetadata > &chunkMetadata) override
 
std::shared_ptr< ChunkMetadatagetMetadata (const SQLTypeInfo &ti) override
 
void updateStats (const int64_t val, const bool is_null) override
 
void updateStats (const double val, const bool is_null) override
 
void updateStats (const int8_t *const src_data, const size_t num_elements) override
 
void updateStatsEncoded (const int8_t *const dst_data, const size_t num_elements) override
 
void updateStats (const std::vector< std::string > *const src_data, const size_t start_idx, const size_t num_elements) override
 
void updateStats (const std::vector< ArrayDatum > *const src_data, const size_t start_idx, const size_t num_elements) override
 
void reduceStats (const Encoder &that) override
 
void copyMetadata (const Encoder *copyFromEncoder) override
 
void writeMetadata (FILE *f) override
 
void readMetadata (FILE *f) override
 
bool resetChunkStats (const ChunkStats &stats) override
 : Reset chunk level stats (min, max, nulls) using new values from the argument. More...
 
void resetChunkStats () override
 
- Public Member Functions inherited from Encoder
 Encoder (Data_Namespace::AbstractBuffer *buffer)
 
virtual ~Encoder ()
 
size_t getNumElems () const
 
void setNumElems (const size_t num_elems)
 

Public Attributes

dataMin
 
dataMax
 
bool has_nulls
 

Private Member Functions

std::shared_ptr< ChunkMetadataappendEncodedOrUnencodedData (int8_t *&src_data, const size_t num_elems_to_append, const SQLTypeInfo &ti, const bool replicating, const int64_t offset, const bool is_encoded)
 
void updateStatsWithAlreadyEncoded (const V &encoded_data)
 
encodeDataAndUpdateStats (const T &unencoded_data)
 

Additional Inherited Members

- Static Public Member Functions inherited from Encoder
static EncoderCreate (Data_Namespace::AbstractBuffer *buffer, const SQLTypeInfo sqlType)
 
- Protected Attributes inherited from Encoder
size_t num_elems_
 
Data_Namespace::AbstractBufferbuffer_
 
DecimalOverflowValidator decimal_overflow_validator_
 
DateDaysOverflowValidator date_days_overflow_validator_
 

Detailed Description

template<typename T, typename V>
class FixedLengthEncoder< T, V >

Definition at line 34 of file FixedLengthEncoder.h.

Constructor & Destructor Documentation

template<typename T, typename V>
FixedLengthEncoder< T, V >::FixedLengthEncoder ( Data_Namespace::AbstractBuffer buffer)
inline

Definition at line 36 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::resetChunkStats().

36  : Encoder(buffer) {
38  }
void resetChunkStats() override
Encoder(Data_Namespace::AbstractBuffer *buffer)
Definition: Encoder.cpp:225

+ Here is the call graph for this function:

Member Function Documentation

template<typename T, typename V>
std::shared_ptr<ChunkMetadata> FixedLengthEncoder< T, V >::appendData ( int8_t *&  src_data,
const size_t  num_elems_to_append,
const SQLTypeInfo ti,
const bool  replicating = false,
const int64_t  offset = -1 
)
inlineoverridevirtual

Append data to the chunk buffer backing this encoder.

Parameters
src_dataSource data for the append
num_elems_to_appendNumber of elements to append
tiSQL Type Info for the column TODO(adb): used?
replicatingPass one value and fill the chunk with it
offsetWrite data starting at a given offset. Default is -1 which indicates an append, an offset of 0 rewrites the chunk up to num_elems_to_append.

Implements Encoder.

Definition at line 75 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::appendEncodedOrUnencodedData().

79  {
81  src_data, num_elems_to_append, ti, replicating, offset, false);
82  }
std::shared_ptr< ChunkMetadata > appendEncodedOrUnencodedData(int8_t *&src_data, const size_t num_elems_to_append, const SQLTypeInfo &ti, const bool replicating, const int64_t offset, const bool is_encoded)

+ Here is the call graph for this function:

template<typename T, typename V>
std::shared_ptr<ChunkMetadata> FixedLengthEncoder< T, V >::appendEncodedData ( const int8_t *  index_data,
int8_t *  data,
const size_t  start_idx,
const size_t  num_elements 
)
inlineoverridevirtual

Append encoded data to the chunk buffer backing this encoder.

Parameters
index_data- (optional) the index data of data to append
data- the data to append
start_idx- the position to start encoding from in the data array
num_elements- the number of elements to encode from the data array
Returns
updated chunk metadata for the chunk buffer backing this encoder

NOTE: index_data must be non-null for varlen encoder types.

Implements Encoder.

Definition at line 66 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::appendEncodedOrUnencodedData().

Referenced by FixedLengthEncoder< T, V >::appendEncodedDataAtIndices().

69  {
70  auto current_data = data + start_idx * sizeof(V);
72  current_data, num_elements, SQLTypeInfo{}, false, -1, true);
73  }
std::shared_ptr< ChunkMetadata > appendEncodedOrUnencodedData(int8_t *&src_data, const size_t num_elems_to_append, const SQLTypeInfo &ti, const bool replicating, const int64_t offset, const bool is_encoded)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

template<typename T, typename V>
std::shared_ptr<ChunkMetadata> FixedLengthEncoder< T, V >::appendEncodedDataAtIndices ( const int8_t *  index_data,
int8_t *  data,
const std::vector< size_t > &  selected_idx 
)
inlineoverridevirtual

Append selected encoded data to the chunk buffer backing this encoder.

Parameters
index_data- (optional) the index data of data to append
data- the data to append
selected_idx- which indices in the encoded data to append
Returns
updated chunk metadata for the chunk buffer backing this encoder

NOTE: index_data must be non-null for varlen encoder types.

Implements Encoder.

Definition at line 49 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::appendEncodedData(), and shared::execute_over_contiguous_indices().

52  {
53  std::shared_ptr<ChunkMetadata> chunk_metadata;
54  // NOTE: the use of `execute_over_contiguous_indices` is an optimization;
55  // it prevents having to copy or move the indexed data and instead performs
56  // an append over contiguous sections of indices.
58  selected_idx, [&](const size_t start_pos, const size_t end_pos) {
59  size_t elem_count = end_pos - start_pos;
60  chunk_metadata =
61  appendEncodedData(nullptr, data, selected_idx[start_pos], elem_count);
62  });
63  return chunk_metadata;
64  }
std::shared_ptr< ChunkMetadata > appendEncodedData(const int8_t *, int8_t *data, const size_t start_idx, const size_t num_elements) override
void execute_over_contiguous_indices(const std::vector< size_t > &indices, std::function< void(const size_t, const size_t)> to_execute)
Definition: Iteration.h:22

+ Here is the call graph for this function:

template<typename T, typename V>
std::shared_ptr<ChunkMetadata> FixedLengthEncoder< T, V >::appendEncodedOrUnencodedData ( int8_t *&  src_data,
const size_t  num_elems_to_append,
const SQLTypeInfo ti,
const bool  replicating,
const int64_t  offset,
const bool  is_encoded 
)
inlineprivate

Definition at line 226 of file FixedLengthEncoder.h.

References Data_Namespace::AbstractBuffer::append(), Encoder::buffer_, CHECK, CHECK_GE, FixedLengthEncoder< T, V >::encodeDataAndUpdateStats(), FixedLengthEncoder< T, V >::getMetadata(), Encoder::num_elems_, Data_Namespace::AbstractBuffer::reserve(), FixedLengthEncoder< T, V >::resetChunkStats(), Data_Namespace::AbstractBuffer::size(), heavydb.dtypes::T, FixedLengthEncoder< T, V >::updateStatsWithAlreadyEncoded(), and Data_Namespace::AbstractBuffer::write().

Referenced by FixedLengthEncoder< T, V >::appendData(), and FixedLengthEncoder< T, V >::appendEncodedData().

232  {
233  if (offset == 0 &&
234  num_elems_to_append >=
235  num_elems_) { // we're rewriting entire buffer so fully recompute metadata
236  resetChunkStats();
237  }
238 
239  CHECK(!is_encoded || !replicating); // do not support replicating of encoded data
240 
241  T* unencoded_data = reinterpret_cast<T*>(src_data);
242  std::vector<V> encoded_data;
243  V* data_to_write = nullptr;
244  if (!is_encoded) {
245  encoded_data.resize(num_elems_to_append);
246  data_to_write = encoded_data.data();
247  for (size_t i = 0; i < num_elems_to_append; ++i) {
248  size_t ri = replicating ? 0 : i;
249  encoded_data[i] = encodeDataAndUpdateStats(unencoded_data[ri]);
250  }
251  } else {
252  data_to_write = reinterpret_cast<V*>(src_data);
253  for (size_t i = 0; i < num_elems_to_append; ++i) {
254  updateStatsWithAlreadyEncoded(data_to_write[i]);
255  }
256  }
257 
258  // assume always CPU_BUFFER?
259  if (offset == -1) {
260  auto append_data_size = num_elems_to_append * sizeof(V);
261  buffer_->reserve(buffer_->size() + append_data_size);
262  num_elems_ += num_elems_to_append;
263  buffer_->append(reinterpret_cast<int8_t*>(data_to_write), append_data_size);
264  if (!replicating) {
265  src_data += num_elems_to_append * sizeof(T);
266  }
267  } else {
268  num_elems_ = offset + num_elems_to_append;
269  CHECK(!replicating);
270  CHECK_GE(offset, 0);
271  buffer_->write(reinterpret_cast<int8_t*>(data_to_write),
272  num_elems_to_append * sizeof(V),
273  static_cast<size_t>(offset));
274  }
275  auto chunk_metadata = std::make_shared<ChunkMetadata>();
276  getMetadata(chunk_metadata);
277  return chunk_metadata;
278  }
size_t num_elems_
Definition: Encoder.h:288
#define CHECK_GE(x, y)
Definition: Logger.h:306
void resetChunkStats() override
Data_Namespace::AbstractBuffer * buffer_
Definition: Encoder.h:290
V encodeDataAndUpdateStats(const T &unencoded_data)
void getMetadata(const std::shared_ptr< ChunkMetadata > &chunkMetadata) override
virtual void write(int8_t *src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int src_device_id=-1)=0
virtual void append(int8_t *src, const size_t num_bytes, const MemoryLevel src_buffer_type=CPU_LEVEL, const int device_id=-1)=0
#define CHECK(condition)
Definition: Logger.h:291
virtual void reserve(size_t num_bytes)=0
void updateStatsWithAlreadyEncoded(const V &encoded_data)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

template<typename T, typename V>
void FixedLengthEncoder< T, V >::copyMetadata ( const Encoder copyFromEncoder)
inlineoverridevirtual

Implements Encoder.

Definition at line 176 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, Encoder::getNumElems(), FixedLengthEncoder< T, V >::has_nulls, and Encoder::num_elems_.

176  {
177  num_elems_ = copyFromEncoder->getNumElems();
178  auto castedEncoder =
179  reinterpret_cast<const FixedLengthEncoder<T, V>*>(copyFromEncoder);
180  dataMin = castedEncoder->dataMin;
181  dataMax = castedEncoder->dataMax;
182  has_nulls = castedEncoder->has_nulls;
183  }
size_t num_elems_
Definition: Encoder.h:288
size_t getNumElems() const
Definition: Encoder.h:284

+ Here is the call graph for this function:

template<typename T, typename V>
V FixedLengthEncoder< T, V >::encodeDataAndUpdateStats ( const T &  unencoded_data)
inlineprivate

Definition at line 289 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, Encoder::decimal_overflow_validator_, logger::ERROR, FixedLengthEncoder< T, V >::has_nulls, LOG, heavydb.dtypes::T, to_string(), and DecimalOverflowValidator::validate().

Referenced by FixedLengthEncoder< T, V >::appendEncodedOrUnencodedData(), and FixedLengthEncoder< T, V >::updateStats().

289  {
290  V encoded_data = static_cast<V>(unencoded_data);
291  if (unencoded_data != encoded_data) {
292  decimal_overflow_validator_.validate(unencoded_data);
293  LOG(ERROR) << "Fixed encoding failed, Unencoded: " +
294  std::to_string(unencoded_data) +
295  " encoded: " + std::to_string(encoded_data);
296  } else {
297  T data = unencoded_data;
298  if (data == std::numeric_limits<V>::min()) {
299  has_nulls = true;
300  } else {
302  dataMin = std::min(dataMin, data);
303  dataMax = std::max(dataMax, data);
304  }
305  }
306  return encoded_data;
307  }
DecimalOverflowValidator decimal_overflow_validator_
Definition: Encoder.h:292
#define LOG(tag)
Definition: Logger.h:285
std::string to_string(char const *&&v)
void validate(T value) const
Definition: Encoder.h:54

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

template<typename T, typename V>
void FixedLengthEncoder< T, V >::getMetadata ( const std::shared_ptr< ChunkMetadata > &  chunkMetadata)
inlineoverridevirtual

Reimplemented from Encoder.

Definition at line 84 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, Encoder::getMetadata(), and FixedLengthEncoder< T, V >::has_nulls.

Referenced by FixedLengthEncoder< T, V >::appendEncodedOrUnencodedData().

84  {
85  Encoder::getMetadata(chunkMetadata); // call on parent class
86  chunkMetadata->fillChunkStats(dataMin, dataMax, has_nulls);
87  }
virtual void getMetadata(const std::shared_ptr< ChunkMetadata > &chunkMetadata)
Definition: Encoder.cpp:231

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

template<typename T, typename V>
std::shared_ptr<ChunkMetadata> FixedLengthEncoder< T, V >::getMetadata ( const SQLTypeInfo ti)
inlineoverridevirtual

Implements Encoder.

Definition at line 90 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, and FixedLengthEncoder< T, V >::has_nulls.

90  {
91  auto chunk_metadata = std::make_shared<ChunkMetadata>(ti, 0, 0, ChunkStats{});
92  chunk_metadata->fillChunkStats(dataMin, dataMax, has_nulls);
93  return chunk_metadata;
94  }
template<typename T, typename V>
size_t FixedLengthEncoder< T, V >::getNumElemsForBytesEncodedDataAtIndices ( const int8_t *  index_data,
const std::vector< size_t > &  selected_idx,
const size_t  byte_limit 
)
inlineoverridevirtual

Compute the maximum number of variable length encoded elements given a byte limit

Parameters
index_data- (optional) index data for the encoded type
selected_idx- which indices in the encoded data to consider
byte_limit- byte limit that must be respected
Returns
the number of elements

NOTE: optional parameters above may be ignored by the implementation, but may or may not be required depending on the encoder type backing the implementation.

Implements Encoder.

Definition at line 40 of file FixedLengthEncoder.h.

References UNREACHABLE.

42  {
43  UNREACHABLE()
44  << "getNumElemsForBytesEncodedDataAtIndices unexpectedly called for non varlen"
45  " encoder";
46  return {};
47  }
#define UNREACHABLE()
Definition: Logger.h:338
template<typename T, typename V>
void FixedLengthEncoder< T, V >::readMetadata ( FILE *  f)
inlineoverridevirtual

Implements Encoder.

Definition at line 193 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, FixedLengthEncoder< T, V >::has_nulls, Encoder::num_elems_, and heavydb.dtypes::T.

193  {
194  // assumes pointer is already in right place
195  fread((int8_t*)&num_elems_, sizeof(size_t), 1, f);
196  fread((int8_t*)&dataMin, 1, sizeof(T), f);
197  fread((int8_t*)&dataMax, 1, sizeof(T), f);
198  fread((int8_t*)&has_nulls, 1, sizeof(bool), f);
199  }
size_t num_elems_
Definition: Encoder.h:288
torch::Tensor f(torch::Tensor x, torch::Tensor W_target, torch::Tensor b_target)
template<typename T, typename V>
void FixedLengthEncoder< T, V >::reduceStats ( const Encoder that)
inlineoverridevirtual

Implements Encoder.

Definition at line 167 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, and FixedLengthEncoder< T, V >::has_nulls.

167  {
168  const auto that_typed = static_cast<const FixedLengthEncoder<T, V>&>(that);
169  if (that_typed.has_nulls) {
170  has_nulls = true;
171  }
172  dataMin = std::min(dataMin, that_typed.dataMin);
173  dataMax = std::max(dataMax, that_typed.dataMax);
174  }
template<typename T, typename V>
bool FixedLengthEncoder< T, V >::resetChunkStats ( const ChunkStats )
inlineoverridevirtual

: Reset chunk level stats (min, max, nulls) using new values from the argument.

Returns
: True if an update occurred and the chunk needs to be flushed. False otherwise. Default false if metadata update is unsupported. Only reset chunk stats if the incoming stats differ from the current stats.

Reimplemented from Encoder.

Definition at line 201 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, ChunkStats::has_nulls, FixedLengthEncoder< T, V >::has_nulls, ChunkStats::max, and ChunkStats::min.

201  {
202  const auto new_min = DatumFetcher::getDatumVal<T>(stats.min);
203  const auto new_max = DatumFetcher::getDatumVal<T>(stats.max);
204 
205  if (dataMin == new_min && dataMax == new_max && has_nulls == stats.has_nulls) {
206  return false;
207  }
208 
209  dataMin = new_min;
210  dataMax = new_max;
211  has_nulls = stats.has_nulls;
212  return true;
213  }
dictionary stats
Definition: report.py:116
template<typename T, typename V>
void FixedLengthEncoder< T, V >::resetChunkStats ( )
inlineoverridevirtual

Resets chunk metadata stats to their default values.

Implements Encoder.

Definition at line 215 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, and FixedLengthEncoder< T, V >::has_nulls.

Referenced by FixedLengthEncoder< T, V >::appendEncodedOrUnencodedData(), and FixedLengthEncoder< T, V >::FixedLengthEncoder().

215  {
216  dataMin = std::numeric_limits<T>::max();
217  dataMax = std::numeric_limits<T>::lowest();
218  has_nulls = false;
219  }

+ Here is the caller graph for this function:

template<typename T, typename V>
void FixedLengthEncoder< T, V >::updateStats ( const int64_t  val,
const bool  is_null 
)
inlineoverridevirtual

Implements Encoder.

Definition at line 97 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, FixedLengthEncoder< T, V >::has_nulls, and heavydb.dtypes::T.

97  {
98  if (is_null) {
99  has_nulls = true;
100  } else {
101  const auto data = static_cast<T>(val);
102  dataMin = std::min(dataMin, data);
103  dataMax = std::max(dataMax, data);
104  }
105  }
CONSTEXPR DEVICE bool is_null(const T &value)
template<typename T, typename V>
void FixedLengthEncoder< T, V >::updateStats ( const double  val,
const bool  is_null 
)
inlineoverridevirtual

Implements Encoder.

Definition at line 108 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, FixedLengthEncoder< T, V >::has_nulls, and heavydb.dtypes::T.

108  {
109  if (is_null) {
110  has_nulls = true;
111  } else {
112  const auto data = static_cast<T>(val);
113  dataMin = std::min(dataMin, data);
114  dataMax = std::max(dataMax, data);
115  }
116  }
CONSTEXPR DEVICE bool is_null(const T &value)
template<typename T, typename V>
void FixedLengthEncoder< T, V >::updateStats ( const int8_t *const  src_data,
const size_t  num_elements 
)
inlineoverridevirtual

Update statistics for data without appending.

Parameters
src_data- the data with which to update statistics
num_elements- the number of elements to scan in the data

Implements Encoder.

Definition at line 118 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::encodeDataAndUpdateStats(), and heavydb.dtypes::T.

118  {
119  const T* unencoded_data = reinterpret_cast<const T*>(src_data);
120  for (size_t i = 0; i < num_elements; ++i) {
121  encodeDataAndUpdateStats(unencoded_data[i]);
122  }
123  }
V encodeDataAndUpdateStats(const T &unencoded_data)

+ Here is the call graph for this function:

template<typename T, typename V>
void FixedLengthEncoder< T, V >::updateStats ( const std::vector< std::string > *const  src_data,
const size_t  start_idx,
const size_t  num_elements 
)
inlineoverridevirtual

Update statistics for string data without appending.

Parameters
src_data- the string data with which to update statistics
start_idx- the offset into src_data to start the update
num_elements- the number of elements to scan in the string data

Implements Encoder.

Definition at line 154 of file FixedLengthEncoder.h.

References UNREACHABLE.

156  {
157  UNREACHABLE();
158  }
#define UNREACHABLE()
Definition: Logger.h:338
template<typename T, typename V>
void FixedLengthEncoder< T, V >::updateStats ( const std::vector< ArrayDatum > *const  src_data,
const size_t  start_idx,
const size_t  num_elements 
)
inlineoverridevirtual

Update statistics for array data without appending.

Parameters
src_data- the array data with which to update statistics
start_idx- the offset into src_data to start the update
num_elements- the number of elements to scan in the array data

Implements Encoder.

Definition at line 160 of file FixedLengthEncoder.h.

References UNREACHABLE.

162  {
163  UNREACHABLE();
164  }
#define UNREACHABLE()
Definition: Logger.h:338
template<typename T, typename V>
void FixedLengthEncoder< T, V >::updateStatsEncoded ( const int8_t *const  dst_data,
const size_t  num_elements 
)
inlineoverridevirtual

Update statistics for encoded data without appending.

Parameters
dst_data- the data with which to update statistics
num_elements- the number of elements to scan in the data

Reimplemented from Encoder.

Definition at line 125 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, Encoder::decimal_overflow_validator_, FixedLengthEncoder< T, V >::has_nulls, logger::init(), threading_serial::parallel_reduce(), and DecimalOverflowValidator::validate().

126  {
127  const V* data = reinterpret_cast<const V*>(dst_data);
128 
130  tbb::blocked_range(size_t(0), num_elements),
131  std::tuple(static_cast<V>(dataMin), static_cast<V>(dataMax), has_nulls),
132  [&](const auto& range, auto init) {
133  auto [min, max, nulls] = init;
134  for (size_t i = range.begin(); i < range.end(); i++) {
135  if (data[i] != std::numeric_limits<V>::min()) {
137  min = std::min(min, data[i]);
138  max = std::max(max, data[i]);
139  } else {
140  nulls = true;
141  }
142  }
143  return std::tuple(min, max, nulls);
144  },
145  [&](auto lhs, auto rhs) {
146  const auto [lhs_min, lhs_max, lhs_nulls] = lhs;
147  const auto [rhs_min, rhs_max, rhs_nulls] = rhs;
148  return std::tuple(std::min(lhs_min, rhs_min),
149  std::max(lhs_max, rhs_max),
150  lhs_nulls || rhs_nulls);
151  });
152  }
DecimalOverflowValidator decimal_overflow_validator_
Definition: Encoder.h:292
void init(LogOptions const &log_opts)
Definition: Logger.cpp:364
Value parallel_reduce(const blocked_range< Int > &range, const Value &identity, const RealBody &real_body, const Reduction &reduction, const Partitioner &p=Partitioner())
Parallel iteration with reduction.
void validate(T value) const
Definition: Encoder.h:54

+ Here is the call graph for this function:

template<typename T, typename V>
void FixedLengthEncoder< T, V >::updateStatsWithAlreadyEncoded ( const V &  encoded_data)
inlineprivate

Definition at line 280 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, and FixedLengthEncoder< T, V >::has_nulls.

Referenced by FixedLengthEncoder< T, V >::appendEncodedOrUnencodedData().

280  {
281  if (encoded_data == std::numeric_limits<V>::min()) {
282  has_nulls = true;
283  } else {
284  dataMin = std::min<T>(dataMin, encoded_data);
285  dataMax = std::max<T>(dataMax, encoded_data);
286  }
287  }

+ Here is the caller graph for this function:

template<typename T, typename V>
void FixedLengthEncoder< T, V >::writeMetadata ( FILE *  f)
inlineoverridevirtual

Implements Encoder.

Definition at line 185 of file FixedLengthEncoder.h.

References FixedLengthEncoder< T, V >::dataMax, FixedLengthEncoder< T, V >::dataMin, FixedLengthEncoder< T, V >::has_nulls, Encoder::num_elems_, and heavydb.dtypes::T.

185  {
186  // assumes pointer is already in right place
187  fwrite((int8_t*)&num_elems_, sizeof(size_t), 1, f);
188  fwrite((int8_t*)&dataMin, sizeof(T), 1, f);
189  fwrite((int8_t*)&dataMax, sizeof(T), 1, f);
190  fwrite((int8_t*)&has_nulls, sizeof(bool), 1, f);
191  }
size_t num_elems_
Definition: Encoder.h:288
torch::Tensor f(torch::Tensor x, torch::Tensor W_target, torch::Tensor b_target)

Member Data Documentation


The documentation for this class was generated from the following file: