OmniSciDB  c0231cc57d
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
FlatBuffer.h
Go to the documentation of this file.
1 #pragma once
2 
3 /*
4  * Copyright 2022 HEAVY.AI, Inc.
5  *
6  * Licensed under the Apache License, Version 2.0 (the "License");
7  * you may not use this file except in compliance with the License.
8  * You may obtain a copy of the License at
9  *
10  * http://www.apache.org/licenses/LICENSE-2.0
11  *
12  * Unless required by applicable law or agreed to in writing, software
13  * distributed under the License is distributed on an "AS IS" BASIS,
14  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15  * See the License for the specific language governing permissions and
16  * limitations under the License.
17  */
18 
19 // clang-format off
20 /*
21  FlatBufferManager provides a storage for a collection of buffers
22  (columns of arrays, columns of strings, etc) that are collected into
23  a single flat buffer so that copying FlatBuffer instances becomes a
24  single buffer copy operation. Flat buffers that store no pointer
25  values can be straightforwardly copied in between different devices.
26 
27  FlatBuffer memory layout specification
28  --------------------------------------
29 
30  The first 8 bytes of the buffer contains a FlatBuffer storage format
31  id (see FlatBufferFormat below) that will determine how the rest of
32  the bytes in the flat buffer will be interpreted. The next 8 bytes
33  of the buffer contains the total size of the flat buffer --- this
34  allows flat buffers passed around by a single pointer value without
35  explicitly specifying the size of the buffer.
36 
37  The memory layout of a flatbuffer is (using units of 8 byte):
38 
39  | <format id> | <flatbuffer size> | <data> ... | <format id> |
40  =
41  |<-- 8-bytes-->|<-- 8-bytes ------>|<-- flatbuffer size minus 24 bytes -->|<-- 8-bytes -->|
42  |<------------------------ flatbuffer size ---------------------------------------------->|
43 
44  where flatbuffer size is specified in bytes.
45 
46  VarlenArray format specification
47  --------------------------------
48 
49  If a FlatBuffer instance uses VarlenArray format (format id is
50  0x766c61) then the <data> part of the FlatBuffer is defined as follows:
51 
52  | <items count> | <dtype metadata kind/buffer size> | <max nof values> | <num specified items> | <offsets> | <varlen array data> |
53  =
54  |<--- 8-bytes -->|<--16-bytes ---------------------->|<-- 8-bytes ----->|<-- 8-bytes ---------->|<--30-bytes-->|<--flatbuffer size minus 96 bytes-->|
55  |<------------------------- flatbuffer size minus 24 bytes ---------------------------------------------------------------------------------------->|
56 
57  where
58 
59  <items count> is the number of items (e.g. varlen arrays) that the
60  flat buffer instance is holding. In the context of columns of
61  arrays, the items count is the number of rows in a column. This
62  is a user-specified parameter.
63 
64  <dtype metadata kind> defines the set of parameters that the dtype
65  of a value depends on. For instance, dtype typically is
66  characterized by its byte size. In the case of text encoding
67  dict, dtype also is parameterized by the string dictionary id.
68 
69  <dtype metadata buffer size> is the byte size of dtype metadata
70  buffer.
71 
72  <max nof values> is the maximum total number of elements in all
73  items that the flat buffer instance can hold. The value defines
74  the size of values buffer. User-specified parameter.
75 
76  <num specified items> is the number of items that has initial
77  value 0 and that is incremented by one on each setItem or
78  setNull call. The flat buffer is completely filled with items
79  when <num specified items> becomes equal to <items count>. Used
80  internally.
81 
82  <offsets> are precomputed offsets of data buffers. Used internally.
83 
84  | <dtype metadata offset> | <values offset> | <compressed_indices offset> | <storage indices offset> |
85  |<-- 8-bytes ------------>|<-- 8-bytes ---->|<-- 8-bytes ---------------->|<-- 8-bytes ------------->|
86 
87  <varlen array data> is
88 
89  | <dtype metadata buffer> | <values> | <compressed indices> | <storage indices> |
90  =
91  |<-- dtype metadata buffer size ->|<-- (max nof values) * 8 bytes -->|<-- (num items + 1) * 8 bytes-->|<-- (num items) * 8 bytes-->|
92  |<------------------------ flatbuffer size minus 88 bytes ------------------------------------------------------------------------>|
93 
94  and
95 
96  - values stores the elements of all items (e.g. the values of all
97  varlen arrays). Item elements are contiguous within an item,
98  however, the storage order of items can be arbitrary.
99 
100  - compressed indices contains the "cumulative sum" of storage
101  indices. Negative entries indicate null items.
102 
103  - storage indices defines the order of specifying items in the flat
104  buffer.
105 
106  For the detailed description of values, compressed_indices, and
107  storage_indices, as well as how empty arrays and null arrays are
108  represented, see https://pearu.github.io/variable_length_arrays.html .
109 
110  FlatBuffer usage
111  ----------------
112 
113  FlatBuffer implements various methods for accessing its content for
114  retriving or storing data. These methods usually are provided as
115  pairs of safe and unsafe methods. Safe methods validate method
116  inputs and return success or error status depending on the
117  validation results. Unsafe methods (that names have "NoValidation"
118  suffix) performs no validation on the inputs and (almost) always
119  return the success status. Use unsafe methods (for efficency) only
120  when one can prove that the inputs are always valid (e.g. indices
121  are in the expected range, buffer memory allocation is sufficient
122  for storing data, etc). Otherwise, use safe methods that will lead
123  to predictable and non-server-crashing behaviour in the case of
124  possibly invalid input data.
125 */
126 // clang-format on
127 
128 #ifdef HAVE_TOSTRING
129 #include <ostream>
130 #endif
131 #include <string.h>
132 
133 // Notice that the format value is used to recognize if a memory
134 // buffer uses some flat buffer format or not. To minimize chances for
135 // false positive test results, use a non-trival integer value when
136 // introducing new formats.
138  VarlenArray = 0x7661726c65634152, // hex repr of 'varlenAR'
139 };
140 
141 inline int64_t _align_to_int64(int64_t addr) {
142  addr += sizeof(int64_t) - 1;
143  return (int64_t)(((uint64_t)addr >> 3) << 3);
144 }
145 
147  enum Status {
148  Success = 0,
157  };
158 
160  FORMAT_ID = 0, // storage format id
161  FLATBUFFER_SIZE, // in bytes
162  ITEMS_COUNT, /* the number of items
163  (e.g. the number of
164  rows in a column) */
165  DTYPE_METADATA_KIND, /* the kind of dtype metadata */
166  DTYPE_METADATA_BUFFER_SIZE, /* the size of dtype metadata buffer */
167  MAX_NOF_VALUES, /* the upper bound to the
168  total number of values
169  in all items */
170  STORAGE_COUNT, /* the number of specified
171  items, incremented by
172  one on each
173  setNull/setItem call */
179  };
180 
181  int8_t* buffer;
182 
183  // Check if a buffer contains FlatBuffer formatted data
184  static bool isFlatBuffer(const void* buffer) {
185  if (buffer) {
186  // warning: assume that buffer size is at least 8 bytes
187  FlatBufferFormat header_format =
188  static_cast<FlatBufferFormat>(((int64_t*)buffer)[VarlenArrayHeader::FORMAT_ID]);
189  if (header_format == FlatBufferFormat::VarlenArray) {
190  int64_t flatbuffer_size = ((int64_t*)buffer)[VarlenArrayHeader::FLATBUFFER_SIZE];
191  if (flatbuffer_size > 0) {
192  FlatBufferFormat footer_format = static_cast<FlatBufferFormat>(
193  ((int64_t*)buffer)[flatbuffer_size / sizeof(int64_t) - 1]);
194  return footer_format == header_format;
195  }
196  }
197  }
198  return false;
199  }
200 
201  // Return the format of FlatBuffer
202  inline FlatBufferFormat format() const {
203  return static_cast<FlatBufferFormat>(
204  ((int64_t*)buffer)[VarlenArrayHeader::FORMAT_ID]);
205  }
206 
207  // Return the allocation size of the the FlatBuffer storage, in bytes
208  inline int64_t flatbufferSize() const {
209  return ((int64_t*)buffer)[VarlenArrayHeader::FLATBUFFER_SIZE];
210  }
211 
212  /* DType MetaData support */
213 
215  SIZE = 1,
217  };
218 
220  int64_t size;
221  };
222 
224  int64_t size;
225  int32_t dict_id;
226  };
227 
229  switch (kind) {
230  case DTypeMetadataKind::SIZE:
231  return _align_to_int64(sizeof(DTypeMetadataSize));
232  case DTypeMetadataKind::SIZE_DICTID:
234  default:
235  return 0;
236  }
237  }
238 
240  return static_cast<DTypeMetadataKind>(
241  ((int64_t*)buffer)[VarlenArrayHeader::DTYPE_METADATA_KIND]);
242  }
243 
244  inline int8_t* getDTypeMetadataBuffer() {
245  return buffer + ((int64_t*)buffer)[VarlenArrayHeader::DTYPE_METADATA_BUFFER_OFFSET];
246  }
247 
248  inline const int8_t* getDTypeMetadataBuffer() const {
249  return buffer +
250  ((const int64_t*)buffer)[VarlenArrayHeader::DTYPE_METADATA_BUFFER_OFFSET];
251  }
252 
253  // dtype size accessors:
254  void setDTypeMetadataSize(int64_t size) {
255  switch (getDTypeMetadataKind()) {
256  case DTypeMetadataKind::SIZE: {
257  DTypeMetadataSize* metadata =
258  reinterpret_cast<DTypeMetadataSize*>(getDTypeMetadataBuffer());
259  metadata->size = size;
260  } break;
261  case DTypeMetadataKind::SIZE_DICTID: {
262  DTypeMetadataSizeDictId* metadata =
263  reinterpret_cast<DTypeMetadataSizeDictId*>(getDTypeMetadataBuffer());
264  metadata->size = size;
265  } break;
266  default:;
267  }
268  }
269 
270  int64_t getDTypeMetadataSize() const {
271  switch (getDTypeMetadataKind()) {
272  case DTypeMetadataKind::SIZE: {
273  const DTypeMetadataSize* metadata =
274  reinterpret_cast<const DTypeMetadataSize*>(getDTypeMetadataBuffer());
275  return metadata->size;
276  }
277  case DTypeMetadataKind::SIZE_DICTID: {
278  const DTypeMetadataSizeDictId* metadata =
279  reinterpret_cast<const DTypeMetadataSizeDictId*>(getDTypeMetadataBuffer());
280  return metadata->size;
281  }
282  default:;
283  }
284  return 0;
285  }
286 
287  // dtype dict id accessors:
288  void setDTypeMetadataDictId(int32_t dict_id) {
289  switch (getDTypeMetadataKind()) {
290  case DTypeMetadataKind::SIZE:
291  break;
292  case DTypeMetadataKind::SIZE_DICTID: {
293  DTypeMetadataSizeDictId* metadata =
294  reinterpret_cast<DTypeMetadataSizeDictId*>(getDTypeMetadataBuffer());
295  metadata->dict_id = dict_id;
296  } break;
297  default:;
298  }
299  }
300 
301  int32_t getDTypeMetadataDictId() const {
302  switch (getDTypeMetadataKind()) {
303  case DTypeMetadataKind::SIZE:
304  break;
305  case DTypeMetadataKind::SIZE_DICTID: {
306  const DTypeMetadataSizeDictId* metadata =
307  reinterpret_cast<const DTypeMetadataSizeDictId*>(getDTypeMetadataBuffer());
308  return metadata->dict_id;
309  }
310  default:;
311  }
312  return 0;
313  }
314 
315  // VarlenArray support:
316  static int64_t get_VarlenArray_flatbuffer_size(int64_t items_count,
317  int64_t max_nof_values,
318  int64_t dtype_size,
319  DTypeMetadataKind dtype_metadata_kind) {
320  const int64_t VarlenArray_buffer_header_size =
321  VarlenArrayHeader::EOFHEADER * sizeof(int64_t);
322  const int64_t dtype_metadata_buffer_size =
323  getDTypeMetadataBufferSize(dtype_metadata_kind);
324  const int64_t values_buffer_size = _align_to_int64(dtype_size * max_nof_values);
325  return (VarlenArray_buffer_header_size // see above
326  + dtype_metadata_buffer_size // size of dtype metadata buffer in bytes,
327  // aligned to int64
328  + values_buffer_size // size of values buffer, aligned to int64
329  + (items_count + 1 // size of compressed_indices buffer
330  + items_count // size of storage_indices buffer
331  + 1 // footer format id
332  ) * sizeof(int64_t));
333  }
334 
335  // Initialize FlatBuffer for VarlenArray storage
336  void initializeVarlenArray(int64_t items_count,
337  int64_t max_nof_values,
338  int64_t dtype_size,
339  DTypeMetadataKind dtype_metadata_kind) {
340  const int64_t VarlenArray_buffer_header_size =
341  VarlenArrayHeader::EOFHEADER * sizeof(int64_t);
342  const int64_t values_buffer_size = _align_to_int64(dtype_size * max_nof_values);
343  const int64_t compressed_indices_buffer_size = (items_count + 1) * sizeof(int64_t);
344  const int64_t dtype_metadata_buffer_size =
345  getDTypeMetadataBufferSize(dtype_metadata_kind);
346  const int64_t flatbuffer_size = get_VarlenArray_flatbuffer_size(
347  items_count, max_nof_values, dtype_size, dtype_metadata_kind);
348  ((int64_t*)buffer)[VarlenArrayHeader::FORMAT_ID] =
349  static_cast<int64_t>(FlatBufferFormat::VarlenArray);
350  ((int64_t*)buffer)[VarlenArrayHeader::FLATBUFFER_SIZE] = flatbuffer_size;
351  ((int64_t*)buffer)[VarlenArrayHeader::ITEMS_COUNT] = items_count;
352  ((int64_t*)buffer)[VarlenArrayHeader::MAX_NOF_VALUES] = max_nof_values;
353  ((int64_t*)buffer)[VarlenArrayHeader::DTYPE_METADATA_KIND] =
354  static_cast<int64_t>(dtype_metadata_kind);
355  ((int64_t*)buffer)[VarlenArrayHeader::DTYPE_METADATA_BUFFER_SIZE] =
356  dtype_metadata_buffer_size;
357  ((int64_t*)buffer)[VarlenArrayHeader::STORAGE_COUNT] = 0;
358  ((int64_t*)buffer)[VarlenArrayHeader::DTYPE_METADATA_BUFFER_OFFSET] =
359  VarlenArray_buffer_header_size;
360  ((int64_t*)buffer)[VarlenArrayHeader::VALUES_OFFSET] =
361  ((const int64_t*)buffer)[VarlenArrayHeader::DTYPE_METADATA_BUFFER_OFFSET] +
362  dtype_metadata_buffer_size;
363  ((int64_t*)buffer)[VarlenArrayHeader::COMPRESSED_INDICES_OFFSET] =
364  ((const int64_t*)buffer)[VarlenArrayHeader::VALUES_OFFSET] + values_buffer_size;
365  ((int64_t*)buffer)[VarlenArrayHeader::STORAGE_INDICES_OFFSET] =
366  ((const int64_t*)buffer)[VarlenArrayHeader::COMPRESSED_INDICES_OFFSET] +
367  compressed_indices_buffer_size;
368 
369  setDTypeMetadataSize(dtype_size);
370 
371  // initialize indices buffers
372  int64_t* compressed_indices = VarlenArray_compressed_indices();
373  int64_t* storage_indices = VarlenArray_storage_indices();
374  for (int i = 0; i < items_count; i++) {
375  compressed_indices[i] = 0;
376  storage_indices[i] = -1;
377  }
378  compressed_indices[items_count] = 0;
379  // store footer format id in the last 8-bytes of the flatbuffer:
380  ((int64_t*)buffer)[flatbuffer_size / sizeof(int64_t) - 1] =
381  static_cast<int64_t>(FlatBufferFormat::VarlenArray);
382  }
383 
384  // Return the number of items
385  inline int64_t itemsCount() const {
387  return ((int64_t*)buffer)[VarlenArrayHeader::ITEMS_COUNT];
388  }
389  return -1; // invalid value
390  }
391 
392  // Return the size of dtype of the item elements, in bytes
393  inline int64_t dtypeSize() const { return getDTypeMetadataSize(); }
394 
395  // Return the upper bound to the total number of values in all items
396  inline int64_t VarlenArray_max_nof_values() const {
397  return ((int64_t*)buffer)[VarlenArrayHeader::MAX_NOF_VALUES];
398  }
399 
400  // Return the size of values buffer in bytes
401  inline int64_t VarlenArray_values_buffer_size() const {
403  }
404 
405  // Return the number of specified items
406  inline int64_t& VarlenArray_storage_count() {
407  return ((int64_t*)buffer)[VarlenArrayHeader::STORAGE_COUNT];
408  }
409 
410  // Return the pointer to values buffer
411  inline int8_t* VarlenArray_values() {
412  return buffer + ((const int64_t*)buffer)[VarlenArrayHeader::VALUES_OFFSET];
413  }
414 
415  inline const int8_t* VarlenArray_values() const {
416  return buffer + ((const int64_t*)buffer)[VarlenArrayHeader::VALUES_OFFSET];
417  }
418 
419  // Return the pointer to compressed indices buffer
420  inline int64_t* VarlenArray_compressed_indices() {
421  return reinterpret_cast<int64_t*>(
422  buffer + ((const int64_t*)buffer)[VarlenArrayHeader::COMPRESSED_INDICES_OFFSET]);
423  }
424  inline const int64_t* VarlenArray_compressed_indices() const {
425  return reinterpret_cast<const int64_t*>(
426  buffer + ((const int64_t*)buffer)[VarlenArrayHeader::COMPRESSED_INDICES_OFFSET]);
427  }
428 
429  // Return the pointer to storage indices buffer
430  inline int64_t* VarlenArray_storage_indices() {
431  return reinterpret_cast<int64_t*>(
432  buffer + ((const int64_t*)buffer)[VarlenArrayHeader::STORAGE_INDICES_OFFSET]);
433  }
434  inline const int64_t* VarlenArray_storage_indices() const {
435  return reinterpret_cast<const int64_t*>(
436  buffer + ((const int64_t*)buffer)[VarlenArrayHeader::STORAGE_INDICES_OFFSET]);
437  }
438 
439  // Set a new item with index and size (in bytes) and initialize its
440  // elements from source buffer. The item values will be
441  // uninitialized when source buffer is nullptr. If dest != nullptr
442  // then the item's buffer pointer will be stored in *dest.
443  Status setItem(int64_t index,
444  const int8_t* src,
445  int64_t size,
446  int8_t** dest = nullptr) {
448  if (index < 0 || index >= itemsCount()) {
449  return IndexError;
450  }
451  int64_t& storage_count = VarlenArray_storage_count();
452  int64_t* compressed_indices = VarlenArray_compressed_indices();
453  int64_t* storage_indices = VarlenArray_storage_indices();
454  const int64_t itemsize = dtypeSize();
455  if (size % itemsize != 0) {
456  return SizeError; // size must be multiple of itemsize. Perhaps size is not in
457  // bytes?
458  }
459  if (storage_indices[index] >= 0) {
461  }
462  const int64_t cindex = compressed_indices[storage_count];
463  const int64_t values_buffer_size = VarlenArray_values_buffer_size();
464  const int64_t csize = cindex * itemsize;
465  if (csize + size > values_buffer_size) {
467  }
468  return setItemNoValidation(index, src, size, dest);
469  }
470  return UnknownFormatError;
471  }
472 
473  // Same as setItem but performs no input validation
475  const int8_t* src,
476  int64_t size,
477  int8_t** dest) {
478  int64_t& storage_count = VarlenArray_storage_count();
479  int64_t* storage_indices = VarlenArray_storage_indices();
480  int64_t* compressed_indices = VarlenArray_compressed_indices();
481  int8_t* values = VarlenArray_values();
482  const int64_t itemsize = dtypeSize();
483  const int64_t values_count = size / itemsize;
484  const int64_t cindex = compressed_indices[storage_count];
485  const int64_t csize = cindex * itemsize;
486  storage_indices[index] = storage_count;
487  compressed_indices[storage_count + 1] = cindex + values_count;
488  if (size > 0 && src != nullptr && memcpy(values + csize, src, size) == nullptr) {
489  return MemoryError;
490  }
491  if (dest != nullptr) {
492  *dest = values + csize;
493  }
494  storage_count++;
495  return Success;
496  }
497 
498  // Set a new item with index and size but without initializing item
499  // elements. The buffer pointer of the new item will be stored in
500  // *dest if dest != nullptr. Inputs are not validated!
501  Status setEmptyItemNoValidation(int64_t index, int64_t size, int8_t** dest) {
502  return setItemNoValidation(index, nullptr, size, dest);
503  }
504 
505  Status concatItem(int64_t index, const int8_t* src, int64_t size) {
507  if (index < 0 || index >= itemsCount()) {
508  return IndexError;
509  }
510  int64_t next_storage_count = VarlenArray_storage_count();
511  int64_t storage_count = next_storage_count - 1;
512  int64_t* compressed_indices = VarlenArray_compressed_indices();
513  int64_t* storage_indices = VarlenArray_storage_indices();
514  int8_t* values = VarlenArray_values();
515  int64_t itemsize = dtypeSize();
516  int64_t storage_index = storage_indices[index];
517 
518  if (storage_index == -1) { // unspecified, so setting the item
519  return setItem(index, src, size, nullptr);
520  }
521  if (size % itemsize != 0) {
522  return SizeError;
523  }
524  if (storage_index != storage_count) {
525  return IndexError; // index does not correspond to the last set
526  // item, only the last item can be
527  // concatenated
528  }
529  if (compressed_indices[storage_index] < 0) {
530  return NotImplemnentedError; // todo: support concat to null when last
531  }
532  int64_t values_count =
533  compressed_indices[next_storage_count] - compressed_indices[storage_index];
534  int64_t extra_values_count = size / itemsize;
535  compressed_indices[next_storage_count] += extra_values_count;
536  int8_t* ptr = values + compressed_indices[storage_index] * itemsize;
537  if (size > 0 && src != nullptr &&
538  memcpy(ptr + values_count * itemsize, src, size) == nullptr) {
539  return MemoryError;
540  }
541  return Success;
542  }
543  return UnknownFormatError;
544  }
545 
546  // Set item with index as a null item
547  Status setNull(int64_t index) {
549  if (index < 0 || index >= itemsCount()) {
550  return IndexError;
551  }
552  int64_t* storage_indices = VarlenArray_storage_indices();
553  if (storage_indices[index] >= 0) {
555  }
556  return setNullNoValidation(index);
557  }
558  return UnknownFormatError;
559  }
560 
561  // Same as setNull but performs no input validation
562  Status setNullNoValidation(int64_t index) {
563  int64_t& storage_count = VarlenArray_storage_count();
564  int64_t* storage_indices = VarlenArray_storage_indices();
565  int64_t* compressed_indices = VarlenArray_compressed_indices();
566  const int64_t cindex = compressed_indices[storage_count];
567  storage_indices[index] = storage_count;
568  compressed_indices[storage_count] = -(cindex + 1);
569  compressed_indices[storage_count + 1] = cindex;
570  storage_count++;
571  return Success;
572  }
573 
574  // Check if the item is unspecified or null.
575  Status isNull(int64_t index, bool& is_null) const {
577  if (index < 0 || index >= itemsCount()) {
578  return IndexError;
579  }
580  const int64_t* compressed_indices = VarlenArray_compressed_indices();
581  const int64_t* storage_indices = VarlenArray_storage_indices();
582  const int64_t storage_index = storage_indices[index];
583  if (storage_index < 0) {
584  return ItemUnspecifiedError;
585  }
586  is_null = (compressed_indices[storage_index] < 0);
587  return Success;
588  }
589  return UnknownFormatError;
590  }
591 
592  // Get item at index by storing its size (in bytes), values buffer,
593  // and nullity information to the corresponding pointer
594  // arguments.
595  Status getItem(int64_t index, int64_t& size, int8_t*& dest, bool& is_null) {
597  if (index < 0 || index >= itemsCount()) {
598  return IndexError;
599  }
600  int8_t* values = VarlenArray_values();
601  const int64_t* compressed_indices = VarlenArray_compressed_indices();
602  const int64_t* storage_indices = VarlenArray_storage_indices();
603  const int64_t storage_index = storage_indices[index];
604  if (storage_index < 0) {
605  return ItemUnspecifiedError;
606  }
607  const int64_t cindex = compressed_indices[storage_index];
608  if (cindex < 0) {
609  // null varlen array
610  size = 0;
611  dest = nullptr;
612  is_null = true;
613  } else {
614  const int64_t dtypesize = dtypeSize();
615  const int64_t next_cindex = compressed_indices[storage_index + 1];
616  const int64_t length =
617  (next_cindex < 0 ? -(next_cindex + 1) - cindex : next_cindex - cindex);
618  size = length * dtypesize;
619  dest = values + cindex * dtypesize;
620  is_null = false;
621  }
622  return Success;
623  } else {
624  return UnknownFormatError;
625  }
626  }
627 
628 #ifdef HAVE_TOSTRING
629  std::string toString() const {
630  if (buffer == nullptr) {
631  return ::typeName(this) + "[UNINITIALIZED]";
632  }
633  switch (format()) {
635  const int64_t* buf = reinterpret_cast<const int64_t*>(buffer);
636  std::vector<int64_t> v(buf, buf + flatbufferSize() / sizeof(int64_t));
637  return ::typeName(this) + ::toString(v);
638  }
639  default:;
640  }
641  return ::typeName(this) + "[UNKNOWN FORMAT]";
642  }
643 #endif
644 };
645 
646 #ifdef HAVE_TOSTRING
647 inline std::ostream& operator<<(std::ostream& os,
648  FlatBufferManager::Status const status) {
649  switch (status) {
651  os << "Success";
652  break;
654  os << "IndexError";
655  break;
657  os << "SizeError";
658  break;
660  os << "ItemAlreadySpecifiedError";
661  break;
663  os << "ItemUnspecifiedError";
664  break;
666  os << "ValuesBufferTooSmallError";
667  break;
669  os << "MemoryError";
670  break;
672  os << "UnknownFormatError";
673  break;
675  os << "NotImplemnentedError";
676  break;
677  default:
678  os << "[Unknown FlatBufferManager::Status value]";
679  }
680  return os;
681 }
682 
683 inline std::string toString(const FlatBufferManager::Status& status) {
684  std::ostringstream ss;
685  ss << status;
686  return ss.str();
687 }
688 #endif
void initializeVarlenArray(int64_t items_count, int64_t max_nof_values, int64_t dtype_size, DTypeMetadataKind dtype_metadata_kind)
Definition: FlatBuffer.h:336
int32_t getDTypeMetadataDictId() const
Definition: FlatBuffer.h:301
Status isNull(int64_t index, bool &is_null) const
Definition: FlatBuffer.h:575
Status concatItem(int64_t index, const int8_t *src, int64_t size)
Definition: FlatBuffer.h:505
std::ostream & operator<<(std::ostream &os, const SessionInfo &session_info)
Definition: SessionInfo.cpp:57
int64_t dtypeSize() const
Definition: FlatBuffer.h:393
Status setItem(int64_t index, const int8_t *src, int64_t size, int8_t **dest=nullptr)
Definition: FlatBuffer.h:443
static int64_t get_VarlenArray_flatbuffer_size(int64_t items_count, int64_t max_nof_values, int64_t dtype_size, DTypeMetadataKind dtype_metadata_kind)
Definition: FlatBuffer.h:316
DTypeMetadataKind getDTypeMetadataKind() const
Definition: FlatBuffer.h:239
int64_t getDTypeMetadataSize() const
Definition: FlatBuffer.h:270
std::string toString(const QueryDescriptionType &type)
Definition: Types.h:64
const int64_t * VarlenArray_storage_indices() const
Definition: FlatBuffer.h:434
int8_t * VarlenArray_values()
Definition: FlatBuffer.h:411
int64_t & VarlenArray_storage_count()
Definition: FlatBuffer.h:406
CONSTEXPR DEVICE bool is_null(const T &value)
const int64_t * VarlenArray_compressed_indices() const
Definition: FlatBuffer.h:424
static bool isFlatBuffer(const void *buffer)
Definition: FlatBuffer.h:184
int64_t VarlenArray_max_nof_values() const
Definition: FlatBuffer.h:396
int64_t VarlenArray_values_buffer_size() const
Definition: FlatBuffer.h:401
FlatBufferFormat format() const
Definition: FlatBuffer.h:202
Status getItem(int64_t index, int64_t &size, int8_t *&dest, bool &is_null)
Definition: FlatBuffer.h:595
int8_t * getDTypeMetadataBuffer()
Definition: FlatBuffer.h:244
Status setNull(int64_t index)
Definition: FlatBuffer.h:547
Status setEmptyItemNoValidation(int64_t index, int64_t size, int8_t **dest)
Definition: FlatBuffer.h:501
std::string typeName(const T *v)
Definition: toString.h:103
int64_t flatbufferSize() const
Definition: FlatBuffer.h:208
void setDTypeMetadataDictId(int32_t dict_id)
Definition: FlatBuffer.h:288
int64_t * VarlenArray_storage_indices()
Definition: FlatBuffer.h:430
int64_t _align_to_int64(int64_t addr)
Definition: FlatBuffer.h:141
int64_t itemsCount() const
Definition: FlatBuffer.h:385
const int8_t * getDTypeMetadataBuffer() const
Definition: FlatBuffer.h:248
int64_t * VarlenArray_compressed_indices()
Definition: FlatBuffer.h:420
static int64_t getDTypeMetadataBufferSize(DTypeMetadataKind kind)
Definition: FlatBuffer.h:228
Status setNullNoValidation(int64_t index)
Definition: FlatBuffer.h:562
void setDTypeMetadataSize(int64_t size)
Definition: FlatBuffer.h:254
FlatBufferFormat
Definition: FlatBuffer.h:137
const int8_t * VarlenArray_values() const
Definition: FlatBuffer.h:415
Status setItemNoValidation(int64_t index, const int8_t *src, int64_t size, int8_t **dest)
Definition: FlatBuffer.h:474