OmniSciDB  a987f07e93
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
foreign_storage::anonymous_namespace{FileReader.cpp} Namespace Reference


void adjust_eof (size_t &read_size, const size_t buffer_size, char *buffer, const char line_delim)
size_t offset_to_index (const std::vector< size_t > &cumulative_sizes, size_t byte_offset)
size_t get_data_size (size_t file_size, size_t header_size)

Function Documentation

void foreign_storage::anonymous_namespace{FileReader.cpp}::adjust_eof ( size_t &  read_size,
const size_t  buffer_size,
char *  buffer,
const char  line_delim 

Adds an end of line character (specified by the line_delim parameter) to provided buffer, if this is the last read buffer and if the buffer does not already end with an end of line character. This allows for appropriate parsing by the file_buffer_parser utility functions, which expect the end of rows to be indicated by end of line characters in the buffer. Also removes extra EOL that may be inserted at the EOF that will not be present if the file is appended to

Definition at line 37 of file FileReader.cpp.

References CHECK.

Referenced by foreign_storage::MultiFileReader::read(), and foreign_storage::CompressedFileReader::readInternal().

40  {
41  if (read_size == 0 || buffer[read_size - 1] != line_delim) {
42  CHECK(buffer_size > read_size);
43  static_cast<char*>(buffer)[read_size] = line_delim;
44  read_size++;
45  } else if (read_size > 1 && buffer[read_size - 2] == line_delim) {
46  // Extra newline may have been due to the file encoding
47  // and may disappear during an append
48  read_size--;
49  }
50 }
#define CHECK(condition)
Definition: Logger.h:289

+ Here is the caller graph for this function:

size_t foreign_storage::anonymous_namespace{FileReader.cpp}::get_data_size ( size_t  file_size,
size_t  header_size 

Definition at line 66 of file FileReader.cpp.

Referenced by foreign_storage::SingleTextFileReader::checkForMoreRows(), and foreign_storage::SingleTextFileReader::SingleTextFileReader().

66  {
67  // Add 1 byte for possible need to insert a newline
68  return file_size - header_size + 1;
69 }
size_t file_size(const int fd)
Definition: heavyai_fs.cpp:33

+ Here is the caller graph for this function:

size_t foreign_storage::anonymous_namespace{FileReader.cpp}::offset_to_index ( const std::vector< size_t > &  cumulative_sizes,
size_t  byte_offset 
cumulative_sizesSize of each file + all previous files
byte_offsetbyte offset into the fileset from the initial scan
the file index for a given byte offset

Definition at line 57 of file FileReader.cpp.

References gpu_enabled::upper_bound().

Referenced by foreign_storage::CompressedFileReader::readRegion(), and foreign_storage::MultiFileReader::readRegion().

57  {
58  auto iterator =
59  std::upper_bound(cumulative_sizes.begin(), cumulative_sizes.end(), byte_offset);
60  if (iterator == cumulative_sizes.end()) {
61  throw std::runtime_error{"Invalid offset into cumulative_sizes"};
62  }
63  return iterator - cumulative_sizes.begin();
64 }
DEVICE auto upper_bound(ARGS &&...args)
Definition: gpu_enabled.h:123

+ Here is the call graph for this function:

+ Here is the caller graph for this function: