OmniSciDB  471d68cefb
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
foreign_storage::anonymous_namespace{FileReader.cpp} Namespace Reference

Functions

void adjust_eof (size_t &read_size, const size_t buffer_size, char *buffer, const char line_delim)
 
size_t offset_to_index (const std::vector< size_t > &cumulative_sizes, size_t byte_offset)
 
size_t get_data_size (size_t file_size, size_t header_size)
 
bool is_compressed_file (const std::string &location)
 

Function Documentation

void foreign_storage::anonymous_namespace{FileReader.cpp}::adjust_eof ( size_t &  read_size,
const size_t  buffer_size,
char *  buffer,
const char  line_delim 
)

Adds an end of line character (specified by the line_delim parameter) to provided buffer, if this is the last read buffer and if the buffer does not already end with an end of line character. This allows for appropriate parsing by the file_buffer_parser utility functions, which expect the end of rows to be indicated by end of line characters in the buffer. Also removes extra EOL that may be inserted at the EOF that will not be present if the file is appended to

Definition at line 36 of file FileReader.cpp.

References CHECK.

Referenced by foreign_storage::MultiFileReader::read(), and foreign_storage::CompressedFileReader::readInternal().

39  {
40  if (read_size == 0 || buffer[read_size - 1] != line_delim) {
41  CHECK(buffer_size > read_size);
42  static_cast<char*>(buffer)[read_size] = line_delim;
43  read_size++;
44  } else if (read_size > 1 && buffer[read_size - 2] == line_delim) {
45  // Extra newline may have been due to the file encoding
46  // and may disappear during an append
47  read_size--;
48  }
49 }
#define CHECK(condition)
Definition: Logger.h:209

+ Here is the caller graph for this function:

size_t foreign_storage::anonymous_namespace{FileReader.cpp}::get_data_size ( size_t  file_size,
size_t  header_size 
)

Definition at line 65 of file FileReader.cpp.

Referenced by foreign_storage::SingleTextFileReader::checkForMoreRows(), and foreign_storage::SingleTextFileReader::SingleTextFileReader().

65  {
66  // Add 1 byte for possible need to insert a newline
67  return file_size - header_size + 1;
68 }
size_t file_size(const int fd)
Definition: omnisci_fs.cpp:31

+ Here is the caller graph for this function:

bool foreign_storage::anonymous_namespace{FileReader.cpp}::is_compressed_file ( const std::string &  location)

Definition at line 605 of file FileReader.cpp.

References shared::contains().

Referenced by foreign_storage::LocalMultiFileReader::insertFile(), and foreign_storage::LocalMultiFileReader::LocalMultiFileReader().

605  {
606  const std::vector<std::string> compressed_exts = {
607  ".zip", ".gz", ".tar", ".rar", ".bz2", ".7z", ".tgz"};
608  return shared::contains(compressed_exts, boost::filesystem::extension(location));
609 }
bool contains(const T &container, const U &element)
Definition: misc.h:188

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t foreign_storage::anonymous_namespace{FileReader.cpp}::offset_to_index ( const std::vector< size_t > &  cumulative_sizes,
size_t  byte_offset 
)
Parameters
cumulative_sizesSize of each file + all previous files
byte_offsetbyte offset into the fileset from the initial scan
Returns
the file index for a given byte offset

Definition at line 56 of file FileReader.cpp.

References gpu_enabled::upper_bound().

Referenced by foreign_storage::CompressedFileReader::readRegion(), and foreign_storage::MultiFileReader::readRegion().

56  {
57  auto iterator =
58  std::upper_bound(cumulative_sizes.begin(), cumulative_sizes.end(), byte_offset);
59  if (iterator == cumulative_sizes.end()) {
60  throw std::runtime_error{"Invalid offset into cumulative_sizes"};
61  }
62  return iterator - cumulative_sizes.begin();
63 }
DEVICE auto upper_bound(ARGS &&...args)
Definition: gpu_enabled.h:123

+ Here is the call graph for this function:

+ Here is the caller graph for this function: