foreign_storage::anonymous_namespace{FileReader.cpp} Namespace Reference

Functions
void	adjust_eof (size_t &read_size, const size_t buffer_size, char *buffer, const char line_delim)
size_t	offset_to_index (const std::vector< size_t > &cumulative_sizes, size_t byte_offset)
size_t	get_data_size (size_t file_size, size_t header_size)

Function Documentation

void foreign_storage::anonymous_namespace{FileReader.cpp}::adjust_eof	(	size_t &	read_size,
		const size_t	buffer_size,
		char *	buffer,
		const char	line_delim
	)

Adds an end of line character (specified by the line_delim parameter) to provided buffer, if this is the last read buffer and if the buffer does not already end with an end of line character. This allows for appropriate parsing by the file_buffer_parser utility functions, which expect the end of rows to be indicated by end of line characters in the buffer. Also removes extra EOL that may be inserted at the EOF that will not be present if the file is appended to

Definition at line 37 of file FileReader.cpp.

References CHECK.

Referenced by foreign_storage::MultiFileReader::read(), and foreign_storage::CompressedFileReader::readInternal().

                                       {
  if (read_size == 0 || buffer[read_size - 1] != line_delim) {
    CHECK(buffer_size > read_size);
    static_cast<char*>(buffer)[read_size] = line_delim;
    read_size++;
  } else if (read_size > 1 && buffer[read_size - 2] == line_delim) {
    // Extra newline may have been due to the file encoding
    // and may disappear during an append
    read_size--;
  }
}

Here is the caller graph for this function:

size_t foreign_storage::anonymous_namespace{FileReader.cpp}::get_data_size	(	size_t	file_size,
		size_t	header_size
	)

Definition at line 66 of file FileReader.cpp.

Referenced by foreign_storage::SingleTextFileReader::checkForMoreRows(), and foreign_storage::SingleTextFileReader::SingleTextFileReader().

                                                           {
  // Add 1 byte for possible need to insert a newline
  return file_size - header_size + 1;
}

Here is the caller graph for this function:

size_t foreign_storage::anonymous_namespace{FileReader.cpp}::offset_to_index	(	const std::vector< size_t > &	cumulative_sizes,
		size_t	byte_offset
	)

Parameters

cumulative_sizes	Size of each file + all previous files
byte_offset	byte offset into the fileset from the initial scan

Returns

the file index for a given byte offset

Definition at line 57 of file FileReader.cpp.

References gpu_enabled::upper_bound().

Referenced by foreign_storage::CompressedFileReader::readRegion(), and foreign_storage::MultiFileReader::readRegion().

                                                                                      {
  auto iterator =
      std::upper_bound(cumulative_sizes.begin(), cumulative_sizes.end(), byte_offset);
  if (iterator == cumulative_sizes.end()) {
    throw std::runtime_error{"Invalid offset into cumulative_sizes"};
  }
  return iterator - cumulative_sizes.begin();
}

Here is the call graph for this function:

Here is the caller graph for this function: