File_Namespace Namespace Reference

Classes
struct	readThreadDS
class	FileBuffer
	Represents/provides access to contiguous data stored in the file system. More...
struct	FileInfo
struct	FileMetadata
struct	StorageStats
class	FileMgr
struct	FileMgrParams
class	GlobalFileMgr
struct	Page
	A logical page (Page) belongs to a file on disk. More...
struct	EpochedPage
struct	MultiPage
	The MultiPage stores versions of the same logical page in a deque. More...
struct	HeaderInfo
	Stores Pair of ChunkKey and Page id and version, in a pair with a Page struct itself (File id and Page num) More...

Typedefs
using	PageSizeFileMMap = std::multimap< size_t, int32_t >
	Maps logical page sizes to files. More...
using	Chunk = FileBuffer
	A Chunk is the fundamental unit of execution in Map-D. More...
using	ChunkKeyToChunkMap = std::map< ChunkKey, FileBuffer * >
	Maps ChunkKeys (unique ids for Chunks) to Chunk objects. More...

Functions
static size_t	readForThread (FileBuffer *fileBuffer, const readThreadDS threadDS)
bool	headerCompare (const HeaderInfo &firstElem, const HeaderInfo &secondElem)
FILE *	create (const std::string &basePath, const int fileId, const size_t pageSize, const size_t numPages)
FILE *	create (const std::string &fullPath, const size_t requestedFileSize)
FILE *	open (int fileId)
	Opens/creates the file with the given id; returns NULL on error. More...
FILE *	open (const std::string &path)
void	close (FILE *f)
	Closes the file pointed to by the FILE pointer. More...
bool	removeFile (const std::string basePath, const std::string filename)
	Deletes the file pointed to by the FILE pointer. More...
size_t	read (FILE *f, const size_t offset, const size_t size, int8_t *buf)
	Reads the specified number of bytes from the offset position in file f into buf. More...
size_t	write (FILE *f, const size_t offset, const size_t size, int8_t *buf)
	Writes the specified number of bytes to the offset position in file f from buf. More...
size_t	append (FILE *f, const size_t size, int8_t *buf)
	Appends the specified number of bytes to the end of the file f from buf. More...
size_t	readPage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf)
	Reads the specified page from the file f into buf. More...
size_t	readPartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t readSize, const size_t pageNum, int8_t *buf)
size_t	writePage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf)
	Writes a page from buf to the file. More...
size_t	writePartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t writeSize, const size_t pageNum, int8_t *buf)
size_t	appendPage (FILE *f, const size_t pageSize, int8_t *buf)
	Appends a page from buf to the file. More...
size_t	fileSize (FILE *f)
	Returns the size of the specified file. More...
void	renameForDelete (const std::string directoryName)
	Renames a directory to DELETE_ME_<EPOCH>_<oldname>. More...

Variables
constexpr int32_t	DELETE_CONTINGENT = -1
	A FileInfo type has a file pointer and metadata about a file. More...
constexpr int32_t	ROLLOFF_CONTINGENT = -2

Typedef Documentation

using File_Namespace::Chunk = typedef FileBuffer

A Chunk is the fundamental unit of execution in Map-D.

Chunk A chunk is composed of logical pages. These pages can exist across multiple files managed by the file manager.

The collection of pages is implemented as a FileBuffer object, which is composed of a vector of MultiPage objects, one for each logical page of the file buffer.

Definition at line 70 of file FileMgr.h.

using File_Namespace::ChunkKeyToChunkMap = typedef std::map<ChunkKey, FileBuffer*>

Maps ChunkKeys (unique ids for Chunks) to Chunk objects.

ChunkKeyToChunkMap The file system can store multiple chunks across multiple files. With that in mind, the challenge is to be able to reconstruct the pages that compose a chunk upon request. A chunk key (ChunkKey) uniquely identifies a chunk, and so ChunkKeyToChunkMap maps chunk keys to Chunk types, which are vectors of MultiPage* pointers (logical pages).

Definition at line 82 of file FileMgr.h.

using File_Namespace::PageSizeFileMMap = typedef std::multimap<size_t, int32_t>

Maps logical page sizes to files.

PageSizeFileMMap The file manager uses this type in order to quickly find files of a certain page size. A multimap is used to associate the key (page size) with values (file identifiers of files having the matching page size).

Definition at line 58 of file FileMgr.h.

Function Documentation

size_t File_Namespace::append	(	FILE *	f,
		const size_t	size,
		int8_t *	buf
	)

Appends the specified number of bytes to the end of the file f from buf.

Parameters

f	Pointer to the FILE.
n	The number of bytes to append to the file.
buf	The source buffer containing the data to be appended.
err	If not NULL, will hold an error code should an error occur.

Returns

size_t The number of bytes written.

Definition at line 141 of file File.cpp.

References fileSize(), and write().

Referenced by concat(), concat_with(), com.mapd.utility.SQLImporter::createMapDTable(), ArrowResultSetConverter::getArrowBatch(), com.omnisci.jdbc.OmniSciPreparedStatement::getQuery(), com.mapd.parser.server.ExtensionFunctionSignatureParser::join(), run_benchmark::read_query_files(), run_benchmark_arrow::run_query(), run_benchmark::run_query(), com.omnisci.jdbc.OmniSciEscapeFunctions::singleArgumentFunctionCall(), com.mapd.parser.server.ExtensionFunction::toJson(), and com.omnisci.jdbc.OmniSciArray::toString().

                                                       {
  return write(f, fileSize(f), size, buf);
}

Here is the call graph for this function:

Here is the caller graph for this function:

size_t File_Namespace::appendPage	(	FILE *	f,
		const size_t	pageSize,
		int8_t *	buf
	)

Appends a page from buf to the file.

Parameters

f	Pointer to the FILE.
pageSize	The logical page size of the file.
buf	The source buffer from where data is being read.
err	If not NULL, will hold an error code should an error occur.

Returns

size_t The number of bytes appended (should be equal to pageSize).

Definition at line 171 of file File.cpp.

References fileSize(), and write().

                                                               {
  return write(f, fileSize(f), pageSize, buf);
}

Here is the call graph for this function:

void File_Namespace::close

(

FILE *

f

)

Closes the file pointed to by the FILE pointer.

Parameters

f	Pointer to the FILE.

Definition at line 107 of file File.cpp.

References CHECK, and CHECK_EQ.

Referenced by File_Namespace::FileMgr::closeRemovePhysical(), File_Namespace::FileMgr::readVersionFromDisk(), File_Namespace::FileMgr::writeAndSyncVersionToDisk(), File_Namespace::FileInfo::~FileInfo(), and File_Namespace::FileMgr::~FileMgr().

                    {
  CHECK(f);
  CHECK_EQ(fflush(f), 0);
  CHECK_EQ(fclose(f), 0);
}

Here is the caller graph for this function:

FILE * File_Namespace::create	(	const std::string &	basePath,
		const int	fileId,
		const size_t	pageSize,
		const size_t	numPages
	)

Definition at line 40 of file File.cpp.

References generate_TableFunctionsFactory_init::f, logger::FATAL, fileSize(), omnisci::fopen(), LOG, MAPD_FILE_EXT, and to_string().

Referenced by File_Namespace::FileMgr::createEpochFile(), File_Namespace::FileMgr::createFile(), kafka_insert(), org.apache.calcite.sql2rel.SqlToRelConverter::SqlToRelConverter(), and File_Namespace::FileMgr::writeAndSyncVersionToDisk().

                                    {
  std::string path(basePath + "/" + std::to_string(fileId) + "." +
                   std::to_string(pageSize) +
                   std::string(MAPD_FILE_EXT));  // MAPD_FILE_EXT has preceding "."
  if (numPages < 1 || pageSize < 1) {
    LOG(FATAL) << "Error trying to create file '" << path
               << "', Number of pages and page size must be positive integers. numPages "
               << numPages << " pageSize " << pageSize;
  }
  FILE* f = omnisci::fopen(path.c_str(), "w+b");
  if (f == nullptr) {
    LOG(FATAL) << "Error trying to create file '" << path
               << "', the error was: " << std::strerror(errno);
    ;
  }
  fseek(f, static_cast<long>((pageSize * numPages) - 1), SEEK_SET);
  fputc(EOF, f);
  fseek(f, 0, SEEK_SET);  // rewind
  if (fileSize(f) != pageSize * numPages) {
    LOG(FATAL) << "Error trying to create file '" << path << "', file size "
               << fileSize(f) << " does not equal pageSize * numPages "
               << pageSize * numPages;
  }

  return f;
}

Here is the call graph for this function:

Here is the caller graph for this function:

FILE * File_Namespace::create	(	const std::string &	fullPath,
		const size_t	requestedFileSize
	)

Definition at line 70 of file File.cpp.

References generate_TableFunctionsFactory_init::f, logger::FATAL, fileSize(), omnisci::fopen(), and LOG.

                                                                        {
  FILE* f = omnisci::fopen(fullPath.c_str(), "w+b");
  if (f == nullptr) {
    LOG(FATAL) << "Error trying to create file '" << fullPath
               << "', the error was:  " << std::strerror(errno);
    ;
  }
  fseek(f, static_cast<long>(requestedFileSize - 1), SEEK_SET);
  fputc(EOF, f);
  fseek(f, 0, SEEK_SET);  // rewind
  if (fileSize(f) != requestedFileSize) {
    LOG(FATAL) << "Error trying to create file '" << fullPath << "', file size "
               << fileSize(f) << " does not equal requestedFileSize "
               << requestedFileSize;
  }
  return f;
}

Here is the call graph for this function:

size_t File_Namespace::fileSize

(

FILE *

f

)

Returns the size of the specified file.

Todo:

There may be an issue casting to size_t from long.

Parameters

f	A pointer to the file.

Returns

size_t The number of bytes of the file.

Definition at line 176 of file File.cpp.

Referenced by append(), appendPage(), create(), and File_Namespace::FileMgr::getTotalFileSize().

                         {
  fseek(f, 0, SEEK_END);
  size_t size = (size_t)ftell(f);
  fseek(f, 0, SEEK_SET);
  return size;
}

Here is the caller graph for this function:

bool File_Namespace::headerCompare	(	const HeaderInfo &	firstElem,
		const HeaderInfo &	secondElem
	)

Definition at line 53 of file FileMgr.cpp.

References File_Namespace::HeaderInfo::chunkKey, File_Namespace::HeaderInfo::pageId, and File_Namespace::HeaderInfo::versionEpoch.

Referenced by File_Namespace::FileMgr::init().

                                                                              {
  // HeaderInfo.first is a pair of Chunk key with a vector containing
  // pageId and version
  if (firstElem.chunkKey != secondElem.chunkKey) {
    return firstElem.chunkKey < secondElem.chunkKey;
  }
  if (firstElem.pageId != secondElem.pageId) {
    return firstElem.pageId < secondElem.pageId;
  }
  return firstElem.versionEpoch < secondElem.versionEpoch;
}

Here is the caller graph for this function:

FILE * File_Namespace::open

(

int

fileId

)

Opens/creates the file with the given id; returns NULL on error.

Parameters

fileId

The id of the file to open.

Returns

FILE* A pointer to a FILE pointer, or NULL on error.

Definition at line 88 of file File.cpp.

References generate_TableFunctionsFactory_init::f, logger::FATAL, omnisci::fopen(), LOG, MAPD_FILE_EXT, and to_string().

Referenced by File_Namespace::FileMgr::openAndReadEpochFile(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileMgr::openExistingFile(), File_Namespace::FileMgr::readVersionFromDisk(), and File_Namespace::FileMgr::writeAndSyncVersionToDisk().

                       {
  std::string s(std::to_string(fileId) + std::string(MAPD_FILE_EXT));
  FILE* f = omnisci::fopen(s.c_str(), "r+b");  // opens existing file for updates
  if (f == nullptr) {
    LOG(FATAL) << "Error trying to open file '" << s
               << "', the error was: " << std::strerror(errno);
  }
  return f;
}

Here is the call graph for this function:

Here is the caller graph for this function:

FILE * File_Namespace::open

(

const std::string &

path

)

Definition at line 98 of file File.cpp.

References generate_TableFunctionsFactory_init::f, logger::FATAL, omnisci::fopen(), and LOG.

                                  {
  FILE* f = omnisci::fopen(path.c_str(), "r+b");  // opens existing file for updates
  if (f == nullptr) {
    LOG(FATAL) << "Error trying to open file '" << path
               << "', the errno was: " << std::strerror(errno);
  }
  return f;
}

Here is the call graph for this function:

size_t File_Namespace::read	(	FILE *	f,
		const size_t	offset,
		const size_t	size,
		int8_t *	buf
	)

Reads the specified number of bytes from the offset position in file f into buf.

Parameters

f	Pointer to the FILE.
offset	The location within the file from which to read.
n	The number of bytes to be read.
buf	The destination buffer to where data is being read from the file.
err	If not NULL, will hold an error code should an error occur.

Returns

size_t The number of bytes read.

Definition at line 118 of file File.cpp.

References CHECK_EQ.

Referenced by File_Namespace::FileMgr::openAndReadEpochFile(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileInfo::read(), readPage(), readPartialPage(), and File_Namespace::FileMgr::readVersionFromDisk().

                                                                          {
  // read "size" bytes from the offset location in the file into the buffer
  CHECK_EQ(fseek(f, static_cast<long>(offset), SEEK_SET), 0);
  size_t bytesRead = fread(buf, sizeof(int8_t), size, f);
  CHECK_EQ(bytesRead, sizeof(int8_t) * size);
  return bytesRead;
}

Here is the caller graph for this function:

static size_t File_Namespace::readForThread ( FileBuffer *  fileBuffer, const readThreadDS  threadDS  )

static

Definition at line 224 of file FileBuffer.cpp.

References CHECK, File_Namespace::FileMgr::getFileInfoForFileId(), File_Namespace::readThreadDS::multiPages, File_Namespace::FileBuffer::pageDataSize(), File_Namespace::FileBuffer::pageSize(), File_Namespace::FileInfo::read(), File_Namespace::FileBuffer::reservedHeaderSize(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.

Referenced by File_Namespace::FileBuffer::read().

                                                                                 {
  size_t startPage = threadDS.t_startPage;  // start reading at startPage, including it
  size_t endPage = threadDS.t_endPage;      // stop reading at endPage, not including it
  int8_t* curPtr = threadDS.t_curPtr;
  size_t bytesLeft = threadDS.t_bytesLeft;
  size_t totalBytesRead = 0;
  bool isFirstPage = threadDS.t_isFirstPage;

  // Traverse the logical pages
  for (size_t pageNum = startPage; pageNum < endPage; ++pageNum) {
    CHECK(threadDS.multiPages[pageNum].pageSize == fileBuffer->pageSize());
    Page page = threadDS.multiPages[pageNum].current().page;

    FileInfo* fileInfo = threadDS.t_fm->getFileInfoForFileId(page.fileId);
    CHECK(fileInfo);

    // Read the page into the destination (dst) buffer at its
    // current (cur) location
    size_t bytesRead = 0;
    if (isFirstPage) {
      bytesRead = fileInfo->read(
          page.pageNum * fileBuffer->pageSize() + threadDS.t_startPageOffset +
              fileBuffer->reservedHeaderSize(),
          min(fileBuffer->pageDataSize() - threadDS.t_startPageOffset, bytesLeft),
          curPtr);
      isFirstPage = false;
    } else {
      bytesRead = fileInfo->read(
          page.pageNum * fileBuffer->pageSize() + fileBuffer->reservedHeaderSize(),
          min(fileBuffer->pageDataSize(), bytesLeft),
          curPtr);
    }
    curPtr += bytesRead;
    bytesLeft -= bytesRead;
    totalBytesRead += bytesRead;
  }
  CHECK(bytesLeft == 0);

  return (totalBytesRead);
}

Here is the call graph for this function:

Here is the caller graph for this function:

size_t File_Namespace::readPage	(	FILE *	f,
		const size_t	pageSize,
		const size_t	pageNum,
		int8_t *	buf
	)

Reads the specified page from the file f into buf.

Parameters

f	Pointer to the FILE.
pageSize	The logical page size of the file.
pageNum	The page number from where data is being read.
buf	The destination buffer to where data is being written.
err	If not NULL, will hold an error code should an error occur.

Returns

size_t The number of bytes read (should be equal to pageSize).

Definition at line 145 of file File.cpp.

References read().

                                                                                   {
  return read(f, pageNum * pageSize, pageSize, buf);
}

Here is the call graph for this function:

size_t File_Namespace::readPartialPage	(	FILE *	f,
		const size_t	pageSize,
		const size_t	offset,
		const size_t	readSize,
		const size_t	pageNum,
		int8_t *	buf
	)

Definition at line 149 of file File.cpp.

References read().

                                    {
  return read(f, pageNum * pageSize + offset, readSize, buf);
}

Here is the call graph for this function:

bool File_Namespace::removeFile	(	const std::string	basePath,
		const std::string	filename
	)

Deletes the file pointed to by the FILE pointer.

Parameters

basePath	The base path (directory) of the file.
f	Pointer to the FILE.

Returns

mapd_err_t Returns an error code when unable to close the file properly.

Definition at line 113 of file File.cpp.

References logger::filename().

                                                                    {
  const std::string filePath = basePath + filename;
  return remove(filePath.c_str()) == 0;
}

Here is the call graph for this function:

void File_Namespace::renameForDelete

(

const std::string

directoryName

)

Renames a directory to DELETE_ME_<EPOCH>_<oldname>.

Parameters

directoryName

name of directory

Definition at line 188 of file File.cpp.

References logger::FATAL, LOG, and to_string().

Referenced by File_Namespace::FileMgr::closeRemovePhysical(), Catalog_Namespace::Catalog::delDictionary(), Catalog_Namespace::Catalog::doTruncateTable(), and Catalog_Namespace::Catalog::removeTableFromMap().

                                                    {
  boost::system::error_code ec;
  boost::filesystem::path directoryPath(directoryName);
  using namespace std::chrono;
  milliseconds ms = duration_cast<milliseconds>(system_clock::now().time_since_epoch());

  if (boost::filesystem::exists(directoryPath) &&
      boost::filesystem::is_directory(directoryPath)) {
    boost::filesystem::path newDirectoryPath(directoryName + "_" +
                                             std::to_string(ms.count()) + "_DELETE_ME");
    boost::filesystem::rename(directoryPath, newDirectoryPath, ec);

    if (ec.value() == boost::system::errc::success) {
      std::thread th([newDirectoryPath]() {
        boost::system::error_code ec;
        boost::filesystem::remove_all(newDirectoryPath, ec);
        // We dont check error on remove here as we cant log the
        // issue fromdetached thrad, its not safe to LOG from here
        // This is under investigation as clang detects TSAN issue data race
        // the main system wide file_delete_thread will clean up any missed files
      });
      // let it run free so we can return
      // if it fails the file_delete_thread in DBHandler will clean up
      th.detach();

      return;
    }

    LOG(FATAL) << "Failed to rename file " << directoryName << " to "
               << directoryName + "_" + std::to_string(ms.count()) + "_DELETE_ME  Error: "
               << ec;
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

size_t File_Namespace::write	(	FILE *	f,
		const size_t	offset,
		const size_t	size,
		int8_t *	buf
	)

Writes the specified number of bytes to the offset position in file f from buf.

Parameters

f	Pointer to the FILE.
offset	The location within the file where data is being written.
size	The number of bytes to write to the file.
buf	The source buffer containing the data to be written.
err	If not NULL, will hold an error code should an error occur.

Returns

size_t The number of bytes written.

Definition at line 126 of file File.cpp.

References logger::FATAL, and LOG.

Referenced by StringDictionary::addStorageCapacity(), append(), appendPage(), File_Namespace::FileInfo::freePage(), import_export::DataStreamSink::import_compressed(), File_Namespace::FileInfo::initNewFile(), File_Namespace::FileInfo::openExistingFile(), CommandLineOptions::validate(), File_Namespace::FileInfo::write(), File_Namespace::FileMgr::writeAndSyncEpochToDisk(), File_Namespace::FileMgr::writeAndSyncVersionToDisk(), writePage(), and writePartialPage().

                                                                           {
  // write size bytes from the buffer to the offset location in the file
  if (fseek(f, static_cast<long>(offset), SEEK_SET) != 0) {
    LOG(FATAL)
        << "Error trying to write to file (during positioning seek) the error was: "
        << std::strerror(errno);
  }
  size_t bytesWritten = fwrite(buf, sizeof(int8_t), size, f);
  if (bytesWritten != sizeof(int8_t) * size) {
    LOG(FATAL) << "Error trying to write to file (during fwrite) the error was: "
               << std::strerror(errno);
  }
  return bytesWritten;
}

Here is the caller graph for this function:

size_t File_Namespace::writePage	(	FILE *	f,
		const size_t	pageSize,
		const size_t	pageNum,
		int8_t *	buf
	)

Writes a page from buf to the file.

Parameters

f	Pointer to the FILE.
pageSize	The logical page size of the file.
pageNum	The page number to where data is being written.
buf	The source buffer from where data is being read.
err	If not NULL, will hold an error code should an error occur.

Returns

size_t The number of bytes written (should be equal to pageSize).

Definition at line 158 of file File.cpp.

References write().

                                                                                    {
  return write(f, pageNum * pageSize, pageSize, buf);
}

Here is the call graph for this function:

size_t File_Namespace::writePartialPage	(	FILE *	f,
		const size_t	pageSize,
		const size_t	offset,
		const size_t	writeSize,
		const size_t	pageNum,
		int8_t *	buf
	)

Definition at line 162 of file File.cpp.

References write().

                                     {
  return write(f, pageNum * pageSize + offset, writeSize, buf);
}

Here is the call graph for this function:

Variable Documentation

constexpr int32_t File_Namespace::DELETE_CONTINGENT = -1

A FileInfo type has a file pointer and metadata about a file.

FileInfo A file info structure wraps around a file pointer in order to contain additional information/metadata about the file that is pertinent to the file manager.

The free pages (freePages) within a file must be tracked, and this is implemented using a basic STL set. The set ensures that no duplicate pages are included, and that the pages are sorted, faciliating the obtaining of consecutive free pages by a constant time pop operation, which may reduce the cost of DBMS disk accesses.

Helper functions are provided: size(), available(), and used().

Definition at line 51 of file FileInfo.h.

Referenced by File_Namespace::FileInfo::freePage(), and File_Namespace::FileInfo::openExistingFile().

constexpr int32_t File_Namespace::ROLLOFF_CONTINGENT = -2

Definition at line 52 of file FileInfo.h.

Referenced by File_Namespace::FileInfo::freePage(), and File_Namespace::FileInfo::openExistingFile().