Classes
class	FileBuffer
	Represents/provides access to contiguous data stored in the file system. More...

struct	FileInfo

class	FileMgr

class	GlobalFileMgr

struct	HeaderInfo
	Stores Pair of ChunkKey and Page id and version, in a pair with a Page struct itself (File id and Page num) More...

struct	MultiPage
	The MultiPage stores versions of the same logical page in a deque. More...

struct	Page
	A logical page (Page) belongs to a file on disk. More...

struct	readThreadDS

Typedefs
using	PageSizeFileMMap = std::multimap< size_t, int >
	Maps logical page sizes to files. More...

using	Chunk = FileBuffer
	A Chunk is the fundamental unit of execution in Map-D. More...

using	ChunkKeyToChunkMap = std::map< ChunkKey, FileBuffer * >
	Maps ChunkKeys (unique ids for Chunks) to Chunk objects. More...

Functions
static size_t	readForThread (FileBuffer *fileBuffer, const readThreadDS threadDS)

bool	headerCompare (const HeaderInfo &firstElem, const HeaderInfo &secondElem)

FILE *	create (const std::string &basePath, const int fileId, const size_t pageSize, const size_t numPages)

FILE *	create (const std::string &fullPath, const size_t requestedFileSize)

FILE *	open (int fileId)
	Opens/creates the file with the given id; returns NULL on error. More...

FILE *	open (const std::string &path)

void	close (FILE *f)
	Closes the file pointed to by the FILE pointer. More...

bool	removeFile (const std::string basePath, const std::string filename)
	Deletes the file pointed to by the FILE pointer. More...

size_t	read (FILE f, const size_t offset, const size_t size, int8_t buf)
	Reads the specified number of bytes from the offset position in file f into buf. More...

size_t	write (FILE f, const size_t offset, const size_t size, int8_t buf)
	Writes the specified number of bytes to the offset position in file f from buf. More...

size_t	append (FILE f, const size_t size, int8_t buf)
	Appends the specified number of bytes to the end of the file f from buf. More...

size_t	readPage (FILE f, const size_t pageSize, const size_t pageNum, int8_t buf)
	Reads the specified page from the file f into buf. More...

size_t	readPartialPage (FILE f, const size_t pageSize, const size_t offset, const size_t readSize, const size_t pageNum, int8_t buf)

size_t	writePage (FILE f, const size_t pageSize, const size_t pageNum, int8_t buf)
	Writes a page from buf to the file. More...

size_t	writePartialPage (FILE f, const size_t pageSize, const size_t offset, const size_t writeSize, const size_t pageNum, int8_t buf)

size_t	appendPage (FILE f, const size_t pageSize, int8_t buf)
	Appends a page from buf to the file. More...

size_t	fileSize (FILE *f)
	Returns the size of the specified file. More...

void	renameForDelete (const std::string directoryName)
	Renames a directory to DELETE_ME_<EPOCH>_<oldname>. More...

Typedef Documentation

◆ Chunk

using File_Namespace::Chunk = typedef FileBuffer

A Chunk is the fundamental unit of execution in Map-D.

Chunk A chunk is composed of logical pages. These pages can exist across multiple files managed by the file manager.

The collection of pages is implemented as a FileBuffer object, which is composed of a vector of MultiPage objects, one for each logical page of the file buffer.

Definition at line 67 of file FileMgr.h.

◆ ChunkKeyToChunkMap

using File_Namespace::ChunkKeyToChunkMap = typedef std::map<ChunkKey, FileBuffer*>

Maps ChunkKeys (unique ids for Chunks) to Chunk objects.

ChunkKeyToChunkMap The file system can store multiple chunks across multiple files. With that in mind, the challenge is to be able to reconstruct the pages that compose a chunk upon request. A chunk key (ChunkKey) uniquely identifies a chunk, and so ChunkKeyToChunkMap maps chunk keys to Chunk types, which are vectors of MultiPage* pointers (logical pages).

Definition at line 79 of file FileMgr.h.

◆ PageSizeFileMMap

using File_Namespace::PageSizeFileMMap = typedef std::multimap<size_t, int>

Maps logical page sizes to files.

PageSizeFileMMap The file manager uses this type in order to quickly find files of a certain page size. A multimap is used to associate the key (page size) with values (file identifiers of files having the matching page size).

Definition at line 55 of file FileMgr.h.

Function Documentation

◆ append()

size_t File_Namespace::append	(	FILE *	f,
		const size_t	size,
		int8_t *	buf
	)

Appends the specified number of bytes to the end of the file f from buf.

Parameters

f	Pointer to the FILE.
n	The number of bytes to append to the file.
buf	The source buffer containing the data to be appended.
err	If not NULL, will hold an error code should an error occur.

Returns: size_t The number of bytes written.

Definition at line 140 of file File.cpp.

References fileSize(), and write().

Referenced by ArrowCsvForeignStorage::ArrowCsvForeignStorage(), analyze_benchmark.BenchAnalyzer::collectMissingQueries(), analyze_benchmark.PrettyPrint::collectMissingQueries(), concat(), concat_with(), com.mapd.utility.SQLImporter::createMapDTable(), ArrowResultSetConverter::getArrowBatch(), run_benchmark::read_query_files(), run_benchmark_arrow::run_query(), run_benchmark::run_query(), com.omnisci.jdbc.OmniSciArray::toString(), ResultSet::updateStorageEntryCount(), and PersistentForeignStorageInterface::~PersistentForeignStorageInterface().

                                                        {
   return write(f, fileSize(f), size, buf);
 }

Here is the call graph for this function:

Here is the caller graph for this function:

◆ appendPage()

size_t File_Namespace::appendPage	(	FILE *	f,
		const size_t	pageSize,
		int8_t *	buf
	)

Appends a page from buf to the file.

Parameters

f	Pointer to the FILE.
pageSize	The logical page size of the file.
buf	The source buffer from where data is being read.
err	If not NULL, will hold an error code should an error occur.

Returns: size_t The number of bytes appended (should be equal to pageSize).

Definition at line 170 of file File.cpp.

References fileSize(), and write().

                                                                {
   return write(f, fileSize(f), pageSize, buf);
 }

Here is the call graph for this function:

◆ close()

void File_Namespace::close ( FILE * f )

Closes the file pointed to by the FILE pointer.

Parameters

f	Pointer to the FILE.

Definition at line 106 of file File.cpp.

References CHECK, and CHECK_EQ.

Referenced by File_Namespace::FileMgr::closeRemovePhysical(), import_export::DataStreamSink::import_compressed(), CommandLineOptions::validate(), File_Namespace::FileInfo::~FileInfo(), File_Namespace::FileMgr::~FileMgr(), and StringDictionary::~StringDictionary().

                     {
   CHECK(f);
   CHECK_EQ(fflush(f), 0);
   CHECK_EQ(fclose(f), 0);
 }

Here is the caller graph for this function:

◆ create() [1/2]

FILE * File_Namespace::create	(	const std::string &	basePath,
		const int	fileId,
		const size_t	pageSize,
		const size_t	numPages
	)

Definition at line 39 of file File.cpp.

References logger::FATAL, fileSize(), LOG, MAPD_FILE_EXT, and to_string().

Referenced by File_Namespace::FileMgr::createDBMetaFile(), File_Namespace::FileMgr::createEpochFile(), and File_Namespace::FileMgr::createFile().

                                     {
   std::string path(basePath + "/" + std::to_string(fileId) + "." +
                    std::to_string(pageSize) +
                    std::string(MAPD_FILE_EXT));  // MAPD_FILE_EXT has preceding "."
   if (numPages < 1 || pageSize < 1) {
     LOG(FATAL) << "Error trying to create file '" << path
                << "', Number of pages and page size must be positive integers. numPages "
                << numPages << " pageSize " << pageSize;
   }
   FILE* f = fopen(path.c_str(), "w+b");
   if (f == nullptr) {
     LOG(FATAL) << "Error trying to create file '" << path
                << "', the error was: " << std::strerror(errno);
     ;
   }
   fseek(f, static_cast<long>((pageSize * numPages) - 1), SEEK_SET);
   fputc(EOF, f);
   fseek(f, 0, SEEK_SET);  // rewind
   if (fileSize(f) != pageSize * numPages) {
     LOG(FATAL) << "Error trying to create file '" << path << "', file size "
                << fileSize(f) << " does not equal pageSize * numPages "
                << pageSize * numPages;
   }
 
   return f;
 }

Here is the call graph for this function:

Here is the caller graph for this function:

◆ create() [2/2]

FILE * File_Namespace::create	(	const std::string &	fullPath,
		const size_t	requestedFileSize
	)

Definition at line 69 of file File.cpp.

References logger::FATAL, fileSize(), and LOG.

                                                                         {
   FILE* f = fopen(fullPath.c_str(), "w+b");
   if (f == nullptr) {
     LOG(FATAL) << "Error trying to create file '" << fullPath
                << "', the error was:  " << std::strerror(errno);
     ;
   }
   fseek(f, static_cast<long>(requestedFileSize - 1), SEEK_SET);
   fputc(EOF, f);
   fseek(f, 0, SEEK_SET);  // rewind
   if (fileSize(f) != requestedFileSize) {
     LOG(FATAL) << "Error trying to create file '" << fullPath << "', file size "
                << fileSize(f) << " does not equal requestedFileSize "
                << requestedFileSize;
   }
   return f;
 }

Here is the call graph for this function:

◆ fileSize()

size_t File_Namespace::fileSize ( FILE * f )

Returns the size of the specified file.

Todo:: There may be an issue casting to size_t from long.

Parameters

f	A pointer to the file.

Returns: size_t The number of bytes of the file.

Definition at line 175 of file File.cpp.

Referenced by append(), appendPage(), create(), and File_Namespace::FileMgr::init().

                          {
   fseek(f, 0, SEEK_END);
   size_t size = (size_t)ftell(f);
   fseek(f, 0, SEEK_SET);
   return size;
 }

Here is the caller graph for this function:

◆ headerCompare()

bool File_Namespace::headerCompare	(	const HeaderInfo &	firstElem,
		const HeaderInfo &	secondElem
	)

Definition at line 50 of file FileMgr.cpp.

References File_Namespace::HeaderInfo::chunkKey, File_Namespace::HeaderInfo::pageId, and File_Namespace::HeaderInfo::versionEpoch.

Referenced by File_Namespace::FileMgr::init().

                                                                               {
   // HeaderInfo.first is a pair of Chunk key with a vector containing
   // pageId and version
   if (firstElem.chunkKey != secondElem.chunkKey) {
     return firstElem.chunkKey < secondElem.chunkKey;
   }
   if (firstElem.pageId != secondElem.pageId) {
     return firstElem.pageId < secondElem.pageId;
   }
   return firstElem.versionEpoch < secondElem.versionEpoch;
 
   /*
   if (firstElem.first.first != secondElem.first.first)
       return firstElem.first.first < secondElem.first.first;
   return firstElem.first.second < secondElem.first.second;
   */
 }

Here is the caller graph for this function:

◆ open() [1/2]

FILE * File_Namespace::open ( int fileId )

Opens/creates the file with the given id; returns NULL on error.

Parameters

fileId The id of the file to open.

Returns: FILE* A pointer to a FILE pointer, or NULL on error.

Definition at line 87 of file File.cpp.

References logger::FATAL, LOG, MAPD_FILE_EXT, and to_string().

Referenced by anonymous_namespace{StringDictionary.cpp}::checked_open(), create_table.SyntheticTable::generateData(), analyze_benchmark.BenchmarkLoader::load(), File_Namespace::FileMgr::openDBMetaFile(), File_Namespace::FileMgr::openEpochFile(), File_Namespace::FileMgr::openExistingFile(), run_benchmark::read_query_files(), run_benchmark::send_results_db(), run_benchmark::send_results_file_json(), run_benchmark::send_results_jenkins_bench(), and CommandLineOptions::validate().

                        {
   std::string s(std::to_string(fileId) + std::string(MAPD_FILE_EXT));
   FILE* f = fopen(s.c_str(), "r+b");  // opens existing file for updates
   if (f == nullptr) {
     LOG(FATAL) << "Error trying to open file '" << s
                << "', the error was: " << std::strerror(errno);
   }
   return f;
 }

Here is the call graph for this function:

Here is the caller graph for this function:

◆ open() [2/2]

FILE * File_Namespace::open ( const std::string & path )

Definition at line 97 of file File.cpp.

References logger::FATAL, and LOG.

                                   {
   FILE* f = fopen(path.c_str(), "r+b");  // opens existing file for updates
   if (f == nullptr) {
     LOG(FATAL) << "Error trying to open file '" << path
                << "', the errno was: " << std::strerror(errno);
   }
   return f;
 }

◆ read()

size_t File_Namespace::read	(	FILE *	f,
		const size_t	offset,
		const size_t	size,
		int8_t *	buf
	)

Reads the specified number of bytes from the offset position in file f into buf.

Parameters

f	Pointer to the FILE.
offset	The location within the file from which to read.
n	The number of bytes to be read.
buf	The destination buffer to where data is being read from the file.
err	If not NULL, will hold an error code should an error occur.

Returns: size_t The number of bytes read.

Definition at line 117 of file File.cpp.

References CHECK_EQ.

Referenced by ArrowCsvForeignStorage::ArrowCsvForeignStorage(), File_Namespace::FileMgr::openDBMetaFile(), File_Namespace::FileMgr::openEpochFile(), File_Namespace::FileInfo::read(), readPage(), readPartialPage(), and PersistentForeignStorageInterface::~PersistentForeignStorageInterface().

                                                                           {
   // read "size" bytes from the offset location in the file into the buffer
   CHECK_EQ(fseek(f, static_cast<long>(offset), SEEK_SET), 0);
   size_t bytesRead = fread(buf, sizeof(int8_t), size, f);
   CHECK_EQ(bytesRead, sizeof(int8_t) * size);
   return bytesRead;
 }

Here is the caller graph for this function:

◆ readForThread()

static size_t File_Namespace::readForThread	(	FileBuffer *	fileBuffer,
		const readThreadDS	threadDS
	)

static

Definition at line 201 of file FileBuffer.cpp.

References CHECK, File_Namespace::FileMgr::getFileInfoForFileId(), File_Namespace::readThreadDS::multiPages, File_Namespace::FileBuffer::pageDataSize(), File_Namespace::FileBuffer::pageSize(), File_Namespace::FileInfo::read(), File_Namespace::FileBuffer::reservedHeaderSize(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.

Referenced by File_Namespace::FileBuffer::read().

                                                                                  {
   size_t startPage = threadDS.t_startPage;  // start reading at startPage, including it
   size_t endPage = threadDS.t_endPage;      // stop reading at endPage, not including it
   int8_t* curPtr = threadDS.t_curPtr;
   size_t bytesLeft = threadDS.t_bytesLeft;
   size_t totalBytesRead = 0;
   bool isFirstPage = threadDS.t_isFirstPage;
 
   // Traverse the logical pages
   for (size_t pageNum = startPage; pageNum < endPage; ++pageNum) {
     CHECK(threadDS.multiPages[pageNum].pageSize == fileBuffer->pageSize());
     Page page = threadDS.multiPages[pageNum].current();
 
     FileInfo* fileInfo = threadDS.t_fm->getFileInfoForFileId(page.fileId);
     CHECK(fileInfo);
 
     // Read the page into the destination (dst) buffer at its
     // current (cur) location
     size_t bytesRead = 0;
     if (isFirstPage) {
       bytesRead = fileInfo->read(
           page.pageNum * fileBuffer->pageSize() + threadDS.t_startPageOffset +
               fileBuffer->reservedHeaderSize(),
           min(fileBuffer->pageDataSize() - threadDS.t_startPageOffset, bytesLeft),
           curPtr);
       isFirstPage = false;
     } else {
       bytesRead = fileInfo->read(
           page.pageNum * fileBuffer->pageSize() + fileBuffer->reservedHeaderSize(),
           min(fileBuffer->pageDataSize(), bytesLeft),
           curPtr);
     }
     curPtr += bytesRead;
     bytesLeft -= bytesRead;
     totalBytesRead += bytesRead;
   }
   CHECK(bytesLeft == 0);
 
   return (totalBytesRead);
 }

Here is the call graph for this function:

Here is the caller graph for this function:

◆ readPage()

size_t File_Namespace::readPage	(	FILE *	f,
		const size_t	pageSize,
		const size_t	pageNum,
		int8_t *	buf
	)

Reads the specified page from the file f into buf.

Parameters

f	Pointer to the FILE.
pageSize	The logical page size of the file.
pageNum	The page number from where data is being read.
buf	The destination buffer to where data is being written.
err	If not NULL, will hold an error code should an error occur.

Returns: size_t The number of bytes read (should be equal to pageSize).

Definition at line 144 of file File.cpp.

References read().

                                                                                    {
   return read(f, pageNum * pageSize, pageSize, buf);
 }

Here is the call graph for this function:

◆ readPartialPage()

size_t File_Namespace::readPartialPage	(	FILE *	f,
		const size_t	pageSize,
		const size_t	offset,
		const size_t	readSize,
		const size_t	pageNum,
		int8_t *	buf
	)

Definition at line 148 of file File.cpp.

References read().

                                     {
   return read(f, pageNum * pageSize + offset, readSize, buf);
 }

Here is the call graph for this function:

◆ removeFile()

bool File_Namespace::removeFile	(	const std::string	basePath,
		const std::string	filename
	)

Deletes the file pointed to by the FILE pointer.

Parameters

basePath	The base path (directory) of the file.
f	Pointer to the FILE.

Returns: mapd_err_t Returns an error code when unable to close the file properly.

Definition at line 112 of file File.cpp.

References logger::filename().

                                                                     {
   const std::string filePath = basePath + filename;
   return remove(filePath.c_str()) == 0;
 }

Here is the call graph for this function:

◆ renameForDelete()

void File_Namespace::renameForDelete ( const std::string directoryName )

Renames a directory to DELETE_ME_<EPOCH>_<oldname>.

Parameters

directoryName name of directory

Definition at line 187 of file File.cpp.

References logger::ERROR, error_code, logger::FATAL, LOG, and to_string().

Referenced by File_Namespace::FileMgr::closeRemovePhysical(), Catalog_Namespace::Catalog::delDictionary(), Catalog_Namespace::Catalog::doTruncateTable(), and Catalog_Namespace::Catalog::removeTableFromMap().

                                                     {
   boost::system::error_code ec;
   boost::filesystem::path directoryPath(directoryName);
   using namespace std::chrono;
   milliseconds ms = duration_cast<milliseconds>(system_clock::now().time_since_epoch());
 
   if (boost::filesystem::exists(directoryPath) &&
       boost::filesystem::is_directory(directoryPath)) {
     boost::filesystem::path newDirectoryPath(directoryName + "_" +
                                              std::to_string(ms.count()) + "_DELETE_ME");
     boost::filesystem::rename(directoryPath, newDirectoryPath, ec);
 
     if (ec.value() == boost::system::errc::success) {
       std::thread th([newDirectoryPath]() {
         boost::system::error_code ec;
         boost::filesystem::remove_all(newDirectoryPath, ec);
         if (ec.value() != boost::system::errc::success) {
           LOG(ERROR) << "Failed to remove directory " << newDirectoryPath << " error was "
                      << ec;
         }
       });
       // let it run free so we can return
       // if it fails the file_delete_thread in DBHandler will clean up
       th.detach();
 
       return;
     }
 
     LOG(FATAL) << "Failed to rename file " << directoryName << " to "
                << directoryName + "_" + std::to_string(ms.count()) + "_DELETE_ME  Error: "
                << ec;
   }
 }

Here is the call graph for this function:

Here is the caller graph for this function:

◆ write()

size_t File_Namespace::write	(	FILE *	f,
		const size_t	offset,
		const size_t	size,
		int8_t *	buf
	)

Writes the specified number of bytes to the offset position in file f from buf.

Parameters

f	Pointer to the FILE.
offset	The location within the file where data is being written.
size	The number of bytes to write to the file.
buf	The source buffer containing the data to be written.
err	If not NULL, will hold an error code should an error occur.

Returns: size_t The number of bytes written.

Definition at line 125 of file File.cpp.

References logger::FATAL, and LOG.

                                                                            {
   // write size bytes from the buffer to the offset location in the file
   if (fseek(f, static_cast<long>(offset), SEEK_SET) != 0) {
     LOG(FATAL)
         << "Error trying to write to file (during positioning seek) the error was: "
         << std::strerror(errno);
   }
   size_t bytesWritten = fwrite(buf, sizeof(int8_t), size, f);
   if (bytesWritten != sizeof(int8_t) * size) {
     LOG(FATAL) << "Error trying to write to file (during fwrite) the error was: "
                << std::strerror(errno);
   }
   return bytesWritten;
 }

Here is the caller graph for this function:

◆ writePage()

size_t File_Namespace::writePage	(	FILE *	f,
		const size_t	pageSize,
		const size_t	pageNum,
		int8_t *	buf
	)

Writes a page from buf to the file.

Parameters

f	Pointer to the FILE.
pageSize	The logical page size of the file.
pageNum	The page number to where data is being written.
buf	The source buffer from where data is being read.
err	If not NULL, will hold an error code should an error occur.

Returns: size_t The number of bytes written (should be equal to pageSize).

Definition at line 157 of file File.cpp.

References write().

                                                                                     {
   return write(f, pageNum * pageSize, pageSize, buf);
 }

Here is the call graph for this function:

◆ writePartialPage()

size_t File_Namespace::writePartialPage	(	FILE *	f,
		const size_t	pageSize,
		const size_t	offset,
		const size_t	writeSize,
		const size_t	pageNum,
		int8_t *	buf
	)

Definition at line 161 of file File.cpp.

References write().

                                      {
   return write(f, pageNum * pageSize + offset, writeSize, buf);
 }

Here is the call graph for this function:

Classes

Typedefs

Functions

Typedef Documentation

◆ Chunk

◆ ChunkKeyToChunkMap

◆ PageSizeFileMMap

Function Documentation

◆ append()

◆ appendPage()

◆ close()

◆ create() [1/2]

◆ create() [2/2]

◆ fileSize()

◆ headerCompare()

◆ open() [1/2]

◆ open() [2/2]

◆ read()

◆ readForThread()

◆ readPage()

◆ readPartialPage()

◆ removeFile()

◆ renameForDelete()

◆ write()

◆ writePage()

◆ writePartialPage()