OmniSciDB
94e8789169
|
Classes | |
struct | readThreadDS |
class | FileBuffer |
Represents/provides access to contiguous data stored in the file system. More... | |
struct | FileInfo |
struct | FileMetadata |
struct | StorageStats |
class | FileMgr |
struct | FileMgrParams |
class | GlobalFileMgr |
struct | Page |
A logical page (Page) belongs to a file on disk. More... | |
struct | EpochedPage |
struct | MultiPage |
The MultiPage stores versions of the same logical page in a deque. More... | |
struct | HeaderInfo |
Stores Pair of ChunkKey and Page id and version, in a pair with a Page struct itself (File id and Page num) More... | |
Typedefs | |
using | PageSizeFileMMap = std::multimap< size_t, int32_t > |
Maps logical page sizes to files. More... | |
using | Chunk = FileBuffer |
A Chunk is the fundamental unit of execution in Map-D. More... | |
using | ChunkKeyToChunkMap = std::map< ChunkKey, FileBuffer * > |
Maps ChunkKeys (unique ids for Chunks) to Chunk objects. More... | |
Functions | |
static size_t | readForThread (FileBuffer *fileBuffer, const readThreadDS threadDS) |
bool | headerCompare (const HeaderInfo &firstElem, const HeaderInfo &secondElem) |
FILE * | create (const std::string &basePath, const int fileId, const size_t pageSize, const size_t numPages) |
FILE * | create (const std::string &fullPath, const size_t requestedFileSize) |
FILE * | open (int fileId) |
Opens/creates the file with the given id; returns NULL on error. More... | |
FILE * | open (const std::string &path) |
void | close (FILE *f) |
Closes the file pointed to by the FILE pointer. More... | |
bool | removeFile (const std::string basePath, const std::string filename) |
Deletes the file pointed to by the FILE pointer. More... | |
size_t | read (FILE *f, const size_t offset, const size_t size, int8_t *buf) |
Reads the specified number of bytes from the offset position in file f into buf. More... | |
size_t | write (FILE *f, const size_t offset, const size_t size, int8_t *buf) |
Writes the specified number of bytes to the offset position in file f from buf. More... | |
size_t | append (FILE *f, const size_t size, int8_t *buf) |
Appends the specified number of bytes to the end of the file f from buf. More... | |
size_t | readPage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf) |
Reads the specified page from the file f into buf. More... | |
size_t | readPartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t readSize, const size_t pageNum, int8_t *buf) |
size_t | writePage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf) |
Writes a page from buf to the file. More... | |
size_t | writePartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t writeSize, const size_t pageNum, int8_t *buf) |
size_t | appendPage (FILE *f, const size_t pageSize, int8_t *buf) |
Appends a page from buf to the file. More... | |
size_t | fileSize (FILE *f) |
Returns the size of the specified file. More... | |
void | renameForDelete (const std::string directoryName) |
Renames a directory to DELETE_ME_<EPOCH>_<oldname>. More... | |
Variables | |
constexpr int32_t | DELETE_CONTINGENT = -1 |
A FileInfo type has a file pointer and metadata about a file. More... | |
constexpr int32_t | ROLLOFF_CONTINGENT = -2 |
using File_Namespace::Chunk = typedef FileBuffer |
A Chunk is the fundamental unit of execution in Map-D.
Chunk A chunk is composed of logical pages. These pages can exist across multiple files managed by the file manager.
The collection of pages is implemented as a FileBuffer object, which is composed of a vector of MultiPage objects, one for each logical page of the file buffer.
using File_Namespace::ChunkKeyToChunkMap = typedef std::map<ChunkKey, FileBuffer*> |
Maps ChunkKeys (unique ids for Chunks) to Chunk objects.
ChunkKeyToChunkMap The file system can store multiple chunks across multiple files. With that in mind, the challenge is to be able to reconstruct the pages that compose a chunk upon request. A chunk key (ChunkKey) uniquely identifies a chunk, and so ChunkKeyToChunkMap maps chunk keys to Chunk types, which are vectors of MultiPage* pointers (logical pages).
using File_Namespace::PageSizeFileMMap = typedef std::multimap<size_t, int32_t> |
Maps logical page sizes to files.
PageSizeFileMMap The file manager uses this type in order to quickly find files of a certain page size. A multimap is used to associate the key (page size) with values (file identifiers of files having the matching page size).
size_t File_Namespace::append | ( | FILE * | f, |
const size_t | size, | ||
int8_t * | buf | ||
) |
Appends the specified number of bytes to the end of the file f from buf.
f | Pointer to the FILE. |
n | The number of bytes to append to the file. |
buf | The source buffer containing the data to be appended. |
err | If not NULL, will hold an error code should an error occur. |
Definition at line 141 of file File.cpp.
References fileSize(), and write().
Referenced by concat(), concat_with(), com.mapd.utility.SQLImporter::createMapDTable(), ArrowResultSetConverter::getArrowBatch(), com.omnisci.jdbc.OmniSciPreparedStatement::getQuery(), com.mapd.parser.server.ExtensionFunctionSignatureParser::join(), run_benchmark::read_query_files(), run_benchmark_arrow::run_query(), run_benchmark::run_query(), com.omnisci.jdbc.OmniSciEscapeFunctions::singleArgumentFunctionCall(), com.mapd.parser.server.ExtensionFunction::toJson(), and com.omnisci.jdbc.OmniSciArray::toString().
size_t File_Namespace::appendPage | ( | FILE * | f, |
const size_t | pageSize, | ||
int8_t * | buf | ||
) |
Appends a page from buf to the file.
f | Pointer to the FILE. |
pageSize | The logical page size of the file. |
buf | The source buffer from where data is being read. |
err | If not NULL, will hold an error code should an error occur. |
Definition at line 171 of file File.cpp.
References fileSize(), and write().
void File_Namespace::close | ( | FILE * | f | ) |
Closes the file pointed to by the FILE pointer.
f | Pointer to the FILE. |
Definition at line 107 of file File.cpp.
References CHECK, and CHECK_EQ.
Referenced by File_Namespace::FileMgr::closeRemovePhysical(), File_Namespace::FileMgr::readVersionFromDisk(), File_Namespace::FileMgr::writeAndSyncVersionToDisk(), File_Namespace::FileInfo::~FileInfo(), and File_Namespace::FileMgr::~FileMgr().
FILE * File_Namespace::create | ( | const std::string & | basePath, |
const int | fileId, | ||
const size_t | pageSize, | ||
const size_t | numPages | ||
) |
Definition at line 40 of file File.cpp.
References generate_TableFunctionsFactory_init::f, logger::FATAL, fileSize(), omnisci::fopen(), LOG, MAPD_FILE_EXT, and to_string().
Referenced by File_Namespace::FileMgr::createEpochFile(), File_Namespace::FileMgr::createFile(), kafka_insert(), org.apache.calcite.sql2rel.SqlToRelConverter::SqlToRelConverter(), and File_Namespace::FileMgr::writeAndSyncVersionToDisk().
FILE * File_Namespace::create | ( | const std::string & | fullPath, |
const size_t | requestedFileSize | ||
) |
Definition at line 70 of file File.cpp.
References generate_TableFunctionsFactory_init::f, logger::FATAL, fileSize(), omnisci::fopen(), and LOG.
size_t File_Namespace::fileSize | ( | FILE * | f | ) |
Returns the size of the specified file.
f | A pointer to the file. |
Definition at line 176 of file File.cpp.
Referenced by append(), appendPage(), create(), and File_Namespace::FileMgr::getTotalFileSize().
bool File_Namespace::headerCompare | ( | const HeaderInfo & | firstElem, |
const HeaderInfo & | secondElem | ||
) |
Definition at line 53 of file FileMgr.cpp.
References File_Namespace::HeaderInfo::chunkKey, File_Namespace::HeaderInfo::pageId, and File_Namespace::HeaderInfo::versionEpoch.
Referenced by File_Namespace::FileMgr::init().
FILE * File_Namespace::open | ( | int | fileId | ) |
Opens/creates the file with the given id; returns NULL on error.
fileId | The id of the file to open. |
Definition at line 88 of file File.cpp.
References generate_TableFunctionsFactory_init::f, logger::FATAL, omnisci::fopen(), LOG, MAPD_FILE_EXT, and to_string().
Referenced by File_Namespace::FileMgr::openAndReadEpochFile(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileMgr::openExistingFile(), File_Namespace::FileMgr::readVersionFromDisk(), and File_Namespace::FileMgr::writeAndSyncVersionToDisk().
FILE * File_Namespace::open | ( | const std::string & | path | ) |
Definition at line 98 of file File.cpp.
References generate_TableFunctionsFactory_init::f, logger::FATAL, omnisci::fopen(), and LOG.
size_t File_Namespace::read | ( | FILE * | f, |
const size_t | offset, | ||
const size_t | size, | ||
int8_t * | buf | ||
) |
Reads the specified number of bytes from the offset position in file f into buf.
f | Pointer to the FILE. |
offset | The location within the file from which to read. |
n | The number of bytes to be read. |
buf | The destination buffer to where data is being read from the file. |
err | If not NULL, will hold an error code should an error occur. |
Definition at line 118 of file File.cpp.
References CHECK_EQ.
Referenced by File_Namespace::FileMgr::openAndReadEpochFile(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileInfo::read(), readPage(), readPartialPage(), and File_Namespace::FileMgr::readVersionFromDisk().
|
static |
Definition at line 224 of file FileBuffer.cpp.
References CHECK, File_Namespace::FileMgr::getFileInfoForFileId(), File_Namespace::readThreadDS::multiPages, File_Namespace::FileBuffer::pageDataSize(), File_Namespace::FileBuffer::pageSize(), File_Namespace::FileInfo::read(), File_Namespace::FileBuffer::reservedHeaderSize(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.
Referenced by File_Namespace::FileBuffer::read().
size_t File_Namespace::readPage | ( | FILE * | f, |
const size_t | pageSize, | ||
const size_t | pageNum, | ||
int8_t * | buf | ||
) |
Reads the specified page from the file f into buf.
f | Pointer to the FILE. |
pageSize | The logical page size of the file. |
pageNum | The page number from where data is being read. |
buf | The destination buffer to where data is being written. |
err | If not NULL, will hold an error code should an error occur. |
Definition at line 145 of file File.cpp.
References read().
size_t File_Namespace::readPartialPage | ( | FILE * | f, |
const size_t | pageSize, | ||
const size_t | offset, | ||
const size_t | readSize, | ||
const size_t | pageNum, | ||
int8_t * | buf | ||
) |
Definition at line 149 of file File.cpp.
References read().
bool File_Namespace::removeFile | ( | const std::string | basePath, |
const std::string | filename | ||
) |
Deletes the file pointed to by the FILE pointer.
basePath | The base path (directory) of the file. |
f | Pointer to the FILE. |
Definition at line 113 of file File.cpp.
References logger::filename().
void File_Namespace::renameForDelete | ( | const std::string | directoryName | ) |
Renames a directory to DELETE_ME_<EPOCH>_<oldname>.
directoryName | name of directory |
Definition at line 188 of file File.cpp.
References logger::FATAL, LOG, and to_string().
Referenced by File_Namespace::FileMgr::closeRemovePhysical(), Catalog_Namespace::Catalog::delDictionary(), Catalog_Namespace::Catalog::doTruncateTable(), and Catalog_Namespace::Catalog::removeTableFromMap().
size_t File_Namespace::write | ( | FILE * | f, |
const size_t | offset, | ||
const size_t | size, | ||
int8_t * | buf | ||
) |
Writes the specified number of bytes to the offset position in file f from buf.
f | Pointer to the FILE. |
offset | The location within the file where data is being written. |
size | The number of bytes to write to the file. |
buf | The source buffer containing the data to be written. |
err | If not NULL, will hold an error code should an error occur. |
Definition at line 126 of file File.cpp.
References logger::FATAL, and LOG.
Referenced by StringDictionary::addStorageCapacity(), append(), appendPage(), File_Namespace::FileInfo::freePage(), import_export::DataStreamSink::import_compressed(), File_Namespace::FileInfo::initNewFile(), File_Namespace::FileInfo::openExistingFile(), CommandLineOptions::validate(), File_Namespace::FileInfo::write(), File_Namespace::FileMgr::writeAndSyncEpochToDisk(), File_Namespace::FileMgr::writeAndSyncVersionToDisk(), writePage(), and writePartialPage().
size_t File_Namespace::writePage | ( | FILE * | f, |
const size_t | pageSize, | ||
const size_t | pageNum, | ||
int8_t * | buf | ||
) |
Writes a page from buf to the file.
f | Pointer to the FILE. |
pageSize | The logical page size of the file. |
pageNum | The page number to where data is being written. |
buf | The source buffer from where data is being read. |
err | If not NULL, will hold an error code should an error occur. |
Definition at line 158 of file File.cpp.
References write().
size_t File_Namespace::writePartialPage | ( | FILE * | f, |
const size_t | pageSize, | ||
const size_t | offset, | ||
const size_t | writeSize, | ||
const size_t | pageNum, | ||
int8_t * | buf | ||
) |
Definition at line 162 of file File.cpp.
References write().
constexpr int32_t File_Namespace::DELETE_CONTINGENT = -1 |
A FileInfo type has a file pointer and metadata about a file.
FileInfo A file info structure wraps around a file pointer in order to contain additional information/metadata about the file that is pertinent to the file manager.
The free pages (freePages) within a file must be tracked, and this is implemented using a basic STL set. The set ensures that no duplicate pages are included, and that the pages are sorted, faciliating the obtaining of consecutive free pages by a constant time pop operation, which may reduce the cost of DBMS disk accesses.
Helper functions are provided: size(), available(), and used().
Definition at line 51 of file FileInfo.h.
Referenced by File_Namespace::FileInfo::freePage(), and File_Namespace::FileInfo::openExistingFile().
constexpr int32_t File_Namespace::ROLLOFF_CONTINGENT = -2 |
Definition at line 52 of file FileInfo.h.
Referenced by File_Namespace::FileInfo::freePage(), and File_Namespace::FileInfo::openExistingFile().