OmniSciDB  04ee39c94c
File_Namespace Namespace Reference

Classes

class  FileBuffer
 Represents/provides access to contiguous data stored in the file system. More...
 
struct  FileInfo
 
class  FileMgr
 
class  GlobalFileMgr
 
struct  HeaderInfo
 Stores Pair of ChunkKey and Page id and version, in a pair with a Page struct itself (File id and Page num) More...
 
struct  MultiPage
 The MultiPage stores versions of the same logical page in a deque. More...
 
struct  Page
 A logical page (Page) belongs to a file on disk. More...
 
struct  readThreadDS
 

Typedefs

typedef std::multimap< size_t, int > PageSizeFileMMap
 Maps logical page sizes to files. More...
 
using Chunk = FileBuffer
 A Chunk is the fundamental unit of execution in Map-D. More...
 
typedef std::map< ChunkKey, FileBuffer * > ChunkKeyToChunkMap
 Maps ChunkKeys (unique ids for Chunks) to Chunk objects. More...
 

Functions

static size_t readForThread (FileBuffer *fileBuffer, const readThreadDS threadDS)
 
bool headerCompare (const HeaderInfo &firstElem, const HeaderInfo &secondElem)
 
FILE * create (const std::string &basePath, const int fileId, const size_t pageSize, const size_t numPages)
 
FILE * create (const std::string &fullPath, const size_t requestedFileSize)
 
FILE * open (int fileId)
 Opens/creates the file with the given id; returns NULL on error. More...
 
FILE * open (const std::string &path)
 
void close (FILE *f)
 Closes the file pointed to by the FILE pointer. More...
 
bool removeFile (const std::string basePath, const std::string filename)
 Deletes the file pointed to by the FILE pointer. More...
 
size_t read (FILE *f, const size_t offset, const size_t size, int8_t *buf)
 Reads the specified number of bytes from the offset position in file f into buf. More...
 
size_t write (FILE *f, const size_t offset, const size_t size, int8_t *buf)
 Writes the specified number of bytes to the offset position in file f from buf. More...
 
size_t append (FILE *f, const size_t size, int8_t *buf)
 Appends the specified number of bytes to the end of the file f from buf. More...
 
size_t readPage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf)
 Reads the specified page from the file f into buf. More...
 
size_t readPartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t readSize, const size_t pageNum, int8_t *buf)
 
size_t writePage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf)
 Writes a page from buf to the file. More...
 
size_t writePartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t writeSize, const size_t pageNum, int8_t *buf)
 
size_t appendPage (FILE *f, const size_t pageSize, int8_t *buf)
 Appends a page from buf to the file. More...
 
size_t fileSize (FILE *f)
 Returns the size of the specified file. More...
 
void renameForDelete (const std::string directoryName)
 Renames a directory to DELETE_ME_<EPOCH>_<oldname>. More...
 

Typedef Documentation

◆ Chunk

A Chunk is the fundamental unit of execution in Map-D.

Chunk A chunk is composed of logical pages. These pages can exist across multiple files managed by the file manager.

The collection of pages is implemented as a FileBuffer object, which is composed of a vector of MultiPage objects, one for each logical page of the file buffer.

Definition at line 68 of file FileMgr.h.

◆ ChunkKeyToChunkMap

Maps ChunkKeys (unique ids for Chunks) to Chunk objects.

ChunkKeyToChunkMap The file system can store multiple chunks across multiple files. With that in mind, the challenge is to be able to reconstruct the pages that compose a chunk upon request. A chunk key (ChunkKey) uniquely identifies a chunk, and so ChunkKeyToChunkMap maps chunk keys to Chunk types, which are vectors of MultiPage* pointers (logical pages).

Definition at line 80 of file FileMgr.h.

◆ PageSizeFileMMap

typedef std::multimap<size_t, int> File_Namespace::PageSizeFileMMap

Maps logical page sizes to files.

PageSizeFileMMap The file manager uses this type in order to quickly find files of a certain page size. A multimap is used to associate the key (page size) with values (file identifiers of files having the matching page size).

Definition at line 47 of file FileMgr.h.

Function Documentation

◆ append()

size_t File_Namespace::append ( FILE *  f,
const size_t  size,
int8_t *  buf 
)

Appends the specified number of bytes to the end of the file f from buf.

Parameters
fPointer to the FILE.
nThe number of bytes to append to the file.
bufThe source buffer containing the data to be appended.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written.

Definition at line 136 of file File.cpp.

References fileSize(), and write().

Referenced by analyze-benchmark.BenchAnalyzer::collectMissingQueries(), analyze-benchmark.PrettyPrint::collectMissingQueries(), com.mapd.utility.SQLImporter::createMapDTable(), ArrowResultSetConverter::getArrowBatch(), and ResultSet::updateStorageEntryCount().

136  {
137  return write(f, fileSize(f), size, buf);
138 }
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:171
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:121
+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ appendPage()

size_t File_Namespace::appendPage ( FILE *  f,
const size_t  pageSize,
int8_t *  buf 
)

Appends a page from buf to the file.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
bufThe source buffer from where data is being read.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes appended (should be equal to pageSize).

Definition at line 166 of file File.cpp.

References fileSize(), and write().

166  {
167  return write(f, fileSize(f), pageSize, buf);
168 }
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:171
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:121
+ Here is the call graph for this function:

◆ close()

void File_Namespace::close ( FILE *  f)

Closes the file pointed to by the FILE pointer.

Parameters
fPointer to the FILE.

Definition at line 102 of file File.cpp.

References CHECK, and CHECK_EQ.

Referenced by File_Namespace::FileMgr::closeRemovePhysical(), Importer_NS::DataStreamSink::import_compressed(), MapDProgramOptions::parse_command_line(), File_Namespace::FileInfo::~FileInfo(), and StringDictionary::~StringDictionary().

102  {
103  CHECK(f);
104  CHECK_EQ(fflush(f), 0);
105  CHECK_EQ(fclose(f), 0);
106 }
#define CHECK_EQ(x, y)
Definition: Logger.h:195
#define CHECK(condition)
Definition: Logger.h:187
+ Here is the caller graph for this function:

◆ create() [1/2]

FILE * File_Namespace::create ( const std::string &  basePath,
const int  fileId,
const size_t  pageSize,
const size_t  numPages 
)

Definition at line 35 of file File.cpp.

References logger::FATAL, fileSize(), LOG, MAPD_FILE_EXT, and to_string().

Referenced by File_Namespace::FileMgr::createDBMetaFile(), File_Namespace::FileMgr::createEpochFile(), and File_Namespace::FileMgr::createFile().

38  {
39  std::string path(basePath + "/" + std::to_string(fileId) + "." +
40  std::to_string(pageSize) +
41  std::string(MAPD_FILE_EXT)); // MAPD_FILE_EXT has preceding "."
42  if (numPages < 1 || pageSize < 1) {
43  LOG(FATAL) << "Error trying to create file '" << path
44  << "', Number of pages and page size must be positive integers. numPages "
45  << numPages << " pageSize " << pageSize;
46  }
47  FILE* f = fopen(path.c_str(), "w+b");
48  if (f == nullptr) {
49  LOG(FATAL) << "Error trying to create file '" << path
50  << "', the error was: " << std::strerror(errno);
51  ;
52  }
53  fseek(f, (pageSize * numPages) - 1, SEEK_SET);
54  fputc(EOF, f);
55  fseek(f, 0, SEEK_SET); // rewind
56  if (fileSize(f) != pageSize * numPages) {
57  LOG(FATAL) << "Error trying to create file '" << path << "', file size "
58  << fileSize(f) << " does not equal pageSize * numPages "
59  << pageSize * numPages;
60  }
61 
62  return f;
63 }
#define LOG(tag)
Definition: Logger.h:182
#define MAPD_FILE_EXT
Definition: File.h:26
std::string to_string(char const *&&v)
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:171
+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ create() [2/2]

FILE * File_Namespace::create ( const std::string &  fullPath,
const size_t  requestedFileSize 
)

Definition at line 65 of file File.cpp.

References logger::FATAL, fileSize(), and LOG.

65  {
66  FILE* f = fopen(fullPath.c_str(), "w+b");
67  if (f == nullptr) {
68  LOG(FATAL) << "Error trying to create file '" << fullPath
69  << "', the error was: " << std::strerror(errno);
70  ;
71  }
72  fseek(f, requestedFileSize - 1, SEEK_SET);
73  fputc(EOF, f);
74  fseek(f, 0, SEEK_SET); // rewind
75  if (fileSize(f) != requestedFileSize) {
76  LOG(FATAL) << "Error trying to create file '" << fullPath << "', file size "
77  << fileSize(f) << " does not equal requestedFileSize "
78  << requestedFileSize;
79  }
80  return f;
81 }
#define LOG(tag)
Definition: Logger.h:182
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:171
+ Here is the call graph for this function:

◆ fileSize()

size_t File_Namespace::fileSize ( FILE *  f)

Returns the size of the specified file.

Todo:
There may be an issue casting to size_t from long.
Parameters
fA pointer to the file.
Returns
size_t The number of bytes of the file.

Definition at line 171 of file File.cpp.

Referenced by append(), appendPage(), create(), and File_Namespace::FileMgr::init().

171  {
172  fseek(f, 0, SEEK_END);
173  size_t size = (size_t)ftell(f);
174  fseek(f, 0, SEEK_SET);
175  return size;
176 }
+ Here is the caller graph for this function:

◆ headerCompare()

bool File_Namespace::headerCompare ( const HeaderInfo firstElem,
const HeaderInfo secondElem 
)

Definition at line 47 of file FileMgr.cpp.

References File_Namespace::HeaderInfo::chunkKey, File_Namespace::HeaderInfo::pageId, and File_Namespace::HeaderInfo::versionEpoch.

Referenced by File_Namespace::FileMgr::init().

47  {
48  // HeaderInfo.first is a pair of Chunk key with a vector containing
49  // pageId and version
50  if (firstElem.chunkKey != secondElem.chunkKey) {
51  return firstElem.chunkKey < secondElem.chunkKey;
52  }
53  if (firstElem.pageId != secondElem.pageId) {
54  return firstElem.pageId < secondElem.pageId;
55  }
56  return firstElem.versionEpoch < secondElem.versionEpoch;
57 
58  /*
59  if (firstElem.first.first != secondElem.first.first)
60  return firstElem.first.first < secondElem.first.first;
61  return firstElem.first.second < secondElem.first.second;
62  */
63 }
+ Here is the caller graph for this function:

◆ open() [1/2]

FILE * File_Namespace::open ( int  fileId)

Opens/creates the file with the given id; returns NULL on error.

Parameters
fileIdThe id of the file to open.
Returns
FILE* A pointer to a FILE pointer, or NULL on error.

Definition at line 83 of file File.cpp.

References logger::FATAL, LOG, MAPD_FILE_EXT, and to_string().

Referenced by anonymous_namespace{StringDictionary.cpp}::checked_open(), analyze-benchmark.BenchmarkLoader::load(), File_Namespace::FileMgr::openDBMetaFile(), File_Namespace::FileMgr::openEpochFile(), File_Namespace::FileMgr::openExistingFile(), and MapDProgramOptions::parse_command_line().

83  {
84  std::string s(std::to_string(fileId) + std::string(MAPD_FILE_EXT));
85  FILE* f = fopen(s.c_str(), "r+b"); // opens existing file for updates
86  if (f == nullptr) {
87  LOG(FATAL) << "Error trying to open file '" << s
88  << "', the error was: " << std::strerror(errno);
89  }
90  return f;
91 }
#define LOG(tag)
Definition: Logger.h:182
#define MAPD_FILE_EXT
Definition: File.h:26
std::string to_string(char const *&&v)
+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ open() [2/2]

FILE * File_Namespace::open ( const std::string &  path)

Definition at line 93 of file File.cpp.

References logger::FATAL, and LOG.

93  {
94  FILE* f = fopen(path.c_str(), "r+b"); // opens existing file for updates
95  if (f == nullptr) {
96  LOG(FATAL) << "Error trying to open file '" << path
97  << "', the errno was: " << std::strerror(errno);
98  }
99  return f;
100 }
#define LOG(tag)
Definition: Logger.h:182

◆ read()

size_t File_Namespace::read ( FILE *  f,
const size_t  offset,
const size_t  size,
int8_t *  buf 
)

Reads the specified number of bytes from the offset position in file f into buf.

Parameters
fPointer to the FILE.
offsetThe location within the file from which to read.
nThe number of bytes to be read.
bufThe destination buffer to where data is being read from the file.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes read.

Definition at line 113 of file File.cpp.

References CHECK_EQ.

Referenced by File_Namespace::FileMgr::openDBMetaFile(), File_Namespace::FileMgr::openEpochFile(), File_Namespace::FileInfo::read(), readPage(), and readPartialPage().

113  {
114  // read "size" bytes from the offset location in the file into the buffer
115  CHECK_EQ(fseek(f, offset, SEEK_SET), 0);
116  size_t bytesRead = fread(buf, sizeof(int8_t), size, f);
117  CHECK_EQ(bytesRead, sizeof(int8_t) * size);
118  return bytesRead;
119 }
#define CHECK_EQ(x, y)
Definition: Logger.h:195
+ Here is the caller graph for this function:

◆ readForThread()

static size_t File_Namespace::readForThread ( FileBuffer fileBuffer,
const readThreadDS  threadDS 
)
static

Definition at line 195 of file FileBuffer.cpp.

References CHECK, File_Namespace::FileMgr::getFileInfoForFileId(), File_Namespace::readThreadDS::multiPages, File_Namespace::FileBuffer::pageDataSize(), File_Namespace::FileBuffer::pageSize(), File_Namespace::FileInfo::read(), File_Namespace::FileBuffer::reservedHeaderSize(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.

Referenced by File_Namespace::FileBuffer::read().

195  {
196  size_t startPage = threadDS.t_startPage; // start reading at startPage, including it
197  size_t endPage = threadDS.t_endPage; // stop reading at endPage, not including it
198  int8_t* curPtr = threadDS.t_curPtr;
199  size_t bytesLeft = threadDS.t_bytesLeft;
200  size_t totalBytesRead = 0;
201  bool isFirstPage = threadDS.t_isFirstPage;
202 
203  // Traverse the logical pages
204  for (size_t pageNum = startPage; pageNum < endPage; ++pageNum) {
205  CHECK(threadDS.multiPages[pageNum].pageSize == fileBuffer->pageSize());
206  Page page = threadDS.multiPages[pageNum].current();
207 
208  FileInfo* fileInfo = threadDS.t_fm->getFileInfoForFileId(page.fileId);
209  CHECK(fileInfo);
210 
211  // Read the page into the destination (dst) buffer at its
212  // current (cur) location
213  size_t bytesRead = 0;
214  if (isFirstPage) {
215  bytesRead = fileInfo->read(
216  page.pageNum * fileBuffer->pageSize() + threadDS.t_startPageOffset +
217  fileBuffer->reservedHeaderSize(),
218  min(fileBuffer->pageDataSize() - threadDS.t_startPageOffset, bytesLeft),
219  curPtr);
220  isFirstPage = false;
221  } else {
222  bytesRead = fileInfo->read(
223  page.pageNum * fileBuffer->pageSize() + fileBuffer->reservedHeaderSize(),
224  min(fileBuffer->pageDataSize(), bytesLeft),
225  curPtr);
226  }
227  curPtr += bytesRead;
228  bytesLeft -= bytesRead;
229  totalBytesRead += bytesRead;
230  }
231  CHECK(bytesLeft == 0);
232 
233  return (totalBytesRead);
234 }
#define CHECK(condition)
Definition: Logger.h:187
+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ readPage()

size_t File_Namespace::readPage ( FILE *  f,
const size_t  pageSize,
const size_t  pageNum,
int8_t *  buf 
)

Reads the specified page from the file f into buf.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
pageNumThe page number from where data is being read.
bufThe destination buffer to where data is being written.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes read (should be equal to pageSize).

Definition at line 140 of file File.cpp.

References read().

140  {
141  return read(f, pageNum * pageSize, pageSize, buf);
142 }
size_t read(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Reads the specified number of bytes from the offset position in file f into buf.
Definition: File.cpp:113
+ Here is the call graph for this function:

◆ readPartialPage()

size_t File_Namespace::readPartialPage ( FILE *  f,
const size_t  pageSize,
const size_t  offset,
const size_t  readSize,
const size_t  pageNum,
int8_t *  buf 
)

Definition at line 144 of file File.cpp.

References read().

149  {
150  return read(f, pageNum * pageSize + offset, readSize, buf);
151 }
size_t read(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Reads the specified number of bytes from the offset position in file f into buf.
Definition: File.cpp:113
+ Here is the call graph for this function:

◆ removeFile()

bool File_Namespace::removeFile ( const std::string  basePath,
const std::string  filename 
)

Deletes the file pointed to by the FILE pointer.

Parameters
basePathThe base path (directory) of the file.
fPointer to the FILE.
Returns
mapd_err_t Returns an error code when unable to close the file properly.

Definition at line 108 of file File.cpp.

108  {
109  const std::string filePath = basePath + filename;
110  return remove(filePath.c_str()) == 0;
111 }

◆ renameForDelete()

void File_Namespace::renameForDelete ( const std::string  directoryName)

Renames a directory to DELETE_ME_<EPOCH>_<oldname>.

Parameters
directoryNamename of directory

Definition at line 183 of file File.cpp.

References logger::ERROR, error_code, logger::FATAL, LOG, and to_string().

Referenced by File_Namespace::FileMgr::closeRemovePhysical(), Catalog_Namespace::Catalog::delDictionary(), Catalog_Namespace::Catalog::doTruncateTable(), and Catalog_Namespace::Catalog::removeTableFromMap().

183  {
185  boost::filesystem::path directoryPath(directoryName);
186  using namespace std::chrono;
187  milliseconds ms = duration_cast<milliseconds>(system_clock::now().time_since_epoch());
188 
189  if (boost::filesystem::exists(directoryPath) &&
190  boost::filesystem::is_directory(directoryPath)) {
191  boost::filesystem::path newDirectoryPath(directoryName + "_" +
192  std::to_string(ms.count()) + "_DELETE_ME");
193  boost::filesystem::rename(directoryPath, newDirectoryPath, ec);
194 
195  if (ec.value() == boost::system::errc::success) {
196  std::thread th([newDirectoryPath]() {
198  boost::filesystem::remove_all(newDirectoryPath, ec);
199  if (ec.value() != boost::system::errc::success) {
200  LOG(ERROR) << "Failed to remove directory " << newDirectoryPath << " error was "
201  << ec;
202  }
203  });
204  // let it run free so we can return
205  // if it fails the file_delete_thread in MapDHandler will clean up
206  th.detach();
207 
208  return;
209  }
210 
211  LOG(FATAL) << "Failed to rename file " << directoryName << " to "
212  << directoryName + "_" + std::to_string(ms.count()) + "_DELETE_ME Error: "
213  << ec;
214  }
215 }
#define LOG(tag)
Definition: Logger.h:182
std::string to_string(char const *&&v)
const int8_t const int64_t const uint64_t const int32_t const int64_t int64_t uint32_t const int64_t int32_t * error_code
+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ write()

size_t File_Namespace::write ( FILE *  f,
const size_t  offset,
const size_t  size,
int8_t *  buf 
)

Writes the specified number of bytes to the offset position in file f from buf.

Parameters
fPointer to the FILE.
offsetThe location within the file where data is being written.
sizeThe number of bytes to write to the file.
bufThe source buffer containing the data to be written.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written.

Definition at line 121 of file File.cpp.

References logger::FATAL, and LOG.

Referenced by StringDictionary::addStorageCapacity(), append(), appendPage(), File_Namespace::FileMgr::createDBMetaFile(), File_Namespace::FileMgr::createEpochFile(), File_Namespace::FileInfo::freePage(), Importer_NS::DataStreamSink::import_compressed(), File_Namespace::FileInfo::initNewFile(), File_Namespace::FileInfo::openExistingFile(), MapDProgramOptions::parse_command_line(), File_Namespace::FileInfo::write(), File_Namespace::FileMgr::writeAndSyncDBMetaToDisk(), File_Namespace::FileMgr::writeAndSyncEpochToDisk(), writePage(), and writePartialPage().

121  {
122  // write size bytes from the buffer to the offset location in the file
123  if (fseek(f, offset, SEEK_SET) != 0) {
124  LOG(FATAL)
125  << "Error trying to write to file (during positioning seek) the error was: "
126  << std::strerror(errno);
127  }
128  size_t bytesWritten = fwrite(buf, sizeof(int8_t), size, f);
129  if (bytesWritten != sizeof(int8_t) * size) {
130  LOG(FATAL) << "Error trying to write to file (during fwrite) the error was: "
131  << std::strerror(errno);
132  }
133  return bytesWritten;
134 }
#define LOG(tag)
Definition: Logger.h:182
+ Here is the caller graph for this function:

◆ writePage()

size_t File_Namespace::writePage ( FILE *  f,
const size_t  pageSize,
const size_t  pageNum,
int8_t *  buf 
)

Writes a page from buf to the file.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
pageNumThe page number to where data is being written.
bufThe source buffer from where data is being read.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written (should be equal to pageSize).

Definition at line 153 of file File.cpp.

References write().

153  {
154  return write(f, pageNum * pageSize, pageSize, buf);
155 }
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:121
+ Here is the call graph for this function:

◆ writePartialPage()

size_t File_Namespace::writePartialPage ( FILE *  f,
const size_t  pageSize,
const size_t  offset,
const size_t  writeSize,
const size_t  pageNum,
int8_t *  buf 
)

Definition at line 157 of file File.cpp.

References write().

162  {
163  return write(f, pageNum * pageSize + offset, writeSize, buf);
164 }
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:121
+ Here is the call graph for this function: