OmniSciDB  addbbd5075
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
File_Namespace Namespace Reference

Classes

struct  readThreadDS
 
class  FileBuffer
 Represents/provides access to contiguous data stored in the file system. More...
 
struct  FileInfo
 
class  FileMgr
 
class  GlobalFileMgr
 
struct  Page
 A logical page (Page) belongs to a file on disk. More...
 
struct  MultiPage
 The MultiPage stores versions of the same logical page in a deque. More...
 
struct  HeaderInfo
 Stores Pair of ChunkKey and Page id and version, in a pair with a Page struct itself (File id and Page num) More...
 

Typedefs

typedef std::multimap< size_t,
int > 
PageSizeFileMMap
 Maps logical page sizes to files. More...
 
using Chunk = FileBuffer
 A Chunk is the fundamental unit of execution in Map-D. More...
 
typedef std::map< ChunkKey,
FileBuffer * > 
ChunkKeyToChunkMap
 Maps ChunkKeys (unique ids for Chunks) to Chunk objects. More...
 

Functions

static size_t readForThread (FileBuffer *fileBuffer, const readThreadDS threadDS)
 
bool headerCompare (const HeaderInfo &firstElem, const HeaderInfo &secondElem)
 
FILE * create (const std::string &basePath, const int fileId, const size_t pageSize, const size_t numPages)
 
FILE * create (const std::string &fullPath, const size_t requestedFileSize)
 
FILE * open (int fileId)
 Opens/creates the file with the given id; returns NULL on error. More...
 
FILE * open (const std::string &path)
 
void close (FILE *f)
 Closes the file pointed to by the FILE pointer. More...
 
bool removeFile (const std::string basePath, const std::string filename)
 Deletes the file pointed to by the FILE pointer. More...
 
size_t read (FILE *f, const size_t offset, const size_t size, int8_t *buf)
 Reads the specified number of bytes from the offset position in file f into buf. More...
 
size_t write (FILE *f, const size_t offset, const size_t size, int8_t *buf)
 Writes the specified number of bytes to the offset position in file f from buf. More...
 
size_t append (FILE *f, const size_t size, int8_t *buf)
 Appends the specified number of bytes to the end of the file f from buf. More...
 
size_t readPage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf)
 Reads the specified page from the file f into buf. More...
 
size_t readPartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t readSize, const size_t pageNum, int8_t *buf)
 
size_t writePage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf)
 Writes a page from buf to the file. More...
 
size_t writePartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t writeSize, const size_t pageNum, int8_t *buf)
 
size_t appendPage (FILE *f, const size_t pageSize, int8_t *buf)
 Appends a page from buf to the file. More...
 
size_t fileSize (FILE *f)
 Returns the size of the specified file. More...
 
void renameForDelete (const std::string directoryName)
 Renames a directory to DELETE_ME_<EPOCH>_<oldname>. More...
 

Typedef Documentation

A Chunk is the fundamental unit of execution in Map-D.

Chunk A chunk is composed of logical pages. These pages can exist across multiple files managed by the file manager.

The collection of pages is implemented as a FileBuffer object, which is composed of a vector of MultiPage objects, one for each logical page of the file buffer.

Definition at line 67 of file FileMgr.h.

Maps ChunkKeys (unique ids for Chunks) to Chunk objects.

ChunkKeyToChunkMap The file system can store multiple chunks across multiple files. With that in mind, the challenge is to be able to reconstruct the pages that compose a chunk upon request. A chunk key (ChunkKey) uniquely identifies a chunk, and so ChunkKeyToChunkMap maps chunk keys to Chunk types, which are vectors of MultiPage* pointers (logical pages).

Definition at line 79 of file FileMgr.h.

typedef std::multimap<size_t, int> File_Namespace::PageSizeFileMMap

Maps logical page sizes to files.

PageSizeFileMMap The file manager uses this type in order to quickly find files of a certain page size. A multimap is used to associate the key (page size) with values (file identifiers of files having the matching page size).

Definition at line 46 of file FileMgr.h.

Function Documentation

size_t File_Namespace::append ( FILE *  f,
const size_t  size,
int8_t *  buf 
)

Appends the specified number of bytes to the end of the file f from buf.

Parameters
fPointer to the FILE.
nThe number of bytes to append to the file.
bufThe source buffer containing the data to be appended.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written.

Definition at line 135 of file File.cpp.

References fileSize(), and write().

Referenced by com.mapd.utility.SQLImporter::createMapDTable(), ArrowResultSetConverter::getArrowBatch(), com.omnisci.jdbc.OmniSciPreparedStatement::getQuery(), com.mapd.parser.server.ExtensionFunctionSignatureParser::join(), run_benchmark::read_query_files(), run_benchmark_arrow::run_query(), run_benchmark::run_query(), and com.mapd.parser.server.ExtensionFunction::toJson().

135  {
136  return write(f, fileSize(f), size, buf);
137 }
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:170
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:120

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::appendPage ( FILE *  f,
const size_t  pageSize,
int8_t *  buf 
)

Appends a page from buf to the file.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
bufThe source buffer from where data is being read.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes appended (should be equal to pageSize).

Definition at line 165 of file File.cpp.

References fileSize(), and write().

165  {
166  return write(f, fileSize(f), pageSize, buf);
167 }
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:170
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:120

+ Here is the call graph for this function:

void File_Namespace::close ( FILE *  f)

Closes the file pointed to by the FILE pointer.

Parameters
fPointer to the FILE.

Definition at line 101 of file File.cpp.

References CHECK(), and CHECK_EQ.

Referenced by File_Namespace::FileMgr::closeRemovePhysical(), Importer_NS::DataStreamSink::import_compressed(), MapDProgramOptions::validate(), File_Namespace::FileInfo::~FileInfo(), and StringDictionary::~StringDictionary().

101  {
102  CHECK(f);
103  CHECK_EQ(fflush(f), 0);
104  CHECK_EQ(fclose(f), 0);
105 }
#define CHECK_EQ(x, y)
Definition: Logger.h:201
CHECK(cgen_state)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

FILE * File_Namespace::create ( const std::string &  basePath,
const int  fileId,
const size_t  pageSize,
const size_t  numPages 
)

Definition at line 34 of file File.cpp.

References logger::FATAL, fileSize(), LOG, MAPD_FILE_EXT, and to_string().

Referenced by File_Namespace::FileMgr::createDBMetaFile(), File_Namespace::FileMgr::createEpochFile(), File_Namespace::FileMgr::createFile(), kafka_insert(), and org.apache.calcite.sql2rel.SqlToRelConverter::SqlToRelConverter().

37  {
38  std::string path(basePath + "/" + std::to_string(fileId) + "." +
39  std::to_string(pageSize) +
40  std::string(MAPD_FILE_EXT)); // MAPD_FILE_EXT has preceding "."
41  if (numPages < 1 || pageSize < 1) {
42  LOG(FATAL) << "Error trying to create file '" << path
43  << "', Number of pages and page size must be positive integers. numPages "
44  << numPages << " pageSize " << pageSize;
45  }
46  FILE* f = fopen(path.c_str(), "w+b");
47  if (f == nullptr) {
48  LOG(FATAL) << "Error trying to create file '" << path
49  << "', the error was: " << std::strerror(errno);
50  ;
51  }
52  fseek(f, (pageSize * numPages) - 1, SEEK_SET);
53  fputc(EOF, f);
54  fseek(f, 0, SEEK_SET); // rewind
55  if (fileSize(f) != pageSize * numPages) {
56  LOG(FATAL) << "Error trying to create file '" << path << "', file size "
57  << fileSize(f) << " does not equal pageSize * numPages "
58  << pageSize * numPages;
59  }
60 
61  return f;
62 }
#define LOG(tag)
Definition: Logger.h:188
#define MAPD_FILE_EXT
Definition: File.h:26
std::string to_string(char const *&&v)
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:170

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

FILE * File_Namespace::create ( const std::string &  fullPath,
const size_t  requestedFileSize 
)

Definition at line 64 of file File.cpp.

References logger::FATAL, fileSize(), and LOG.

64  {
65  FILE* f = fopen(fullPath.c_str(), "w+b");
66  if (f == nullptr) {
67  LOG(FATAL) << "Error trying to create file '" << fullPath
68  << "', the error was: " << std::strerror(errno);
69  ;
70  }
71  fseek(f, requestedFileSize - 1, SEEK_SET);
72  fputc(EOF, f);
73  fseek(f, 0, SEEK_SET); // rewind
74  if (fileSize(f) != requestedFileSize) {
75  LOG(FATAL) << "Error trying to create file '" << fullPath << "', file size "
76  << fileSize(f) << " does not equal requestedFileSize "
77  << requestedFileSize;
78  }
79  return f;
80 }
#define LOG(tag)
Definition: Logger.h:188
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:170

+ Here is the call graph for this function:

size_t File_Namespace::fileSize ( FILE *  f)

Returns the size of the specified file.

Todo:
There may be an issue casting to size_t from long.
Parameters
fA pointer to the file.
Returns
size_t The number of bytes of the file.

Definition at line 170 of file File.cpp.

Referenced by append(), appendPage(), create(), and File_Namespace::FileMgr::init().

170  {
171  fseek(f, 0, SEEK_END);
172  size_t size = (size_t)ftell(f);
173  fseek(f, 0, SEEK_SET);
174  return size;
175 }

+ Here is the caller graph for this function:

bool File_Namespace::headerCompare ( const HeaderInfo &  firstElem,
const HeaderInfo &  secondElem 
)

Definition at line 49 of file FileMgr.cpp.

References File_Namespace::HeaderInfo::chunkKey, File_Namespace::HeaderInfo::pageId, and File_Namespace::HeaderInfo::versionEpoch.

Referenced by File_Namespace::FileMgr::init().

49  {
50  // HeaderInfo.first is a pair of Chunk key with a vector containing
51  // pageId and version
52  if (firstElem.chunkKey != secondElem.chunkKey) {
53  return firstElem.chunkKey < secondElem.chunkKey;
54  }
55  if (firstElem.pageId != secondElem.pageId) {
56  return firstElem.pageId < secondElem.pageId;
57  }
58  return firstElem.versionEpoch < secondElem.versionEpoch;
59 
60  /*
61  if (firstElem.first.first != secondElem.first.first)
62  return firstElem.first.first < secondElem.first.first;
63  return firstElem.first.second < secondElem.first.second;
64  */
65 }

+ Here is the caller graph for this function:

FILE * File_Namespace::open ( int  fileId)

Opens/creates the file with the given id; returns NULL on error.

Parameters
fileIdThe id of the file to open.
Returns
FILE* A pointer to a FILE pointer, or NULL on error.

Definition at line 82 of file File.cpp.

References logger::FATAL, LOG, MAPD_FILE_EXT, and to_string().

Referenced by anonymous_namespace{StringDictionary.cpp}::checked_open(), create_table.SyntheticTable::generateData(), analyze_benchmark.BenchmarkLoader::load(), File_Namespace::FileMgr::openDBMetaFile(), File_Namespace::FileMgr::openEpochFile(), File_Namespace::FileMgr::openExistingFile(), run_benchmark::read_query_files(), run_benchmark::send_results_db(), run_benchmark::send_results_file_json(), run_benchmark::send_results_jenkins_bench(), and MapDProgramOptions::validate().

82  {
83  std::string s(std::to_string(fileId) + std::string(MAPD_FILE_EXT));
84  FILE* f = fopen(s.c_str(), "r+b"); // opens existing file for updates
85  if (f == nullptr) {
86  LOG(FATAL) << "Error trying to open file '" << s
87  << "', the error was: " << std::strerror(errno);
88  }
89  return f;
90 }
#define LOG(tag)
Definition: Logger.h:188
#define MAPD_FILE_EXT
Definition: File.h:26
std::string to_string(char const *&&v)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

FILE * File_Namespace::open ( const std::string &  path)

Definition at line 92 of file File.cpp.

References logger::FATAL, and LOG.

92  {
93  FILE* f = fopen(path.c_str(), "r+b"); // opens existing file for updates
94  if (f == nullptr) {
95  LOG(FATAL) << "Error trying to open file '" << path
96  << "', the errno was: " << std::strerror(errno);
97  }
98  return f;
99 }
#define LOG(tag)
Definition: Logger.h:188
size_t File_Namespace::read ( FILE *  f,
const size_t  offset,
const size_t  size,
int8_t *  buf 
)

Reads the specified number of bytes from the offset position in file f into buf.

Parameters
fPointer to the FILE.
offsetThe location within the file from which to read.
nThe number of bytes to be read.
bufThe destination buffer to where data is being read from the file.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes read.

Definition at line 112 of file File.cpp.

References CHECK_EQ.

Referenced by File_Namespace::FileMgr::openDBMetaFile(), File_Namespace::FileMgr::openEpochFile(), File_Namespace::FileInfo::read(), readPage(), and readPartialPage().

112  {
113  // read "size" bytes from the offset location in the file into the buffer
114  CHECK_EQ(fseek(f, offset, SEEK_SET), 0);
115  size_t bytesRead = fread(buf, sizeof(int8_t), size, f);
116  CHECK_EQ(bytesRead, sizeof(int8_t) * size);
117  return bytesRead;
118 }
#define CHECK_EQ(x, y)
Definition: Logger.h:201

+ Here is the caller graph for this function:

static size_t File_Namespace::readForThread ( FileBuffer *  fileBuffer,
const readThreadDS  threadDS 
)
static

Definition at line 197 of file FileBuffer.cpp.

References CHECK(), File_Namespace::FileMgr::getFileInfoForFileId(), File_Namespace::readThreadDS::multiPages, File_Namespace::FileBuffer::pageDataSize(), File_Namespace::FileBuffer::pageSize(), File_Namespace::FileInfo::read(), File_Namespace::FileBuffer::reservedHeaderSize(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.

Referenced by File_Namespace::FileBuffer::read().

197  {
198  size_t startPage = threadDS.t_startPage; // start reading at startPage, including it
199  size_t endPage = threadDS.t_endPage; // stop reading at endPage, not including it
200  int8_t* curPtr = threadDS.t_curPtr;
201  size_t bytesLeft = threadDS.t_bytesLeft;
202  size_t totalBytesRead = 0;
203  bool isFirstPage = threadDS.t_isFirstPage;
204 
205  // Traverse the logical pages
206  for (size_t pageNum = startPage; pageNum < endPage; ++pageNum) {
207  CHECK(threadDS.multiPages[pageNum].pageSize == fileBuffer->pageSize());
208  Page page = threadDS.multiPages[pageNum].current();
209 
210  FileInfo* fileInfo = threadDS.t_fm->getFileInfoForFileId(page.fileId);
211  CHECK(fileInfo);
212 
213  // Read the page into the destination (dst) buffer at its
214  // current (cur) location
215  size_t bytesRead = 0;
216  if (isFirstPage) {
217  bytesRead = fileInfo->read(
218  page.pageNum * fileBuffer->pageSize() + threadDS.t_startPageOffset +
219  fileBuffer->reservedHeaderSize(),
220  min(fileBuffer->pageDataSize() - threadDS.t_startPageOffset, bytesLeft),
221  curPtr);
222  isFirstPage = false;
223  } else {
224  bytesRead = fileInfo->read(
225  page.pageNum * fileBuffer->pageSize() + fileBuffer->reservedHeaderSize(),
226  min(fileBuffer->pageDataSize(), bytesLeft),
227  curPtr);
228  }
229  curPtr += bytesRead;
230  bytesLeft -= bytesRead;
231  totalBytesRead += bytesRead;
232  }
233  CHECK(bytesLeft == 0);
234 
235  return (totalBytesRead);
236 }
CHECK(cgen_state)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::readPage ( FILE *  f,
const size_t  pageSize,
const size_t  pageNum,
int8_t *  buf 
)

Reads the specified page from the file f into buf.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
pageNumThe page number from where data is being read.
bufThe destination buffer to where data is being written.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes read (should be equal to pageSize).

Definition at line 139 of file File.cpp.

References read().

139  {
140  return read(f, pageNum * pageSize, pageSize, buf);
141 }
size_t read(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Reads the specified number of bytes from the offset position in file f into buf.
Definition: File.cpp:112

+ Here is the call graph for this function:

size_t File_Namespace::readPartialPage ( FILE *  f,
const size_t  pageSize,
const size_t  offset,
const size_t  readSize,
const size_t  pageNum,
int8_t *  buf 
)

Definition at line 143 of file File.cpp.

References read().

148  {
149  return read(f, pageNum * pageSize + offset, readSize, buf);
150 }
size_t read(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Reads the specified number of bytes from the offset position in file f into buf.
Definition: File.cpp:112

+ Here is the call graph for this function:

bool File_Namespace::removeFile ( const std::string  basePath,
const std::string  filename 
)

Deletes the file pointed to by the FILE pointer.

Parameters
basePathThe base path (directory) of the file.
fPointer to the FILE.
Returns
mapd_err_t Returns an error code when unable to close the file properly.

Definition at line 107 of file File.cpp.

References logger::filename().

107  {
108  const std::string filePath = basePath + filename;
109  return remove(filePath.c_str()) == 0;
110 }
std::string filename(char const *path)
Definition: Logger.cpp:58

+ Here is the call graph for this function:

void File_Namespace::renameForDelete ( const std::string  directoryName)

Renames a directory to DELETE_ME_<EPOCH>_<oldname>.

Parameters
directoryNamename of directory

Definition at line 182 of file File.cpp.

References logger::ERROR, error_code, logger::FATAL, LOG, and to_string().

Referenced by File_Namespace::FileMgr::closeRemovePhysical(), Catalog_Namespace::Catalog::delDictionary(), Catalog_Namespace::Catalog::doTruncateTable(), and Catalog_Namespace::Catalog::removeTableFromMap().

182  {
184  boost::filesystem::path directoryPath(directoryName);
185  using namespace std::chrono;
186  milliseconds ms = duration_cast<milliseconds>(system_clock::now().time_since_epoch());
187 
188  if (boost::filesystem::exists(directoryPath) &&
189  boost::filesystem::is_directory(directoryPath)) {
190  boost::filesystem::path newDirectoryPath(directoryName + "_" +
191  std::to_string(ms.count()) + "_DELETE_ME");
192  boost::filesystem::rename(directoryPath, newDirectoryPath, ec);
193 
194  if (ec.value() == boost::system::errc::success) {
195  std::thread th([newDirectoryPath]() {
197  boost::filesystem::remove_all(newDirectoryPath, ec);
198  if (ec.value() != boost::system::errc::success) {
199  LOG(ERROR) << "Failed to remove directory " << newDirectoryPath << " error was "
200  << ec;
201  }
202  });
203  // let it run free so we can return
204  // if it fails the file_delete_thread in MapDHandler will clean up
205  th.detach();
206 
207  return;
208  }
209 
210  LOG(FATAL) << "Failed to rename file " << directoryName << " to "
211  << directoryName + "_" + std::to_string(ms.count()) + "_DELETE_ME Error: "
212  << ec;
213  }
214 }
#define LOG(tag)
Definition: Logger.h:188
std::string to_string(char const *&&v)
const int8_t const int64_t const uint64_t const int32_t const int64_t int64_t uint32_t const int64_t int32_t * error_code

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::write ( FILE *  f,
const size_t  offset,
const size_t  size,
int8_t *  buf 
)

Writes the specified number of bytes to the offset position in file f from buf.

Parameters
fPointer to the FILE.
offsetThe location within the file where data is being written.
sizeThe number of bytes to write to the file.
bufThe source buffer containing the data to be written.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written.

Definition at line 120 of file File.cpp.

References logger::FATAL, and LOG.

Referenced by StringDictionary::addStorageCapacity(), append(), appendPage(), File_Namespace::FileMgr::createDBMetaFile(), File_Namespace::FileMgr::createEpochFile(), File_Namespace::FileInfo::freePage(), Importer_NS::DataStreamSink::import_compressed(), File_Namespace::FileInfo::initNewFile(), File_Namespace::FileInfo::openExistingFile(), MapDProgramOptions::validate(), File_Namespace::FileInfo::write(), File_Namespace::FileMgr::writeAndSyncDBMetaToDisk(), File_Namespace::FileMgr::writeAndSyncEpochToDisk(), writePage(), and writePartialPage().

120  {
121  // write size bytes from the buffer to the offset location in the file
122  if (fseek(f, offset, SEEK_SET) != 0) {
123  LOG(FATAL)
124  << "Error trying to write to file (during positioning seek) the error was: "
125  << std::strerror(errno);
126  }
127  size_t bytesWritten = fwrite(buf, sizeof(int8_t), size, f);
128  if (bytesWritten != sizeof(int8_t) * size) {
129  LOG(FATAL) << "Error trying to write to file (during fwrite) the error was: "
130  << std::strerror(errno);
131  }
132  return bytesWritten;
133 }
#define LOG(tag)
Definition: Logger.h:188

+ Here is the caller graph for this function:

size_t File_Namespace::writePage ( FILE *  f,
const size_t  pageSize,
const size_t  pageNum,
int8_t *  buf 
)

Writes a page from buf to the file.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
pageNumThe page number to where data is being written.
bufThe source buffer from where data is being read.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written (should be equal to pageSize).

Definition at line 152 of file File.cpp.

References write().

152  {
153  return write(f, pageNum * pageSize, pageSize, buf);
154 }
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:120

+ Here is the call graph for this function:

size_t File_Namespace::writePartialPage ( FILE *  f,
const size_t  pageSize,
const size_t  offset,
const size_t  writeSize,
const size_t  pageNum,
int8_t *  buf 
)

Definition at line 156 of file File.cpp.

References write().

161  {
162  return write(f, pageNum * pageSize + offset, writeSize, buf);
163 }
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:120

+ Here is the call graph for this function: