OmniSciDB  a987f07e93
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
File_Namespace Namespace Reference

Namespaces

 anonymous_namespace{FileMgr.cpp}
 

Classes

struct  DiskCacheConfig
 
class  TableFileMgr
 
class  CachingFileBuffer
 
class  CachingFileMgr
 A FileMgr capable of limiting it's size and storing data from multiple tables in a shared directory. For any table that supports DiskCaching, the CachingFileMgr must contain either metadata for all table chunks, or for none (the cache is either has no knowledge of that table, or has complete knowledge of that table). Any data chunk within a table may or may not be contained within the cache. More...
 
class  CachingGlobalFileMgr
 
struct  readThreadDS
 
class  FileBuffer
 Represents/provides access to contiguous data stored in the file system. More...
 
struct  FileInfo
 
struct  FileMetadata
 
struct  StorageStats
 
struct  OpenFilesResult
 
struct  PageMapping
 
class  FileMgr
 
struct  FileMgrParams
 
class  GlobalFileMgr
 
struct  Page
 A logical page (Page) belongs to a file on disk. More...
 
struct  EpochedPage
 
struct  MultiPage
 The MultiPage stores versions of the same logical page in a deque. More...
 
struct  HeaderInfo
 Stores Pair of ChunkKey and Page id and version, in a pair with a Page struct itself (File id and Page num) More...
 

Typedefs

using PageSizeFileMMap = std::multimap< size_t, int32_t >
 Maps logical page sizes to files. More...
 
using Chunk = FileBuffer
 A Chunk is the fundamental unit of execution in Map-D. More...
 
using ChunkKeyToChunkMap = std::map< ChunkKey, FileBuffer * >
 Maps ChunkKeys (unique ids for Chunks) to Chunk objects. More...
 
using TablePair = std::pair< const int32_t, const int32_t >
 
using PageHeaderSizeType = int32_t
 

Enumerations

enum  DiskCacheLevel { DiskCacheLevel::none, DiskCacheLevel::fsi, DiskCacheLevel::non_fsi, DiskCacheLevel::all }
 

Functions

std::string get_dir_name_for_table (int db_id, int tb_id)
 
static size_t readForThread (FileBuffer *fileBuffer, const readThreadDS threadDS)
 
bool is_page_deleted_with_checkpoint (int32_t table_epoch, int32_t page_epoch, int32_t contingent)
 
bool is_page_deleted_without_checkpoint (int32_t table_epoch, int32_t page_epoch, int32_t contingent)
 
std::string get_data_file_path (const std::string &base_path, int file_id, size_t page_size)
 
std::string get_legacy_data_file_path (const std::string &new_data_file_path)
 
FILE * create (const std::string &basePath, const int fileId, const size_t pageSize, const size_t numPages)
 
FILE * create (const std::string &fullPath, const size_t requestedFileSize)
 
FILE * open (int fileId)
 Opens/creates the file with the given id; returns NULL on error. More...
 
FILE * open (const std::string &path)
 
void close (FILE *f)
 Closes the file pointed to by the FILE pointer. More...
 
bool removeFile (const std::string basePath, const std::string filename)
 Deletes the file pointed to by the FILE pointer. More...
 
size_t read (FILE *f, const size_t offset, const size_t size, int8_t *buf)
 Reads the specified number of bytes from the offset position in file f into buf. More...
 
size_t write (FILE *f, const size_t offset, const size_t size, const int8_t *buf)
 Writes the specified number of bytes to the offset position in file f from buf. More...
 
size_t append (FILE *f, const size_t size, const int8_t *buf)
 Appends the specified number of bytes to the end of the file f from buf. More...
 
size_t readPage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf)
 Reads the specified page from the file f into buf. More...
 
size_t readPartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t readSize, const size_t pageNum, int8_t *buf)
 
size_t writePage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf)
 Writes a page from buf to the file. More...
 
size_t writePartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t writeSize, const size_t pageNum, int8_t *buf)
 
size_t appendPage (FILE *f, const size_t pageSize, int8_t *buf)
 Appends a page from buf to the file. More...
 
size_t fileSize (FILE *f)
 Returns the size of the specified file. More...
 
void renameForDelete (const std::string directoryName)
 Renames a directory to DELETE_ME_<EPOCH>_<oldname>. More...
 

Variables

constexpr int32_t DELETE_CONTINGENT = -1
 A FileInfo type has a file pointer and metadata about a file. More...
 
constexpr int32_t ROLLOFF_CONTINGENT = -2
 
constexpr auto kLegacyDataFileExtension {".mapd"}
 

Typedef Documentation

A Chunk is the fundamental unit of execution in Map-D.

Chunk A chunk is composed of logical pages. These pages can exist across multiple files managed by the file manager.

The collection of pages is implemented as a FileBuffer object, which is composed of a vector of MultiPage objects, one for each logical page of the file buffer.

Definition at line 73 of file FileMgr.h.

Maps ChunkKeys (unique ids for Chunks) to Chunk objects.

ChunkKeyToChunkMap The file system can store multiple chunks across multiple files. With that in mind, the challenge is to be able to reconstruct the pages that compose a chunk upon request. A chunk key (ChunkKey) uniquely identifies a chunk, and so ChunkKeyToChunkMap maps chunk keys to Chunk types, which are vectors of MultiPage* pointers (logical pages).

Definition at line 85 of file FileMgr.h.

using File_Namespace::PageHeaderSizeType = typedef int32_t

Definition at line 127 of file FileMgr.h.

using File_Namespace::PageSizeFileMMap = typedef std::multimap<size_t, int32_t>

Maps logical page sizes to files.

PageSizeFileMMap The file manager uses this type in order to quickly find files of a certain page size. A multimap is used to associate the key (page size) with values (file identifiers of files having the matching page size).

Definition at line 61 of file FileMgr.h.

using File_Namespace::TablePair = typedef std::pair<const int32_t, const int32_t>

TablePair Pair detailing the id for a database and table (first two entries in a ChunkKey).

Definition at line 91 of file FileMgr.h.

Enumeration Type Documentation

Function Documentation

size_t File_Namespace::append ( FILE *  f,
const size_t  size,
const int8_t *  buf 
)

Appends the specified number of bytes to the end of the file f from buf.

Parameters
fPointer to the FILE.
nThe number of bytes to append to the file.
bufThe source buffer containing the data to be appended.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written.

Definition at line 168 of file File.cpp.

References logger::FATAL, fileSize(), g_read_only, LOG, and write().

Referenced by benchmarks.GoogleBenchmark::aggregateBenchmarks(), run_benchmark::benchmark(), concat(), concat_with(), com.mapd.utility.SQLImporter::createDBTable(), com.mapd.calcite.parser.HeavyDBSqlOperatorTable.ExtTableFunction::getExtendedSignature(), ai.heavy.jdbc.HeavyAIPreparedStatement::getQuery(), com.mapd.parser.server.ExtensionFunctionSignatureParser::join(), run_benchmark::read_query_files(), run_benchmark_arrow::run_query(), run_benchmark::run_query(), ai.heavy.jdbc.HeavyAIEscapeFunctions::singleArgumentFunctionCall(), com.mapd.parser.server.ExtensionFunction::toJson(), and ai.heavy.jdbc.HeavyAIArray::toString().

168  {
169  if (g_read_only) {
170  LOG(FATAL) << "Error trying to append file '" << f << "', running readonly";
171  }
172  return write(f, fileSize(f), size, buf);
173 }
#define LOG(tag)
Definition: Logger.h:283
constexpr double f
Definition: Utm.h:31
size_t write(FILE *f, const size_t offset, const size_t size, const int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:150
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:215
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::appendPage ( FILE *  f,
const size_t  pageSize,
int8_t *  buf 
)

Appends a page from buf to the file.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
bufThe source buffer from where data is being read.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes appended (should be equal to pageSize).

Definition at line 207 of file File.cpp.

References logger::FATAL, fileSize(), g_read_only, LOG, and write().

207  {
208  if (g_read_only) {
209  LOG(FATAL) << "Error trying to appendPage file '" << f << "', running readonly";
210  }
211  return write(f, fileSize(f), pageSize, buf);
212 }
#define LOG(tag)
Definition: Logger.h:283
constexpr double f
Definition: Utm.h:31
size_t write(FILE *f, const size_t offset, const size_t size, const int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:150
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:215
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

void File_Namespace::close ( FILE *  f)

Closes the file pointed to by the FILE pointer.

Parameters
fPointer to the FILE.

Definition at line 128 of file File.cpp.

References CHECK, and CHECK_EQ.

Referenced by File_Namespace::FileMgr::clearFileInfos(), File_Namespace::FileMgr::closePhysicalUnlocked(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileMgr::readVersionFromDisk(), File_Namespace::FileMgr::writeAndSyncVersionToDisk(), File_Namespace::FileInfo::~FileInfo(), File_Namespace::FileMgr::~FileMgr(), and File_Namespace::TableFileMgr::~TableFileMgr().

128  {
129  CHECK(f);
130  CHECK_EQ(fflush(f), 0);
131  CHECK_EQ(fclose(f), 0);
132 }
#define CHECK_EQ(x, y)
Definition: Logger.h:297
constexpr double f
Definition: Utm.h:31
#define CHECK(condition)
Definition: Logger.h:289

+ Here is the caller graph for this function:

FILE * File_Namespace::create ( const std::string &  basePath,
const int  fileId,
const size_t  pageSize,
const size_t  numPages 
)

Definition at line 57 of file File.cpp.

References anonymous_namespace{Utm.h}::f, logger::FATAL, nvtx_helpers::anonymous_namespace{nvtx_helpers.cpp}::filename(), fileSize(), heavyai::fopen(), get_data_file_path(), get_legacy_data_file_path(), and LOG.

Referenced by File_Namespace::FileMgr::createEpochFile(), File_Namespace::FileMgr::createFile(), UdfCompiler::generateAST(), kafka_insert(), org.apache.calcite.sql2rel.SqlToRelConverter::SqlToRelConverter(), org.apache.calcite.sql.validate.SqlValidatorImpl::SqlValidatorImpl(), File_Namespace::TableFileMgr::TableFileMgr(), and File_Namespace::FileMgr::writeAndSyncVersionToDisk().

60  {
61  auto path = get_data_file_path(basePath, fileId, pageSize);
62  if (numPages < 1 || pageSize < 1) {
63  LOG(FATAL) << "Error trying to create file '" << path
64  << "', Number of pages and page size must be positive integers. numPages "
65  << numPages << " pageSize " << pageSize;
66  }
67  FILE* f = heavyai::fopen(path.c_str(), "w+b");
68  if (f == nullptr) {
69  LOG(FATAL) << "Error trying to create file '" << path
70  << "', the error was: " << std::strerror(errno);
71  }
72  fseek(f, static_cast<long>((pageSize * numPages) - 1), SEEK_SET);
73  fputc(EOF, f);
74  fseek(f, 0, SEEK_SET); // rewind
75  if (fileSize(f) != pageSize * numPages) {
76  LOG(FATAL) << "Error trying to create file '" << path << "', file size "
77  << fileSize(f) << " does not equal pageSize * numPages "
78  << pageSize * numPages;
79  }
80  boost::filesystem::create_symlink(boost::filesystem::canonical(path).filename(),
82  return f;
83 }
std::string get_legacy_data_file_path(const std::string &new_data_file_path)
Definition: File.cpp:51
#define LOG(tag)
Definition: Logger.h:283
constexpr double f
Definition: Utm.h:31
::FILE * fopen(const char *filename, const char *mode)
Definition: heavyai_fs.cpp:74
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:215
std::string get_data_file_path(const std::string &base_path, int file_id, size_t page_size)
Definition: File.cpp:44

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

FILE * File_Namespace::create ( const std::string &  fullPath,
const size_t  requestedFileSize 
)

Definition at line 85 of file File.cpp.

References anonymous_namespace{Utm.h}::f, logger::FATAL, fileSize(), heavyai::fopen(), g_read_only, and LOG.

85  {
86  if (g_read_only) {
87  LOG(FATAL) << "Error trying to create file '" << fullPath
88  << "', not allowed read only ";
89  }
90  FILE* f = heavyai::fopen(fullPath.c_str(), "w+b");
91  if (f == nullptr) {
92  LOG(FATAL) << "Error trying to create file '" << fullPath
93  << "', the error was: " << std::strerror(errno);
94  ;
95  }
96  fseek(f, static_cast<long>(requestedFileSize - 1), SEEK_SET);
97  fputc(EOF, f);
98  fseek(f, 0, SEEK_SET); // rewind
99  if (fileSize(f) != requestedFileSize) {
100  LOG(FATAL) << "Error trying to create file '" << fullPath << "', file size "
101  << fileSize(f) << " does not equal requestedFileSize "
102  << requestedFileSize;
103  }
104  return f;
105 }
#define LOG(tag)
Definition: Logger.h:283
constexpr double f
Definition: Utm.h:31
::FILE * fopen(const char *filename, const char *mode)
Definition: heavyai_fs.cpp:74
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:215
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

size_t File_Namespace::fileSize ( FILE *  f)

Returns the size of the specified file.

Todo:
There may be an issue casting to size_t from long.
Parameters
fA pointer to the file.
Returns
size_t The number of bytes of the file.

Definition at line 215 of file File.cpp.

Referenced by append(), appendPage(), and create().

215  {
216  fseek(f, 0, SEEK_END);
217  size_t size = (size_t)ftell(f);
218  fseek(f, 0, SEEK_SET);
219  return size;
220 }
constexpr double f
Definition: Utm.h:31

+ Here is the caller graph for this function:

std::string File_Namespace::get_data_file_path ( const std::string &  base_path,
int  file_id,
size_t  page_size 
)

Definition at line 44 of file File.cpp.

References DATA_FILE_EXT, and to_string().

Referenced by create(), and File_Namespace::FileMgr::deleteEmptyFiles().

46  {
47  return base_path + "/" + std::to_string(file_id) + "." + std::to_string(page_size) +
48  std::string(DATA_FILE_EXT); // DATA_FILE_EXT has preceding "."
49 }
#define DATA_FILE_EXT
Definition: File.h:25
std::string to_string(char const *&&v)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string File_Namespace::get_dir_name_for_table ( int  db_id,
int  tb_id 
)
inline

Definition at line 84 of file CachingFileMgr.h.

Referenced by File_Namespace::CachingFileMgr::getTableFileMgrPath().

84  {
85  std::stringstream file_name;
86  file_name << "table_" << db_id << "_" << tb_id << "/";
87  return file_name.str();
88 }

+ Here is the caller graph for this function:

std::string File_Namespace::get_legacy_data_file_path ( const std::string &  new_data_file_path)

Definition at line 51 of file File.cpp.

References kLegacyDataFileExtension.

Referenced by create(), and File_Namespace::FileMgr::deleteEmptyFiles().

51  {
52  auto legacy_path = boost::filesystem::canonical(new_data_file_path);
53  legacy_path.replace_extension(kLegacyDataFileExtension);
54  return legacy_path.string();
55 }
constexpr auto kLegacyDataFileExtension
Definition: File.h:36

+ Here is the caller graph for this function:

bool File_Namespace::is_page_deleted_with_checkpoint ( int32_t  table_epoch,
int32_t  page_epoch,
int32_t  contingent 
)

Definition at line 271 of file FileInfo.cpp.

References DELETE_CONTINGENT, and ROLLOFF_CONTINGENT.

Referenced by anonymous_namespace{TableArchiver.cpp}::rewrite_column_ids_in_page_headers(), and File_Namespace::FileMgr::updatePageIfDeleted().

273  {
274  const bool delete_contingent =
275  (contingent == DELETE_CONTINGENT || contingent == ROLLOFF_CONTINGENT);
276  // Check if page was deleted with a checkpointed epoch
277  if (delete_contingent && (table_epoch >= page_epoch)) {
278  return true;
279  }
280  return false;
281 }
constexpr int32_t DELETE_CONTINGENT
A FileInfo type has a file pointer and metadata about a file.
Definition: FileInfo.h:51
constexpr int32_t ROLLOFF_CONTINGENT
Definition: FileInfo.h:52

+ Here is the caller graph for this function:

bool File_Namespace::is_page_deleted_without_checkpoint ( int32_t  table_epoch,
int32_t  page_epoch,
int32_t  contingent 
)

Definition at line 283 of file FileInfo.cpp.

References DELETE_CONTINGENT, and ROLLOFF_CONTINGENT.

Referenced by File_Namespace::FileMgr::updatePageIfDeleted().

285  {
286  const bool delete_contingent =
287  (contingent == DELETE_CONTINGENT || contingent == ROLLOFF_CONTINGENT);
288  // Check if page was deleted but the epoch was not yet checkpointed.
289  if (delete_contingent && (table_epoch < page_epoch)) {
290  return true;
291  }
292  return false;
293 }
constexpr int32_t DELETE_CONTINGENT
A FileInfo type has a file pointer and metadata about a file.
Definition: FileInfo.h:51
constexpr int32_t ROLLOFF_CONTINGENT
Definition: FileInfo.h:52

+ Here is the caller graph for this function:

FILE * File_Namespace::open ( int  fileId)

Opens/creates the file with the given id; returns NULL on error.

Parameters
fileIdThe id of the file to open.
Returns
FILE* A pointer to a FILE pointer, or NULL on error.

Definition at line 107 of file File.cpp.

References DATA_FILE_EXT, anonymous_namespace{Utm.h}::f, logger::FATAL, heavyai::fopen(), g_read_only, LOG, and to_string().

Referenced by File_Namespace::FileMgr::openAndReadEpochFile(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileMgr::openExistingFile(), File_Namespace::FileMgr::readVersionFromDisk(), File_Namespace::TableFileMgr::TableFileMgr(), and File_Namespace::FileMgr::writeAndSyncVersionToDisk().

107  {
108  std::string s(std::to_string(fileId) + std::string(DATA_FILE_EXT));
109  FILE* f = heavyai::fopen(
110  s.c_str(), g_read_only ? "rb" : "r+b"); // opens existing file for updates
111  if (f == nullptr) {
112  LOG(FATAL) << "Error trying to open file '" << s
113  << "', the error was: " << std::strerror(errno);
114  }
115  return f;
116 }
#define LOG(tag)
Definition: Logger.h:283
#define DATA_FILE_EXT
Definition: File.h:25
constexpr double f
Definition: Utm.h:31
std::string to_string(char const *&&v)
::FILE * fopen(const char *filename, const char *mode)
Definition: heavyai_fs.cpp:74
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

FILE * File_Namespace::open ( const std::string &  path)

Definition at line 118 of file File.cpp.

References anonymous_namespace{Utm.h}::f, logger::FATAL, heavyai::fopen(), g_read_only, and LOG.

118  {
119  FILE* f = heavyai::fopen(
120  path.c_str(), g_read_only ? "rb" : "r+b"); // opens existing file for updates
121  if (f == nullptr) {
122  LOG(FATAL) << "Error trying to open file '" << path
123  << "', the errno was: " << std::strerror(errno);
124  }
125  return f;
126 }
#define LOG(tag)
Definition: Logger.h:283
constexpr double f
Definition: Utm.h:31
::FILE * fopen(const char *filename, const char *mode)
Definition: heavyai_fs.cpp:74
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

size_t File_Namespace::read ( FILE *  f,
const size_t  offset,
const size_t  size,
int8_t *  buf 
)

Reads the specified number of bytes from the offset position in file f into buf.

Parameters
fPointer to the FILE.
offsetThe location within the file from which to read.
nThe number of bytes to be read.
bufThe destination buffer to where data is being read from the file.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes read.

Definition at line 142 of file File.cpp.

References CHECK_EQ.

Referenced by File_Namespace::FileMgr::openAndReadEpochFile(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileInfo::read(), readPage(), readPartialPage(), File_Namespace::FileMgr::readVersionFromDisk(), heavyai::safe_read(), and File_Namespace::TableFileMgr::TableFileMgr().

142  {
143  // read "size" bytes from the offset location in the file into the buffer
144  CHECK_EQ(fseek(f, static_cast<long>(offset), SEEK_SET), 0);
145  size_t bytesRead = fread(buf, sizeof(int8_t), size, f);
146  CHECK_EQ(bytesRead, sizeof(int8_t) * size);
147  return bytesRead;
148 }
#define CHECK_EQ(x, y)
Definition: Logger.h:297
constexpr double f
Definition: Utm.h:31

+ Here is the caller graph for this function:

static size_t File_Namespace::readForThread ( FileBuffer *  fileBuffer,
const readThreadDS  threadDS 
)
static

Definition at line 245 of file FileBuffer.cpp.

References CHECK, File_Namespace::FileMgr::getFileInfoForFileId(), File_Namespace::readThreadDS::multiPages, File_Namespace::FileBuffer::pageDataSize(), File_Namespace::FileBuffer::pageSize(), File_Namespace::FileInfo::read(), File_Namespace::FileBuffer::reservedHeaderSize(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.

Referenced by File_Namespace::FileBuffer::read().

245  {
246  size_t startPage = threadDS.t_startPage; // start reading at startPage, including it
247  size_t endPage = threadDS.t_endPage; // stop reading at endPage, not including it
248  int8_t* curPtr = threadDS.t_curPtr;
249  size_t bytesLeft = threadDS.t_bytesLeft;
250  size_t totalBytesRead = 0;
251  bool isFirstPage = threadDS.t_isFirstPage;
252 
253  // Traverse the logical pages
254  for (size_t pageNum = startPage; pageNum < endPage; ++pageNum) {
255  CHECK(threadDS.multiPages[pageNum].pageSize == fileBuffer->pageSize());
256  Page page = threadDS.multiPages[pageNum].current().page;
257 
258  FileInfo* fileInfo = threadDS.t_fm->getFileInfoForFileId(page.fileId);
259  CHECK(fileInfo);
260 
261  // Read the page into the destination (dst) buffer at its
262  // current (cur) location
263  size_t bytesRead = 0;
264  if (isFirstPage) {
265  bytesRead = fileInfo->read(
266  page.pageNum * fileBuffer->pageSize() + threadDS.t_startPageOffset +
267  fileBuffer->reservedHeaderSize(),
268  min(fileBuffer->pageDataSize() - threadDS.t_startPageOffset, bytesLeft),
269  curPtr);
270  isFirstPage = false;
271  } else {
272  bytesRead = fileInfo->read(
273  page.pageNum * fileBuffer->pageSize() + fileBuffer->reservedHeaderSize(),
274  min(fileBuffer->pageDataSize(), bytesLeft),
275  curPtr);
276  }
277  curPtr += bytesRead;
278  bytesLeft -= bytesRead;
279  totalBytesRead += bytesRead;
280  }
281  CHECK(bytesLeft == 0);
282 
283  return (totalBytesRead);
284 }
#define CHECK(condition)
Definition: Logger.h:289

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::readPage ( FILE *  f,
const size_t  pageSize,
const size_t  pageNum,
int8_t *  buf 
)

Reads the specified page from the file f into buf.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
pageNumThe page number from where data is being read.
bufThe destination buffer to where data is being written.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes read (should be equal to pageSize).

Definition at line 175 of file File.cpp.

References read().

175  {
176  return read(f, pageNum * pageSize, pageSize, buf);
177 }
constexpr double f
Definition: Utm.h:31
size_t read(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Reads the specified number of bytes from the offset position in file f into buf.
Definition: File.cpp:142

+ Here is the call graph for this function:

size_t File_Namespace::readPartialPage ( FILE *  f,
const size_t  pageSize,
const size_t  offset,
const size_t  readSize,
const size_t  pageNum,
int8_t *  buf 
)

Definition at line 179 of file File.cpp.

References read().

184  {
185  return read(f, pageNum * pageSize + offset, readSize, buf);
186 }
constexpr double f
Definition: Utm.h:31
size_t read(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Reads the specified number of bytes from the offset position in file f into buf.
Definition: File.cpp:142

+ Here is the call graph for this function:

bool File_Namespace::removeFile ( const std::string  basePath,
const std::string  filename 
)

Deletes the file pointed to by the FILE pointer.

Parameters
basePathThe base path (directory) of the file.
fPointer to the FILE.
Returns
mapd_err_t Returns an error code when unable to close the file properly.

Definition at line 134 of file File.cpp.

References logger::FATAL, nvtx_helpers::anonymous_namespace{nvtx_helpers.cpp}::filename(), g_read_only, and LOG.

134  {
135  if (g_read_only) {
136  LOG(FATAL) << "Error trying to remove file '" << filename << "', running readonly";
137  }
138  const std::string filePath = basePath + filename;
139  return remove(filePath.c_str()) == 0;
140 }
#define LOG(tag)
Definition: Logger.h:283
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

void File_Namespace::renameForDelete ( const std::string  directoryName)

Renames a directory to DELETE_ME_<EPOCH>_<oldname>.

Parameters
directoryNamename of directory

Definition at line 227 of file File.cpp.

References logger::ERROR, logger::FATAL, LOG, and to_string().

Referenced by File_Namespace::FileMgr::closeRemovePhysical(), Catalog_Namespace::Catalog::delDictionary(), Catalog_Namespace::Catalog::doTruncateTable(), and Catalog_Namespace::Catalog::removeTableFromMap().

227  {
228  boost::system::error_code ec;
229  boost::filesystem::path directoryPath(directoryName);
230  using namespace std::chrono;
231  milliseconds ms = duration_cast<milliseconds>(system_clock::now().time_since_epoch());
232 
233  if (boost::filesystem::exists(directoryPath) &&
234  boost::filesystem::is_directory(directoryPath)) {
235  boost::filesystem::path newDirectoryPath(directoryName + "_" +
236  std::to_string(ms.count()) + "_DELETE_ME");
237  boost::filesystem::rename(directoryPath, newDirectoryPath, ec);
238 
239 #ifdef _WIN32
240  // On Windows we sometimes fail to rename a directory with System: 5 error
241  // code (access denied). An attempt to stop in debugger and look for opened
242  // handles for some of directory content shows no opened handles and actually
243  // allows renaming to execute successfully. It's not clear why, but a short
244  // pause allows to rename directory successfully. Until reasons are known,
245  // use this retry loop as a workaround.
246  int tries = 10;
247  while (ec.value() != boost::system::errc::success && tries) {
248  LOG(ERROR) << "Failed to rename directory " << directoryPath << " error was " << ec
249  << " (" << tries << " attempts left)";
250  std::this_thread::sleep_for(std::chrono::milliseconds(100 / tries));
251  tries--;
252  boost::filesystem::rename(directoryPath, newDirectoryPath, ec);
253  }
254 #endif
255 
256  if (ec.value() == boost::system::errc::success) {
257  std::thread th([newDirectoryPath]() {
258  boost::system::error_code ec;
259  boost::filesystem::remove_all(newDirectoryPath, ec);
260  // We dont check error on remove here as we cant log the
261  // issue fromdetached thrad, its not safe to LOG from here
262  // This is under investigation as clang detects TSAN issue data race
263  // the main system wide file_delete_thread will clean up any missed files
264  });
265  // let it run free so we can return
266  // if it fails the file_delete_thread in DBHandler will clean up
267  th.detach();
268 
269  return;
270  }
271 
272  LOG(FATAL) << "Failed to rename file " << directoryName << " to "
273  << directoryName + "_" + std::to_string(ms.count()) + "_DELETE_ME Error: "
274  << ec;
275  }
276 }
#define LOG(tag)
Definition: Logger.h:283
std::string to_string(char const *&&v)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::write ( FILE *  f,
const size_t  offset,
const size_t  size,
const int8_t *  buf 
)

Writes the specified number of bytes to the offset position in file f from buf.

Parameters
fPointer to the FILE.
offsetThe location within the file where data is being written.
sizeThe number of bytes to write to the file.
bufThe source buffer containing the data to be written.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written.

Definition at line 150 of file File.cpp.

References logger::FATAL, g_read_only, and LOG.

Referenced by StringDictionary::addStorageCapacity(), append(), appendPage(), File_Namespace::FileInfo::freePage(), File_Namespace::FileInfo::freePageImmediate(), lockmgr::TableLockMgrImpl< TableDataLockMgr >::getClusterTableMutex(), import_export::DataStreamSink::import_compressed(), File_Namespace::FileInfo::initNewFile(), File_Namespace::FileInfo::recoverPage(), heavyai::safe_write(), File_Namespace::FileInfo::write(), File_Namespace::TableFileMgr::writeAndSyncEpochToDisk(), File_Namespace::FileMgr::writeAndSyncEpochToDisk(), File_Namespace::FileMgr::writeAndSyncVersionToDisk(), writePage(), and writePartialPage().

150  {
151  if (g_read_only) {
152  LOG(FATAL) << "Error trying to write file '" << f << "', running readonly";
153  }
154  // write size bytes from the buffer to the offset location in the file
155  if (fseek(f, static_cast<long>(offset), SEEK_SET) != 0) {
156  LOG(FATAL)
157  << "Error trying to write to file (during positioning seek) the error was: "
158  << std::strerror(errno);
159  }
160  size_t bytesWritten = fwrite(buf, sizeof(int8_t), size, f);
161  if (bytesWritten != sizeof(int8_t) * size) {
162  LOG(FATAL) << "Error trying to write to file (during fwrite) the error was: "
163  << std::strerror(errno);
164  }
165  return bytesWritten;
166 }
#define LOG(tag)
Definition: Logger.h:283
constexpr double f
Definition: Utm.h:31
bool g_read_only
Definition: File.cpp:40

+ Here is the caller graph for this function:

size_t File_Namespace::writePage ( FILE *  f,
const size_t  pageSize,
const size_t  pageNum,
int8_t *  buf 
)

Writes a page from buf to the file.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
pageNumThe page number to where data is being written.
bufThe source buffer from where data is being read.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written (should be equal to pageSize).

Definition at line 188 of file File.cpp.

References logger::FATAL, g_read_only, LOG, and write().

188  {
189  if (g_read_only) {
190  LOG(FATAL) << "Error trying to writePage file '" << f << "', running readonly";
191  }
192  return write(f, pageNum * pageSize, pageSize, buf);
193 }
#define LOG(tag)
Definition: Logger.h:283
constexpr double f
Definition: Utm.h:31
size_t write(FILE *f, const size_t offset, const size_t size, const int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:150
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

size_t File_Namespace::writePartialPage ( FILE *  f,
const size_t  pageSize,
const size_t  offset,
const size_t  writeSize,
const size_t  pageNum,
int8_t *  buf 
)

Definition at line 195 of file File.cpp.

References logger::FATAL, g_read_only, LOG, and write().

200  {
201  if (g_read_only) {
202  LOG(FATAL) << "Error trying to writePartialPage file '" << f << "', running readonly";
203  }
204  return write(f, pageNum * pageSize + offset, writeSize, buf);
205 }
#define LOG(tag)
Definition: Logger.h:283
constexpr double f
Definition: Utm.h:31
size_t write(FILE *f, const size_t offset, const size_t size, const int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:150
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

Variable Documentation

constexpr int32_t File_Namespace::DELETE_CONTINGENT = -1

A FileInfo type has a file pointer and metadata about a file.

FileInfo A file info structure wraps around a file pointer in order to contain additional information/metadata about the file that is pertinent to the file manager.

The free pages (freePages) within a file must be tracked, and this is implemented using a basic STL set. The set ensures that no duplicate pages are included, and that the pages are sorted, faciliating the obtaining of consecutive free pages by a constant time pop operation, which may reduce the cost of DBMS disk accesses.

Helper functions are provided: size(), available(), and used().

Definition at line 51 of file FileInfo.h.

Referenced by File_Namespace::FileInfo::freePage(), is_page_deleted_with_checkpoint(), is_page_deleted_without_checkpoint(), and File_Namespace::CachingFileMgr::updatePageIfDeleted().

constexpr auto File_Namespace::kLegacyDataFileExtension {".mapd"}
constexpr int32_t File_Namespace::ROLLOFF_CONTINGENT = -2