OmniSciDB  c1a53651b2
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
File_Namespace Namespace Reference

Namespaces

 anonymous_namespace{FileMgr.cpp}
 

Classes

struct  DiskCacheConfig
 
class  TableFileMgr
 
class  CachingFileBuffer
 
class  CachingFileMgr
 A FileMgr capable of limiting it's size and storing data from multiple tables in a shared directory. For any table that supports DiskCaching, the CachingFileMgr must contain either metadata for all table chunks, or for none (the cache is either has no knowledge of that table, or has complete knowledge of that table). Any data chunk within a table may or may not be contained within the cache. More...
 
class  CachingGlobalFileMgr
 
struct  readThreadDS
 
class  FileBuffer
 Represents/provides access to contiguous data stored in the file system. More...
 
struct  FileInfo
 
struct  FileMetadata
 
struct  StorageStats
 
struct  OpenFilesResult
 
struct  PageMapping
 
class  FileMgr
 
struct  FileMgrParams
 
class  GlobalFileMgr
 
struct  Page
 A logical page (Page) belongs to a file on disk. More...
 
struct  EpochedPage
 
struct  MultiPage
 The MultiPage stores versions of the same logical page in a deque. More...
 
struct  HeaderInfo
 Stores Pair of ChunkKey and Page id and version, in a pair with a Page struct itself (File id and Page num) More...
 

Typedefs

using PageSizeFileMMap = std::multimap< size_t, int32_t >
 Maps logical page sizes to files. More...
 
using Chunk = FileBuffer
 A Chunk is the fundamental unit of execution in Map-D. More...
 
using ChunkKeyToChunkMap = std::map< ChunkKey, FileBuffer * >
 Maps ChunkKeys (unique ids for Chunks) to Chunk objects. More...
 
using TablePair = std::pair< const int32_t, const int32_t >
 
using PageHeaderSizeType = int32_t
 

Enumerations

enum  DiskCacheLevel { DiskCacheLevel::none, DiskCacheLevel::fsi, DiskCacheLevel::non_fsi, DiskCacheLevel::all }
 

Functions

std::string get_dir_name_for_table (int db_id, int tb_id)
 
static size_t readForThread (FileBuffer *fileBuffer, const readThreadDS threadDS)
 
bool is_page_deleted_with_checkpoint (int32_t table_epoch, int32_t page_epoch, int32_t contingent)
 
bool is_page_deleted_without_checkpoint (int32_t table_epoch, int32_t page_epoch, int32_t contingent)
 
std::string get_data_file_path (const std::string &base_path, int file_id, size_t page_size)
 
std::string get_legacy_data_file_path (const std::string &new_data_file_path)
 
std::pair< FILE *, std::string > create (const std::string &basePath, const int fileId, const size_t pageSize, const size_t numPages)
 
FILE * create (const std::string &fullPath, const size_t requestedFileSize)
 
FILE * open (int fileId)
 Opens/creates the file with the given id; returns NULL on error. More...
 
FILE * open (const std::string &path)
 
void close (FILE *f)
 Closes the file pointed to by the FILE pointer. More...
 
bool removeFile (const std::string basePath, const std::string filename)
 Deletes the file pointed to by the FILE pointer. More...
 
size_t read (FILE *f, const size_t offset, const size_t size, int8_t *buf, const std::string &file_path)
 Reads the specified number of bytes from the offset position in file f into buf. More...
 
size_t write (FILE *f, const size_t offset, const size_t size, const int8_t *buf)
 Writes the specified number of bytes to the offset position in file f from buf. More...
 
size_t append (FILE *f, const size_t size, const int8_t *buf)
 Appends the specified number of bytes to the end of the file f from buf. More...
 
size_t readPage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf, const std::string &file_path)
 Reads the specified page from the file f into buf. More...
 
size_t readPartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t readSize, const size_t pageNum, int8_t *buf, const std::string &file_path)
 
size_t writePage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf)
 Writes a page from buf to the file. More...
 
size_t writePartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t writeSize, const size_t pageNum, int8_t *buf)
 
size_t appendPage (FILE *f, const size_t pageSize, int8_t *buf)
 Appends a page from buf to the file. More...
 
size_t fileSize (FILE *f)
 Returns the size of the specified file. More...
 
void renameForDelete (const std::string directoryName)
 Renames a directory to DELETE_ME_<EPOCH>_<oldname>. More...
 

Variables

constexpr int32_t DELETE_CONTINGENT = -1
 A FileInfo type has a file pointer and metadata about a file. More...
 
constexpr int32_t ROLLOFF_CONTINGENT = -2
 
constexpr auto kLegacyDataFileExtension {".mapd"}
 

Typedef Documentation

A Chunk is the fundamental unit of execution in Map-D.

Chunk A chunk is composed of logical pages. These pages can exist across multiple files managed by the file manager.

The collection of pages is implemented as a FileBuffer object, which is composed of a vector of MultiPage objects, one for each logical page of the file buffer.

Definition at line 73 of file FileMgr.h.

Maps ChunkKeys (unique ids for Chunks) to Chunk objects.

ChunkKeyToChunkMap The file system can store multiple chunks across multiple files. With that in mind, the challenge is to be able to reconstruct the pages that compose a chunk upon request. A chunk key (ChunkKey) uniquely identifies a chunk, and so ChunkKeyToChunkMap maps chunk keys to Chunk types, which are vectors of MultiPage* pointers (logical pages).

Definition at line 85 of file FileMgr.h.

using File_Namespace::PageHeaderSizeType = typedef int32_t

Definition at line 127 of file FileMgr.h.

using File_Namespace::PageSizeFileMMap = typedef std::multimap<size_t, int32_t>

Maps logical page sizes to files.

PageSizeFileMMap The file manager uses this type in order to quickly find files of a certain page size. A multimap is used to associate the key (page size) with values (file identifiers of files having the matching page size).

Definition at line 61 of file FileMgr.h.

using File_Namespace::TablePair = typedef std::pair<const int32_t, const int32_t>

TablePair Pair detailing the id for a database and table (first two entries in a ChunkKey).

Definition at line 91 of file FileMgr.h.

Enumeration Type Documentation

Function Documentation

size_t File_Namespace::append ( FILE *  f,
const size_t  size,
const int8_t *  buf 
)

Appends the specified number of bytes to the end of the file f from buf.

Parameters
fPointer to the FILE.
nThe number of bytes to append to the file.
bufThe source buffer containing the data to be appended.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written.

Definition at line 178 of file File.cpp.

References logger::FATAL, fileSize(), g_read_only, LOG, and write().

Referenced by benchmarks.GoogleBenchmark::aggregateBenchmarks(), run_benchmark::benchmark(), concat(), concat_with(), com.mapd.utility.SQLImporter::createDBTable(), com.mapd.calcite.parser.HeavyDBSqlOperatorTable.ExtTableFunction::getExtendedSignature(), ai.heavy.jdbc.HeavyAIPreparedStatement::getQuery(), com.mapd.parser.server.ExtensionFunctionSignatureParser::join(), run_benchmark::read_query_files(), run_benchmark_arrow::run_query(), run_benchmark::run_query(), ai.heavy.jdbc.HeavyAIEscapeFunctions::singleArgumentFunctionCall(), com.mapd.parser.server.ExtensionFunction::toJson(), and ai.heavy.jdbc.HeavyAIArray::toString().

178  {
179  if (g_read_only) {
180  LOG(FATAL) << "Error trying to append file '" << f << "', running readonly";
181  }
182  return write(f, fileSize(f), size, buf);
183 }
#define LOG(tag)
Definition: Logger.h:285
constexpr double f
Definition: Utm.h:31
size_t write(FILE *f, const size_t offset, const size_t size, const int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:160
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:230
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::appendPage ( FILE *  f,
const size_t  pageSize,
int8_t *  buf 
)

Appends a page from buf to the file.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
bufThe source buffer from where data is being read.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes appended (should be equal to pageSize).

Definition at line 222 of file File.cpp.

References logger::FATAL, fileSize(), g_read_only, LOG, and write().

222  {
223  if (g_read_only) {
224  LOG(FATAL) << "Error trying to appendPage file '" << f << "', running readonly";
225  }
226  return write(f, fileSize(f), pageSize, buf);
227 }
#define LOG(tag)
Definition: Logger.h:285
constexpr double f
Definition: Utm.h:31
size_t write(FILE *f, const size_t offset, const size_t size, const int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:160
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:230
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

void File_Namespace::close ( FILE *  f)

Closes the file pointed to by the FILE pointer.

Parameters
fPointer to the FILE.

Definition at line 128 of file File.cpp.

References CHECK, and CHECK_EQ.

Referenced by File_Namespace::FileMgr::clearFileInfos(), File_Namespace::FileMgr::closePhysicalUnlocked(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileMgr::readVersionFromDisk(), File_Namespace::FileMgr::writeAndSyncVersionToDisk(), File_Namespace::FileInfo::~FileInfo(), File_Namespace::FileMgr::~FileMgr(), and File_Namespace::TableFileMgr::~TableFileMgr().

128  {
129  CHECK(f);
130  CHECK_EQ(fflush(f), 0);
131  CHECK_EQ(fclose(f), 0);
132 }
#define CHECK_EQ(x, y)
Definition: Logger.h:301
constexpr double f
Definition: Utm.h:31
#define CHECK(condition)
Definition: Logger.h:291

+ Here is the caller graph for this function:

std::pair< FILE *, std::string > File_Namespace::create ( const std::string &  basePath,
const int  fileId,
const size_t  pageSize,
const size_t  numPages 
)

Definition at line 57 of file File.cpp.

References anonymous_namespace{Utm.h}::f, logger::FATAL, nvtx_helpers::anonymous_namespace{nvtx_helpers.cpp}::filename(), fileSize(), heavyai::fopen(), get_data_file_path(), get_legacy_data_file_path(), and LOG.

Referenced by File_Namespace::FileMgr::createEpochFile(), File_Namespace::FileMgr::createFile(), UdfCompiler::generateAST(), kafka_insert(), org.apache.calcite.sql2rel.SqlToRelConverter::SqlToRelConverter(), org.apache.calcite.sql.validate.SqlValidatorImpl::SqlValidatorImpl(), File_Namespace::TableFileMgr::TableFileMgr(), and File_Namespace::FileMgr::writeAndSyncVersionToDisk().

60  {
61  auto path = get_data_file_path(basePath, fileId, pageSize);
62  if (numPages < 1 || pageSize < 1) {
63  LOG(FATAL) << "Error trying to create file '" << path
64  << "', Number of pages and page size must be positive integers. numPages "
65  << numPages << " pageSize " << pageSize;
66  }
67  FILE* f = heavyai::fopen(path.c_str(), "w+b");
68  if (f == nullptr) {
69  LOG(FATAL) << "Error trying to create file '" << path
70  << "', the error was: " << std::strerror(errno);
71  }
72  fseek(f, static_cast<long>((pageSize * numPages) - 1), SEEK_SET);
73  fputc(EOF, f);
74  fseek(f, 0, SEEK_SET); // rewind
75  if (fileSize(f) != pageSize * numPages) {
76  LOG(FATAL) << "Error trying to create file '" << path << "', file size "
77  << fileSize(f) << " does not equal pageSize * numPages "
78  << pageSize * numPages;
79  }
80  boost::filesystem::create_symlink(boost::filesystem::canonical(path).filename(),
82  return {f, path};
83 }
std::string get_legacy_data_file_path(const std::string &new_data_file_path)
Definition: File.cpp:51
#define LOG(tag)
Definition: Logger.h:285
constexpr double f
Definition: Utm.h:31
::FILE * fopen(const char *filename, const char *mode)
Definition: heavyai_fs.cpp:74
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:230
std::string get_data_file_path(const std::string &base_path, int file_id, size_t page_size)
Definition: File.cpp:44

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

FILE * File_Namespace::create ( const std::string &  fullPath,
const size_t  requestedFileSize 
)

Definition at line 85 of file File.cpp.

References anonymous_namespace{Utm.h}::f, logger::FATAL, fileSize(), heavyai::fopen(), g_read_only, and LOG.

85  {
86  if (g_read_only) {
87  LOG(FATAL) << "Error trying to create file '" << fullPath
88  << "', not allowed read only ";
89  }
90  FILE* f = heavyai::fopen(fullPath.c_str(), "w+b");
91  if (f == nullptr) {
92  LOG(FATAL) << "Error trying to create file '" << fullPath
93  << "', the error was: " << std::strerror(errno);
94  ;
95  }
96  fseek(f, static_cast<long>(requestedFileSize - 1), SEEK_SET);
97  fputc(EOF, f);
98  fseek(f, 0, SEEK_SET); // rewind
99  if (fileSize(f) != requestedFileSize) {
100  LOG(FATAL) << "Error trying to create file '" << fullPath << "', file size "
101  << fileSize(f) << " does not equal requestedFileSize "
102  << requestedFileSize;
103  }
104  return f;
105 }
#define LOG(tag)
Definition: Logger.h:285
constexpr double f
Definition: Utm.h:31
::FILE * fopen(const char *filename, const char *mode)
Definition: heavyai_fs.cpp:74
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:230
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

size_t File_Namespace::fileSize ( FILE *  f)

Returns the size of the specified file.

Todo:
There may be an issue casting to size_t from long.
Parameters
fA pointer to the file.
Returns
size_t The number of bytes of the file.

Definition at line 230 of file File.cpp.

Referenced by append(), appendPage(), and create().

230  {
231  fseek(f, 0, SEEK_END);
232  size_t size = (size_t)ftell(f);
233  fseek(f, 0, SEEK_SET);
234  return size;
235 }
constexpr double f
Definition: Utm.h:31

+ Here is the caller graph for this function:

std::string File_Namespace::get_data_file_path ( const std::string &  base_path,
int  file_id,
size_t  page_size 
)

Definition at line 44 of file File.cpp.

References DATA_FILE_EXT, and to_string().

Referenced by create(), and File_Namespace::FileMgr::deleteEmptyFiles().

46  {
47  return base_path + "/" + std::to_string(file_id) + "." + std::to_string(page_size) +
48  std::string(DATA_FILE_EXT); // DATA_FILE_EXT has preceding "."
49 }
#define DATA_FILE_EXT
Definition: File.h:25
std::string to_string(char const *&&v)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string File_Namespace::get_dir_name_for_table ( int  db_id,
int  tb_id 
)
inline

Definition at line 84 of file CachingFileMgr.h.

Referenced by File_Namespace::CachingFileMgr::getTableFileMgrPath().

84  {
85  std::stringstream file_name;
86  file_name << "table_" << db_id << "_" << tb_id << "/";
87  return file_name.str();
88 }

+ Here is the caller graph for this function:

std::string File_Namespace::get_legacy_data_file_path ( const std::string &  new_data_file_path)

Definition at line 51 of file File.cpp.

References kLegacyDataFileExtension.

Referenced by create(), and File_Namespace::FileMgr::deleteEmptyFiles().

51  {
52  auto legacy_path = boost::filesystem::canonical(new_data_file_path);
53  legacy_path.replace_extension(kLegacyDataFileExtension);
54  return legacy_path.string();
55 }
constexpr auto kLegacyDataFileExtension
Definition: File.h:36

+ Here is the caller graph for this function:

bool File_Namespace::is_page_deleted_with_checkpoint ( int32_t  table_epoch,
int32_t  page_epoch,
int32_t  contingent 
)

Definition at line 277 of file FileInfo.cpp.

References DELETE_CONTINGENT, and ROLLOFF_CONTINGENT.

Referenced by anonymous_namespace{TableArchiver.cpp}::rewrite_column_ids_in_page_headers(), and File_Namespace::FileMgr::updatePageIfDeleted().

279  {
280  const bool delete_contingent =
281  (contingent == DELETE_CONTINGENT || contingent == ROLLOFF_CONTINGENT);
282  // Check if page was deleted with a checkpointed epoch
283  if (delete_contingent && (table_epoch >= page_epoch)) {
284  return true;
285  }
286  return false;
287 }
constexpr int32_t DELETE_CONTINGENT
A FileInfo type has a file pointer and metadata about a file.
Definition: FileInfo.h:51
constexpr int32_t ROLLOFF_CONTINGENT
Definition: FileInfo.h:52

+ Here is the caller graph for this function:

bool File_Namespace::is_page_deleted_without_checkpoint ( int32_t  table_epoch,
int32_t  page_epoch,
int32_t  contingent 
)

Definition at line 289 of file FileInfo.cpp.

References DELETE_CONTINGENT, and ROLLOFF_CONTINGENT.

Referenced by File_Namespace::FileMgr::updatePageIfDeleted().

291  {
292  const bool delete_contingent =
293  (contingent == DELETE_CONTINGENT || contingent == ROLLOFF_CONTINGENT);
294  // Check if page was deleted but the epoch was not yet checkpointed.
295  if (delete_contingent && (table_epoch < page_epoch)) {
296  return true;
297  }
298  return false;
299 }
constexpr int32_t DELETE_CONTINGENT
A FileInfo type has a file pointer and metadata about a file.
Definition: FileInfo.h:51
constexpr int32_t ROLLOFF_CONTINGENT
Definition: FileInfo.h:52

+ Here is the caller graph for this function:

FILE * File_Namespace::open ( int  fileId)

Opens/creates the file with the given id; returns NULL on error.

Parameters
fileIdThe id of the file to open.
Returns
FILE* A pointer to a FILE pointer, or NULL on error.

Definition at line 107 of file File.cpp.

References DATA_FILE_EXT, anonymous_namespace{Utm.h}::f, logger::FATAL, heavyai::fopen(), g_read_only, LOG, and to_string().

Referenced by File_Namespace::FileMgr::openAndReadEpochFile(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileMgr::openExistingFile(), File_Namespace::FileMgr::readVersionFromDisk(), File_Namespace::TableFileMgr::TableFileMgr(), and File_Namespace::FileMgr::writeAndSyncVersionToDisk().

107  {
108  std::string s(std::to_string(fileId) + std::string(DATA_FILE_EXT));
109  FILE* f = heavyai::fopen(
110  s.c_str(), g_read_only ? "rb" : "r+b"); // opens existing file for updates
111  if (f == nullptr) {
112  LOG(FATAL) << "Error trying to open file '" << s
113  << "', the error was: " << std::strerror(errno);
114  }
115  return f;
116 }
#define LOG(tag)
Definition: Logger.h:285
#define DATA_FILE_EXT
Definition: File.h:25
constexpr double f
Definition: Utm.h:31
std::string to_string(char const *&&v)
::FILE * fopen(const char *filename, const char *mode)
Definition: heavyai_fs.cpp:74
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

FILE * File_Namespace::open ( const std::string &  path)

Definition at line 118 of file File.cpp.

References anonymous_namespace{Utm.h}::f, logger::FATAL, heavyai::fopen(), g_read_only, and LOG.

118  {
119  FILE* f = heavyai::fopen(
120  path.c_str(), g_read_only ? "rb" : "r+b"); // opens existing file for updates
121  if (f == nullptr) {
122  LOG(FATAL) << "Error trying to open file '" << path
123  << "', the errno was: " << std::strerror(errno);
124  }
125  return f;
126 }
#define LOG(tag)
Definition: Logger.h:285
constexpr double f
Definition: Utm.h:31
::FILE * fopen(const char *filename, const char *mode)
Definition: heavyai_fs.cpp:74
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

size_t File_Namespace::read ( FILE *  f,
const size_t  offset,
const size_t  size,
int8_t *  buf,
const std::string &  file_path 
)

Reads the specified number of bytes from the offset position in file f into buf.

Parameters
fPointer to the FILE.
offsetThe location within the file from which to read.
sizeThe number of bytes to be read.
bufThe destination buffer to where data is being read from the file.
file_pathPath of file to read from.
Returns
size_t The number of bytes read.

Definition at line 142 of file File.cpp.

References CHECK_EQ.

Referenced by File_Namespace::FileMgr::openAndReadEpochFile(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileInfo::read(), readPage(), readPartialPage(), File_Namespace::FileMgr::readVersionFromDisk(), heavyai::safe_read(), and File_Namespace::TableFileMgr::TableFileMgr().

146  {
147  // read "size" bytes from the offset location in the file into the buffer
148  CHECK_EQ(fseek(f, static_cast<long>(offset), SEEK_SET), 0);
149  size_t bytesRead = fread(buf, sizeof(int8_t), size, f);
150  auto expected_bytes_read = sizeof(int8_t) * size;
151  CHECK_EQ(bytesRead, expected_bytes_read)
152  << "Unexpected number of bytes read from file: " << file_path
153  << ". Expected bytes read: " << expected_bytes_read
154  << ", actual bytes read: " << bytesRead << ", offset: " << offset
155  << ", file stream error set: " << (std::ferror(f) ? "true" : "false")
156  << ", EOF reached: " << (std::feof(f) ? "true" : "false");
157  return bytesRead;
158 }
#define CHECK_EQ(x, y)
Definition: Logger.h:301
constexpr double f
Definition: Utm.h:31

+ Here is the caller graph for this function:

static size_t File_Namespace::readForThread ( FileBuffer *  fileBuffer,
const readThreadDS  threadDS 
)
static

Definition at line 245 of file FileBuffer.cpp.

References CHECK, File_Namespace::FileMgr::getFileInfoForFileId(), File_Namespace::readThreadDS::multiPages, File_Namespace::FileBuffer::pageDataSize(), File_Namespace::FileBuffer::pageSize(), File_Namespace::FileInfo::read(), File_Namespace::FileBuffer::reservedHeaderSize(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.

Referenced by File_Namespace::FileBuffer::read().

245  {
246  size_t startPage = threadDS.t_startPage; // start reading at startPage, including it
247  size_t endPage = threadDS.t_endPage; // stop reading at endPage, not including it
248  int8_t* curPtr = threadDS.t_curPtr;
249  size_t bytesLeft = threadDS.t_bytesLeft;
250  size_t totalBytesRead = 0;
251  bool isFirstPage = threadDS.t_isFirstPage;
252 
253  // Traverse the logical pages
254  for (size_t pageNum = startPage; pageNum < endPage; ++pageNum) {
255  CHECK(threadDS.multiPages[pageNum].pageSize == fileBuffer->pageSize());
256  Page page = threadDS.multiPages[pageNum].current().page;
257 
258  FileInfo* fileInfo = threadDS.t_fm->getFileInfoForFileId(page.fileId);
259  CHECK(fileInfo);
260 
261  // Read the page into the destination (dst) buffer at its
262  // current (cur) location
263  size_t bytesRead = 0;
264  if (isFirstPage) {
265  bytesRead = fileInfo->read(
266  page.pageNum * fileBuffer->pageSize() + threadDS.t_startPageOffset +
267  fileBuffer->reservedHeaderSize(),
268  min(fileBuffer->pageDataSize() - threadDS.t_startPageOffset, bytesLeft),
269  curPtr);
270  isFirstPage = false;
271  } else {
272  bytesRead = fileInfo->read(
273  page.pageNum * fileBuffer->pageSize() + fileBuffer->reservedHeaderSize(),
274  min(fileBuffer->pageDataSize(), bytesLeft),
275  curPtr);
276  }
277  curPtr += bytesRead;
278  bytesLeft -= bytesRead;
279  totalBytesRead += bytesRead;
280  }
281  CHECK(bytesLeft == 0);
282 
283  return (totalBytesRead);
284 }
#define CHECK(condition)
Definition: Logger.h:291

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::readPage ( FILE *  f,
const size_t  pageSize,
const size_t  pageNum,
int8_t *  buf,
const std::string &  file_path 
)

Reads the specified page from the file f into buf.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
pageNumThe page number from where data is being read.
bufThe destination buffer to where data is being written.
file_pathPath of file to read from.
Returns
size_t The number of bytes read (should be equal to pageSize).

Definition at line 185 of file File.cpp.

References read().

189  {
190  return read(f, pageNum * pageSize, pageSize, buf, file_path);
191 }
constexpr double f
Definition: Utm.h:31
size_t read(FILE *f, const size_t offset, const size_t size, int8_t *buf, const std::string &file_path)
Reads the specified number of bytes from the offset position in file f into buf.
Definition: File.cpp:142

+ Here is the call graph for this function:

size_t File_Namespace::readPartialPage ( FILE *  f,
const size_t  pageSize,
const size_t  offset,
const size_t  readSize,
const size_t  pageNum,
int8_t *  buf,
const std::string &  file_path 
)

Definition at line 193 of file File.cpp.

References read().

199  {
200  return read(f, pageNum * pageSize + offset, readSize, buf, file_path);
201 }
constexpr double f
Definition: Utm.h:31
size_t read(FILE *f, const size_t offset, const size_t size, int8_t *buf, const std::string &file_path)
Reads the specified number of bytes from the offset position in file f into buf.
Definition: File.cpp:142

+ Here is the call graph for this function:

bool File_Namespace::removeFile ( const std::string  basePath,
const std::string  filename 
)

Deletes the file pointed to by the FILE pointer.

Parameters
basePathThe base path (directory) of the file.
fPointer to the FILE.
Returns
mapd_err_t Returns an error code when unable to close the file properly.

Definition at line 134 of file File.cpp.

References logger::FATAL, nvtx_helpers::anonymous_namespace{nvtx_helpers.cpp}::filename(), g_read_only, and LOG.

134  {
135  if (g_read_only) {
136  LOG(FATAL) << "Error trying to remove file '" << filename << "', running readonly";
137  }
138  const std::string filePath = basePath + filename;
139  return remove(filePath.c_str()) == 0;
140 }
#define LOG(tag)
Definition: Logger.h:285
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

void File_Namespace::renameForDelete ( const std::string  directoryName)

Renames a directory to DELETE_ME_<EPOCH>_<oldname>.

Parameters
directoryNamename of directory

Definition at line 242 of file File.cpp.

References logger::ERROR, logger::FATAL, LOG, and to_string().

Referenced by File_Namespace::FileMgr::closeRemovePhysical(), Catalog_Namespace::Catalog::delDictionary(), Catalog_Namespace::Catalog::doTruncateTable(), and Catalog_Namespace::Catalog::removeTableFromMap().

242  {
243  boost::system::error_code ec;
244  boost::filesystem::path directoryPath(directoryName);
245  using namespace std::chrono;
246  milliseconds ms = duration_cast<milliseconds>(system_clock::now().time_since_epoch());
247 
248  if (boost::filesystem::exists(directoryPath) &&
249  boost::filesystem::is_directory(directoryPath)) {
250  boost::filesystem::path newDirectoryPath(directoryName + "_" +
251  std::to_string(ms.count()) + "_DELETE_ME");
252  boost::filesystem::rename(directoryPath, newDirectoryPath, ec);
253 
254 #ifdef _WIN32
255  // On Windows we sometimes fail to rename a directory with System: 5 error
256  // code (access denied). An attempt to stop in debugger and look for opened
257  // handles for some of directory content shows no opened handles and actually
258  // allows renaming to execute successfully. It's not clear why, but a short
259  // pause allows to rename directory successfully. Until reasons are known,
260  // use this retry loop as a workaround.
261  int tries = 10;
262  while (ec.value() != boost::system::errc::success && tries) {
263  LOG(ERROR) << "Failed to rename directory " << directoryPath << " error was " << ec
264  << " (" << tries << " attempts left)";
265  std::this_thread::sleep_for(std::chrono::milliseconds(100 / tries));
266  tries--;
267  boost::filesystem::rename(directoryPath, newDirectoryPath, ec);
268  }
269 #endif
270 
271  if (ec.value() == boost::system::errc::success) {
272  std::thread th([newDirectoryPath]() {
273  boost::system::error_code ec;
274  boost::filesystem::remove_all(newDirectoryPath, ec);
275  // We dont check error on remove here as we cant log the
276  // issue fromdetached thrad, its not safe to LOG from here
277  // This is under investigation as clang detects TSAN issue data race
278  // the main system wide file_delete_thread will clean up any missed files
279  });
280  // let it run free so we can return
281  // if it fails the file_delete_thread in DBHandler will clean up
282  th.detach();
283 
284  return;
285  }
286 
287  LOG(FATAL) << "Failed to rename file " << directoryName << " to "
288  << directoryName + "_" + std::to_string(ms.count()) + "_DELETE_ME Error: "
289  << ec;
290  }
291 }
#define LOG(tag)
Definition: Logger.h:285
std::string to_string(char const *&&v)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::write ( FILE *  f,
const size_t  offset,
const size_t  size,
const int8_t *  buf 
)

Writes the specified number of bytes to the offset position in file f from buf.

Parameters
fPointer to the FILE.
offsetThe location within the file where data is being written.
sizeThe number of bytes to write to the file.
bufThe source buffer containing the data to be written.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written.

Definition at line 160 of file File.cpp.

References logger::FATAL, g_read_only, and LOG.

Referenced by StringDictionary::addStorageCapacity(), append(), appendPage(), File_Namespace::FileInfo::freePage(), File_Namespace::FileInfo::freePageImmediate(), lockmgr::TableLockMgrImpl< TableDataLockMgr >::getClusterTableMutex(), import_export::DataStreamSink::import_compressed(), File_Namespace::FileInfo::initNewFile(), File_Namespace::FileInfo::recoverPage(), heavyai::safe_write(), File_Namespace::FileInfo::write(), File_Namespace::TableFileMgr::writeAndSyncEpochToDisk(), File_Namespace::FileMgr::writeAndSyncEpochToDisk(), File_Namespace::FileMgr::writeAndSyncVersionToDisk(), writePage(), and writePartialPage().

160  {
161  if (g_read_only) {
162  LOG(FATAL) << "Error trying to write file '" << f << "', running readonly";
163  }
164  // write size bytes from the buffer to the offset location in the file
165  if (fseek(f, static_cast<long>(offset), SEEK_SET) != 0) {
166  LOG(FATAL)
167  << "Error trying to write to file (during positioning seek) the error was: "
168  << std::strerror(errno);
169  }
170  size_t bytesWritten = fwrite(buf, sizeof(int8_t), size, f);
171  if (bytesWritten != sizeof(int8_t) * size) {
172  LOG(FATAL) << "Error trying to write to file (during fwrite) the error was: "
173  << std::strerror(errno);
174  }
175  return bytesWritten;
176 }
#define LOG(tag)
Definition: Logger.h:285
constexpr double f
Definition: Utm.h:31
bool g_read_only
Definition: File.cpp:40

+ Here is the caller graph for this function:

size_t File_Namespace::writePage ( FILE *  f,
const size_t  pageSize,
const size_t  pageNum,
int8_t *  buf 
)

Writes a page from buf to the file.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
pageNumThe page number to where data is being written.
bufThe source buffer from where data is being read.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written (should be equal to pageSize).

Definition at line 203 of file File.cpp.

References logger::FATAL, g_read_only, LOG, and write().

203  {
204  if (g_read_only) {
205  LOG(FATAL) << "Error trying to writePage file '" << f << "', running readonly";
206  }
207  return write(f, pageNum * pageSize, pageSize, buf);
208 }
#define LOG(tag)
Definition: Logger.h:285
constexpr double f
Definition: Utm.h:31
size_t write(FILE *f, const size_t offset, const size_t size, const int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:160
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

size_t File_Namespace::writePartialPage ( FILE *  f,
const size_t  pageSize,
const size_t  offset,
const size_t  writeSize,
const size_t  pageNum,
int8_t *  buf 
)

Definition at line 210 of file File.cpp.

References logger::FATAL, g_read_only, LOG, and write().

215  {
216  if (g_read_only) {
217  LOG(FATAL) << "Error trying to writePartialPage file '" << f << "', running readonly";
218  }
219  return write(f, pageNum * pageSize + offset, writeSize, buf);
220 }
#define LOG(tag)
Definition: Logger.h:285
constexpr double f
Definition: Utm.h:31
size_t write(FILE *f, const size_t offset, const size_t size, const int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:160
bool g_read_only
Definition: File.cpp:40

+ Here is the call graph for this function:

Variable Documentation

constexpr int32_t File_Namespace::DELETE_CONTINGENT = -1

A FileInfo type has a file pointer and metadata about a file.

FileInfo A file info structure wraps around a file pointer in order to contain additional information/metadata about the file that is pertinent to the file manager.

The free pages (freePages) within a file must be tracked, and this is implemented using a basic STL set. The set ensures that no duplicate pages are included, and that the pages are sorted, faciliating the obtaining of consecutive free pages by a constant time pop operation, which may reduce the cost of DBMS disk accesses.

Helper functions are provided: size(), available(), and used().

Definition at line 51 of file FileInfo.h.

Referenced by File_Namespace::FileInfo::freePage(), is_page_deleted_with_checkpoint(), is_page_deleted_without_checkpoint(), and File_Namespace::CachingFileMgr::updatePageIfDeleted().

constexpr auto File_Namespace::kLegacyDataFileExtension {".mapd"}
constexpr int32_t File_Namespace::ROLLOFF_CONTINGENT = -2