File_Namespace Namespace Reference

Namespaces
	anonymous_namespace{FileMgr.cpp}

Classes
struct	DiskCacheConfig
class	TableFileMgr
class	CachingFileBuffer
class	CachingFileMgr
	A FileMgr capable of limiting it's size and storing data from multiple tables in a shared directory. For any table that supports DiskCaching, the CachingFileMgr must contain either metadata for all table chunks, or for none (the cache is either has no knowledge of that table, or has complete knowledge of that table). Any data chunk within a table may or may not be contained within the cache. More...
class	CachingGlobalFileMgr
struct	readThreadDS
class	FileBuffer
	Represents/provides access to contiguous data stored in the file system. More...
struct	FileInfo
struct	FileMetadata
struct	StorageStats
struct	OpenFilesResult
struct	PageMapping
class	FileMgr
struct	FileMgrParams
class	GlobalFileMgr
struct	Page
	A logical page (Page) belongs to a file on disk. More...
struct	EpochedPage
struct	MultiPage
	The MultiPage stores versions of the same logical page in a deque. More...
struct	HeaderInfo
	Stores Pair of ChunkKey and Page id and version, in a pair with a Page struct itself (File id and Page num) More...

Typedefs
using	PageSizeFileMMap = std::multimap< size_t, int32_t >
	Maps logical page sizes to files. More...
using	Chunk = FileBuffer
	A Chunk is the fundamental unit of execution in Map-D. More...
using	ChunkKeyToChunkMap = std::map< ChunkKey, FileBuffer * >
	Maps ChunkKeys (unique ids for Chunks) to Chunk objects. More...
using	TablePair = std::pair< const int32_t, const int32_t >
using	PageHeaderSizeType = int32_t

Enumerations
enum	DiskCacheLevel { DiskCacheLevel::none, DiskCacheLevel::fsi, DiskCacheLevel::non_fsi, DiskCacheLevel::all }

Functions
std::string	get_dir_name_for_table (int db_id, int tb_id)
static size_t	readForThread (FileBuffer *fileBuffer, const readThreadDS threadDS)
bool	is_page_deleted_with_checkpoint (int32_t table_epoch, int32_t page_epoch, int32_t contingent)
bool	is_page_deleted_without_checkpoint (int32_t table_epoch, int32_t page_epoch, int32_t contingent)
std::string	get_data_file_path (const std::string &base_path, int file_id, size_t page_size)
std::string	get_legacy_data_file_path (const std::string &new_data_file_path)
std::pair< FILE *, std::string >	create (const std::string &basePath, const int fileId, const size_t pageSize, const size_t numPages)
FILE *	create (const std::string &fullPath, const size_t requestedFileSize)
FILE *	open (int fileId)
	Opens/creates the file with the given id; returns NULL on error. More...
FILE *	open (const std::string &path)
void	close (FILE *f)
	Closes the file pointed to by the FILE pointer. More...
bool	removeFile (const std::string basePath, const std::string filename)
	Deletes the file pointed to by the FILE pointer. More...
size_t	read (FILE *f, const size_t offset, const size_t size, int8_t *buf, const std::string &file_path)
	Reads the specified number of bytes from the offset position in file f into buf. More...
size_t	write (FILE *f, const size_t offset, const size_t size, const int8_t *buf)
	Writes the specified number of bytes to the offset position in file f from buf. More...
size_t	append (FILE *f, const size_t size, const int8_t *buf)
	Appends the specified number of bytes to the end of the file f from buf. More...
size_t	readPage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf, const std::string &file_path)
	Reads the specified page from the file f into buf. More...
size_t	readPartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t readSize, const size_t pageNum, int8_t *buf, const std::string &file_path)
size_t	writePage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf)
	Writes a page from buf to the file. More...
size_t	writePartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t writeSize, const size_t pageNum, int8_t *buf)
size_t	appendPage (FILE *f, const size_t pageSize, int8_t *buf)
	Appends a page from buf to the file. More...
size_t	fileSize (FILE *f)
	Returns the size of the specified file. More...
void	renameForDelete (const std::string directoryName)
	Renames a directory to DELETE_ME_<EPOCH>_<oldname>. More...

Variables
constexpr int32_t	DELETE_CONTINGENT = -1
	A FileInfo type has a file pointer and metadata about a file. More...
constexpr int32_t	ROLLOFF_CONTINGENT = -2
constexpr auto	kLegacyDataFileExtension {".mapd"}

Typedef Documentation

using File_Namespace::Chunk = typedef FileBuffer

A Chunk is the fundamental unit of execution in Map-D.

Chunk A chunk is composed of logical pages. These pages can exist across multiple files managed by the file manager.

The collection of pages is implemented as a FileBuffer object, which is composed of a vector of MultiPage objects, one for each logical page of the file buffer.

Definition at line 73 of file FileMgr.h.

using File_Namespace::ChunkKeyToChunkMap = typedef std::map<ChunkKey, FileBuffer*>

Maps ChunkKeys (unique ids for Chunks) to Chunk objects.

ChunkKeyToChunkMap The file system can store multiple chunks across multiple files. With that in mind, the challenge is to be able to reconstruct the pages that compose a chunk upon request. A chunk key (ChunkKey) uniquely identifies a chunk, and so ChunkKeyToChunkMap maps chunk keys to Chunk types, which are vectors of MultiPage* pointers (logical pages).

Definition at line 85 of file FileMgr.h.

using File_Namespace::PageHeaderSizeType = typedef int32_t

Definition at line 127 of file FileMgr.h.

using File_Namespace::PageSizeFileMMap = typedef std::multimap<size_t, int32_t>

Maps logical page sizes to files.

PageSizeFileMMap The file manager uses this type in order to quickly find files of a certain page size. A multimap is used to associate the key (page size) with values (file identifiers of files having the matching page size).

Definition at line 61 of file FileMgr.h.

using File_Namespace::TablePair = typedef std::pair<const int32_t, const int32_t>

TablePair Pair detailing the id for a database and table (first two entries in a ChunkKey).

Definition at line 91 of file FileMgr.h.

Enumeration Type Documentation

enum File_Namespace::DiskCacheLevel

strong

Enumerator
none
fsi
non_fsi
all

Definition at line 40 of file CachingFileMgr.h.

{ none, fsi, non_fsi, all };

Function Documentation

size_t File_Namespace::append	(	FILE *	f,
		const size_t	size,
		const int8_t *	buf
	)

Appends the specified number of bytes to the end of the file f from buf.

Parameters

f	Pointer to the FILE.
n	The number of bytes to append to the file.
buf	The source buffer containing the data to be appended.
err	If not NULL, will hold an error code should an error occur.

Returns

size_t The number of bytes written.

Definition at line 178 of file File.cpp.

References logger::FATAL, fileSize(), g_read_only, LOG, and write().

Referenced by benchmarks.GoogleBenchmark::aggregateBenchmarks(), run_benchmark::benchmark(), concat(), concat_with(), com.mapd.utility.SQLImporter::createDBTable(), com.mapd.calcite.parser.HeavyDBSqlOperatorTable.ExtTableFunction::getExtendedSignature(), ai.heavy.jdbc.HeavyAIPreparedStatement::getQuery(), com.mapd.parser.server.ExtensionFunctionSignatureParser::join(), run_benchmark::read_query_files(), run_benchmark_arrow::run_query(), run_benchmark::run_query(), ai.heavy.jdbc.HeavyAIEscapeFunctions::singleArgumentFunctionCall(), com.mapd.parser.server.ExtensionFunction::toJson(), and ai.heavy.jdbc.HeavyAIArray::toString().

                                                             {
  if (g_read_only) {
    LOG(FATAL) << "Error trying to append file '" << f << "', running readonly";
  }
  return write(f, fileSize(f), size, buf);
}

Here is the call graph for this function:

Here is the caller graph for this function:

size_t File_Namespace::appendPage	(	FILE *	f,
		const size_t	pageSize,
		int8_t *	buf
	)

Appends a page from buf to the file.

Parameters

f	Pointer to the FILE.
pageSize	The logical page size of the file.
buf	The source buffer from where data is being read.
err	If not NULL, will hold an error code should an error occur.

Returns

size_t The number of bytes appended (should be equal to pageSize).

Definition at line 222 of file File.cpp.

References logger::FATAL, fileSize(), g_read_only, LOG, and write().

                                                               {
  if (g_read_only) {
    LOG(FATAL) << "Error trying to appendPage file '" << f << "', running readonly";
  }
  return write(f, fileSize(f), pageSize, buf);
}

Here is the call graph for this function:

void File_Namespace::close

(

FILE *

f

)

Closes the file pointed to by the FILE pointer.

Parameters

f	Pointer to the FILE.

Definition at line 128 of file File.cpp.

References CHECK, and CHECK_EQ.

Referenced by File_Namespace::FileMgr::clearFileInfos(), File_Namespace::FileMgr::closePhysicalUnlocked(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileMgr::readVersionFromDisk(), File_Namespace::FileMgr::writeAndSyncVersionToDisk(), File_Namespace::FileInfo::~FileInfo(), File_Namespace::FileMgr::~FileMgr(), and File_Namespace::TableFileMgr::~TableFileMgr().

                    {
  CHECK(f);
  CHECK_EQ(fflush(f), 0);
  CHECK_EQ(fclose(f), 0);
}

Here is the caller graph for this function:

std::pair< FILE *, std::string > File_Namespace::create	(	const std::string &	basePath,
		const int	fileId,
		const size_t	pageSize,
		const size_t	numPages
	)

Definition at line 57 of file File.cpp.

References anonymous_namespace{Utm.h}::f, logger::FATAL, nvtx_helpers::anonymous_namespace{nvtx_helpers.cpp}::filename(), fileSize(), heavyai::fopen(), get_data_file_path(), get_legacy_data_file_path(), and LOG.

Referenced by File_Namespace::FileMgr::createEpochFile(), File_Namespace::FileMgr::createFile(), UdfCompiler::generateAST(), kafka_insert(), org.apache.calcite.sql2rel.SqlToRelConverter::SqlToRelConverter(), org.apache.calcite.sql.validate.SqlValidatorImpl::SqlValidatorImpl(), File_Namespace::TableFileMgr::TableFileMgr(), and File_Namespace::FileMgr::writeAndSyncVersionToDisk().

                                                            {
  auto path = get_data_file_path(basePath, fileId, pageSize);
  if (numPages < 1 || pageSize < 1) {
    LOG(FATAL) << "Error trying to create file '" << path
               << "', Number of pages and page size must be positive integers. numPages "
               << numPages << " pageSize " << pageSize;
  }
  FILE* f = heavyai::fopen(path.c_str(), "w+b");
  if (f == nullptr) {
    LOG(FATAL) << "Error trying to create file '" << path
               << "', the error was: " << std::strerror(errno);
  }
  fseek(f, static_cast<long>((pageSize * numPages) - 1), SEEK_SET);
  fputc(EOF, f);
  fseek(f, 0, SEEK_SET);  // rewind
  if (fileSize(f) != pageSize * numPages) {
    LOG(FATAL) << "Error trying to create file '" << path << "', file size "
               << fileSize(f) << " does not equal pageSize * numPages "
               << pageSize * numPages;
  }
  boost::filesystem::create_symlink(boost::filesystem::canonical(path).filename(),
                                    get_legacy_data_file_path(path));
  return {f, path};
}

Here is the call graph for this function:

Here is the caller graph for this function:

FILE * File_Namespace::create	(	const std::string &	fullPath,
		const size_t	requestedFileSize
	)

Definition at line 85 of file File.cpp.

References anonymous_namespace{Utm.h}::f, logger::FATAL, fileSize(), heavyai::fopen(), g_read_only, and LOG.

                                                                        {
  if (g_read_only) {
    LOG(FATAL) << "Error trying to create file '" << fullPath
               << "', not allowed read only ";
  }
  FILE* f = heavyai::fopen(fullPath.c_str(), "w+b");
  if (f == nullptr) {
    LOG(FATAL) << "Error trying to create file '" << fullPath
               << "', the error was:  " << std::strerror(errno);
    ;
  }
  fseek(f, static_cast<long>(requestedFileSize - 1), SEEK_SET);
  fputc(EOF, f);
  fseek(f, 0, SEEK_SET);  // rewind
  if (fileSize(f) != requestedFileSize) {
    LOG(FATAL) << "Error trying to create file '" << fullPath << "', file size "
               << fileSize(f) << " does not equal requestedFileSize "
               << requestedFileSize;
  }
  return f;
}

Here is the call graph for this function:

size_t File_Namespace::fileSize

(

FILE *

f

)

Returns the size of the specified file.

Todo:

There may be an issue casting to size_t from long.

Parameters

f	A pointer to the file.

Returns

size_t The number of bytes of the file.

Definition at line 230 of file File.cpp.

Referenced by append(), appendPage(), and create().

                         {
  fseek(f, 0, SEEK_END);
  size_t size = (size_t)ftell(f);
  fseek(f, 0, SEEK_SET);
  return size;
}

Here is the caller graph for this function:

std::string File_Namespace::get_data_file_path	(	const std::string &	base_path,
		int	file_id,
		size_t	page_size
	)

Definition at line 44 of file File.cpp.

References DATA_FILE_EXT, and to_string().

Referenced by create(), and File_Namespace::FileMgr::deleteEmptyFiles().

                                                 {
  return base_path + "/" + std::to_string(file_id) + "." + std::to_string(page_size) +
         std::string(DATA_FILE_EXT);  // DATA_FILE_EXT has preceding "."
}

Here is the call graph for this function:

Here is the caller graph for this function:

std::string File_Namespace::get_dir_name_for_table ( int  db_id, int  tb_id  )

inline

Definition at line 84 of file CachingFileMgr.h.

Referenced by File_Namespace::CachingFileMgr::getTableFileMgrPath().

                                                              {
  std::stringstream file_name;
  file_name << "table_" << db_id << "_" << tb_id << "/";
  return file_name.str();
}

Here is the caller graph for this function:

std::string File_Namespace::get_legacy_data_file_path

(

const std::string &

new_data_file_path

)

Definition at line 51 of file File.cpp.

References kLegacyDataFileExtension.

Referenced by create(), and File_Namespace::FileMgr::deleteEmptyFiles().

                                                                         {
  auto legacy_path = boost::filesystem::canonical(new_data_file_path);
  legacy_path.replace_extension(kLegacyDataFileExtension);
  return legacy_path.string();
}

Here is the caller graph for this function:

bool File_Namespace::is_page_deleted_with_checkpoint	(	int32_t	table_epoch,
		int32_t	page_epoch,
		int32_t	contingent
	)

Definition at line 277 of file FileInfo.cpp.

References DELETE_CONTINGENT, and ROLLOFF_CONTINGENT.

Referenced by anonymous_namespace{TableArchiver.cpp}::rewrite_column_ids_in_page_headers(), and File_Namespace::FileMgr::updatePageIfDeleted().

                                                         {
  const bool delete_contingent =
      (contingent == DELETE_CONTINGENT || contingent == ROLLOFF_CONTINGENT);
  // Check if page was deleted with a checkpointed epoch
  if (delete_contingent && (table_epoch >= page_epoch)) {
    return true;
  }
  return false;
}

Here is the caller graph for this function:

bool File_Namespace::is_page_deleted_without_checkpoint	(	int32_t	table_epoch,
		int32_t	page_epoch,
		int32_t	contingent
	)

Definition at line 289 of file FileInfo.cpp.

References DELETE_CONTINGENT, and ROLLOFF_CONTINGENT.

Referenced by File_Namespace::FileMgr::updatePageIfDeleted().

                                                            {
  const bool delete_contingent =
      (contingent == DELETE_CONTINGENT || contingent == ROLLOFF_CONTINGENT);
  // Check if page was deleted but the epoch was not yet checkpointed.
  if (delete_contingent && (table_epoch < page_epoch)) {
    return true;
  }
  return false;
}

Here is the caller graph for this function:

FILE * File_Namespace::open

(

int

fileId

)

Opens/creates the file with the given id; returns NULL on error.

Parameters

fileId

The id of the file to open.

Returns

FILE* A pointer to a FILE pointer, or NULL on error.

Definition at line 107 of file File.cpp.

References DATA_FILE_EXT, anonymous_namespace{Utm.h}::f, logger::FATAL, heavyai::fopen(), g_read_only, LOG, and to_string().

Referenced by File_Namespace::FileMgr::openAndReadEpochFile(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileMgr::openExistingFile(), File_Namespace::FileMgr::readVersionFromDisk(), File_Namespace::TableFileMgr::TableFileMgr(), and File_Namespace::FileMgr::writeAndSyncVersionToDisk().

                       {
  std::string s(std::to_string(fileId) + std::string(DATA_FILE_EXT));
  FILE* f = heavyai::fopen(
      s.c_str(), g_read_only ? "rb" : "r+b");  // opens existing file for updates
  if (f == nullptr) {
    LOG(FATAL) << "Error trying to open file '" << s
               << "', the error was: " << std::strerror(errno);
  }
  return f;
}

Here is the call graph for this function:

Here is the caller graph for this function:

FILE * File_Namespace::open

(

const std::string &

path

)

Definition at line 118 of file File.cpp.

References anonymous_namespace{Utm.h}::f, logger::FATAL, heavyai::fopen(), g_read_only, and LOG.

                                  {
  FILE* f = heavyai::fopen(
      path.c_str(), g_read_only ? "rb" : "r+b");  // opens existing file for updates
  if (f == nullptr) {
    LOG(FATAL) << "Error trying to open file '" << path
               << "', the errno was: " << std::strerror(errno);
  }
  return f;
}

Here is the call graph for this function:

size_t File_Namespace::read	(	FILE *	f,
		const size_t	offset,
		const size_t	size,
		int8_t *	buf,
		const std::string &	file_path
	)

Reads the specified number of bytes from the offset position in file f into buf.

Parameters

f	Pointer to the FILE.
offset	The location within the file from which to read.
size	The number of bytes to be read.
buf	The destination buffer to where data is being read from the file.
file_path	Path of file to read from.

Returns

size_t The number of bytes read.

Definition at line 142 of file File.cpp.

References CHECK_EQ.

Referenced by File_Namespace::FileMgr::openAndReadEpochFile(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileInfo::read(), readPage(), readPartialPage(), File_Namespace::FileMgr::readVersionFromDisk(), heavyai::safe_read(), and File_Namespace::TableFileMgr::TableFileMgr().

                                        {
  // read "size" bytes from the offset location in the file into the buffer
  CHECK_EQ(fseek(f, static_cast<long>(offset), SEEK_SET), 0);
  size_t bytesRead = fread(buf, sizeof(int8_t), size, f);
  auto expected_bytes_read = sizeof(int8_t) * size;
  CHECK_EQ(bytesRead, expected_bytes_read)
      << "Unexpected number of bytes read from file: " << file_path
      << ". Expected bytes read: " << expected_bytes_read
      << ", actual bytes read: " << bytesRead << ", offset: " << offset
      << ", file stream error set: " << (std::ferror(f) ? "true" : "false")
      << ", EOF reached: " << (std::feof(f) ? "true" : "false");
  return bytesRead;
}

Here is the caller graph for this function:

static size_t File_Namespace::readForThread ( FileBuffer *  fileBuffer, const readThreadDS  threadDS  )

static

Definition at line 245 of file FileBuffer.cpp.

References CHECK, File_Namespace::FileMgr::getFileInfoForFileId(), File_Namespace::readThreadDS::multiPages, File_Namespace::FileBuffer::pageDataSize(), File_Namespace::FileBuffer::pageSize(), File_Namespace::FileInfo::read(), File_Namespace::FileBuffer::reservedHeaderSize(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.

Referenced by File_Namespace::FileBuffer::read().

                                                                                 {
  size_t startPage = threadDS.t_startPage;  // start reading at startPage, including it
  size_t endPage = threadDS.t_endPage;      // stop reading at endPage, not including it
  int8_t* curPtr = threadDS.t_curPtr;
  size_t bytesLeft = threadDS.t_bytesLeft;
  size_t totalBytesRead = 0;
  bool isFirstPage = threadDS.t_isFirstPage;

  // Traverse the logical pages
  for (size_t pageNum = startPage; pageNum < endPage; ++pageNum) {
    CHECK(threadDS.multiPages[pageNum].pageSize == fileBuffer->pageSize());
    Page page = threadDS.multiPages[pageNum].current().page;

    FileInfo* fileInfo = threadDS.t_fm->getFileInfoForFileId(page.fileId);
    CHECK(fileInfo);

    // Read the page into the destination (dst) buffer at its
    // current (cur) location
    size_t bytesRead = 0;
    if (isFirstPage) {
      bytesRead = fileInfo->read(
          page.pageNum * fileBuffer->pageSize() + threadDS.t_startPageOffset +
              fileBuffer->reservedHeaderSize(),
          min(fileBuffer->pageDataSize() - threadDS.t_startPageOffset, bytesLeft),
          curPtr);
      isFirstPage = false;
    } else {
      bytesRead = fileInfo->read(
          page.pageNum * fileBuffer->pageSize() + fileBuffer->reservedHeaderSize(),
          min(fileBuffer->pageDataSize(), bytesLeft),
          curPtr);
    }
    curPtr += bytesRead;
    bytesLeft -= bytesRead;
    totalBytesRead += bytesRead;
  }
  CHECK(bytesLeft == 0);

  return (totalBytesRead);
}

Here is the call graph for this function:

Here is the caller graph for this function:

size_t File_Namespace::readPage	(	FILE *	f,
		const size_t	pageSize,
		const size_t	pageNum,
		int8_t *	buf,
		const std::string &	file_path
	)

Reads the specified page from the file f into buf.

Parameters

f	Pointer to the FILE.
pageSize	The logical page size of the file.
pageNum	The page number from where data is being read.
buf	The destination buffer to where data is being written.
file_path	Path of file to read from.

Returns

size_t The number of bytes read (should be equal to pageSize).

Definition at line 185 of file File.cpp.

References read().

                                            {
  return read(f, pageNum * pageSize, pageSize, buf, file_path);
}

Here is the call graph for this function:

size_t File_Namespace::readPartialPage	(	FILE *	f,
		const size_t	pageSize,
		const size_t	offset,
		const size_t	readSize,
		const size_t	pageNum,
		int8_t *	buf,
		const std::string &	file_path
	)

Definition at line 193 of file File.cpp.

References read().

                                                   {
  return read(f, pageNum * pageSize + offset, readSize, buf, file_path);
}

Here is the call graph for this function:

bool File_Namespace::removeFile	(	const std::string	basePath,
		const std::string	filename
	)

Deletes the file pointed to by the FILE pointer.

Parameters

basePath	The base path (directory) of the file.
f	Pointer to the FILE.

Returns

mapd_err_t Returns an error code when unable to close the file properly.

Definition at line 134 of file File.cpp.

References logger::FATAL, nvtx_helpers::anonymous_namespace{nvtx_helpers.cpp}::filename(), g_read_only, and LOG.

                                                                    {
  if (g_read_only) {
    LOG(FATAL) << "Error trying to remove file '" << filename << "', running readonly";
  }
  const std::string filePath = basePath + filename;
  return remove(filePath.c_str()) == 0;
}

Here is the call graph for this function:

void File_Namespace::renameForDelete

(

const std::string

directoryName

)

Renames a directory to DELETE_ME_<EPOCH>_<oldname>.

Parameters

directoryName

name of directory

Definition at line 242 of file File.cpp.

References logger::ERROR, logger::FATAL, LOG, and to_string().

Referenced by File_Namespace::FileMgr::closeRemovePhysical(), Catalog_Namespace::Catalog::delDictionary(), Catalog_Namespace::Catalog::doTruncateTable(), and Catalog_Namespace::Catalog::removeTableFromMap().

                                                    {
  boost::system::error_code ec;
  boost::filesystem::path directoryPath(directoryName);
  using namespace std::chrono;
  milliseconds ms = duration_cast<milliseconds>(system_clock::now().time_since_epoch());

  if (boost::filesystem::exists(directoryPath) &&
      boost::filesystem::is_directory(directoryPath)) {
    boost::filesystem::path newDirectoryPath(directoryName + "_" +
                                             std::to_string(ms.count()) + "_DELETE_ME");
    boost::filesystem::rename(directoryPath, newDirectoryPath, ec);

#ifdef _WIN32
    // On Windows we sometimes fail to rename a directory with System: 5 error
    // code (access denied). An attempt to stop in debugger and look for opened
    // handles for some of directory content shows no opened handles and actually
    // allows renaming to execute successfully. It's not clear why, but a short
    // pause allows to rename directory successfully. Until reasons are known,
    // use this retry loop as a workaround.
    int tries = 10;
    while (ec.value() != boost::system::errc::success && tries) {
      LOG(ERROR) << "Failed to rename directory " << directoryPath << " error was " << ec
                 << " (" << tries << " attempts left)";
      std::this_thread::sleep_for(std::chrono::milliseconds(100 / tries));
      tries--;
      boost::filesystem::rename(directoryPath, newDirectoryPath, ec);
    }
#endif

    if (ec.value() == boost::system::errc::success) {
      std::thread th([newDirectoryPath]() {
        boost::system::error_code ec;
        boost::filesystem::remove_all(newDirectoryPath, ec);
        // We dont check error on remove here as we cant log the
        // issue fromdetached thrad, its not safe to LOG from here
        // This is under investigation as clang detects TSAN issue data race
        // the main system wide file_delete_thread will clean up any missed files
      });
      // let it run free so we can return
      // if it fails the file_delete_thread in DBHandler will clean up
      th.detach();

      return;
    }

    LOG(FATAL) << "Failed to rename file " << directoryName << " to "
               << directoryName + "_" + std::to_string(ms.count()) + "_DELETE_ME  Error: "
               << ec;
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

size_t File_Namespace::write	(	FILE *	f,
		const size_t	offset,
		const size_t	size,
		const int8_t *	buf
	)

Writes the specified number of bytes to the offset position in file f from buf.

Parameters

f	Pointer to the FILE.
offset	The location within the file where data is being written.
size	The number of bytes to write to the file.
buf	The source buffer containing the data to be written.
err	If not NULL, will hold an error code should an error occur.

Returns

size_t The number of bytes written.

Definition at line 160 of file File.cpp.

References logger::FATAL, g_read_only, and LOG.

Referenced by StringDictionary::addStorageCapacity(), append(), appendPage(), File_Namespace::FileInfo::freePage(), File_Namespace::FileInfo::freePageImmediate(), lockmgr::TableLockMgrImpl< TableDataLockMgr >::getClusterTableMutex(), import_export::DataStreamSink::import_compressed(), File_Namespace::FileInfo::initNewFile(), File_Namespace::FileInfo::recoverPage(), heavyai::safe_write(), File_Namespace::FileInfo::write(), File_Namespace::TableFileMgr::writeAndSyncEpochToDisk(), File_Namespace::FileMgr::writeAndSyncEpochToDisk(), File_Namespace::FileMgr::writeAndSyncVersionToDisk(), writePage(), and writePartialPage().

                                                                                 {
  if (g_read_only) {
    LOG(FATAL) << "Error trying to write file '" << f << "', running readonly";
  }
  // write size bytes from the buffer to the offset location in the file
  if (fseek(f, static_cast<long>(offset), SEEK_SET) != 0) {
    LOG(FATAL)
        << "Error trying to write to file (during positioning seek) the error was: "
        << std::strerror(errno);
  }
  size_t bytesWritten = fwrite(buf, sizeof(int8_t), size, f);
  if (bytesWritten != sizeof(int8_t) * size) {
    LOG(FATAL) << "Error trying to write to file (during fwrite) the error was: "
               << std::strerror(errno);
  }
  return bytesWritten;
}

Here is the caller graph for this function:

size_t File_Namespace::writePage	(	FILE *	f,
		const size_t	pageSize,
		const size_t	pageNum,
		int8_t *	buf
	)

Writes a page from buf to the file.

Parameters

f	Pointer to the FILE.
pageSize	The logical page size of the file.
pageNum	The page number to where data is being written.
buf	The source buffer from where data is being read.
err	If not NULL, will hold an error code should an error occur.

Returns

size_t The number of bytes written (should be equal to pageSize).

Definition at line 203 of file File.cpp.

References logger::FATAL, g_read_only, LOG, and write().

                                                                                    {
  if (g_read_only) {
    LOG(FATAL) << "Error trying to writePage file '" << f << "', running readonly";
  }
  return write(f, pageNum * pageSize, pageSize, buf);
}

Here is the call graph for this function:

size_t File_Namespace::writePartialPage	(	FILE *	f,
		const size_t	pageSize,
		const size_t	offset,
		const size_t	writeSize,
		const size_t	pageNum,
		int8_t *	buf
	)

Definition at line 210 of file File.cpp.

References logger::FATAL, g_read_only, LOG, and write().

                                     {
  if (g_read_only) {
    LOG(FATAL) << "Error trying to writePartialPage file '" << f << "', running readonly";
  }
  return write(f, pageNum * pageSize + offset, writeSize, buf);
}

Here is the call graph for this function:

Variable Documentation

constexpr int32_t File_Namespace::DELETE_CONTINGENT = -1

A FileInfo type has a file pointer and metadata about a file.

FileInfo A file info structure wraps around a file pointer in order to contain additional information/metadata about the file that is pertinent to the file manager.

The free pages (freePages) within a file must be tracked, and this is implemented using a basic STL set. The set ensures that no duplicate pages are included, and that the pages are sorted, faciliating the obtaining of consecutive free pages by a constant time pop operation, which may reduce the cost of DBMS disk accesses.

Helper functions are provided: size(), available(), and used().

Definition at line 51 of file FileInfo.h.

Referenced by File_Namespace::FileInfo::freePage(), is_page_deleted_with_checkpoint(), is_page_deleted_without_checkpoint(), and File_Namespace::CachingFileMgr::updatePageIfDeleted().

constexpr auto File_Namespace::kLegacyDataFileExtension {".mapd"}

Definition at line 36 of file File.h.

Referenced by anonymous_namespace{TableArchiver.cpp}::add_data_file_symlinks(), get_legacy_data_file_path(), and File_Namespace::FileMgr::renameAndSymlinkLegacyFiles().

constexpr int32_t File_Namespace::ROLLOFF_CONTINGENT = -2

Definition at line 52 of file FileInfo.h.

Referenced by File_Namespace::FileInfo::freePage(), is_page_deleted_with_checkpoint(), is_page_deleted_without_checkpoint(), and File_Namespace::CachingFileMgr::updatePageIfDeleted().