OmniSciDB  21ac014ffc
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
File_Namespace Namespace Reference

Namespaces

 anonymous_namespace{FileMgr.cpp}
 

Classes

struct  DiskCacheConfig
 
class  TableFileMgr
 
class  CachingFileBuffer
 
class  CachingFileMgr
 A FileMgr capable of limiting it's size and storing data from multiple tables in a shared directory. For any table that supports DiskCaching, the CachingFileMgr must contain either metadata for all table chunks, or for none (the cache is either has no knowledge of that table, or has complete knowledge of that table). Any data chunk within a table may or may not be contained within the cache. More...
 
struct  readThreadDS
 
class  FileBuffer
 Represents/provides access to contiguous data stored in the file system. More...
 
struct  FileInfo
 
struct  FileMetadata
 
struct  StorageStats
 
struct  OpenFilesResult
 
struct  PageMapping
 
class  FileMgr
 
struct  FileMgrParams
 
class  GlobalFileMgr
 
struct  Page
 A logical page (Page) belongs to a file on disk. More...
 
struct  EpochedPage
 
struct  MultiPage
 The MultiPage stores versions of the same logical page in a deque. More...
 
struct  HeaderInfo
 Stores Pair of ChunkKey and Page id and version, in a pair with a Page struct itself (File id and Page num) More...
 

Typedefs

using PageSizeFileMMap = std::multimap< size_t, int32_t >
 Maps logical page sizes to files. More...
 
using Chunk = FileBuffer
 A Chunk is the fundamental unit of execution in Map-D. More...
 
using ChunkKeyToChunkMap = std::map< ChunkKey, FileBuffer * >
 Maps ChunkKeys (unique ids for Chunks) to Chunk objects. More...
 
using TablePair = std::pair< const int32_t, const int32_t >
 
using PageHeaderSizeType = int32_t
 

Enumerations

enum  DiskCacheLevel { DiskCacheLevel::none, DiskCacheLevel::fsi, DiskCacheLevel::non_fsi, DiskCacheLevel::all }
 

Functions

std::string get_dir_name_for_table (int db_id, int tb_id)
 
static size_t readForThread (FileBuffer *fileBuffer, const readThreadDS threadDS)
 
std::string get_data_file_path (const std::string &base_path, int file_id, size_t page_size)
 
FILE * create (const std::string &basePath, const int fileId, const size_t pageSize, const size_t numPages)
 
FILE * create (const std::string &fullPath, const size_t requestedFileSize)
 
FILE * open (int fileId)
 Opens/creates the file with the given id; returns NULL on error. More...
 
FILE * open (const std::string &path)
 
void close (FILE *f)
 Closes the file pointed to by the FILE pointer. More...
 
bool removeFile (const std::string basePath, const std::string filename)
 Deletes the file pointed to by the FILE pointer. More...
 
size_t read (FILE *f, const size_t offset, const size_t size, int8_t *buf)
 Reads the specified number of bytes from the offset position in file f into buf. More...
 
size_t write (FILE *f, const size_t offset, const size_t size, const int8_t *buf)
 Writes the specified number of bytes to the offset position in file f from buf. More...
 
size_t append (FILE *f, const size_t size, const int8_t *buf)
 Appends the specified number of bytes to the end of the file f from buf. More...
 
size_t readPage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf)
 Reads the specified page from the file f into buf. More...
 
size_t readPartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t readSize, const size_t pageNum, int8_t *buf)
 
size_t writePage (FILE *f, const size_t pageSize, const size_t pageNum, int8_t *buf)
 Writes a page from buf to the file. More...
 
size_t writePartialPage (FILE *f, const size_t pageSize, const size_t offset, const size_t writeSize, const size_t pageNum, int8_t *buf)
 
size_t appendPage (FILE *f, const size_t pageSize, int8_t *buf)
 Appends a page from buf to the file. More...
 
size_t fileSize (FILE *f)
 Returns the size of the specified file. More...
 
void renameForDelete (const std::string directoryName)
 Renames a directory to DELETE_ME_<EPOCH>_<oldname>. More...
 

Variables

constexpr int32_t DELETE_CONTINGENT = -1
 A FileInfo type has a file pointer and metadata about a file. More...
 
constexpr int32_t ROLLOFF_CONTINGENT = -2
 

Typedef Documentation

A Chunk is the fundamental unit of execution in Map-D.

Chunk A chunk is composed of logical pages. These pages can exist across multiple files managed by the file manager.

The collection of pages is implemented as a FileBuffer object, which is composed of a vector of MultiPage objects, one for each logical page of the file buffer.

Definition at line 68 of file FileMgr.h.

Maps ChunkKeys (unique ids for Chunks) to Chunk objects.

ChunkKeyToChunkMap The file system can store multiple chunks across multiple files. With that in mind, the challenge is to be able to reconstruct the pages that compose a chunk upon request. A chunk key (ChunkKey) uniquely identifies a chunk, and so ChunkKeyToChunkMap maps chunk keys to Chunk types, which are vectors of MultiPage* pointers (logical pages).

Definition at line 80 of file FileMgr.h.

using File_Namespace::PageHeaderSizeType = typedef int32_t

Definition at line 121 of file FileMgr.h.

using File_Namespace::PageSizeFileMMap = typedef std::multimap<size_t, int32_t>

Maps logical page sizes to files.

PageSizeFileMMap The file manager uses this type in order to quickly find files of a certain page size. A multimap is used to associate the key (page size) with values (file identifiers of files having the matching page size).

Definition at line 56 of file FileMgr.h.

using File_Namespace::TablePair = typedef std::pair<const int32_t, const int32_t>

TablePair Pair detailing the id for a database and table (first two entries in a ChunkKey).

Definition at line 86 of file FileMgr.h.

Enumeration Type Documentation

Function Documentation

size_t File_Namespace::append ( FILE *  f,
const size_t  size,
const int8_t *  buf 
)

Appends the specified number of bytes to the end of the file f from buf.

Parameters
fPointer to the FILE.
nThe number of bytes to append to the file.
bufThe source buffer containing the data to be appended.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written.

Definition at line 159 of file File.cpp.

References logger::FATAL, fileSize(), g_read_only, LOG, and write().

Referenced by concat(), concat_with(), com.mapd.utility.SQLImporter::createMapDTable(), com.omnisci.jdbc.OmniSciPreparedStatement::getQuery(), com.mapd.parser.server.ExtensionFunctionSignatureParser::join(), run_benchmark::read_query_files(), run_benchmark_arrow::run_query(), run_benchmark::run_query(), com.omnisci.jdbc.OmniSciEscapeFunctions::singleArgumentFunctionCall(), com.mapd.parser.server.ExtensionFunction::toJson(), and com.omnisci.jdbc.OmniSciArray::toString().

159  {
160  if (g_read_only) {
161  LOG(FATAL) << "Error trying to append file '" << f << "', running readonly";
162  }
163  return write(f, fileSize(f), size, buf);
164 }
#define LOG(tag)
Definition: Logger.h:200
size_t write(FILE *f, const size_t offset, const size_t size, const int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:141
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:206
bool g_read_only
Definition: File.cpp:38
char * f

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::appendPage ( FILE *  f,
const size_t  pageSize,
int8_t *  buf 
)

Appends a page from buf to the file.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
bufThe source buffer from where data is being read.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes appended (should be equal to pageSize).

Definition at line 198 of file File.cpp.

References logger::FATAL, fileSize(), g_read_only, LOG, and write().

198  {
199  if (g_read_only) {
200  LOG(FATAL) << "Error trying to appendPage file '" << f << "', running readonly";
201  }
202  return write(f, fileSize(f), pageSize, buf);
203 }
#define LOG(tag)
Definition: Logger.h:200
size_t write(FILE *f, const size_t offset, const size_t size, const int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:141
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:206
bool g_read_only
Definition: File.cpp:38
char * f

+ Here is the call graph for this function:

void File_Namespace::close ( FILE *  f)

Closes the file pointed to by the FILE pointer.

Parameters
fPointer to the FILE.

Definition at line 119 of file File.cpp.

References CHECK, and CHECK_EQ.

Referenced by File_Namespace::FileMgr::clearFileInfos(), File_Namespace::FileMgr::closePhysicalUnlocked(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileMgr::readVersionFromDisk(), File_Namespace::FileMgr::writeAndSyncVersionToDisk(), File_Namespace::FileInfo::~FileInfo(), File_Namespace::FileMgr::~FileMgr(), and File_Namespace::TableFileMgr::~TableFileMgr().

119  {
120  CHECK(f);
121  CHECK_EQ(fflush(f), 0);
122  CHECK_EQ(fclose(f), 0);
123 }
#define CHECK_EQ(x, y)
Definition: Logger.h:214
#define CHECK(condition)
Definition: Logger.h:206
char * f

+ Here is the caller graph for this function:

FILE * File_Namespace::create ( const std::string &  basePath,
const int  fileId,
const size_t  pageSize,
const size_t  numPages 
)

Definition at line 49 of file File.cpp.

References f, logger::FATAL, fileSize(), omnisci::fopen(), get_data_file_path(), and LOG.

Referenced by File_Namespace::FileMgr::createEpochFile(), File_Namespace::FileMgr::createFile(), kafka_insert(), org.apache.calcite.sql2rel.SqlToRelConverter::SqlToRelConverter(), File_Namespace::TableFileMgr::TableFileMgr(), and File_Namespace::FileMgr::writeAndSyncVersionToDisk().

52  {
53  auto path = get_data_file_path(basePath, fileId, pageSize);
54  if (numPages < 1 || pageSize < 1) {
55  LOG(FATAL) << "Error trying to create file '" << path
56  << "', Number of pages and page size must be positive integers. numPages "
57  << numPages << " pageSize " << pageSize;
58  }
59  FILE* f = omnisci::fopen(path.c_str(), "w+b");
60  if (f == nullptr) {
61  LOG(FATAL) << "Error trying to create file '" << path
62  << "', the error was: " << std::strerror(errno);
63  }
64  fseek(f, static_cast<long>((pageSize * numPages) - 1), SEEK_SET);
65  fputc(EOF, f);
66  fseek(f, 0, SEEK_SET); // rewind
67  if (fileSize(f) != pageSize * numPages) {
68  LOG(FATAL) << "Error trying to create file '" << path << "', file size "
69  << fileSize(f) << " does not equal pageSize * numPages "
70  << pageSize * numPages;
71  }
72 
73  return f;
74 }
::FILE * fopen(const char *filename, const char *mode)
Definition: omnisci_fs.cpp:72
#define LOG(tag)
Definition: Logger.h:200
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:206
char * f
std::string get_data_file_path(const std::string &base_path, int file_id, size_t page_size)
Definition: File.cpp:42

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

FILE * File_Namespace::create ( const std::string &  fullPath,
const size_t  requestedFileSize 
)

Definition at line 76 of file File.cpp.

References f, logger::FATAL, fileSize(), omnisci::fopen(), g_read_only, and LOG.

76  {
77  if (g_read_only) {
78  LOG(FATAL) << "Error trying to create file '" << fullPath
79  << "', not allowed read only ";
80  }
81  FILE* f = omnisci::fopen(fullPath.c_str(), "w+b");
82  if (f == nullptr) {
83  LOG(FATAL) << "Error trying to create file '" << fullPath
84  << "', the error was: " << std::strerror(errno);
85  ;
86  }
87  fseek(f, static_cast<long>(requestedFileSize - 1), SEEK_SET);
88  fputc(EOF, f);
89  fseek(f, 0, SEEK_SET); // rewind
90  if (fileSize(f) != requestedFileSize) {
91  LOG(FATAL) << "Error trying to create file '" << fullPath << "', file size "
92  << fileSize(f) << " does not equal requestedFileSize "
93  << requestedFileSize;
94  }
95  return f;
96 }
::FILE * fopen(const char *filename, const char *mode)
Definition: omnisci_fs.cpp:72
#define LOG(tag)
Definition: Logger.h:200
size_t fileSize(FILE *f)
Returns the size of the specified file.
Definition: File.cpp:206
bool g_read_only
Definition: File.cpp:38
char * f

+ Here is the call graph for this function:

size_t File_Namespace::fileSize ( FILE *  f)

Returns the size of the specified file.

Todo:
There may be an issue casting to size_t from long.
Parameters
fA pointer to the file.
Returns
size_t The number of bytes of the file.

Definition at line 206 of file File.cpp.

Referenced by append(), appendPage(), and create().

206  {
207  fseek(f, 0, SEEK_END);
208  size_t size = (size_t)ftell(f);
209  fseek(f, 0, SEEK_SET);
210  return size;
211 }
char * f

+ Here is the caller graph for this function:

std::string File_Namespace::get_data_file_path ( const std::string &  base_path,
int  file_id,
size_t  page_size 
)

Definition at line 42 of file File.cpp.

References MAPD_FILE_EXT, and to_string().

Referenced by create(), and File_Namespace::FileMgr::deleteEmptyFiles().

44  {
45  return base_path + "/" + std::to_string(file_id) + "." + std::to_string(page_size) +
46  std::string(MAPD_FILE_EXT); // MAPD_FILE_EXT has preceding "."
47 }
#define MAPD_FILE_EXT
Definition: File.h:25
std::string to_string(char const *&&v)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string File_Namespace::get_dir_name_for_table ( int  db_id,
int  tb_id 
)
inline

Definition at line 79 of file CachingFileMgr.h.

Referenced by File_Namespace::CachingFileMgr::getTableFileMgrPath().

79  {
80  std::stringstream file_name;
81  file_name << "table_" << db_id << "_" << tb_id << "/";
82  return file_name.str();
83 }

+ Here is the caller graph for this function:

FILE * File_Namespace::open ( int  fileId)

Opens/creates the file with the given id; returns NULL on error.

Parameters
fileIdThe id of the file to open.
Returns
FILE* A pointer to a FILE pointer, or NULL on error.

Definition at line 98 of file File.cpp.

References f, logger::FATAL, omnisci::fopen(), g_read_only, LOG, MAPD_FILE_EXT, and to_string().

Referenced by File_Namespace::FileMgr::openAndReadEpochFile(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileMgr::openExistingFile(), File_Namespace::FileMgr::readVersionFromDisk(), File_Namespace::TableFileMgr::TableFileMgr(), and File_Namespace::FileMgr::writeAndSyncVersionToDisk().

98  {
99  std::string s(std::to_string(fileId) + std::string(MAPD_FILE_EXT));
100  FILE* f = omnisci::fopen(
101  s.c_str(), g_read_only ? "rb" : "r+b"); // opens existing file for updates
102  if (f == nullptr) {
103  LOG(FATAL) << "Error trying to open file '" << s
104  << "', the error was: " << std::strerror(errno);
105  }
106  return f;
107 }
::FILE * fopen(const char *filename, const char *mode)
Definition: omnisci_fs.cpp:72
#define LOG(tag)
Definition: Logger.h:200
#define MAPD_FILE_EXT
Definition: File.h:25
std::string to_string(char const *&&v)
bool g_read_only
Definition: File.cpp:38
char * f

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

FILE * File_Namespace::open ( const std::string &  path)

Definition at line 109 of file File.cpp.

References f, logger::FATAL, omnisci::fopen(), g_read_only, and LOG.

109  {
110  FILE* f = omnisci::fopen(
111  path.c_str(), g_read_only ? "rb" : "r+b"); // opens existing file for updates
112  if (f == nullptr) {
113  LOG(FATAL) << "Error trying to open file '" << path
114  << "', the errno was: " << std::strerror(errno);
115  }
116  return f;
117 }
::FILE * fopen(const char *filename, const char *mode)
Definition: omnisci_fs.cpp:72
#define LOG(tag)
Definition: Logger.h:200
bool g_read_only
Definition: File.cpp:38
char * f

+ Here is the call graph for this function:

size_t File_Namespace::read ( FILE *  f,
const size_t  offset,
const size_t  size,
int8_t *  buf 
)

Reads the specified number of bytes from the offset position in file f into buf.

Parameters
fPointer to the FILE.
offsetThe location within the file from which to read.
nThe number of bytes to be read.
bufThe destination buffer to where data is being read from the file.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes read.

Definition at line 133 of file File.cpp.

References CHECK_EQ.

Referenced by File_Namespace::FileMgr::openAndReadEpochFile(), File_Namespace::FileMgr::openAndReadLegacyEpochFile(), File_Namespace::FileInfo::read(), readPage(), readPartialPage(), File_Namespace::FileMgr::readVersionFromDisk(), and File_Namespace::TableFileMgr::TableFileMgr().

133  {
134  // read "size" bytes from the offset location in the file into the buffer
135  CHECK_EQ(fseek(f, static_cast<long>(offset), SEEK_SET), 0);
136  size_t bytesRead = fread(buf, sizeof(int8_t), size, f);
137  CHECK_EQ(bytesRead, sizeof(int8_t) * size);
138  return bytesRead;
139 }
#define CHECK_EQ(x, y)
Definition: Logger.h:214
char * f

+ Here is the caller graph for this function:

static size_t File_Namespace::readForThread ( FileBuffer *  fileBuffer,
const readThreadDS  threadDS 
)
static

Definition at line 242 of file FileBuffer.cpp.

References CHECK, File_Namespace::FileMgr::getFileInfoForFileId(), File_Namespace::readThreadDS::multiPages, File_Namespace::FileBuffer::pageDataSize(), File_Namespace::FileBuffer::pageSize(), File_Namespace::FileInfo::read(), File_Namespace::FileBuffer::reservedHeaderSize(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.

Referenced by File_Namespace::FileBuffer::read().

242  {
243  size_t startPage = threadDS.t_startPage; // start reading at startPage, including it
244  size_t endPage = threadDS.t_endPage; // stop reading at endPage, not including it
245  int8_t* curPtr = threadDS.t_curPtr;
246  size_t bytesLeft = threadDS.t_bytesLeft;
247  size_t totalBytesRead = 0;
248  bool isFirstPage = threadDS.t_isFirstPage;
249 
250  // Traverse the logical pages
251  for (size_t pageNum = startPage; pageNum < endPage; ++pageNum) {
252  CHECK(threadDS.multiPages[pageNum].pageSize == fileBuffer->pageSize());
253  Page page = threadDS.multiPages[pageNum].current().page;
254 
255  FileInfo* fileInfo = threadDS.t_fm->getFileInfoForFileId(page.fileId);
256  CHECK(fileInfo);
257 
258  // Read the page into the destination (dst) buffer at its
259  // current (cur) location
260  size_t bytesRead = 0;
261  if (isFirstPage) {
262  bytesRead = fileInfo->read(
263  page.pageNum * fileBuffer->pageSize() + threadDS.t_startPageOffset +
264  fileBuffer->reservedHeaderSize(),
265  min(fileBuffer->pageDataSize() - threadDS.t_startPageOffset, bytesLeft),
266  curPtr);
267  isFirstPage = false;
268  } else {
269  bytesRead = fileInfo->read(
270  page.pageNum * fileBuffer->pageSize() + fileBuffer->reservedHeaderSize(),
271  min(fileBuffer->pageDataSize(), bytesLeft),
272  curPtr);
273  }
274  curPtr += bytesRead;
275  bytesLeft -= bytesRead;
276  totalBytesRead += bytesRead;
277  }
278  CHECK(bytesLeft == 0);
279 
280  return (totalBytesRead);
281 }
#define CHECK(condition)
Definition: Logger.h:206

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::readPage ( FILE *  f,
const size_t  pageSize,
const size_t  pageNum,
int8_t *  buf 
)

Reads the specified page from the file f into buf.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
pageNumThe page number from where data is being read.
bufThe destination buffer to where data is being written.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes read (should be equal to pageSize).

Definition at line 166 of file File.cpp.

References read().

166  {
167  return read(f, pageNum * pageSize, pageSize, buf);
168 }
size_t read(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Reads the specified number of bytes from the offset position in file f into buf.
Definition: File.cpp:133
char * f

+ Here is the call graph for this function:

size_t File_Namespace::readPartialPage ( FILE *  f,
const size_t  pageSize,
const size_t  offset,
const size_t  readSize,
const size_t  pageNum,
int8_t *  buf 
)

Definition at line 170 of file File.cpp.

References read().

175  {
176  return read(f, pageNum * pageSize + offset, readSize, buf);
177 }
size_t read(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Reads the specified number of bytes from the offset position in file f into buf.
Definition: File.cpp:133
char * f

+ Here is the call graph for this function:

bool File_Namespace::removeFile ( const std::string  basePath,
const std::string  filename 
)

Deletes the file pointed to by the FILE pointer.

Parameters
basePathThe base path (directory) of the file.
fPointer to the FILE.
Returns
mapd_err_t Returns an error code when unable to close the file properly.

Definition at line 125 of file File.cpp.

References logger::FATAL, logger::filename(), g_read_only, and LOG.

125  {
126  if (g_read_only) {
127  LOG(FATAL) << "Error trying to remove file '" << filename << "', running readonly";
128  }
129  const std::string filePath = basePath + filename;
130  return remove(filePath.c_str()) == 0;
131 }
std::string filename(char const *path)
Definition: Logger.cpp:64
#define LOG(tag)
Definition: Logger.h:200
bool g_read_only
Definition: File.cpp:38

+ Here is the call graph for this function:

void File_Namespace::renameForDelete ( const std::string  directoryName)

Renames a directory to DELETE_ME_<EPOCH>_<oldname>.

Parameters
directoryNamename of directory

Definition at line 218 of file File.cpp.

References logger::ERROR, logger::FATAL, LOG, and to_string().

Referenced by File_Namespace::FileMgr::closeRemovePhysical(), Catalog_Namespace::Catalog::delDictionary(), Catalog_Namespace::Catalog::doTruncateTable(), and Catalog_Namespace::Catalog::removeTableFromMap().

218  {
219  boost::system::error_code ec;
220  boost::filesystem::path directoryPath(directoryName);
221  using namespace std::chrono;
222  milliseconds ms = duration_cast<milliseconds>(system_clock::now().time_since_epoch());
223 
224  if (boost::filesystem::exists(directoryPath) &&
225  boost::filesystem::is_directory(directoryPath)) {
226  boost::filesystem::path newDirectoryPath(directoryName + "_" +
227  std::to_string(ms.count()) + "_DELETE_ME");
228  boost::filesystem::rename(directoryPath, newDirectoryPath, ec);
229 
230 #ifdef _WIN32
231  // On Windows we sometimes fail to rename a directory with System: 5 error
232  // code (access denied). An attempt to stop in debugger and look for opened
233  // handles for some of directory content shows no opened handles and actually
234  // allows renaming to execute successfully. It's not clear why, but a short
235  // pause allows to rename directory successfully. Until reasons are known,
236  // use this retry loop as a workaround.
237  int tries = 10;
238  while (ec.value() != boost::system::errc::success && tries) {
239  LOG(ERROR) << "Failed to rename directory " << directoryPath << " error was " << ec
240  << " (" << tries << " attempts left)";
241  std::this_thread::sleep_for(std::chrono::milliseconds(100 / tries));
242  tries--;
243  boost::filesystem::rename(directoryPath, newDirectoryPath, ec);
244  }
245 #endif
246 
247  if (ec.value() == boost::system::errc::success) {
248  std::thread th([newDirectoryPath]() {
249  boost::system::error_code ec;
250  boost::filesystem::remove_all(newDirectoryPath, ec);
251  // We dont check error on remove here as we cant log the
252  // issue fromdetached thrad, its not safe to LOG from here
253  // This is under investigation as clang detects TSAN issue data race
254  // the main system wide file_delete_thread will clean up any missed files
255  });
256  // let it run free so we can return
257  // if it fails the file_delete_thread in DBHandler will clean up
258  th.detach();
259 
260  return;
261  }
262 
263  LOG(FATAL) << "Failed to rename file " << directoryName << " to "
264  << directoryName + "_" + std::to_string(ms.count()) + "_DELETE_ME Error: "
265  << ec;
266  }
267 }
#define LOG(tag)
Definition: Logger.h:200
std::string to_string(char const *&&v)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::write ( FILE *  f,
const size_t  offset,
const size_t  size,
const int8_t *  buf 
)

Writes the specified number of bytes to the offset position in file f from buf.

Parameters
fPointer to the FILE.
offsetThe location within the file where data is being written.
sizeThe number of bytes to write to the file.
bufThe source buffer containing the data to be written.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written.

Definition at line 141 of file File.cpp.

References logger::FATAL, g_read_only, and LOG.

Referenced by StringDictionary::addStorageCapacity(), append(), appendPage(), File_Namespace::FileInfo::freePage(), File_Namespace::FileInfo::freePageImmediate(), import_export::DataStreamSink::import_compressed(), File_Namespace::FileInfo::initNewFile(), File_Namespace::FileInfo::recoverPage(), CommandLineOptions::validate(), File_Namespace::FileInfo::write(), File_Namespace::TableFileMgr::writeAndSyncEpochToDisk(), File_Namespace::FileMgr::writeAndSyncEpochToDisk(), File_Namespace::FileMgr::writeAndSyncVersionToDisk(), writePage(), and writePartialPage().

141  {
142  if (g_read_only) {
143  LOG(FATAL) << "Error trying to write file '" << f << "', running readonly";
144  }
145  // write size bytes from the buffer to the offset location in the file
146  if (fseek(f, static_cast<long>(offset), SEEK_SET) != 0) {
147  LOG(FATAL)
148  << "Error trying to write to file (during positioning seek) the error was: "
149  << std::strerror(errno);
150  }
151  size_t bytesWritten = fwrite(buf, sizeof(int8_t), size, f);
152  if (bytesWritten != sizeof(int8_t) * size) {
153  LOG(FATAL) << "Error trying to write to file (during fwrite) the error was: "
154  << std::strerror(errno);
155  }
156  return bytesWritten;
157 }
#define LOG(tag)
Definition: Logger.h:200
bool g_read_only
Definition: File.cpp:38
char * f

+ Here is the caller graph for this function:

size_t File_Namespace::writePage ( FILE *  f,
const size_t  pageSize,
const size_t  pageNum,
int8_t *  buf 
)

Writes a page from buf to the file.

Parameters
fPointer to the FILE.
pageSizeThe logical page size of the file.
pageNumThe page number to where data is being written.
bufThe source buffer from where data is being read.
errIf not NULL, will hold an error code should an error occur.
Returns
size_t The number of bytes written (should be equal to pageSize).

Definition at line 179 of file File.cpp.

References logger::FATAL, g_read_only, LOG, and write().

179  {
180  if (g_read_only) {
181  LOG(FATAL) << "Error trying to writePage file '" << f << "', running readonly";
182  }
183  return write(f, pageNum * pageSize, pageSize, buf);
184 }
#define LOG(tag)
Definition: Logger.h:200
size_t write(FILE *f, const size_t offset, const size_t size, const int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:141
bool g_read_only
Definition: File.cpp:38
char * f

+ Here is the call graph for this function:

size_t File_Namespace::writePartialPage ( FILE *  f,
const size_t  pageSize,
const size_t  offset,
const size_t  writeSize,
const size_t  pageNum,
int8_t *  buf 
)

Definition at line 186 of file File.cpp.

References logger::FATAL, g_read_only, LOG, and write().

191  {
192  if (g_read_only) {
193  LOG(FATAL) << "Error trying to writePartialPage file '" << f << "', running readonly";
194  }
195  return write(f, pageNum * pageSize + offset, writeSize, buf);
196 }
#define LOG(tag)
Definition: Logger.h:200
size_t write(FILE *f, const size_t offset, const size_t size, const int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:141
bool g_read_only
Definition: File.cpp:38
char * f

+ Here is the call graph for this function:

Variable Documentation

constexpr int32_t File_Namespace::DELETE_CONTINGENT = -1

A FileInfo type has a file pointer and metadata about a file.

FileInfo A file info structure wraps around a file pointer in order to contain additional information/metadata about the file that is pertinent to the file manager.

The free pages (freePages) within a file must be tracked, and this is implemented using a basic STL set. The set ensures that no duplicate pages are included, and that the pages are sorted, faciliating the obtaining of consecutive free pages by a constant time pop operation, which may reduce the cost of DBMS disk accesses.

Helper functions are provided: size(), available(), and used().

Definition at line 51 of file FileInfo.h.

Referenced by File_Namespace::FileInfo::freePage(), File_Namespace::CachingFileMgr::updatePageIfDeleted(), and File_Namespace::FileMgr::updatePageIfDeleted().

constexpr int32_t File_Namespace::ROLLOFF_CONTINGENT = -2