OmniSciDB  85c2d10cdc
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
File_Namespace::FileInfo Struct Reference

#include <FileInfo.h>

+ Collaboration diagram for File_Namespace::FileInfo:

Public Member Functions

 FileInfo (FileMgr *fileMgr, const int32_t fileId, FILE *f, const size_t pageSize, const size_t numPages, const bool init=false)
 Constructor. More...
 
 ~FileInfo ()
 Destructor. More...
 
void initNewFile ()
 Adds all pages to freePages and zeroes first four bytes of header. More...
 
void freePageDeferred (int32_t pageId)
 
void freePage (int32_t pageId, const bool isRolloff)
 
int32_t getFreePage ()
 
size_t write (const size_t offset, const size_t size, int8_t *buf)
 
size_t read (const size_t offset, const size_t size, int8_t *buf)
 
void openExistingFile (std::vector< HeaderInfo > &headerVec, const int32_t fileMgrEpoch)
 
void print (bool pagesummary)
 Prints a summary of the file to stdout. More...
 
size_t size () const
 Returns the number of bytes used by the file. More...
 
int32_t syncToDisk ()
 
size_t available ()
 Returns the number of free bytes available. More...
 
size_t numFreePages ()
 Returns the number of free pages available. More...
 
size_t used ()
 Returns the amount of used bytes; size() - available() More...
 

Public Attributes

FileMgrfileMgr
 
int32_t fileId
 
FILE * f
 unique file identifier (i.e., used for a file name) More...
 
size_t pageSize
 file stream object for the represented file More...
 
size_t numPages
 the fixed size of each page in the file More...
 
bool isDirty {false}
 the number of pages in the file More...
 
std::set< size_t > freePages
 
std::mutex freePagesMutex_
 set of page numbers of free pages More...
 
std::mutex readWriteMutex_
 

Detailed Description

Definition at line 55 of file FileInfo.h.

Constructor & Destructor Documentation

File_Namespace::FileInfo::FileInfo ( FileMgr fileMgr,
const int32_t  fileId,
FILE *  f,
const size_t  pageSize,
const size_t  numPages,
const bool  init = false 
)

Constructor.

Definition at line 28 of file FileInfo.cpp.

References initNewFile().

35  if (init) {
36  initNewFile();
37  }
38 }
size_t pageSize
file stream object for the represented file
Definition: FileInfo.h:59
void init(LogOptions const &log_opts)
Definition: Logger.cpp:280
void initNewFile()
Adds all pages to freePages and zeroes first four bytes of header.
Definition: FileInfo.cpp:47
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:58
size_t numPages
the fixed size of each page in the file
Definition: FileInfo.h:60

+ Here is the call graph for this function:

File_Namespace::FileInfo::~FileInfo ( )

Destructor.

Definition at line 40 of file FileInfo.cpp.

References File_Namespace::close(), and f.

40  {
41  // close file, if applicable
42  if (f) {
43  close(f);
44  }
45 }
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:58
void close(FILE *f)
Closes the file pointed to by the FILE pointer.
Definition: File.cpp:119

+ Here is the call graph for this function:

Member Function Documentation

size_t File_Namespace::FileInfo::available ( )
inline

Returns the number of free bytes available.

Definition at line 99 of file FileInfo.h.

References freePages, and pageSize.

Referenced by print(), and used().

99 { return freePages.size() * pageSize; }
std::set< size_t > freePages
Definition: FileInfo.h:62
size_t pageSize
file stream object for the represented file
Definition: FileInfo.h:59

+ Here is the caller graph for this function:

void File_Namespace::FileInfo::freePage ( int32_t  pageId,
const bool  isRolloff 
)

Definition at line 206 of file FileInfo.cpp.

References CHECK, File_Namespace::DELETE_CONTINGENT, File_Namespace::FileMgr::epoch(), f, fileMgr, File_Namespace::FileMgr::free_page(), freePages, freePagesMutex_, isDirty, pageSize, readWriteMutex_, File_Namespace::ROLLOFF_CONTINGENT, and File_Namespace::write().

Referenced by File_Namespace::FileBuffer::freePage().

206  {
207  std::lock_guard<std::mutex> lock(readWriteMutex_);
208 #define RESILIENT_PAGE_HEADER
209 #ifdef RESILIENT_PAGE_HEADER
210  int epoch_freed_page[2] = {DELETE_CONTINGENT, fileMgr->epoch()};
211  if (isRolloff) {
212  epoch_freed_page[0] = ROLLOFF_CONTINGENT;
213  }
215  pageId * pageSize + sizeof(int32_t),
216  sizeof(epoch_freed_page),
217  (int8_t*)epoch_freed_page);
218  fileMgr->free_page(std::make_pair(this, pageId));
219 #else
220  int32_t zeroVal = 0;
221  int8_t* zeroAddr = reinterpret_cast<int8_t*>(&zeroVal);
222  File_Namespace::write(f, pageId * pageSize, sizeof(int32_t), zeroAddr);
223  std::lock_guard<std::mutex> lock(freePagesMutex_);
224  freePages.insert(pageId);
225 #endif // RESILIENT_PAGE_HEADER
226  isDirty = true;
227 
228 #ifdef ENABLE_CRASH_CORRUPTION_TEST
229  signal(SIGUSR2, sighandler);
230  if (goto_crash)
231  CHECK(pageId % 8 != 4);
232 #endif
233 }
std::mutex readWriteMutex_
Definition: FileInfo.h:64
std::set< size_t > freePages
Definition: FileInfo.h:62
size_t pageSize
file stream object for the represented file
Definition: FileInfo.h:59
constexpr int32_t DELETE_CONTINGENT
A FileInfo type has a file pointer and metadata about a file.
Definition: FileInfo.h:51
std::mutex freePagesMutex_
set of page numbers of free pages
Definition: FileInfo.h:63
constexpr int32_t ROLLOFF_CONTINGENT
Definition: FileInfo.h:52
int32_t epoch()
Returns current value of epoch - should be one greater than recorded at last checkpoint.
Definition: FileMgr.h:273
void free_page(std::pair< FileInfo *, int32_t > &&page)
Definition: FileMgr.cpp:1212
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:58
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:141
#define CHECK(condition)
Definition: Logger.h:197
bool isDirty
the number of pages in the file
Definition: FileInfo.h:61

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileInfo::freePageDeferred ( int32_t  pageId)

Definition at line 191 of file FileInfo.cpp.

References freePages, and freePagesMutex_.

191  {
192  std::lock_guard<std::mutex> lock(freePagesMutex_);
193  freePages.insert(pageId);
194 }
std::set< size_t > freePages
Definition: FileInfo.h:62
std::mutex freePagesMutex_
set of page numbers of free pages
Definition: FileInfo.h:63
int32_t File_Namespace::FileInfo::getFreePage ( )

Definition at line 235 of file FileInfo.cpp.

References freePages, and freePagesMutex_.

Referenced by File_Namespace::FileMgr::copySourcePageForCompaction(), File_Namespace::FileMgr::requestFreePage(), and File_Namespace::FileMgr::requestFreePages().

235  {
236  // returns -1 if there is no free page
237  std::lock_guard<std::mutex> lock(freePagesMutex_);
238  if (freePages.size() == 0) {
239  return -1;
240  }
241  auto pageIt = freePages.begin();
242  int32_t pageNum = *pageIt;
243  freePages.erase(pageIt);
244  return pageNum;
245 }
std::set< size_t > freePages
Definition: FileInfo.h:62
std::mutex freePagesMutex_
set of page numbers of free pages
Definition: FileInfo.h:63

+ Here is the caller graph for this function:

void File_Namespace::FileInfo::initNewFile ( )

Adds all pages to freePages and zeroes first four bytes of header.

Definition at line 47 of file FileInfo.cpp.

References f, freePages, isDirty, numPages, pageSize, and File_Namespace::write().

Referenced by FileInfo().

47  {
48  // initialize pages and free page list
49  // Also zeroes out first four bytes of every header
50 
51  int32_t headerSize = 0;
52  int8_t* headerSizePtr = (int8_t*)(&headerSize);
53  for (size_t pageId = 0; pageId < numPages; ++pageId) {
54  File_Namespace::write(f, pageId * pageSize, sizeof(int32_t), headerSizePtr);
55  freePages.insert(pageId);
56  }
57  isDirty = true;
58 }
std::set< size_t > freePages
Definition: FileInfo.h:62
size_t pageSize
file stream object for the represented file
Definition: FileInfo.h:59
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:58
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:141
size_t numPages
the fixed size of each page in the file
Definition: FileInfo.h:60
bool isDirty
the number of pages in the file
Definition: FileInfo.h:61

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::FileInfo::numFreePages ( )
inline

Returns the number of free pages available.

Definition at line 102 of file FileInfo.h.

References freePages, and freePagesMutex_.

102  {
103  std::lock_guard<std::mutex> lock(freePagesMutex_);
104  return freePages.size();
105  }
std::set< size_t > freePages
Definition: FileInfo.h:62
std::mutex freePagesMutex_
set of page numbers of free pages
Definition: FileInfo.h:63
void File_Namespace::FileInfo::openExistingFile ( std::vector< HeaderInfo > &  headerVec,
const int32_t  fileMgrEpoch 
)

Definition at line 71 of file FileInfo.cpp.

References CHECK_EQ, CHECK_GE, File_Namespace::DELETE_CONTINGENT, f, fileId, fileMgr, freePages, g_read_only, File_Namespace::FileMgr::get_fileMgrKey(), File_Namespace::FileMgr::hasFileMgrKey(), LOG, numPages, pageSize, File_Namespace::ROLLOFF_CONTINGENT, show_chunk(), VLOG, logger::WARNING, and File_Namespace::write().

Referenced by File_Namespace::FileMgr::openExistingFile().

72  {
73  // HeaderInfo is defined in Page.h
74 
75  // Oct 2020: Changing semantics such that fileMgrEpoch should be last checkpointed
76  // epoch, not incremented epoch. This changes some of the gt/gte/lt/lte comparison below
77  ChunkKey oldChunkKey(4);
78  int32_t oldPageId = -99;
79  int32_t oldVersionEpoch = -99;
80  int32_t skipped = 0;
81  for (size_t pageNum = 0; pageNum < numPages; ++pageNum) {
82  int32_t headerSize;
83 
84  constexpr size_t MAX_INTS_TO_READ{10}; // currently use 1+6 ints
85  int32_t ints[MAX_INTS_TO_READ];
86  CHECK_EQ(fseek(f, pageNum * pageSize, SEEK_SET), 0);
87  CHECK_EQ(fread(ints, sizeof(int32_t), MAX_INTS_TO_READ, f), MAX_INTS_TO_READ);
88 
89  headerSize = ints[0];
90  const bool should_delete_deleted =
91  DELETE_CONTINGENT == ints[1] && fileMgrEpoch >= ints[2];
92  const bool should_delete_rolled_off =
93  ROLLOFF_CONTINGENT == ints[1] && fileMgrEpoch >= ints[2];
94  if (should_delete_deleted || should_delete_rolled_off) {
95  int32_t zero{0};
96  File_Namespace::write(f, pageNum * pageSize, sizeof(int32_t), (int8_t*)&zero);
97  headerSize = 0;
98  }
99 
100  if (headerSize != 0) {
101  // headerSize doesn't include headerSize itself
102  // We're tying ourself to headers of ints here
103  size_t numHeaderElems = headerSize / sizeof(int32_t);
104  CHECK_GE(numHeaderElems, size_t(2));
105  // We don't want to read headerSize in our header - so start
106  // reading 4 bytes past it
107 
108  // Derive dbid/tbid if from FileMgr
109  ChunkKey chunkKey(&ints[1], &ints[1 + numHeaderElems - 2]);
110  if (fileMgr->hasFileMgrKey()) {
111  // A regular FileMgr is locked to one table, but a CachingFileMgr can contain
112  // chunks from different tables
113  chunkKey[0] = fileMgr->get_fileMgrKey().first;
114  chunkKey[1] = fileMgr->get_fileMgrKey().second;
115  }
116  // recover page in case a crash failed deletion of this page
117  if (!g_read_only) {
118  if (ints[1] == DELETE_CONTINGENT || ints[1] == ROLLOFF_CONTINGENT) {
120  pageNum * pageSize + sizeof(int32_t),
121  2 * sizeof(int32_t),
122  (int8_t*)&chunkKey[0]);
123  }
124  }
125 
126  // Last two elements of header are always PageId and Version
127  // epoch - these are not in the chunk key so seperate them
128  int32_t pageId = ints[1 + numHeaderElems - 2];
129  int32_t versionEpoch = ints[1 + numHeaderElems - 1];
130  if (chunkKey != oldChunkKey || oldPageId != pageId - (1 + skipped)) {
131  if (skipped > 0) {
132  VLOG(4) << "FId.PSz: " << fileId << "." << pageSize
133  << " Chunk key: " << show_chunk(oldChunkKey)
134  << " Page id from : " << oldPageId << " to : " << oldPageId + skipped
135  << " Epoch: " << oldVersionEpoch;
136  } else if (oldPageId != -99) {
137  VLOG(4) << "FId.PSz: " << fileId << "." << pageSize
138  << " Chunk key: " << show_chunk(oldChunkKey)
139  << " Page id: " << oldPageId << " Epoch: " << oldVersionEpoch;
140  }
141  oldPageId = pageId;
142  oldVersionEpoch = versionEpoch;
143  oldChunkKey = chunkKey;
144  skipped = 0;
145  } else {
146  skipped++;
147  }
148 
149  /* Check if version epoch is equal to
150  * or greater (note: should never be greater)
151  * than FileMgr epoch_ - this means that this
152  * page wasn't checkpointed and thus we should
153  * not use it
154  */
155  if (versionEpoch > fileMgrEpoch) {
156  // First write 0 to first four bytes of
157  // header to mark as free
158  if (!g_read_only) {
159  headerSize = 0;
161  f, pageNum * pageSize, sizeof(int32_t), (int8_t*)&headerSize);
162  // Now add page to free list
163  freePages.insert(pageNum);
164  }
165  LOG(WARNING) << "Was not checkpointed: Chunk key: " << show_chunk(chunkKey)
166  << " Page id: " << pageId << " Epoch: " << versionEpoch
167  << " FileMgrEpoch " << fileMgrEpoch << endl;
168  } else { // page was checkpointed properly
169  Page page(fileId, pageNum);
170  headerVec.emplace_back(chunkKey, pageId, versionEpoch, page);
171  }
172  } else { // no header for this page - insert into free list
173  freePages.insert(pageNum);
174  }
175  }
176  // printlast
177  if (oldPageId != -99) {
178  if (skipped > 0) {
179  VLOG(4) << "FId.PSz: " << fileId << "." << pageSize
180  << " Chunk key: " << show_chunk(oldChunkKey)
181  << " Page id from : " << oldPageId << " to : " << oldPageId + skipped
182  << " Epoch: " << oldVersionEpoch;
183  } else {
184  VLOG(4) << "FId.PSz: " << fileId << "." << pageSize
185  << " Chunk key: " << show_chunk(oldChunkKey) << " Page id: " << oldPageId
186  << " Epoch: " << oldVersionEpoch;
187  }
188  }
189 }
#define CHECK_EQ(x, y)
Definition: Logger.h:205
std::vector< int > ChunkKey
Definition: types.h:37
virtual bool hasFileMgrKey() const
Definition: FileMgr.h:336
#define LOG(tag)
Definition: Logger.h:188
#define CHECK_GE(x, y)
Definition: Logger.h:210
std::string show_chunk(const ChunkKey &key)
Definition: types.h:85
const std::pair< const int32_t, const int32_t > get_fileMgrKey() const
Definition: FileMgr.h:337
std::set< size_t > freePages
Definition: FileInfo.h:62
size_t pageSize
file stream object for the represented file
Definition: FileInfo.h:59
constexpr int32_t DELETE_CONTINGENT
A FileInfo type has a file pointer and metadata about a file.
Definition: FileInfo.h:51
constexpr int32_t ROLLOFF_CONTINGENT
Definition: FileInfo.h:52
bool g_read_only
Definition: File.cpp:38
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:58
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:141
size_t numPages
the fixed size of each page in the file
Definition: FileInfo.h:60
#define VLOG(n)
Definition: Logger.h:291

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileInfo::print ( bool  pagesummary)

Prints a summary of the file to stdout.

Definition at line 247 of file FileInfo.cpp.

References available(), fileId, size(), and used().

247  {
248  std::cout << "File: " << fileId << std::endl;
249  std::cout << "Size: " << size() << std::endl;
250  std::cout << "Used: " << used() << std::endl;
251  std::cout << "Free: " << available() << std::endl;
252  if (!pagesummary) {
253  return;
254  }
255 }
size_t size() const
Returns the number of bytes used by the file.
Definition: FileInfo.h:92
size_t used()
Returns the amount of used bytes; size() - available()
Definition: FileInfo.h:108
size_t available()
Returns the number of free bytes available.
Definition: FileInfo.h:99

+ Here is the call graph for this function:

size_t File_Namespace::FileInfo::read ( const size_t  offset,
const size_t  size,
int8_t *  buf 
)

Definition at line 66 of file FileInfo.cpp.

References f, File_Namespace::read(), and readWriteMutex_.

Referenced by File_Namespace::FileBuffer::copyPage(), File_Namespace::FileMgr::copyPage(), File_Namespace::FileMgr::copyPageWithoutHeaderSize(), and File_Namespace::readForThread().

66  {
67  std::lock_guard<std::mutex> lock(readWriteMutex_);
68  return File_Namespace::read(f, offset, size, buf);
69 }
std::mutex readWriteMutex_
Definition: FileInfo.h:64
size_t read(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Reads the specified number of bytes from the offset position in file f into buf.
Definition: File.cpp:133
size_t size() const
Returns the number of bytes used by the file.
Definition: FileInfo.h:92
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:58

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::FileInfo::size ( ) const
inline

Returns the number of bytes used by the file.

Definition at line 92 of file FileInfo.h.

References numPages, and pageSize.

Referenced by print(), and used().

92 { return pageSize * numPages; }
size_t pageSize
file stream object for the represented file
Definition: FileInfo.h:59
size_t numPages
the fixed size of each page in the file
Definition: FileInfo.h:60

+ Here is the caller graph for this function:

int32_t File_Namespace::FileInfo::syncToDisk ( )

Syncs file to disk via a buffer flush and then a sync (fflush and fsync on posix systems)

Definition at line 256 of file FileInfo.cpp.

References f, logger::FATAL, omnisci::fsync(), isDirty, LOG, and readWriteMutex_.

256  {
257  std::lock_guard<std::mutex> lock(readWriteMutex_);
258  if (isDirty) {
259  if (fflush(f) != 0) {
260  LOG(FATAL) << "Error trying to flush changes to disk, the error was: "
261  << std::strerror(errno);
262  }
263 #ifdef __APPLE__
264  const int32_t sync_result = fcntl(fileno(f), 51);
265 #else
266  const int32_t sync_result = omnisci::fsync(fileno(f));
267 #endif
268  if (sync_result == 0) {
269  isDirty = false;
270  }
271  return sync_result;
272  }
273  return 0; // if file was not dirty and no syncing was needed
274 }
#define LOG(tag)
Definition: Logger.h:188
std::mutex readWriteMutex_
Definition: FileInfo.h:64
int fsync(int fd)
Definition: omnisci_fs.cpp:60
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:58
bool isDirty
the number of pages in the file
Definition: FileInfo.h:61

+ Here is the call graph for this function:

size_t File_Namespace::FileInfo::used ( )
inline

Returns the amount of used bytes; size() - available()

Definition at line 108 of file FileInfo.h.

References available(), and size().

Referenced by print().

108 { return size() - available(); }
size_t size() const
Returns the number of bytes used by the file.
Definition: FileInfo.h:92
size_t available()
Returns the number of free bytes available.
Definition: FileInfo.h:99

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::FileInfo::write ( const size_t  offset,
const size_t  size,
int8_t *  buf 
)

Definition at line 60 of file FileInfo.cpp.

References f, isDirty, readWriteMutex_, and File_Namespace::write().

Referenced by File_Namespace::FileBuffer::append(), File_Namespace::FileBuffer::copyPage(), File_Namespace::FileMgr::copyPage(), File_Namespace::FileMgr::copyPageWithoutHeaderSize(), File_Namespace::FileBuffer::write(), and File_Namespace::FileBuffer::writeHeader().

60  {
61  std::lock_guard<std::mutex> lock(readWriteMutex_);
62  isDirty = true;
63  return File_Namespace::write(f, offset, size, buf);
64 }
std::mutex readWriteMutex_
Definition: FileInfo.h:64
size_t size() const
Returns the number of bytes used by the file.
Definition: FileInfo.h:92
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:58
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:141
bool isDirty
the number of pages in the file
Definition: FileInfo.h:61

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

Member Data Documentation

FILE* File_Namespace::FileInfo::f

unique file identifier (i.e., used for a file name)

Definition at line 58 of file FileInfo.h.

Referenced by freePage(), initNewFile(), openExistingFile(), read(), syncToDisk(), write(), and ~FileInfo().

FileMgr* File_Namespace::FileInfo::fileMgr

Definition at line 56 of file FileInfo.h.

Referenced by freePage(), and openExistingFile().

std::set<size_t> File_Namespace::FileInfo::freePages
std::mutex File_Namespace::FileInfo::freePagesMutex_

set of page numbers of free pages

Definition at line 63 of file FileInfo.h.

Referenced by freePage(), freePageDeferred(), getFreePage(), and numFreePages().

bool File_Namespace::FileInfo::isDirty {false}

the number of pages in the file

Definition at line 61 of file FileInfo.h.

Referenced by freePage(), initNewFile(), syncToDisk(), and write().

size_t File_Namespace::FileInfo::numPages

the fixed size of each page in the file

Definition at line 60 of file FileInfo.h.

Referenced by initNewFile(), openExistingFile(), and size().

size_t File_Namespace::FileInfo::pageSize

file stream object for the represented file

Definition at line 59 of file FileInfo.h.

Referenced by available(), File_Namespace::FileMgr::copyPageWithoutHeaderSize(), freePage(), initNewFile(), openExistingFile(), and size().

std::mutex File_Namespace::FileInfo::readWriteMutex_

Definition at line 64 of file FileInfo.h.

Referenced by freePage(), read(), syncToDisk(), and write().


The documentation for this struct was generated from the following files: