OmniSciDB  b24e664e58
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
File_Namespace::FileInfo Struct Reference

#include <FileInfo.h>

+ Collaboration diagram for File_Namespace::FileInfo:

Public Member Functions

 FileInfo (FileMgr *fileMgr, const int fileId, FILE *f, const size_t pageSize, const size_t numPages, const bool init=false)
 Constructor. More...
 
 ~FileInfo ()
 Destructor. More...
 
void initNewFile ()
 Adds all pages to freePages and zeroes first four bytes of header. More...
 
void freePageDeferred (int pageId)
 
void freePage (int pageId)
 
int getFreePage ()
 
size_t write (const size_t offset, const size_t size, int8_t *buf)
 
size_t read (const size_t offset, const size_t size, int8_t *buf)
 
void openExistingFile (std::vector< HeaderInfo > &headerVec, const int fileMgrEpoch)
 
void print (bool pagesummary)
 Prints a summary of the file to stdout. More...
 
size_t size ()
 Returns the number of bytes used by the file. More...
 
int syncToDisk ()
 
size_t available ()
 Returns the number of free bytes available. More...
 
size_t numFreePages ()
 Returns the number of free pages available. More...
 
size_t used ()
 Returns the amount of used bytes; size() - available() More...
 

Public Attributes

FileMgrfileMgr
 
int fileId
 
FILE * f
 unique file identifier (i.e., used for a file name) More...
 
size_t pageSize
 file stream object for the represented file More...
 
size_t numPages
 the fixed size of each page in the file More...
 
std::set< size_t > freePages
 the number of pages in the file More...
 
std::mutex freePagesMutex_
 set of page numbers of free pages More...
 
std::mutex readWriteMutex_
 

Detailed Description

Definition at line 52 of file FileInfo.h.

Constructor & Destructor Documentation

File_Namespace::FileInfo::FileInfo ( FileMgr fileMgr,
const int  fileId,
FILE *  f,
const size_t  pageSize,
const size_t  numPages,
const bool  init = false 
)

Constructor.

Definition at line 28 of file FileInfo.cpp.

References initNewFile().

35  if (init) {
36  initNewFile();
37  }
38 }
size_t pageSize
file stream object for the represented file
Definition: FileInfo.h:56
void init(LogOptions const &log_opts)
Definition: Logger.cpp:265
void initNewFile()
Adds all pages to freePages and zeroes first four bytes of header.
Definition: FileInfo.cpp:47
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:55
size_t numPages
the fixed size of each page in the file
Definition: FileInfo.h:57

+ Here is the call graph for this function:

File_Namespace::FileInfo::~FileInfo ( )

Destructor.

Definition at line 40 of file FileInfo.cpp.

References File_Namespace::close(), and f.

40  {
41  // close file, if applicable
42  if (f) {
43  close(f);
44  }
45 }
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:55
void close(FILE *f)
Closes the file pointed to by the FILE pointer.
Definition: File.cpp:102

+ Here is the call graph for this function:

Member Function Documentation

size_t File_Namespace::FileInfo::available ( )
inline

Returns the number of free bytes available.

Definition at line 105 of file FileInfo.h.

References freePages, and pageSize.

Referenced by print(), and used().

105 { return freePages.size() * pageSize; }
std::set< size_t > freePages
the number of pages in the file
Definition: FileInfo.h:60
size_t pageSize
file stream object for the represented file
Definition: FileInfo.h:56

+ Here is the caller graph for this function:

void File_Namespace::FileInfo::freePage ( int  pageId)

Definition at line 202 of file FileInfo.cpp.

References CHECK(), DELETE_CONTINGENT, File_Namespace::FileMgr::epoch(), f, fileMgr, File_Namespace::FileMgr::free_page(), freePages, freePagesMutex_, pageSize, and File_Namespace::write().

Referenced by File_Namespace::FileBuffer::freePages().

202  {
203 #define RESILIENT_PAGE_HEADER
204 #ifdef RESILIENT_PAGE_HEADER
205  int epoch_freed_page[2] = {DELETE_CONTINGENT, fileMgr->epoch()};
207  pageId * pageSize + sizeof(int),
208  sizeof(epoch_freed_page),
209  (int8_t*)epoch_freed_page);
210  fileMgr->free_page(std::make_pair(this, pageId));
211 #else
212  int zeroVal = 0;
213  int8_t* zeroAddr = reinterpret_cast<int8_t*>(&zeroVal);
214  File_Namespace::write(f, pageId * pageSize, sizeof(int), zeroAddr);
215  std::lock_guard<std::mutex> lock(freePagesMutex_);
216  freePages.insert(pageId);
217 #endif // RESILIENT_PAGE_HEADER
218 
219 #ifdef ENABLE_CRASH_CORRUPTION_TEST
220  signal(SIGUSR2, sighandler);
221  if (goto_crash)
222  CHECK(pageId % 8 != 4);
223 #endif
224 }
int epoch()
Returns current value of epoch - should be one greater than recorded at last checkpoint.
Definition: FileMgr.h:204
void free_page(std::pair< FileInfo *, int > &&page)
Definition: FileMgr.cpp:997
std::set< size_t > freePages
the number of pages in the file
Definition: FileInfo.h:60
size_t pageSize
file stream object for the represented file
Definition: FileInfo.h:56
CHECK(cgen_state)
std::mutex freePagesMutex_
set of page numbers of free pages
Definition: FileInfo.h:61
#define DELETE_CONTINGENT
A FileInfo type has a file pointer and metadata about a file.
Definition: FileInfo.h:49
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:55
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:121

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileInfo::freePageDeferred ( int  pageId)

Definition at line 187 of file FileInfo.cpp.

References freePages, and freePagesMutex_.

187  {
188  std::lock_guard<std::mutex> lock(freePagesMutex_);
189  freePages.insert(pageId);
190 }
std::set< size_t > freePages
the number of pages in the file
Definition: FileInfo.h:60
std::mutex freePagesMutex_
set of page numbers of free pages
Definition: FileInfo.h:61
int File_Namespace::FileInfo::getFreePage ( )

Definition at line 226 of file FileInfo.cpp.

References freePages, and freePagesMutex_.

Referenced by File_Namespace::FileMgr::requestFreePage(), and File_Namespace::FileMgr::requestFreePages().

226  {
227  // returns -1 if there is no free page
228  std::lock_guard<std::mutex> lock(freePagesMutex_);
229  if (freePages.size() == 0) {
230  return -1;
231  }
232  auto pageIt = freePages.begin();
233  int pageNum = *pageIt;
234  freePages.erase(pageIt);
235  return pageNum;
236 }
std::set< size_t > freePages
the number of pages in the file
Definition: FileInfo.h:60
std::mutex freePagesMutex_
set of page numbers of free pages
Definition: FileInfo.h:61

+ Here is the caller graph for this function:

void File_Namespace::FileInfo::initNewFile ( )

Adds all pages to freePages and zeroes first four bytes of header.

Definition at line 47 of file FileInfo.cpp.

References f, freePages, numPages, pageSize, and File_Namespace::write().

Referenced by FileInfo().

47  {
48  // initialize pages and free page list
49  // Also zeroes out first four bytes of every header
50 
51  int headerSize = 0;
52  int8_t* headerSizePtr = (int8_t*)(&headerSize);
53  for (size_t pageId = 0; pageId < numPages; ++pageId) {
54  File_Namespace::write(f, pageId * pageSize, sizeof(int), headerSizePtr);
55  freePages.insert(pageId);
56  }
57 }
std::set< size_t > freePages
the number of pages in the file
Definition: FileInfo.h:60
size_t pageSize
file stream object for the represented file
Definition: FileInfo.h:56
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:55
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:121
size_t numPages
the fixed size of each page in the file
Definition: FileInfo.h:57

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::FileInfo::numFreePages ( )
inline

Returns the number of free pages available.

Definition at line 108 of file FileInfo.h.

References freePages, and freePagesMutex_.

108  {
109  std::lock_guard<std::mutex> lock(freePagesMutex_);
110  return freePages.size();
111  }
std::set< size_t > freePages
the number of pages in the file
Definition: FileInfo.h:60
std::mutex freePagesMutex_
set of page numbers of free pages
Definition: FileInfo.h:61
void File_Namespace::FileInfo::openExistingFile ( std::vector< HeaderInfo > &  headerVec,
const int  fileMgrEpoch 
)

Definition at line 69 of file FileInfo.cpp.

References CHECK_EQ, CHECK_GE, DELETE_CONTINGENT, File_Namespace::FileMgr::epoch(), f, fileId, fileMgr, freePages, File_Namespace::FileMgr::get_fileMgrKey(), LOG, numPages, pageSize, showChunk(), VLOG, logger::WARNING, and File_Namespace::write().

Referenced by File_Namespace::FileMgr::openExistingFile().

70  {
71  // HeaderInfo is defined in Page.h
72  ChunkKey oldChunkKey(4);
73  int oldPageId = -99;
74  int oldVersionEpoch = -99;
75  int skipped = 0;
76  for (size_t pageNum = 0; pageNum < numPages; ++pageNum) {
77  int headerSize;
78 
79  constexpr size_t MAX_INTS_TO_READ{10}; // currently use 1+6 ints
80  int ints[MAX_INTS_TO_READ];
81  CHECK_EQ(fseek(f, pageNum * pageSize, SEEK_SET), 0);
82  CHECK_EQ(fread(ints, sizeof(int), MAX_INTS_TO_READ, f), MAX_INTS_TO_READ);
83 
84  headerSize = ints[0];
85  if (0 != headerSize) {
86  if (DELETE_CONTINGENT == ints[1]) {
87  if (fileMgr->epoch() > ints[2]) {
88  int zero{0};
89  File_Namespace::write(f, pageNum * pageSize, sizeof(int), (int8_t*)&zero);
90  headerSize = 0;
91  }
92  }
93  }
94 
95  if (headerSize != 0) {
96  // headerSize doesn't include headerSize itself
97  // We're tying ourself to headers of ints here
98  size_t numHeaderElems = headerSize / sizeof(int);
99  CHECK_GE(numHeaderElems, size_t(2));
100  // size_t chunkSize;
101  // We don't want to read headerSize in our header - so start
102  // reading 4 bytes past it
103 
104  // always derive dbid/tbid from FileMgr
105  ChunkKey chunkKey(&ints[1], &ints[1 + numHeaderElems - 2]);
106  chunkKey[0] = fileMgr->get_fileMgrKey().first;
107  chunkKey[1] = fileMgr->get_fileMgrKey().second;
108  // recover page in case a crash failed deletion of this page
109  if (DELETE_CONTINGENT == ints[1]) {
111  f, pageNum * pageSize + sizeof(int), 2 * sizeof(int), (int8_t*)&chunkKey[0]);
112  }
113 
114  // cout << "Chunk key: " << showChunk(chunkKey) << endl;
115  // Last two elements of header are always PageId and Version
116  // epoch - these are not in the chunk key so seperate them
117  int pageId = ints[1 + numHeaderElems - 2];
118  // cout << "Page id: " << pageId << endl;
119  int versionEpoch = ints[1 + numHeaderElems - 1];
120  if (chunkKey != oldChunkKey || oldPageId != pageId - (1 + skipped)) {
121  if (skipped > 0) {
122  VLOG(4) << "FId.PSz: " << fileId << "." << pageSize
123  << " Chunk key: " << showChunk(oldChunkKey)
124  << " Page id from : " << oldPageId << " to : " << oldPageId + skipped
125  << " Epoch: " << oldVersionEpoch;
126  } else if (oldPageId != -99) {
127  VLOG(4) << "FId.PSz: " << fileId << "." << pageSize
128  << " Chunk key: " << showChunk(oldChunkKey) << " Page id: " << oldPageId
129  << " Epoch: " << oldVersionEpoch;
130  }
131  oldPageId = pageId;
132  oldVersionEpoch = versionEpoch;
133  oldChunkKey = chunkKey;
134  skipped = 0;
135  } else {
136  skipped++;
137  }
138  // read(f,pageNum*pageSize+sizeof(int),headerSize-2*sizeof(int),(int8_t
139  // *)(&chunkKey[0])); read(f,pageNum*pageSize+sizeof(int) + headerSize -
140  // 2*sizeof(int),sizeof(int),(int8_t *)(&pageId));
141  // read(f,pageNum*pageSize+sizeof(int) + headerSize -
142  // sizeof(int),sizeof(int),(int8_t *)(&versionEpoch));
143  // read(f,pageNum*pageSize+sizeof(int) + headerSize -
144  // sizeof(size_t),sizeof(size_t),(int8_t *)(&chunkSize));
145 
146  /* Check if version epoch is equal to
147  * or greater (note: should never be greater)
148  * than FileMgr epoch_ - this means that this
149  * page wasn't checkpointed and thus we should
150  * not use it
151  */
152  if (versionEpoch >= fileMgrEpoch) {
153  // First write 0 to first four bytes of
154  // header to mark as free
155  headerSize = 0;
156  File_Namespace::write(f, pageNum * pageSize, sizeof(int), (int8_t*)&headerSize);
157  // Now add page to free list
158  freePages.insert(pageNum);
159  LOG(WARNING) << "Was not checkpointed: Chunk key: " << showChunk(chunkKey)
160  << " Page id: " << pageId << " Epoch: " << versionEpoch
161  << " FileMgrEpoch " << fileMgrEpoch << endl;
162 
163  } else { // page was checkpointed properly
164  Page page(fileId, pageNum);
165  headerVec.emplace_back(chunkKey, pageId, versionEpoch, page);
166  // std::cout << "Inserted into headerVec" << std::endl;
167  }
168  } else { // no header for this page - insert into free list
169  freePages.insert(pageNum);
170  }
171  }
172  // printlast
173  if (oldPageId != -99) {
174  if (skipped > 0) {
175  VLOG(4) << "FId.PSz: " << fileId << "." << pageSize
176  << " Chunk key: " << showChunk(oldChunkKey)
177  << " Page id from : " << oldPageId << " to : " << oldPageId + skipped
178  << " Epoch: " << oldVersionEpoch;
179  } else {
180  VLOG(4) << "FId.PSz: " << fileId << "." << pageSize
181  << " Chunk key: " << showChunk(oldChunkKey) << " Page id: " << oldPageId
182  << " Epoch: " << oldVersionEpoch;
183  }
184  }
185 }
#define CHECK_EQ(x, y)
Definition: Logger.h:198
std::vector< int > ChunkKey
Definition: types.h:35
#define LOG(tag)
Definition: Logger.h:185
#define CHECK_GE(x, y)
Definition: Logger.h:203
int epoch()
Returns current value of epoch - should be one greater than recorded at last checkpoint.
Definition: FileMgr.h:204
std::set< size_t > freePages
the number of pages in the file
Definition: FileInfo.h:60
size_t pageSize
file stream object for the represented file
Definition: FileInfo.h:56
std::string showChunk(const ChunkKey &key)
Definition: types.h:37
#define DELETE_CONTINGENT
A FileInfo type has a file pointer and metadata about a file.
Definition: FileInfo.h:49
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:55
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:121
const std::pair< const int, const int > get_fileMgrKey() const
Definition: FileMgr.h:234
size_t numPages
the fixed size of each page in the file
Definition: FileInfo.h:57
#define VLOG(n)
Definition: Logger.h:280

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileInfo::print ( bool  pagesummary)

Prints a summary of the file to stdout.

Definition at line 238 of file FileInfo.cpp.

References available(), fileId, size(), and used().

238  {
239  std::cout << "File: " << fileId << std::endl;
240  std::cout << "Size: " << size() << std::endl;
241  std::cout << "Used: " << used() << std::endl;
242  std::cout << "Free: " << available() << std::endl;
243  if (!pagesummary) {
244  return;
245  }
246 
247  // for (size_t i = 0; i < pages.size(); ++i) {
248  // // @todo page summary
249  //}
250 }
size_t size()
Returns the number of bytes used by the file.
Definition: FileInfo.h:90
size_t used()
Returns the amount of used bytes; size() - available()
Definition: FileInfo.h:114
size_t available()
Returns the number of free bytes available.
Definition: FileInfo.h:105

+ Here is the call graph for this function:

size_t File_Namespace::FileInfo::read ( const size_t  offset,
const size_t  size,
int8_t *  buf 
)

Definition at line 64 of file FileInfo.cpp.

References f, File_Namespace::read(), and readWriteMutex_.

Referenced by File_Namespace::FileBuffer::copyPage(), File_Namespace::FileMgr::copyPage(), and File_Namespace::readForThread().

64  {
65  std::lock_guard<std::mutex> lock(readWriteMutex_);
66  return File_Namespace::read(f, offset, size, buf);
67 }
std::mutex readWriteMutex_
Definition: FileInfo.h:62
size_t size()
Returns the number of bytes used by the file.
Definition: FileInfo.h:90
size_t read(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Reads the specified number of bytes from the offset position in file f into buf.
Definition: File.cpp:113
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:55

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::FileInfo::size ( )
inline

Returns the number of bytes used by the file.

Definition at line 90 of file FileInfo.h.

References numPages, and pageSize.

Referenced by print(), and used().

90 { return pageSize * numPages; }
size_t pageSize
file stream object for the represented file
Definition: FileInfo.h:56
size_t numPages
the fixed size of each page in the file
Definition: FileInfo.h:57

+ Here is the caller graph for this function:

int File_Namespace::FileInfo::syncToDisk ( )
inline

Definition at line 92 of file FileInfo.h.

References f, logger::FATAL, and LOG.

92  {
93  if (fflush(f) != 0) {
94  LOG(FATAL) << "Error trying to flush changes to disk, the error was: "
95  << std::strerror(errno);
96  }
97 #ifdef __APPLE__
98  return fcntl(fileno(f), 51);
99 #else
100  return fsync(fileno(f));
101 #endif
102  }
#define LOG(tag)
Definition: Logger.h:185
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:55
size_t File_Namespace::FileInfo::used ( )
inline

Returns the amount of used bytes; size() - available()

Definition at line 114 of file FileInfo.h.

References available(), and size().

Referenced by print().

114 { return size() - available(); }
size_t size()
Returns the number of bytes used by the file.
Definition: FileInfo.h:90
size_t available()
Returns the number of free bytes available.
Definition: FileInfo.h:105

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::FileInfo::write ( const size_t  offset,
const size_t  size,
int8_t *  buf 
)

Definition at line 59 of file FileInfo.cpp.

References f, readWriteMutex_, and File_Namespace::write().

Referenced by File_Namespace::FileBuffer::append(), File_Namespace::FileBuffer::copyPage(), File_Namespace::FileMgr::copyPage(), File_Namespace::FileBuffer::write(), and File_Namespace::FileBuffer::writeHeader().

59  {
60  std::lock_guard<std::mutex> lock(readWriteMutex_);
61  return File_Namespace::write(f, offset, size, buf);
62 }
std::mutex readWriteMutex_
Definition: FileInfo.h:62
size_t size()
Returns the number of bytes used by the file.
Definition: FileInfo.h:90
FILE * f
unique file identifier (i.e., used for a file name)
Definition: FileInfo.h:55
size_t write(FILE *f, const size_t offset, const size_t size, int8_t *buf)
Writes the specified number of bytes to the offset position in file f from buf.
Definition: File.cpp:121

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

Member Data Documentation

FILE* File_Namespace::FileInfo::f

unique file identifier (i.e., used for a file name)

Definition at line 55 of file FileInfo.h.

Referenced by freePage(), initNewFile(), openExistingFile(), read(), syncToDisk(), write(), and ~FileInfo().

int File_Namespace::FileInfo::fileId
FileMgr* File_Namespace::FileInfo::fileMgr

Definition at line 53 of file FileInfo.h.

Referenced by freePage(), and openExistingFile().

std::set<size_t> File_Namespace::FileInfo::freePages

the number of pages in the file

Definition at line 60 of file FileInfo.h.

Referenced by available(), freePage(), freePageDeferred(), getFreePage(), initNewFile(), numFreePages(), and openExistingFile().

std::mutex File_Namespace::FileInfo::freePagesMutex_

set of page numbers of free pages

Definition at line 61 of file FileInfo.h.

Referenced by freePage(), freePageDeferred(), getFreePage(), and numFreePages().

size_t File_Namespace::FileInfo::numPages

the fixed size of each page in the file

Definition at line 57 of file FileInfo.h.

Referenced by initNewFile(), openExistingFile(), and size().

size_t File_Namespace::FileInfo::pageSize

file stream object for the represented file

Definition at line 56 of file FileInfo.h.

Referenced by available(), freePage(), initNewFile(), openExistingFile(), and size().

std::mutex File_Namespace::FileInfo::readWriteMutex_

Definition at line 62 of file FileInfo.h.

Referenced by read(), and write().


The documentation for this struct was generated from the following files: