OmniSciDB  fe05a0c208
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
File_Namespace::FileBuffer Class Reference

Represents/provides access to contiguous data stored in the file system. More...

#include <FileBuffer.h>

+ Inheritance diagram for File_Namespace::FileBuffer:
+ Collaboration diagram for File_Namespace::FileBuffer:

Public Member Functions

 FileBuffer (FileMgr *fm, const size_t pageSize, const ChunkKey &chunkKey, const size_t initialSize=0)
 Constructs a FileBuffer object. More...
 
 FileBuffer (FileMgr *fm, const size_t pageSize, const ChunkKey &chunkKey, const SQLTypeInfo sqlType, const size_t initialSize=0)
 
 FileBuffer (FileMgr *fm, const ChunkKey &chunkKey, const std::vector< HeaderInfo >::const_iterator &headerStartIt, const std::vector< HeaderInfo >::const_iterator &headerEndIt)
 
virtual ~FileBuffer () override
 Destructor. More...
 
Page addNewMultiPage (const int32_t epoch)
 
void reserve (const size_t numBytes) override
 
void freeMetadataPages ()
 
size_t freeChunkPages ()
 
void freePages ()
 
void freePagesBeforeEpoch (const int32_t targetEpoch)
 
void read (int8_t *const dst, const size_t numBytes=0, const size_t offset=0, const MemoryLevel dstMemoryLevel=CPU_LEVEL, const int32_t deviceId=-1) override
 
void write (int8_t *src, const size_t numBytes, const size_t offset=0, const MemoryLevel srcMemoryLevel=CPU_LEVEL, const int32_t deviceId=-1) override
 Writes the contents of source (src) into new versions of the affected logical pages. More...
 
void append (int8_t *src, const size_t numBytes, const MemoryLevel srcMemoryLevel=CPU_LEVEL, const int32_t deviceId=-1) override
 
void copyPage (Page &srcPage, Page &destPage, const size_t numBytes, const size_t offset=0)
 
Data_Namespace::MemoryLevel getType () const override
 
int8_t * getMemoryPtr () override
 Not implemented for FileMgr – throws a runtime_error. More...
 
size_t pageCount () const override
 Returns the number of pages in the FileBuffer. More...
 
size_t pageSize () const override
 Returns the size in bytes of each page in the FileBuffer. More...
 
virtual size_t pageDataSize () const
 Returns the size in bytes of the data portion of each page in the FileBuffer. More...
 
virtual size_t reservedHeaderSize () const
 
virtual std::vector< MultiPagegetMultiPage () const
 Returns vector of MultiPages in the FileBuffer. More...
 
MultiPage getMetadataPage () const
 
size_t reservedSize () const override
 Returns the total number of bytes allocated for the FileBuffer. More...
 
size_t numMetadataPages () const
 Returns the total number of used bytes in the FileBuffer. More...
 
size_t numChunkPages () const
 
std::string dump () const
 
bool isMissingPages () const
 
void freePage (const Page &page)
 
- Public Member Functions inherited from Data_Namespace::AbstractBuffer
 AbstractBuffer (const int device_id)
 
 AbstractBuffer (const int device_id, const SQLTypeInfo sql_type)
 
virtual ~AbstractBuffer ()
 
virtual void read (int8_t *const dst, const size_t num_bytes, const size_t offset=0, const MemoryLevel dst_buffer_type=CPU_LEVEL, const int dst_device_id=-1)=0
 
virtual void write (int8_t *src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int src_device_id=-1)=0
 
virtual void append (int8_t *src, const size_t num_bytes, const MemoryLevel src_buffer_type=CPU_LEVEL, const int device_id=-1)=0
 
virtual void setMemoryPtr (int8_t *new_ptr)
 
virtual int pin ()
 
virtual int unPin ()
 
virtual int getPinCount ()
 
size_t size () const
 
int getDeviceId () const
 
bool isDirty () const
 
bool isAppended () const
 
bool isUpdated () const
 
bool hasEncoder () const
 
SQLTypeInfo getSqlType () const
 
void setSqlType (const SQLTypeInfo &sql_type)
 
EncodergetEncoder () const
 
void setDirty ()
 
void setUpdated ()
 
void setAppended ()
 
void setSize (const size_t size)
 
void clearDirtyBits ()
 
void initEncoder (const SQLTypeInfo &tmp_sql_type)
 
void syncEncoder (const AbstractBuffer *src_buffer)
 
void copyTo (AbstractBuffer *destination_buffer, const size_t num_bytes=0)
 
void resetToEmpty ()
 

Static Public Attributes

static constexpr size_t headerBufferOffset_ = 32
 

Private Member Functions

void writeHeader (Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
 Write header writes header at top of page in format. More...
 
void writeMetadata (const int32_t epoch)
 
void readMetadata (const Page &page)
 
void calcHeaderBuffer ()
 
void freePage (const Page &page, const bool isRolloff)
 
void freePagesBeforeEpochForMultiPage (MultiPage &multiPage, const int32_t targetEpoch, const int32_t currentEpoch)
 
void initMetadataAndPageDataSize ()
 
int32_t getFileMgrEpoch ()
 

Private Attributes

FileMgrfm_
 
MultiPage metadataPages_
 
std::vector< MultiPagemultiPages_
 
size_t pageSize_
 
size_t pageDataSize_
 
size_t reservedHeaderSize_
 
ChunkKey chunkKey_
 

Friends

class FileMgr
 
class CachingFileMgr
 

Additional Inherited Members

- Protected Attributes inherited from Data_Namespace::AbstractBuffer
std::unique_ptr< Encoderencoder_
 
SQLTypeInfo sql_type_
 
size_t size_
 
int device_id_
 

Detailed Description

Represents/provides access to contiguous data stored in the file system.

The FileBuffer consists of logical pages, which can map to any identically-sized page in any file of the underlying file system. A page's metadata (file and page number) are stored in MultiPage objects, and each MultiPage includes page metadata for multiple versions of the same page.

Note that a "Chunk" is brought into a FileBuffer by the FileMgr.

Note(s): Forbid Copying Idiom 4.1

Definition at line 58 of file FileBuffer.h.

Constructor & Destructor Documentation

File_Namespace::FileBuffer::FileBuffer ( FileMgr fm,
const size_t  pageSize,
const ChunkKey chunkKey,
const size_t  initialSize = 0 
)

Constructs a FileBuffer object.

Definition at line 38 of file FileBuffer.cpp.

References calcHeaderBuffer(), CHECK, fm_, pageDataSize_, pageSize_, and reservedHeaderSize_.

42  : AbstractBuffer(fm->getDeviceId())
43  , fm_(fm)
46  , chunkKey_(chunkKey) {
47  // Create a new FileBuffer
48  CHECK(fm_);
52  //@todo reintroduce initialSize - need to develop easy way of
53  // differentiating these pre-allocated pages from "written-to" pages
54  /*
55  if (initalSize > 0) {
56  // should expand to initialSize bytes
57  size_t initialNumPages = (initalSize + pageSize_ -1) / pageSize_;
58  int32_t epoch = fm_->epoch();
59  for (size_t pageNum = 0; pageNum < initialNumPages; ++pageNum) {
60  Page page = addNewMultiPage(epoch);
61  writeHeader(page,pageNum,epoch);
62  }
63  }
64  */
65 }
#define METADATA_PAGE_SIZE
Definition: FileBuffer.h:37
AbstractBuffer(const int device_id)
size_t pageSize() const override
Returns the size in bytes of each page in the FileBuffer.
Definition: FileBuffer.h:134
#define CHECK(condition)
Definition: Logger.h:203

+ Here is the call graph for this function:

File_Namespace::FileBuffer::FileBuffer ( FileMgr fm,
const size_t  pageSize,
const ChunkKey chunkKey,
const SQLTypeInfo  sqlType,
const size_t  initialSize = 0 
)

Definition at line 67 of file FileBuffer.cpp.

References calcHeaderBuffer(), CHECK, fm_, pageDataSize_, pageSize_, and reservedHeaderSize_.

72  : AbstractBuffer(fm->getDeviceId(), sqlType)
73  , fm_(fm)
76  , chunkKey_(chunkKey) {
77  CHECK(fm_);
80 }
#define METADATA_PAGE_SIZE
Definition: FileBuffer.h:37
AbstractBuffer(const int device_id)
size_t pageSize() const override
Returns the size in bytes of each page in the FileBuffer.
Definition: FileBuffer.h:134
#define CHECK(condition)
Definition: Logger.h:203

+ Here is the call graph for this function:

File_Namespace::FileBuffer::FileBuffer ( FileMgr fm,
const ChunkKey chunkKey,
const std::vector< HeaderInfo >::const_iterator &  headerStartIt,
const std::vector< HeaderInfo >::const_iterator &  headerEndIt 
)

Definition at line 82 of file FileBuffer.cpp.

References calcHeaderBuffer(), CHECK, File_Namespace::FileMgr::failOnReadError(), logger::FATAL, fm_, initMetadataAndPageDataSize(), LOG, metadataPages_, multiPages_, pageSize_, File_Namespace::MultiPage::push(), and show_chunk().

86  : AbstractBuffer(fm->getDeviceId())
87  , fm_(fm)
89  , pageSize_(0)
90  , chunkKey_(chunkKey) {
91  // We are being assigned an existing FileBuffer on disk
92 
93  CHECK(fm_);
95  int32_t lastPageId = -1;
96  int32_t curPageId = 0;
97  for (auto vecIt = headerStartIt; vecIt != headerEndIt; ++vecIt) {
98  curPageId = vecIt->pageId;
99 
100  // We only want to read last metadata page
101  if (curPageId == -1) { // stats page
102  metadataPages_.push(vecIt->page, vecIt->versionEpoch);
103  } else {
104  if (curPageId != lastPageId) {
105  // protect from bad data on disk, and give diagnostics
106  if (fm->failOnReadError()) {
107  if (curPageId != lastPageId + 1) {
108  LOG(FATAL) << "Failure reading DB file " << show_chunk(chunkKey)
109  << " Current page " << curPageId << " last page " << lastPageId
110  << " epoch " << vecIt->versionEpoch;
111  }
112  }
113  if (lastPageId == -1) { // If we are on first real page
115  }
116  MultiPage multiPage(pageSize_);
117  multiPages_.push_back(multiPage);
118  lastPageId = curPageId;
119  }
120  multiPages_.back().push(vecIt->page, vecIt->versionEpoch);
121  }
122  }
123  if (curPageId == -1) { // meaning there was only a metadata page
125  }
126 }
#define METADATA_PAGE_SIZE
Definition: FileBuffer.h:37
#define LOG(tag)
Definition: Logger.h:194
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197
std::string show_chunk(const ChunkKey &key)
Definition: types.h:85
AbstractBuffer(const int device_id)
void push(const Page &page, const int epoch)
Pushes a new page with epoch value.
Definition: Page.h:102
#define CHECK(condition)
Definition: Logger.h:203

+ Here is the call graph for this function:

File_Namespace::FileBuffer::~FileBuffer ( )
overridevirtual

Destructor.

Definition at line 128 of file FileBuffer.cpp.

128  {
129  // need to free pages
130  // NOP
131 }

Member Function Documentation

Page File_Namespace::FileBuffer::addNewMultiPage ( const int32_t  epoch)

Definition at line 405 of file FileBuffer.cpp.

References fm_, multiPages_, pageSize_, File_Namespace::MultiPage::push(), and File_Namespace::FileMgr::requestFreePage().

Referenced by append(), reserve(), and write().

405  {
406  Page page = fm_->requestFreePage(pageSize_, false);
407  MultiPage multiPage(pageSize_);
408  multiPage.push(page, epoch);
409  multiPages_.emplace_back(multiPage);
410  return page;
411 }
Page requestFreePage(size_t pagesize, const bool isMetadata)
Definition: FileMgr.cpp:861
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::append ( int8_t *  src,
const size_t  numBytes,
const MemoryLevel  srcMemoryLevel = CPU_LEVEL,
const int32_t  deviceId = -1 
)
override

Definition at line 489 of file FileBuffer.cpp.

References addNewMultiPage(), CHECK, File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), getFileMgrEpoch(), multiPages_, pageDataSize_, File_Namespace::Page::pageNum, pageSize_, reservedHeaderSize_, Data_Namespace::AbstractBuffer::setAppended(), Data_Namespace::AbstractBuffer::size_, File_Namespace::FileInfo::write(), and writeHeader().

492  {
493  setAppended();
494 
495  size_t startPage = size_ / pageDataSize_;
496  size_t startPageOffset = size_ % pageDataSize_;
497  size_t numPagesToWrite =
498  (numBytes + startPageOffset + pageDataSize_ - 1) / pageDataSize_;
499  size_t bytesLeft = numBytes;
500  int8_t* curPtr = src; // a pointer to the current location in dst being written to
501  size_t initialNumPages = multiPages_.size();
502  size_ = size_ + numBytes;
503  auto epoch = getFileMgrEpoch();
504  for (size_t pageNum = startPage; pageNum < startPage + numPagesToWrite; ++pageNum) {
505  Page page;
506  if (pageNum >= initialNumPages) {
507  page = addNewMultiPage(epoch);
508  writeHeader(page, pageNum, epoch);
509  } else {
510  // we already have a new page at current
511  // epoch for this page - just grab this page
512  page = multiPages_[pageNum].current().page;
513  }
514  CHECK(page.fileId >= 0); // make sure page was initialized
515  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
516  size_t bytesWritten;
517  if (pageNum == startPage) {
518  bytesWritten = fileInfo->write(
519  page.pageNum * pageSize_ + startPageOffset + reservedHeaderSize_,
520  min(pageDataSize_ - startPageOffset, bytesLeft),
521  curPtr);
522  } else {
523  bytesWritten = fileInfo->write(page.pageNum * pageSize_ + reservedHeaderSize_,
524  min(pageDataSize_, bytesLeft),
525  curPtr);
526  }
527  curPtr += bytesWritten;
528  bytesLeft -= bytesWritten;
529  }
530  CHECK(bytesLeft == 0);
531 }
Page addNewMultiPage(const int32_t epoch)
Definition: FileBuffer.cpp:405
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197
size_t write(const size_t offset, const size_t size, int8_t *buf)
Definition: FileInfo.cpp:60
FileInfo * getFileInfoForFileId(const int32_t fileId)
Definition: FileMgr.h:220
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:413
#define CHECK(condition)
Definition: Logger.h:203

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::calcHeaderBuffer ( )
private

Definition at line 143 of file FileBuffer.cpp.

References chunkKey_, headerBufferOffset_, and reservedHeaderSize_.

Referenced by FileBuffer().

143  {
144  // 3 * sizeof(int32_t) is for headerSize, for pageId and versionEpoch
145  // sizeof(size_t) is for chunkSize
146  reservedHeaderSize_ = (chunkKey_.size() + 3) * sizeof(int32_t);
147  size_t headerMod = reservedHeaderSize_ % headerBufferOffset_;
148  if (headerMod > 0) {
150  }
151 }
static constexpr size_t headerBufferOffset_
Definition: FileBuffer.h:169

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::copyPage ( Page srcPage,
Page destPage,
const size_t  numBytes,
const size_t  offset = 0 
)

Definition at line 385 of file FileBuffer.cpp.

References CHECK, CHECK_LE, checked_malloc(), File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), pageDataSize_, File_Namespace::Page::pageNum, pageSize_, File_Namespace::FileInfo::read(), reservedHeaderSize_, and File_Namespace::FileInfo::write().

Referenced by write().

388  {
389  // FILE *srcFile = fm_->files_[srcPage.fileId]->f;
390  // FILE *destFile = fm_->files_[destPage.fileId]->f;
391  CHECK_LE(offset + numBytes, pageDataSize_);
392  FileInfo* srcFileInfo = fm_->getFileInfoForFileId(srcPage.fileId);
393  FileInfo* destFileInfo = fm_->getFileInfoForFileId(destPage.fileId);
394 
395  int8_t* buffer = reinterpret_cast<int8_t*>(checked_malloc(numBytes));
396  size_t bytesRead = srcFileInfo->read(
397  srcPage.pageNum * pageSize_ + offset + reservedHeaderSize_, numBytes, buffer);
398  CHECK(bytesRead == numBytes);
399  size_t bytesWritten = destFileInfo->write(
400  destPage.pageNum * pageSize_ + offset + reservedHeaderSize_, numBytes, buffer);
401  CHECK(bytesWritten == numBytes);
402  free(buffer);
403 }
void * checked_malloc(const size_t size)
Definition: checked_alloc.h:45
FileInfo * getFileInfoForFileId(const int32_t fileId)
Definition: FileMgr.h:220
#define CHECK_LE(x, y)
Definition: Logger.h:214
size_t read(const size_t offset, const size_t size, int8_t *buf)
Definition: FileInfo.cpp:66
#define CHECK(condition)
Definition: Logger.h:203

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string File_Namespace::FileBuffer::dump ( ) const

Definition at line 628 of file FileBuffer.cpp.

References chunkKey_, Data_Namespace::AbstractBuffer::hasEncoder(), show_chunk(), and Data_Namespace::AbstractBuffer::size_.

628  {
629  std::stringstream ss;
630  ss << "chunk_key = " << show_chunk(chunkKey_) << "\n";
631  ss << "has_encoder = " << (hasEncoder() ? "true\n" : "false\n");
632  ss << "size_ = " << size_ << "\n";
633  return ss.str();
634 }
std::string show_chunk(const ChunkKey &key)
Definition: types.h:85

+ Here is the call graph for this function:

size_t File_Namespace::FileBuffer::freeChunkPages ( )

Definition at line 173 of file FileBuffer.cpp.

References freePage(), and multiPages_.

Referenced by foreign_storage::ForeignStorageCache::eraseChunk(), foreign_storage::ForeignStorageCache::evictChunkByIterator(), freePages(), and freePagesBeforeEpoch().

173  {
174  size_t num_pages_freed = multiPages_.size();
175  for (auto multiPageIt = multiPages_.begin(); multiPageIt != multiPages_.end();
176  ++multiPageIt) {
177  for (auto pageIt = multiPageIt->pageVersions.begin();
178  pageIt != multiPageIt->pageVersions.end();
179  ++pageIt) {
180  freePage(pageIt->page, false /* isRolloff */);
181  }
182  }
183  multiPages_.clear();
184  return num_pages_freed;
185 }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197
void freePage(const Page &page)
Definition: FileBuffer.cpp:153

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::freeMetadataPages ( )

Definition at line 162 of file FileBuffer.cpp.

References freePage(), metadataPages_, File_Namespace::MultiPage::pageVersions, and File_Namespace::MultiPage::pop().

Referenced by freePages().

162  {
163  for (auto metaPageIt = metadataPages_.pageVersions.begin();
164  metaPageIt != metadataPages_.pageVersions.end();
165  ++metaPageIt) {
166  freePage(metaPageIt->page, false /* isRolloff */);
167  }
168  while (metadataPages_.pageVersions.size() > 0) {
170  }
171 }
void pop()
Purges the oldest Page.
Definition: Page.h:110
std::deque< EpochedPage > pageVersions
Definition: Page.h:81
void freePage(const Page &page)
Definition: FileBuffer.cpp:153

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::freePage ( const Page page)

Definition at line 153 of file FileBuffer.cpp.

Referenced by freeChunkPages(), freeMetadataPages(), and freePagesBeforeEpochForMultiPage().

153  {
154  freePage(page, false);
155 }
void freePage(const Page &page)
Definition: FileBuffer.cpp:153

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::freePage ( const Page page,
const bool  isRolloff 
)
private

Definition at line 157 of file FileBuffer.cpp.

References File_Namespace::Page::fileId, fm_, File_Namespace::FileInfo::freePage(), File_Namespace::FileMgr::getFileInfoForFileId(), getFileMgrEpoch(), and File_Namespace::Page::pageNum.

157  {
158  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
159  fileInfo->freePage(page.pageNum, isRolloff, getFileMgrEpoch());
160 }
void freePage(int32_t pageId, const bool isRolloff, int32_t epoch)
Definition: FileInfo.cpp:178
FileInfo * getFileInfoForFileId(const int32_t fileId)
Definition: FileMgr.h:220

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::freePages ( )

Definition at line 187 of file FileBuffer.cpp.

References freeChunkPages(), and freeMetadataPages().

187  {
189  freeChunkPages();
190 }

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::freePagesBeforeEpoch ( const int32_t  targetEpoch)

Definition at line 202 of file FileBuffer.cpp.

References CHECK_LE, chunkKey_, fm_, freeChunkPages(), freePagesBeforeEpochForMultiPage(), getFileMgrEpoch(), metadataPages_, multiPages_, pageSize_, File_Namespace::MultiPage::pageVersions, and Data_Namespace::AbstractBuffer::size_.

202  {
203  // This method is only safe to be called within a checkpoint, after the sync and epoch
204  // increment where a failure at any point32_t in the process would lead to a safe
205  // rollback
206  auto currentEpoch = getFileMgrEpoch();
207  CHECK_LE(targetEpoch, currentEpoch);
208  freePagesBeforeEpochForMultiPage(metadataPages_, targetEpoch, currentEpoch);
209  for (auto& multiPage : multiPages_) {
210  freePagesBeforeEpochForMultiPage(multiPage, targetEpoch, currentEpoch);
211  }
212 
213  // Check if all buffer pages can be freed
214  if (size_ == 0) {
215  size_t max_historical_buffer_size{0};
216  for (auto& epoch_page : metadataPages_.pageVersions) {
217  // Create buffer that is used to get the buffer size at the epoch version
218  FileBuffer buffer{fm_, pageSize_, chunkKey_};
219  buffer.readMetadata(epoch_page.page);
220  max_historical_buffer_size = std::max(max_historical_buffer_size, buffer.size());
221  }
222 
223  // Free all chunk pages, if none of the old chunk versions has any data
224  if (max_historical_buffer_size == 0) {
225  freeChunkPages();
226  }
227  }
228 }
void freePagesBeforeEpochForMultiPage(MultiPage &multiPage, const int32_t targetEpoch, const int32_t currentEpoch)
Definition: FileBuffer.cpp:192
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197
std::deque< EpochedPage > pageVersions
Definition: Page.h:81
#define CHECK_LE(x, y)
Definition: Logger.h:214
FileBuffer(FileMgr *fm, const size_t pageSize, const ChunkKey &chunkKey, const size_t initialSize=0)
Constructs a FileBuffer object.
Definition: FileBuffer.cpp:38

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::freePagesBeforeEpochForMultiPage ( MultiPage multiPage,
const int32_t  targetEpoch,
const int32_t  currentEpoch 
)
private

Definition at line 192 of file FileBuffer.cpp.

References freePage(), and File_Namespace::MultiPage::freePagesBeforeEpoch().

Referenced by freePagesBeforeEpoch().

194  {
195  std::vector<EpochedPage> epochedPagesToFree =
196  multiPage.freePagesBeforeEpoch(targetEpoch, currentEpoch);
197  for (const auto& epochedPageToFree : epochedPagesToFree) {
198  freePage(epochedPageToFree.page, true /* isRolloff */);
199  }
200 }
void freePage(const Page &page)
Definition: FileBuffer.cpp:153

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

int32_t File_Namespace::FileBuffer::getFileMgrEpoch ( )
private

Definition at line 623 of file FileBuffer.cpp.

References chunkKey_, File_Namespace::FileMgr::epoch(), fm_, and get_table_prefix().

Referenced by append(), freePage(), freePagesBeforeEpoch(), reserve(), and write().

623  {
624  auto [db_id, tb_id] = get_table_prefix(chunkKey_);
625  return fm_->epoch(db_id, tb_id);
626 }
virtual int32_t epoch(int32_t db_id, int32_t tb_id) const
Returns current value of epoch - should be one greater than recorded at last checkpoint. Because FileMgr only contains buffers from one table we can just return the FileMgr&#39;s epoch instead of finding a table-specific epoch.
Definition: FileMgr.h:271
std::pair< int, int > get_table_prefix(const ChunkKey &key)
Definition: types.h:57

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

int8_t* File_Namespace::FileBuffer::getMemoryPtr ( )
inlineoverridevirtual

Not implemented for FileMgr – throws a runtime_error.

Implements Data_Namespace::AbstractBuffer.

Definition at line 125 of file FileBuffer.h.

References logger::FATAL, and LOG.

125  {
126  LOG(FATAL) << "Operation not supported.";
127  return nullptr; // satisfy return-type warning
128  }
#define LOG(tag)
Definition: Logger.h:194
MultiPage File_Namespace::FileBuffer::getMetadataPage ( ) const
inline

Definition at line 145 of file FileBuffer.h.

145 { return metadataPages_; }
virtual std::vector<MultiPage> File_Namespace::FileBuffer::getMultiPage ( ) const
inlinevirtual

Returns vector of MultiPages in the FileBuffer.

Definition at line 144 of file FileBuffer.h.

Referenced by File_Namespace::FileMgr::init(), and read().

144 { return multiPages_; }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197

+ Here is the caller graph for this function:

Data_Namespace::MemoryLevel File_Namespace::FileBuffer::getType ( ) const
inlineoverridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 122 of file FileBuffer.h.

References Data_Namespace::DISK_LEVEL.

void File_Namespace::FileBuffer::initMetadataAndPageDataSize ( )
private

Definition at line 636 of file FileBuffer.cpp.

References CHECK, File_Namespace::MultiPage::current(), File_Namespace::Page::fileId, metadataPages_, File_Namespace::EpochedPage::page, pageDataSize_, pageSize_, readMetadata(), and reservedHeaderSize_.

Referenced by FileBuffer().

636  {
637  CHECK(metadataPages_.current().page.fileId != -1); // was initialized
640 }
int32_t fileId
Definition: Page.h:47
#define CHECK(condition)
Definition: Logger.h:203
void readMetadata(const Page &page)
Definition: FileBuffer.cpp:432
EpochedPage current() const
Returns a reference to the most recent version of the page.
Definition: Page.h:94

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

bool File_Namespace::FileBuffer::isMissingPages ( ) const

Definition at line 642 of file FileBuffer.cpp.

References multiPages_, pageDataSize_, and Data_Namespace::AbstractBuffer::size().

642  {
643  // Detect the case where a page is missing by comparing the amount of pages read
644  // with the metadata size.
645  return ((size() + pageDataSize_ - 1) / pageDataSize_ != multiPages_.size());
646 }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197

+ Here is the call graph for this function:

size_t File_Namespace::FileBuffer::numChunkPages ( ) const
inline

Definition at line 154 of file FileBuffer.h.

154  {
155  size_t total_size = 0;
156  for (const auto& multi_page : multiPages_) {
157  total_size += multi_page.pageVersions.size();
158  }
159  return total_size;
160  }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197
size_t File_Namespace::FileBuffer::numMetadataPages ( ) const
inline

Returns the total number of used bytes in the FileBuffer.

Definition at line 153 of file FileBuffer.h.

153 { return metadataPages_.pageVersions.size(); };
std::deque< EpochedPage > pageVersions
Definition: Page.h:81
size_t File_Namespace::FileBuffer::pageCount ( ) const
inlineoverridevirtual

Returns the number of pages in the FileBuffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 131 of file FileBuffer.h.

131 { return multiPages_.size(); }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197
virtual size_t File_Namespace::FileBuffer::pageDataSize ( ) const
inlinevirtual

Returns the size in bytes of the data portion of each page in the FileBuffer.

Definition at line 137 of file FileBuffer.h.

Referenced by File_Namespace::FileMgr::init(), and File_Namespace::readForThread().

137 { return pageDataSize_; }

+ Here is the caller graph for this function:

size_t File_Namespace::FileBuffer::pageSize ( ) const
inlineoverridevirtual

Returns the size in bytes of each page in the FileBuffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 134 of file FileBuffer.h.

Referenced by File_Namespace::FileMgr::init(), File_Namespace::readForThread(), and writeHeader().

134 { return pageSize_; }

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::read ( int8_t *const  dst,
const size_t  numBytes = 0,
const size_t  offset = 0,
const MemoryLevel  dstMemoryLevel = CPU_LEVEL,
const int32_t  deviceId = -1 
)
override

Definition at line 282 of file FileBuffer.cpp.

References CHECK, Data_Namespace::CPU_LEVEL, logger::FATAL, fm_, getMultiPage(), File_Namespace::FileMgr::getNumReaderThreads(), i, LOG, File_Namespace::readThreadDS::multiPages, multiPages_, pageDataSize_, File_Namespace::readForThread(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.

286  {
287  if (dstBufferType != CPU_LEVEL) {
288  LOG(FATAL) << "Unsupported Buffer type";
289  }
290 
291  // variable declarations
292  size_t startPage = offset / pageDataSize_;
293  size_t startPageOffset = offset % pageDataSize_;
294  size_t numPagesToRead =
295  (numBytes + startPageOffset + pageDataSize_ - 1) / pageDataSize_;
296  /*
297  if (startPage + numPagesToRead > multiPages_.size()) {
298  cout << "Start page: " << startPage << endl;
299  cout << "Num pages to read: " << numPagesToRead << endl;
300  cout << "Num multipages: " << multiPages_.size() << endl;
301  cout << "Offset: " << offset << endl;
302  cout << "Num bytes: " << numBytes << endl;
303  }
304  */
305 
306  CHECK(startPage + numPagesToRead <= multiPages_.size());
307 
308  size_t numPagesPerThread = 0;
309  size_t numBytesCurrent = numBytes; // total number of bytes still to be read
310  size_t bytesRead = 0; // total number of bytes already being read
311  size_t bytesLeftForThread = 0; // number of bytes to be read in the thread
312  size_t numExtraPages = 0; // extra pages to be assigned one per thread as needed
313  size_t numThreads = fm_->getNumReaderThreads();
314  std::vector<readThreadDS>
315  threadDSArr; // array of threadDS, needed to avoid racing conditions
316 
317  if (numPagesToRead > numThreads) {
318  numPagesPerThread = numPagesToRead / numThreads;
319  numExtraPages = numPagesToRead - (numThreads * numPagesPerThread);
320  } else {
321  numThreads = numPagesToRead;
322  numPagesPerThread = 1;
323  }
324 
325  /* set threadDS for the first thread */
326  readThreadDS threadDS;
327  threadDS.t_fm = fm_;
328  threadDS.t_startPage = offset / pageDataSize_;
329  if (numExtraPages > 0) {
330  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread + 1;
331  numExtraPages--;
332  } else {
333  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread;
334  }
335  threadDS.t_curPtr = dst;
336  threadDS.t_startPageOffset = offset % pageDataSize_;
337  threadDS.t_isFirstPage = true;
338 
339  bytesLeftForThread = min(((threadDS.t_endPage - threadDS.t_startPage) * pageDataSize_ -
340  threadDS.t_startPageOffset),
341  numBytesCurrent);
342  threadDS.t_bytesLeft = bytesLeftForThread;
343  threadDS.multiPages = getMultiPage();
344 
345  if (numThreads == 1) {
346  bytesRead += readForThread(this, threadDS);
347  } else {
348  std::vector<std::future<size_t>> threads;
349 
350  for (size_t i = 0; i < numThreads; i++) {
351  threadDSArr.push_back(threadDS);
352  threads.push_back(
353  std::async(std::launch::async, readForThread, this, threadDSArr[i]));
354 
355  // calculate elements of threadDS
356  threadDS.t_fm = fm_;
357  threadDS.t_isFirstPage = false;
358  threadDS.t_curPtr += bytesLeftForThread;
359  threadDS.t_startPage +=
360  threadDS.t_endPage -
361  threadDS.t_startPage; // based on # of pages read on previous iteration
362  if (numExtraPages > 0) {
363  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread + 1;
364  numExtraPages--;
365  } else {
366  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread;
367  }
368  numBytesCurrent -= bytesLeftForThread;
369  bytesLeftForThread = min(
370  ((threadDS.t_endPage - threadDS.t_startPage) * pageDataSize_), numBytesCurrent);
371  threadDS.t_bytesLeft = bytesLeftForThread;
372  threadDS.multiPages = getMultiPage();
373  }
374 
375  for (auto& p : threads) {
376  p.wait();
377  }
378  for (auto& p : threads) {
379  bytesRead += p.get();
380  }
381  }
382  CHECK(bytesRead == numBytes);
383 }
virtual std::vector< MultiPage > getMultiPage() const
Returns vector of MultiPages in the FileBuffer.
Definition: FileBuffer.h:144
static size_t readForThread(FileBuffer *fileBuffer, const readThreadDS threadDS)
Definition: FileBuffer.cpp:241
#define LOG(tag)
Definition: Logger.h:194
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197
#define CHECK(condition)
Definition: Logger.h:203
size_t getNumReaderThreads()
Returns number of threads defined by parameter num-reader-threads which should be used during initial...
Definition: FileMgr.h:304

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::readMetadata ( const Page page)
private

Definition at line 432 of file FileBuffer.cpp.

References CHECK, Data_Namespace::AbstractBuffer::encoder_, f, File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileForFileId(), Data_Namespace::AbstractBuffer::initEncoder(), METADATA_PAGE_SIZE, METADATA_VERSION, NUM_METADATA, File_Namespace::Page::pageNum, pageSize_, reservedHeaderSize_, SQLTypeInfo::set_comp_param(), SQLTypeInfo::set_compression(), SQLTypeInfo::set_dimension(), SQLTypeInfo::set_notnull(), SQLTypeInfo::set_scale(), SQLTypeInfo::set_size(), SQLTypeInfo::set_subtype(), SQLTypeInfo::set_type(), Data_Namespace::AbstractBuffer::size_, Data_Namespace::AbstractBuffer::sql_type_, and setup::version.

Referenced by initMetadataAndPageDataSize().

432  {
433  FILE* f = fm_->getFileForFileId(page.fileId);
434  fseek(f, page.pageNum * METADATA_PAGE_SIZE + reservedHeaderSize_, SEEK_SET);
435  fread((int8_t*)&pageSize_, sizeof(size_t), 1, f);
436  fread((int8_t*)&size_, sizeof(size_t), 1, f);
437  vector<int32_t> typeData(
438  NUM_METADATA); // assumes we will encode hasEncoder, bufferType,
439  // encodingType, encodingBits all as int
440  fread((int8_t*)&(typeData[0]), sizeof(int32_t), typeData.size(), f);
441  int32_t version = typeData[0];
442  CHECK(version == METADATA_VERSION); // add backward compatibility code here
443  bool has_encoder = static_cast<bool>(typeData[1]);
444  if (has_encoder) {
445  sql_type_.set_type(static_cast<SQLTypes>(typeData[2]));
446  sql_type_.set_subtype(static_cast<SQLTypes>(typeData[3]));
447  sql_type_.set_dimension(typeData[4]);
448  sql_type_.set_scale(typeData[5]);
449  sql_type_.set_notnull(static_cast<bool>(typeData[6]));
450  sql_type_.set_compression(static_cast<EncodingType>(typeData[7]));
451  sql_type_.set_comp_param(typeData[8]);
452  sql_type_.set_size(typeData[9]);
454  encoder_->readMetadata(f);
455  }
456 }
void set_compression(EncodingType c)
Definition: sqltypes.h:414
void set_size(int s)
Definition: sqltypes.h:412
#define METADATA_PAGE_SIZE
Definition: FileBuffer.h:37
HOST DEVICE void set_subtype(SQLTypes st)
Definition: sqltypes.h:405
void initEncoder(const SQLTypeInfo &tmp_sql_type)
void set_scale(int s)
Definition: sqltypes.h:409
FILE * getFileForFileId(const int32_t fileId)
Returns FILE pointer associated with requested fileId.
Definition: FileMgr.cpp:974
void set_comp_param(int p)
Definition: sqltypes.h:415
string version
Definition: setup.in.py:73
#define NUM_METADATA
Definition: FileBuffer.h:35
void set_dimension(int d)
Definition: sqltypes.h:406
std::unique_ptr< Encoder > encoder_
void set_notnull(bool n)
Definition: sqltypes.h:411
#define CHECK(condition)
Definition: Logger.h:203
char * f
#define METADATA_VERSION
Definition: FileBuffer.h:36
HOST DEVICE void set_type(SQLTypes t)
Definition: sqltypes.h:404

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::reserve ( const size_t  numBytes)
overridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 133 of file FileBuffer.cpp.

References addNewMultiPage(), getFileMgrEpoch(), multiPages_, pageSize_, and writeHeader().

133  {
134  size_t numPagesRequested = (numBytes + pageSize_ - 1) / pageSize_;
135  size_t numCurrentPages = multiPages_.size();
136  auto epoch = getFileMgrEpoch();
137  for (size_t pageNum = numCurrentPages; pageNum < numPagesRequested; ++pageNum) {
138  Page page = addNewMultiPage(epoch);
139  writeHeader(page, pageNum, epoch);
140  }
141 }
Page addNewMultiPage(const int32_t epoch)
Definition: FileBuffer.cpp:405
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:413

+ Here is the call graph for this function:

virtual size_t File_Namespace::FileBuffer::reservedHeaderSize ( ) const
inlinevirtual

Returns the size in bytes of the reserved header portion of each page in the FileBuffer.

Definition at line 141 of file FileBuffer.h.

Referenced by File_Namespace::FileMgr::init(), and File_Namespace::readForThread().

141 { return reservedHeaderSize_; }

+ Here is the caller graph for this function:

size_t File_Namespace::FileBuffer::reservedSize ( ) const
inlineoverridevirtual

Returns the total number of bytes allocated for the FileBuffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 148 of file FileBuffer.h.

148 { return multiPages_.size() * pageSize_; }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197
void File_Namespace::FileBuffer::write ( int8_t *  src,
const size_t  numBytes,
const size_t  offset = 0,
const MemoryLevel  srcMemoryLevel = CPU_LEVEL,
const int32_t  deviceId = -1 
)
override

Writes the contents of source (src) into new versions of the affected logical pages.

This method will write the contents of source (src) into new version of the affected logical pages. New pages are only appended if the value of epoch (in FileMgr)

Definition at line 533 of file FileBuffer.cpp.

References addNewMultiPage(), CHECK, copyPage(), Data_Namespace::CPU_LEVEL, File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), getFileMgrEpoch(), multiPages_, pageDataSize_, File_Namespace::Page::pageNum, pageSize_, File_Namespace::FileMgr::requestFreePage(), reservedHeaderSize_, Data_Namespace::AbstractBuffer::setAppended(), Data_Namespace::AbstractBuffer::setDirty(), Data_Namespace::AbstractBuffer::setUpdated(), Data_Namespace::AbstractBuffer::size_, File_Namespace::FileInfo::write(), and writeHeader().

537  {
538  CHECK(srcBufferType == CPU_LEVEL) << "Unsupported Buffer type";
539 
540  bool tempIsAppended = false;
541  setDirty();
542  if (offset < size_) {
543  setUpdated();
544  }
545  if (offset + numBytes > size_) {
546  tempIsAppended = true; // because is_appended_ could have already been true - to
547  // avoid rewriting header
548  setAppended();
549  size_ = offset + numBytes;
550  }
551 
552  size_t startPage = offset / pageDataSize_;
553  size_t startPageOffset = offset % pageDataSize_;
554  size_t numPagesToWrite =
555  (numBytes + startPageOffset + pageDataSize_ - 1) / pageDataSize_;
556  size_t bytesLeft = numBytes;
557  int8_t* curPtr = src; // a pointer to the current location in dst being written to
558  size_t initialNumPages = multiPages_.size();
559  auto epoch = getFileMgrEpoch();
560 
561  if (startPage >
562  initialNumPages) { // means there is a gap we need to allocate pages for
563  for (size_t pageNum = initialNumPages; pageNum < startPage; ++pageNum) {
564  Page page = addNewMultiPage(epoch);
565  writeHeader(page, pageNum, epoch);
566  }
567  }
568  for (size_t pageNum = startPage; pageNum < startPage + numPagesToWrite; ++pageNum) {
569  Page page;
570  if (pageNum >= initialNumPages) {
571  page = addNewMultiPage(epoch);
572  writeHeader(page, pageNum, epoch);
573  } else if (multiPages_[pageNum].current().epoch <
574  epoch) { // need to create new page b/c this current one lags epoch and we
575  // can't overwrite it also need to copy if we are on first or
576  // last page
577  Page lastPage = multiPages_[pageNum].current().page;
578  page = fm_->requestFreePage(pageSize_, false);
579  multiPages_[pageNum].push(page, epoch);
580  if (pageNum == startPage && startPageOffset > 0) {
581  // copyPage takes care of header offset so don't worry
582  // about it
583  copyPage(lastPage, page, startPageOffset, 0);
584  }
585  if (pageNum == (startPage + numPagesToWrite - 1) &&
586  bytesLeft > 0) { // bytesLeft should always > 0
587  copyPage(lastPage,
588  page,
589  pageDataSize_ - bytesLeft,
590  bytesLeft); // these would be empty if we're appending but we won't
591  // worry about it right now
592  }
593  writeHeader(page, pageNum, epoch);
594  } else {
595  // we already have a new page at current
596  // epoch for this page - just grab this page
597  page = multiPages_[pageNum].current().page;
598  }
599  CHECK(page.fileId >= 0); // make sure page was initialized
600  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
601  size_t bytesWritten;
602  if (pageNum == startPage) {
603  bytesWritten = fileInfo->write(
604  page.pageNum * pageSize_ + startPageOffset + reservedHeaderSize_,
605  min(pageDataSize_ - startPageOffset, bytesLeft),
606  curPtr);
607  } else {
608  bytesWritten = fileInfo->write(page.pageNum * pageSize_ + reservedHeaderSize_,
609  min(pageDataSize_, bytesLeft),
610  curPtr);
611  }
612  curPtr += bytesWritten;
613  bytesLeft -= bytesWritten;
614  if (tempIsAppended && pageNum == startPage + numPagesToWrite - 1) { // if last page
615  //@todo below can lead to undefined - we're overwriting num
616  // bytes valid at checkpoint
617  writeHeader(page, 0, multiPages_[0].current().epoch, true);
618  }
619  }
620  CHECK(bytesLeft == 0);
621 }
Page requestFreePage(size_t pagesize, const bool isMetadata)
Definition: FileMgr.cpp:861
Page addNewMultiPage(const int32_t epoch)
Definition: FileBuffer.cpp:405
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:197
size_t write(const size_t offset, const size_t size, int8_t *buf)
Definition: FileInfo.cpp:60
FileInfo * getFileInfoForFileId(const int32_t fileId)
Definition: FileMgr.h:220
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:413
#define CHECK(condition)
Definition: Logger.h:203
void copyPage(Page &srcPage, Page &destPage, const size_t numBytes, const size_t offset=0)
Definition: FileBuffer.cpp:385

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::writeHeader ( Page page,
const int32_t  pageId,
const int32_t  epoch,
const bool  writeMetadata = false 
)
private

Write header writes header at top of page in format.

Definition at line 413 of file FileBuffer.cpp.

References chunkKey_, gpu_enabled::copy(), File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), METADATA_PAGE_SIZE, File_Namespace::Page::pageNum, pageSize(), pageSize_, and File_Namespace::FileInfo::write().

Referenced by append(), File_Namespace::FileMgr::init(), reserve(), write(), and writeMetadata().

416  {
417  int32_t intHeaderSize = chunkKey_.size() + 3; // does not include chunkSize
418  vector<int32_t> header(intHeaderSize);
419  // in addition to chunkkey we need size of header, pageId, version
420  header[0] =
421  (intHeaderSize - 1) * sizeof(int32_t); // don't need to include size of headerSize
422  // value - sizeof(size_t) is for chunkSize
423  std::copy(chunkKey_.begin(), chunkKey_.end(), header.begin() + 1);
424  header[intHeaderSize - 2] = pageId;
425  header[intHeaderSize - 1] = epoch;
426  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
428  fileInfo->write(
429  page.pageNum * pageSize, (intHeaderSize) * sizeof(int32_t), (int8_t*)&header[0]);
430 }
#define METADATA_PAGE_SIZE
Definition: FileBuffer.h:37
void writeMetadata(const int32_t epoch)
Definition: FileBuffer.cpp:458
DEVICE auto copy(ARGS &&...args)
Definition: gpu_enabled.h:51
size_t pageSize() const override
Returns the size in bytes of each page in the FileBuffer.
Definition: FileBuffer.h:134
FileInfo * getFileInfoForFileId(const int32_t fileId)
Definition: FileMgr.h:220

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::writeMetadata ( const int32_t  epoch)
private

Definition at line 458 of file FileBuffer.cpp.

References Data_Namespace::AbstractBuffer::encoder_, f, File_Namespace::Page::fileId, fm_, SQLTypeInfo::get_comp_param(), SQLTypeInfo::get_compression(), SQLTypeInfo::get_dimension(), SQLTypeInfo::get_notnull(), SQLTypeInfo::get_scale(), SQLTypeInfo::get_size(), SQLTypeInfo::get_subtype(), SQLTypeInfo::get_type(), File_Namespace::FileMgr::getFileForFileId(), Data_Namespace::AbstractBuffer::hasEncoder(), METADATA_PAGE_SIZE, METADATA_VERSION, metadataPages_, NUM_METADATA, File_Namespace::Page::pageNum, pageSize_, File_Namespace::MultiPage::push(), File_Namespace::FileMgr::requestFreePage(), reservedHeaderSize_, Data_Namespace::AbstractBuffer::size_, Data_Namespace::AbstractBuffer::sql_type_, and writeHeader().

458  {
459  // Right now stats page is size_ (in bytes), bufferType, encodingType,
460  // encodingDataType, numElements
461  Page page = fm_->requestFreePage(METADATA_PAGE_SIZE, true);
462  writeHeader(page, -1, epoch, true);
463  FILE* f = fm_->getFileForFileId(page.fileId);
464  fseek(f, page.pageNum * METADATA_PAGE_SIZE + reservedHeaderSize_, SEEK_SET);
465  fwrite((int8_t*)&pageSize_, sizeof(size_t), 1, f);
466  fwrite((int8_t*)&size_, sizeof(size_t), 1, f);
467  vector<int32_t> typeData(
468  NUM_METADATA); // assumes we will encode hasEncoder, bufferType,
469  // encodingType, encodingBits all as int32_t
470  typeData[0] = METADATA_VERSION;
471  typeData[1] = static_cast<int32_t>(hasEncoder());
472  if (hasEncoder()) {
473  typeData[2] = static_cast<int32_t>(sql_type_.get_type());
474  typeData[3] = static_cast<int32_t>(sql_type_.get_subtype());
475  typeData[4] = sql_type_.get_dimension();
476  typeData[5] = sql_type_.get_scale();
477  typeData[6] = static_cast<int32_t>(sql_type_.get_notnull());
478  typeData[7] = static_cast<int32_t>(sql_type_.get_compression());
479  typeData[8] = sql_type_.get_comp_param();
480  typeData[9] = sql_type_.get_size();
481  }
482  fwrite((int8_t*)&(typeData[0]), sizeof(int32_t), typeData.size(), f);
483  if (hasEncoder()) { // redundant
484  encoder_->writeMetadata(f);
485  }
486  metadataPages_.push(page, epoch);
487 }
HOST DEVICE SQLTypes get_subtype() const
Definition: sqltypes.h:315
#define METADATA_PAGE_SIZE
Definition: FileBuffer.h:37
HOST DEVICE int get_size() const
Definition: sqltypes.h:324
Page requestFreePage(size_t pagesize, const bool isMetadata)
Definition: FileMgr.cpp:861
HOST DEVICE int get_scale() const
Definition: sqltypes.h:319
HOST DEVICE SQLTypes get_type() const
Definition: sqltypes.h:314
FILE * getFileForFileId(const int32_t fileId)
Returns FILE pointer associated with requested fileId.
Definition: FileMgr.cpp:974
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:413
HOST DEVICE EncodingType get_compression() const
Definition: sqltypes.h:322
#define NUM_METADATA
Definition: FileBuffer.h:35
void push(const Page &page, const int epoch)
Pushes a new page with epoch value.
Definition: Page.h:102
HOST DEVICE int get_dimension() const
Definition: sqltypes.h:316
std::unique_ptr< Encoder > encoder_
HOST DEVICE int get_comp_param() const
Definition: sqltypes.h:323
char * f
HOST DEVICE bool get_notnull() const
Definition: sqltypes.h:321
#define METADATA_VERSION
Definition: FileBuffer.h:36

+ Here is the call graph for this function:

Friends And Related Function Documentation

friend class CachingFileMgr
friend

Definition at line 60 of file FileBuffer.h.

friend class FileMgr
friend

Definition at line 59 of file FileBuffer.h.

Member Data Documentation

ChunkKey File_Namespace::FileBuffer::chunkKey_
private
constexpr size_t File_Namespace::FileBuffer::headerBufferOffset_ = 32
static

Definition at line 169 of file FileBuffer.h.

Referenced by calcHeaderBuffer().

MultiPage File_Namespace::FileBuffer::metadataPages_
private
std::vector<MultiPage> File_Namespace::FileBuffer::multiPages_
private
size_t File_Namespace::FileBuffer::pageDataSize_
private
size_t File_Namespace::FileBuffer::pageSize_
private
size_t File_Namespace::FileBuffer::reservedHeaderSize_
private

The documentation for this class was generated from the following files: