OmniSciDB  471d68cefb
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
File_Namespace::FileBuffer Class Reference

Represents/provides access to contiguous data stored in the file system. More...

#include <FileBuffer.h>

+ Inheritance diagram for File_Namespace::FileBuffer:
+ Collaboration diagram for File_Namespace::FileBuffer:

Public Member Functions

 FileBuffer (FileMgr *fm, const size_t pageSize, const ChunkKey &chunkKey, const size_t initialSize=0)
 Constructs a FileBuffer object. More...
 
 FileBuffer (FileMgr *fm, const size_t pageSize, const ChunkKey &chunkKey, const SQLTypeInfo sqlType, const size_t initialSize=0)
 
 FileBuffer (FileMgr *fm, const ChunkKey &chunkKey, const std::vector< HeaderInfo >::const_iterator &headerStartIt, const std::vector< HeaderInfo >::const_iterator &headerEndIt)
 
 ~FileBuffer () override
 Destructor. More...
 
Page addNewMultiPage (const int32_t epoch)
 
void reserve (const size_t numBytes) override
 
size_t freeMetadataPages ()
 
size_t freeChunkPages ()
 
size_t freePages ()
 
void freePagesBeforeEpoch (const int32_t targetEpoch)
 
void read (int8_t *const dst, const size_t numBytes=0, const size_t offset=0, const MemoryLevel dstMemoryLevel=CPU_LEVEL, const int32_t deviceId=-1) override
 
void write (int8_t *src, const size_t numBytes, const size_t offset=0, const MemoryLevel srcMemoryLevel=CPU_LEVEL, const int32_t deviceId=-1) override
 Writes the contents of source (src) into new versions of the affected logical pages. More...
 
void append (int8_t *src, const size_t numBytes, const MemoryLevel srcMemoryLevel=CPU_LEVEL, const int32_t deviceId=-1) override
 
void copyPage (Page &srcPage, Page &destPage, const size_t numBytes, const size_t offset=0)
 
Data_Namespace::MemoryLevel getType () const override
 
int8_t * getMemoryPtr () override
 Not implemented for FileMgr – throws a runtime_error. More...
 
size_t pageCount () const override
 Returns the number of pages in the FileBuffer. More...
 
bool hasDataPages () const
 
size_t pageSize () const override
 Returns the size in bytes of each page in the FileBuffer. More...
 
virtual size_t pageDataSize () const
 Returns the size in bytes of the data portion of each page in the FileBuffer. More...
 
virtual size_t reservedHeaderSize () const
 
virtual std::vector< MultiPagegetMultiPage () const
 Returns vector of MultiPages in the FileBuffer. More...
 
MultiPage getMetadataPage () const
 
size_t reservedSize () const override
 Returns the total number of bytes allocated for the FileBuffer. More...
 
size_t numMetadataPages () const
 Returns the total number of used bytes in the FileBuffer. More...
 
bool isMissingPages () const
 
size_t numChunkPages () const
 
std::string dump () const
 
void freePage (const Page &page)
 
- Public Member Functions inherited from Data_Namespace::AbstractBuffer
 AbstractBuffer (const int device_id)
 
 AbstractBuffer (const int device_id, const SQLTypeInfo sql_type)
 
virtual ~AbstractBuffer ()
 
virtual void read (int8_t *const dst, const size_t num_bytes, const size_t offset=0, const MemoryLevel dst_buffer_type=CPU_LEVEL, const int dst_device_id=-1)=0
 
virtual void write (int8_t *src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int src_device_id=-1)=0
 
virtual void append (int8_t *src, const size_t num_bytes, const MemoryLevel src_buffer_type=CPU_LEVEL, const int device_id=-1)=0
 
virtual void setMemoryPtr (int8_t *new_ptr)
 
virtual int pin ()
 
virtual int unPin ()
 
virtual int getPinCount ()
 
size_t size () const
 
int getDeviceId () const
 
bool isDirty () const
 
bool isAppended () const
 
bool isUpdated () const
 
bool hasEncoder () const
 
SQLTypeInfo getSqlType () const
 
void setSqlType (const SQLTypeInfo &sql_type)
 
EncodergetEncoder () const
 
void setDirty ()
 
void setUpdated ()
 
void setAppended ()
 
void setSize (const size_t size)
 
void clearDirtyBits ()
 
void initEncoder (const SQLTypeInfo &tmp_sql_type)
 
void syncEncoder (const AbstractBuffer *src_buffer)
 
void copyTo (AbstractBuffer *destination_buffer, const size_t num_bytes=0)
 
void resetToEmpty ()
 

Static Public Attributes

static constexpr size_t headerBufferOffset_ = 32
 

Private Member Functions

void writeHeader (Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
 Write header writes header at top of page in format. More...
 
void writeMetadata (const int32_t epoch)
 
void readMetadata (const Page &page)
 
void calcHeaderBuffer ()
 
void freePage (const Page &page, const bool isRolloff)
 
void freePagesBeforeEpochForMultiPage (MultiPage &multiPage, const int32_t targetEpoch, const int32_t currentEpoch)
 
void initMetadataAndPageDataSize ()
 
int32_t getFileMgrEpoch ()
 

Private Attributes

FileMgrfm_
 
MultiPage metadataPages_
 
std::vector< MultiPagemultiPages_
 
size_t pageSize_
 
size_t pageDataSize_
 
size_t reservedHeaderSize_
 
ChunkKey chunkKey_
 

Friends

class FileMgr
 
class CachingFileMgr
 

Additional Inherited Members

- Protected Attributes inherited from Data_Namespace::AbstractBuffer
std::unique_ptr< Encoderencoder_
 
SQLTypeInfo sql_type_
 
size_t size_
 
int device_id_
 

Detailed Description

Represents/provides access to contiguous data stored in the file system.

The FileBuffer consists of logical pages, which can map to any identically-sized page in any file of the underlying file system. A page's metadata (file and page number) are stored in MultiPage objects, and each MultiPage includes page metadata for multiple versions of the same page.

Note that a "Chunk" is brought into a FileBuffer by the FileMgr.

Note(s): Forbid Copying Idiom 4.1

Definition at line 58 of file FileBuffer.h.

Constructor & Destructor Documentation

File_Namespace::FileBuffer::FileBuffer ( FileMgr fm,
const size_t  pageSize,
const ChunkKey chunkKey,
const size_t  initialSize = 0 
)

Constructs a FileBuffer object.

Definition at line 38 of file FileBuffer.cpp.

References calcHeaderBuffer(), CHECK, CHECK_GT, fm_, pageDataSize_, pageSize_, and reservedHeaderSize_.

42  : AbstractBuffer(fm->getDeviceId())
43  , fm_(fm)
46  , chunkKey_(chunkKey) {
47  // Create a new FileBuffer
48  CHECK(fm_);
52  //@todo reintroduce initialSize - need to develop easy way of
53  // differentiating these pre-allocated pages from "written-to" pages
54  /*
55  if (initalSize > 0) {
56  // should expand to initialSize bytes
57  size_t initialNumPages = (initalSize + pageSize_ -1) / pageSize_;
58  int32_t epoch = fm_->epoch();
59  for (size_t pageNum = 0; pageNum < initialNumPages; ++pageNum) {
60  Page page = addNewMultiPage(epoch);
61  writeHeader(page,pageNum,epoch);
62  }
63  }
64  */
65 }
#define METADATA_PAGE_SIZE
Definition: FileBuffer.h:37
#define CHECK_GT(x, y)
Definition: Logger.h:221
AbstractBuffer(const int device_id)
size_t pageSize() const override
Returns the size in bytes of each page in the FileBuffer.
Definition: FileBuffer.h:138
#define CHECK(condition)
Definition: Logger.h:209

+ Here is the call graph for this function:

File_Namespace::FileBuffer::FileBuffer ( FileMgr fm,
const size_t  pageSize,
const ChunkKey chunkKey,
const SQLTypeInfo  sqlType,
const size_t  initialSize = 0 
)

Definition at line 67 of file FileBuffer.cpp.

References calcHeaderBuffer(), CHECK, fm_, pageDataSize_, pageSize_, and reservedHeaderSize_.

72  : AbstractBuffer(fm->getDeviceId(), sqlType)
73  , fm_(fm)
76  , chunkKey_(chunkKey) {
77  CHECK(fm_);
80 }
#define METADATA_PAGE_SIZE
Definition: FileBuffer.h:37
AbstractBuffer(const int device_id)
size_t pageSize() const override
Returns the size in bytes of each page in the FileBuffer.
Definition: FileBuffer.h:138
#define CHECK(condition)
Definition: Logger.h:209

+ Here is the call graph for this function:

File_Namespace::FileBuffer::FileBuffer ( FileMgr fm,
const ChunkKey chunkKey,
const std::vector< HeaderInfo >::const_iterator &  headerStartIt,
const std::vector< HeaderInfo >::const_iterator &  headerEndIt 
)

Definition at line 82 of file FileBuffer.cpp.

References calcHeaderBuffer(), CHECK, File_Namespace::FileMgr::failOnReadError(), logger::FATAL, fm_, initMetadataAndPageDataSize(), LOG, metadataPages_, multiPages_, pageSize_, File_Namespace::MultiPage::push(), and show_chunk().

86  : AbstractBuffer(fm->getDeviceId())
87  , fm_(fm)
89  , pageSize_(0)
90  , chunkKey_(chunkKey) {
91  // We are being assigned an existing FileBuffer on disk
92 
93  CHECK(fm_);
95  int32_t lastPageId = -1;
96  int32_t curPageId = 0;
97  for (auto vecIt = headerStartIt; vecIt != headerEndIt; ++vecIt) {
98  curPageId = vecIt->pageId;
99 
100  // We only want to read last metadata page
101  if (curPageId == -1) { // stats page
102  metadataPages_.push(vecIt->page, vecIt->versionEpoch);
103  } else {
104  if (curPageId != lastPageId) {
105  // protect from bad data on disk, and give diagnostics
106  if (fm->failOnReadError()) {
107  if (curPageId != lastPageId + 1) {
108  LOG(FATAL) << "Failure reading DB file " << show_chunk(chunkKey)
109  << " Current page " << curPageId << " last page " << lastPageId
110  << " epoch " << vecIt->versionEpoch;
111  }
112  }
113  if (lastPageId == -1) { // If we are on first real page
115  }
116  MultiPage multiPage(pageSize_);
117  multiPages_.push_back(multiPage);
118  lastPageId = curPageId;
119  }
120  multiPages_.back().push(vecIt->page, vecIt->versionEpoch);
121  }
122  }
123  if (curPageId == -1) { // meaning there was only a metadata page
125  }
126 }
#define METADATA_PAGE_SIZE
Definition: FileBuffer.h:37
#define LOG(tag)
Definition: Logger.h:203
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194
std::string show_chunk(const ChunkKey &key)
Definition: types.h:86
AbstractBuffer(const int device_id)
void push(const Page &page, const int epoch)
Pushes a new page with epoch value.
Definition: Page.h:102
#define CHECK(condition)
Definition: Logger.h:209

+ Here is the call graph for this function:

File_Namespace::FileBuffer::~FileBuffer ( )
override

Destructor.

Definition at line 128 of file FileBuffer.cpp.

128  {
129  // need to free pages
130  // NOP
131 }

Member Function Documentation

Page File_Namespace::FileBuffer::addNewMultiPage ( const int32_t  epoch)

Definition at line 406 of file FileBuffer.cpp.

References fm_, multiPages_, pageSize_, File_Namespace::MultiPage::push(), and File_Namespace::FileMgr::requestFreePage().

Referenced by append(), reserve(), and write().

406  {
407  Page page = fm_->requestFreePage(pageSize_, false);
408  MultiPage multiPage(pageSize_);
409  multiPage.push(page, epoch);
410  multiPages_.emplace_back(multiPage);
411  return page;
412 }
virtual Page requestFreePage(size_t pagesize, const bool isMetadata)
Definition: FileMgr.cpp:851
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::append ( int8_t *  src,
const size_t  numBytes,
const MemoryLevel  srcMemoryLevel = CPU_LEVEL,
const int32_t  deviceId = -1 
)
override

Definition at line 490 of file FileBuffer.cpp.

References addNewMultiPage(), CHECK, File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), getFileMgrEpoch(), multiPages_, pageDataSize_, File_Namespace::Page::pageNum, pageSize_, reservedHeaderSize_, Data_Namespace::AbstractBuffer::setAppended(), Data_Namespace::AbstractBuffer::size_, File_Namespace::FileInfo::write(), and writeHeader().

493  {
494  setAppended();
495 
496  size_t startPage = size_ / pageDataSize_;
497  size_t startPageOffset = size_ % pageDataSize_;
498  size_t numPagesToWrite =
499  (numBytes + startPageOffset + pageDataSize_ - 1) / pageDataSize_;
500  size_t bytesLeft = numBytes;
501  int8_t* curPtr = src; // a pointer to the current location in dst being written to
502  size_t initialNumPages = multiPages_.size();
503  size_ = size_ + numBytes;
504  auto epoch = getFileMgrEpoch();
505  for (size_t pageNum = startPage; pageNum < startPage + numPagesToWrite; ++pageNum) {
506  Page page;
507  if (pageNum >= initialNumPages) {
508  page = addNewMultiPage(epoch);
509  writeHeader(page, pageNum, epoch);
510  } else {
511  // we already have a new page at current
512  // epoch for this page - just grab this page
513  page = multiPages_[pageNum].current().page;
514  }
515  CHECK(page.fileId >= 0); // make sure page was initialized
516  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
517  size_t bytesWritten;
518  if (pageNum == startPage) {
519  bytesWritten = fileInfo->write(
520  page.pageNum * pageSize_ + startPageOffset + reservedHeaderSize_,
521  min(pageDataSize_ - startPageOffset, bytesLeft),
522  curPtr);
523  } else {
524  bytesWritten = fileInfo->write(page.pageNum * pageSize_ + reservedHeaderSize_,
525  min(pageDataSize_, bytesLeft),
526  curPtr);
527  }
528  curPtr += bytesWritten;
529  bytesLeft -= bytesWritten;
530  }
531  CHECK(bytesLeft == 0);
532 }
size_t write(const size_t offset, const size_t size, const int8_t *buf)
Definition: FileInfo.cpp:60
Page addNewMultiPage(const int32_t epoch)
Definition: FileBuffer.cpp:406
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:414
#define CHECK(condition)
Definition: Logger.h:209
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:218

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::calcHeaderBuffer ( )
private

Definition at line 143 of file FileBuffer.cpp.

References chunkKey_, headerBufferOffset_, and reservedHeaderSize_.

Referenced by FileBuffer().

143  {
144  // 3 * sizeof(int32_t) is for headerSize, for pageId and versionEpoch
145  // sizeof(size_t) is for chunkSize
146  reservedHeaderSize_ = (chunkKey_.size() + 3) * sizeof(int32_t);
147  size_t headerMod = reservedHeaderSize_ % headerBufferOffset_;
148  if (headerMod > 0) {
150  }
151 }
static constexpr size_t headerBufferOffset_
Definition: FileBuffer.h:166

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::copyPage ( Page srcPage,
Page destPage,
const size_t  numBytes,
const size_t  offset = 0 
)

Definition at line 386 of file FileBuffer.cpp.

References CHECK, CHECK_LE, checked_malloc(), File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), pageDataSize_, File_Namespace::Page::pageNum, pageSize_, File_Namespace::FileInfo::read(), reservedHeaderSize_, and File_Namespace::FileInfo::write().

Referenced by write().

389  {
390  // FILE *srcFile = fm_->files_[srcPage.fileId]->f;
391  // FILE *destFile = fm_->files_[destPage.fileId]->f;
392  CHECK_LE(offset + numBytes, pageDataSize_);
393  FileInfo* srcFileInfo = fm_->getFileInfoForFileId(srcPage.fileId);
394  FileInfo* destFileInfo = fm_->getFileInfoForFileId(destPage.fileId);
395 
396  int8_t* buffer = reinterpret_cast<int8_t*>(checked_malloc(numBytes));
397  size_t bytesRead = srcFileInfo->read(
398  srcPage.pageNum * pageSize_ + offset + reservedHeaderSize_, numBytes, buffer);
399  CHECK(bytesRead == numBytes);
400  size_t bytesWritten = destFileInfo->write(
401  destPage.pageNum * pageSize_ + offset + reservedHeaderSize_, numBytes, buffer);
402  CHECK(bytesWritten == numBytes);
403  free(buffer);
404 }
void * checked_malloc(const size_t size)
Definition: checked_alloc.h:45
#define CHECK_LE(x, y)
Definition: Logger.h:220
size_t read(const size_t offset, const size_t size, int8_t *buf)
Definition: FileInfo.cpp:66
#define CHECK(condition)
Definition: Logger.h:209
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:218

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string File_Namespace::FileBuffer::dump ( ) const

Definition at line 629 of file FileBuffer.cpp.

References chunkKey_, Data_Namespace::AbstractBuffer::hasEncoder(), show_chunk(), and Data_Namespace::AbstractBuffer::size_.

629  {
630  std::stringstream ss;
631  ss << "chunk_key = " << show_chunk(chunkKey_) << "\n";
632  ss << "has_encoder = " << (hasEncoder() ? "true\n" : "false\n");
633  ss << "size_ = " << size_ << "\n";
634  return ss.str();
635 }
std::string show_chunk(const ChunkKey &key)
Definition: types.h:86

+ Here is the call graph for this function:

size_t File_Namespace::FileBuffer::freeChunkPages ( )

Definition at line 175 of file FileBuffer.cpp.

References freePage(), and multiPages_.

Referenced by freePages(), and freePagesBeforeEpoch().

175  {
176  size_t num_pages_freed = multiPages_.size();
177  for (auto multiPageIt = multiPages_.begin(); multiPageIt != multiPages_.end();
178  ++multiPageIt) {
179  for (auto pageIt = multiPageIt->pageVersions.begin();
180  pageIt != multiPageIt->pageVersions.end();
181  ++pageIt) {
182  freePage(pageIt->page, false /* isRolloff */);
183  }
184  }
185  multiPages_.clear();
186  return num_pages_freed;
187 }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194
void freePage(const Page &page)
Definition: FileBuffer.cpp:153

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::FileBuffer::freeMetadataPages ( )

Definition at line 162 of file FileBuffer.cpp.

References freePage(), metadataPages_, File_Namespace::MultiPage::pageVersions, and File_Namespace::MultiPage::pop().

Referenced by freePages().

162  {
163  size_t num_pages_freed = metadataPages_.pageVersions.size();
164  for (auto metaPageIt = metadataPages_.pageVersions.begin();
165  metaPageIt != metadataPages_.pageVersions.end();
166  ++metaPageIt) {
167  freePage(metaPageIt->page, false /* isRolloff */);
168  }
169  while (metadataPages_.pageVersions.size() > 0) {
171  }
172  return num_pages_freed;
173 }
void pop()
Purges the oldest Page.
Definition: Page.h:110
std::deque< EpochedPage > pageVersions
Definition: Page.h:81
void freePage(const Page &page)
Definition: FileBuffer.cpp:153

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::freePage ( const Page page)

Definition at line 153 of file FileBuffer.cpp.

Referenced by freeChunkPages(), freeMetadataPages(), and freePagesBeforeEpochForMultiPage().

153  {
154  freePage(page, false);
155 }
void freePage(const Page &page)
Definition: FileBuffer.cpp:153

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::freePage ( const Page page,
const bool  isRolloff 
)
private

Definition at line 157 of file FileBuffer.cpp.

References File_Namespace::Page::fileId, fm_, File_Namespace::FileInfo::freePage(), File_Namespace::FileMgr::getFileInfoForFileId(), getFileMgrEpoch(), and File_Namespace::Page::pageNum.

157  {
158  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
159  fileInfo->freePage(page.pageNum, isRolloff, getFileMgrEpoch());
160 }
void freePage(int32_t pageId, const bool isRolloff, int32_t epoch)
Definition: FileInfo.cpp:178
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:218

+ Here is the call graph for this function:

size_t File_Namespace::FileBuffer::freePages ( )

Definition at line 189 of file FileBuffer.cpp.

References freeChunkPages(), and freeMetadataPages().

189  {
190  return freeMetadataPages() + freeChunkPages();
191 }

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::freePagesBeforeEpoch ( const int32_t  targetEpoch)

Definition at line 203 of file FileBuffer.cpp.

References CHECK_LE, chunkKey_, fm_, freeChunkPages(), freePagesBeforeEpochForMultiPage(), getFileMgrEpoch(), metadataPages_, multiPages_, pageSize_, File_Namespace::MultiPage::pageVersions, and Data_Namespace::AbstractBuffer::size_.

203  {
204  // This method is only safe to be called within a checkpoint, after the sync and epoch
205  // increment where a failure at any point32_t in the process would lead to a safe
206  // rollback
207  auto currentEpoch = getFileMgrEpoch();
208  CHECK_LE(targetEpoch, currentEpoch);
209  freePagesBeforeEpochForMultiPage(metadataPages_, targetEpoch, currentEpoch);
210  for (auto& multiPage : multiPages_) {
211  freePagesBeforeEpochForMultiPage(multiPage, targetEpoch, currentEpoch);
212  }
213 
214  // Check if all buffer pages can be freed
215  if (size_ == 0) {
216  size_t max_historical_buffer_size{0};
217  for (auto& epoch_page : metadataPages_.pageVersions) {
218  // Create buffer that is used to get the buffer size at the epoch version
219  FileBuffer buffer{fm_, pageSize_, chunkKey_};
220  buffer.readMetadata(epoch_page.page);
221  max_historical_buffer_size = std::max(max_historical_buffer_size, buffer.size());
222  }
223 
224  // Free all chunk pages, if none of the old chunk versions has any data
225  if (max_historical_buffer_size == 0) {
226  freeChunkPages();
227  }
228  }
229 }
void freePagesBeforeEpochForMultiPage(MultiPage &multiPage, const int32_t targetEpoch, const int32_t currentEpoch)
Definition: FileBuffer.cpp:193
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194
std::deque< EpochedPage > pageVersions
Definition: Page.h:81
#define CHECK_LE(x, y)
Definition: Logger.h:220
FileBuffer(FileMgr *fm, const size_t pageSize, const ChunkKey &chunkKey, const size_t initialSize=0)
Constructs a FileBuffer object.
Definition: FileBuffer.cpp:38

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::freePagesBeforeEpochForMultiPage ( MultiPage multiPage,
const int32_t  targetEpoch,
const int32_t  currentEpoch 
)
private

Definition at line 193 of file FileBuffer.cpp.

References freePage(), and File_Namespace::MultiPage::freePagesBeforeEpoch().

Referenced by freePagesBeforeEpoch().

195  {
196  std::vector<EpochedPage> epochedPagesToFree =
197  multiPage.freePagesBeforeEpoch(targetEpoch, currentEpoch);
198  for (const auto& epochedPageToFree : epochedPagesToFree) {
199  freePage(epochedPageToFree.page, true /* isRolloff */);
200  }
201 }
void freePage(const Page &page)
Definition: FileBuffer.cpp:153

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

int32_t File_Namespace::FileBuffer::getFileMgrEpoch ( )
private

Definition at line 624 of file FileBuffer.cpp.

References chunkKey_, File_Namespace::FileMgr::epoch(), fm_, and get_table_prefix().

Referenced by append(), freePage(), freePagesBeforeEpoch(), reserve(), and write().

624  {
625  auto [db_id, tb_id] = get_table_prefix(chunkKey_);
626  return fm_->epoch(db_id, tb_id);
627 }
virtual int32_t epoch(int32_t db_id, int32_t tb_id) const
Returns current value of epoch - should be one greater than recorded at last checkpoint. Because FileMgr only contains buffers from one table we can just return the FileMgr&#39;s epoch instead of finding a table-specific epoch.
Definition: FileMgr.h:271
std::pair< int, int > get_table_prefix(const ChunkKey &key)
Definition: types.h:58

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

int8_t* File_Namespace::FileBuffer::getMemoryPtr ( )
inlineoverridevirtual

Not implemented for FileMgr – throws a runtime_error.

Implements Data_Namespace::AbstractBuffer.

Definition at line 125 of file FileBuffer.h.

References logger::FATAL, and LOG.

125  {
126  LOG(FATAL) << "Operation not supported.";
127  return nullptr; // satisfy return-type warning
128  }
#define LOG(tag)
Definition: Logger.h:203
MultiPage File_Namespace::FileBuffer::getMetadataPage ( ) const
inline

Definition at line 149 of file FileBuffer.h.

149 { return metadataPages_; }
virtual std::vector<MultiPage> File_Namespace::FileBuffer::getMultiPage ( ) const
inlinevirtual

Returns vector of MultiPages in the FileBuffer.

Definition at line 148 of file FileBuffer.h.

Referenced by read().

148 { return multiPages_; }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194

+ Here is the caller graph for this function:

Data_Namespace::MemoryLevel File_Namespace::FileBuffer::getType ( ) const
inlineoverridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 122 of file FileBuffer.h.

References Data_Namespace::DISK_LEVEL.

bool File_Namespace::FileBuffer::hasDataPages ( ) const
inline

Returns whether or not a buffer has data pages. It is possible for a buffer to represent metadata (have a size and encode) but not contain actual data.

Definition at line 135 of file FileBuffer.h.

135 { return pageCount() > 0; }
size_t pageCount() const override
Returns the number of pages in the FileBuffer.
Definition: FileBuffer.h:131
void File_Namespace::FileBuffer::initMetadataAndPageDataSize ( )
private

Definition at line 637 of file FileBuffer.cpp.

References CHECK, File_Namespace::MultiPage::current(), File_Namespace::Page::fileId, metadataPages_, File_Namespace::EpochedPage::page, pageDataSize_, pageSize_, readMetadata(), and reservedHeaderSize_.

Referenced by FileBuffer().

637  {
638  CHECK(metadataPages_.current().page.fileId != -1); // was initialized
641 }
int32_t fileId
Definition: Page.h:47
#define CHECK(condition)
Definition: Logger.h:209
void readMetadata(const Page &page)
Definition: FileBuffer.cpp:433
EpochedPage current() const
Returns a reference to the most recent version of the page.
Definition: Page.h:94

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

bool File_Namespace::FileBuffer::isMissingPages ( ) const

Definition at line 643 of file FileBuffer.cpp.

References multiPages_, pageDataSize_, and Data_Namespace::AbstractBuffer::size().

643  {
644  // Detect the case where a page is missing by comparing the amount of pages read
645  // with the metadata size.
646  return ((size() + pageDataSize_ - 1) / pageDataSize_ != multiPages_.size());
647 }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194

+ Here is the call graph for this function:

size_t File_Namespace::FileBuffer::numChunkPages ( ) const

Definition at line 649 of file FileBuffer.cpp.

References multiPages_.

649  {
650  size_t total_size = 0;
651  for (const auto& multi_page : multiPages_) {
652  total_size += multi_page.pageVersions.size();
653  }
654  return total_size;
655 }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194
size_t File_Namespace::FileBuffer::numMetadataPages ( ) const
inline

Returns the total number of used bytes in the FileBuffer.

Definition at line 157 of file FileBuffer.h.

157 { return metadataPages_.pageVersions.size(); };
std::deque< EpochedPage > pageVersions
Definition: Page.h:81
size_t File_Namespace::FileBuffer::pageCount ( ) const
inlineoverridevirtual

Returns the number of pages in the FileBuffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 131 of file FileBuffer.h.

131 { return multiPages_.size(); }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194
virtual size_t File_Namespace::FileBuffer::pageDataSize ( ) const
inlinevirtual

Returns the size in bytes of the data portion of each page in the FileBuffer.

Definition at line 141 of file FileBuffer.h.

Referenced by File_Namespace::readForThread().

141 { return pageDataSize_; }

+ Here is the caller graph for this function:

size_t File_Namespace::FileBuffer::pageSize ( ) const
inlineoverridevirtual

Returns the size in bytes of each page in the FileBuffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 138 of file FileBuffer.h.

Referenced by File_Namespace::readForThread(), and writeHeader().

138 { return pageSize_; }

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::read ( int8_t *const  dst,
const size_t  numBytes = 0,
const size_t  offset = 0,
const MemoryLevel  dstMemoryLevel = CPU_LEVEL,
const int32_t  deviceId = -1 
)
override

Definition at line 283 of file FileBuffer.cpp.

References threading_serial::async(), CHECK, Data_Namespace::CPU_LEVEL, logger::FATAL, fm_, getMultiPage(), File_Namespace::FileMgr::getNumReaderThreads(), i, LOG, File_Namespace::readThreadDS::multiPages, multiPages_, pageDataSize_, File_Namespace::readForThread(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.

287  {
288  if (dstBufferType != CPU_LEVEL) {
289  LOG(FATAL) << "Unsupported Buffer type";
290  }
291 
292  // variable declarations
293  size_t startPage = offset / pageDataSize_;
294  size_t startPageOffset = offset % pageDataSize_;
295  size_t numPagesToRead =
296  (numBytes + startPageOffset + pageDataSize_ - 1) / pageDataSize_;
297  /*
298  if (startPage + numPagesToRead > multiPages_.size()) {
299  cout << "Start page: " << startPage << endl;
300  cout << "Num pages to read: " << numPagesToRead << endl;
301  cout << "Num multipages: " << multiPages_.size() << endl;
302  cout << "Offset: " << offset << endl;
303  cout << "Num bytes: " << numBytes << endl;
304  }
305  */
306 
307  CHECK(startPage + numPagesToRead <= multiPages_.size());
308 
309  size_t numPagesPerThread = 0;
310  size_t numBytesCurrent = numBytes; // total number of bytes still to be read
311  size_t bytesRead = 0; // total number of bytes already being read
312  size_t bytesLeftForThread = 0; // number of bytes to be read in the thread
313  size_t numExtraPages = 0; // extra pages to be assigned one per thread as needed
314  size_t numThreads = fm_->getNumReaderThreads();
315  std::vector<readThreadDS>
316  threadDSArr; // array of threadDS, needed to avoid racing conditions
317 
318  if (numPagesToRead > numThreads) {
319  numPagesPerThread = numPagesToRead / numThreads;
320  numExtraPages = numPagesToRead - (numThreads * numPagesPerThread);
321  } else {
322  numThreads = numPagesToRead;
323  numPagesPerThread = 1;
324  }
325 
326  /* set threadDS for the first thread */
327  readThreadDS threadDS;
328  threadDS.t_fm = fm_;
329  threadDS.t_startPage = offset / pageDataSize_;
330  if (numExtraPages > 0) {
331  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread + 1;
332  numExtraPages--;
333  } else {
334  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread;
335  }
336  threadDS.t_curPtr = dst;
337  threadDS.t_startPageOffset = offset % pageDataSize_;
338  threadDS.t_isFirstPage = true;
339 
340  bytesLeftForThread = min(((threadDS.t_endPage - threadDS.t_startPage) * pageDataSize_ -
341  threadDS.t_startPageOffset),
342  numBytesCurrent);
343  threadDS.t_bytesLeft = bytesLeftForThread;
344  threadDS.multiPages = getMultiPage();
345 
346  if (numThreads == 1) {
347  bytesRead += readForThread(this, threadDS);
348  } else {
349  std::vector<std::future<size_t>> threads;
350 
351  for (size_t i = 0; i < numThreads; i++) {
352  threadDSArr.push_back(threadDS);
353  threads.push_back(
354  std::async(std::launch::async, readForThread, this, threadDSArr[i]));
355 
356  // calculate elements of threadDS
357  threadDS.t_fm = fm_;
358  threadDS.t_isFirstPage = false;
359  threadDS.t_curPtr += bytesLeftForThread;
360  threadDS.t_startPage +=
361  threadDS.t_endPage -
362  threadDS.t_startPage; // based on # of pages read on previous iteration
363  if (numExtraPages > 0) {
364  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread + 1;
365  numExtraPages--;
366  } else {
367  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread;
368  }
369  numBytesCurrent -= bytesLeftForThread;
370  bytesLeftForThread = min(
371  ((threadDS.t_endPage - threadDS.t_startPage) * pageDataSize_), numBytesCurrent);
372  threadDS.t_bytesLeft = bytesLeftForThread;
373  threadDS.multiPages = getMultiPage();
374  }
375 
376  for (auto& p : threads) {
377  p.wait();
378  }
379  for (auto& p : threads) {
380  bytesRead += p.get();
381  }
382  }
383  CHECK(bytesRead == numBytes);
384 }
virtual std::vector< MultiPage > getMultiPage() const
Returns vector of MultiPages in the FileBuffer.
Definition: FileBuffer.h:148
static size_t readForThread(FileBuffer *fileBuffer, const readThreadDS threadDS)
Definition: FileBuffer.cpp:242
#define LOG(tag)
Definition: Logger.h:203
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194
future< Result > async(Fn &&fn, Args &&...args)
#define CHECK(condition)
Definition: Logger.h:209
size_t getNumReaderThreads()
Returns number of threads defined by parameter num-reader-threads which should be used during initial...
Definition: FileMgr.h:306

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::readMetadata ( const Page page)
private

Definition at line 433 of file FileBuffer.cpp.

References CHECK, Data_Namespace::AbstractBuffer::encoder_, f, File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileForFileId(), Data_Namespace::AbstractBuffer::initEncoder(), METADATA_PAGE_SIZE, METADATA_VERSION, NUM_METADATA, File_Namespace::Page::pageNum, pageSize_, reservedHeaderSize_, SQLTypeInfo::set_comp_param(), SQLTypeInfo::set_compression(), SQLTypeInfo::set_dimension(), SQLTypeInfo::set_notnull(), SQLTypeInfo::set_scale(), SQLTypeInfo::set_size(), SQLTypeInfo::set_subtype(), SQLTypeInfo::set_type(), Data_Namespace::AbstractBuffer::size_, Data_Namespace::AbstractBuffer::sql_type_, and setup::version.

Referenced by initMetadataAndPageDataSize().

433  {
434  FILE* f = fm_->getFileForFileId(page.fileId);
435  fseek(f, page.pageNum * METADATA_PAGE_SIZE + reservedHeaderSize_, SEEK_SET);
436  fread((int8_t*)&pageSize_, sizeof(size_t), 1, f);
437  fread((int8_t*)&size_, sizeof(size_t), 1, f);
438  vector<int32_t> typeData(
439  NUM_METADATA); // assumes we will encode hasEncoder, bufferType,
440  // encodingType, encodingBits all as int
441  fread((int8_t*)&(typeData[0]), sizeof(int32_t), typeData.size(), f);
442  int32_t version = typeData[0];
443  CHECK(version == METADATA_VERSION); // add backward compatibility code here
444  bool has_encoder = static_cast<bool>(typeData[1]);
445  if (has_encoder) {
446  sql_type_.set_type(static_cast<SQLTypes>(typeData[2]));
447  sql_type_.set_subtype(static_cast<SQLTypes>(typeData[3]));
448  sql_type_.set_dimension(typeData[4]);
449  sql_type_.set_scale(typeData[5]);
450  sql_type_.set_notnull(static_cast<bool>(typeData[6]));
451  sql_type_.set_compression(static_cast<EncodingType>(typeData[7]));
452  sql_type_.set_comp_param(typeData[8]);
453  sql_type_.set_size(typeData[9]);
455  encoder_->readMetadata(f);
456  }
457 }
void set_compression(EncodingType c)
Definition: sqltypes.h:429
void set_size(int s)
Definition: sqltypes.h:427
#define METADATA_PAGE_SIZE
Definition: FileBuffer.h:37
HOST DEVICE void set_subtype(SQLTypes st)
Definition: sqltypes.h:420
void initEncoder(const SQLTypeInfo &tmp_sql_type)
void set_scale(int s)
Definition: sqltypes.h:424
FILE * getFileForFileId(const int32_t fileId)
Returns FILE pointer associated with requested fileId.
Definition: FileMgr.cpp:964
void set_comp_param(int p)
Definition: sqltypes.h:430
string version
Definition: setup.in.py:73
#define NUM_METADATA
Definition: FileBuffer.h:35
void set_dimension(int d)
Definition: sqltypes.h:421
std::unique_ptr< Encoder > encoder_
void set_notnull(bool n)
Definition: sqltypes.h:426
#define CHECK(condition)
Definition: Logger.h:209
char * f
#define METADATA_VERSION
Definition: FileBuffer.h:36
HOST DEVICE void set_type(SQLTypes t)
Definition: sqltypes.h:419

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::reserve ( const size_t  numBytes)
overridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 133 of file FileBuffer.cpp.

References addNewMultiPage(), getFileMgrEpoch(), multiPages_, pageSize_, and writeHeader().

133  {
134  size_t numPagesRequested = (numBytes + pageSize_ - 1) / pageSize_;
135  size_t numCurrentPages = multiPages_.size();
136  auto epoch = getFileMgrEpoch();
137  for (size_t pageNum = numCurrentPages; pageNum < numPagesRequested; ++pageNum) {
138  Page page = addNewMultiPage(epoch);
139  writeHeader(page, pageNum, epoch);
140  }
141 }
Page addNewMultiPage(const int32_t epoch)
Definition: FileBuffer.cpp:406
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:414

+ Here is the call graph for this function:

virtual size_t File_Namespace::FileBuffer::reservedHeaderSize ( ) const
inlinevirtual

Returns the size in bytes of the reserved header portion of each page in the FileBuffer.

Definition at line 145 of file FileBuffer.h.

Referenced by File_Namespace::readForThread().

145 { return reservedHeaderSize_; }

+ Here is the caller graph for this function:

size_t File_Namespace::FileBuffer::reservedSize ( ) const
inlineoverridevirtual

Returns the total number of bytes allocated for the FileBuffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 152 of file FileBuffer.h.

152 { return multiPages_.size() * pageSize_; }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194
void File_Namespace::FileBuffer::write ( int8_t *  src,
const size_t  numBytes,
const size_t  offset = 0,
const MemoryLevel  srcMemoryLevel = CPU_LEVEL,
const int32_t  deviceId = -1 
)
override

Writes the contents of source (src) into new versions of the affected logical pages.

This method will write the contents of source (src) into new version of the affected logical pages. New pages are only appended if the value of epoch (in FileMgr)

Definition at line 534 of file FileBuffer.cpp.

References addNewMultiPage(), CHECK, copyPage(), Data_Namespace::CPU_LEVEL, File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), getFileMgrEpoch(), multiPages_, pageDataSize_, File_Namespace::Page::pageNum, pageSize_, File_Namespace::FileMgr::requestFreePage(), reservedHeaderSize_, Data_Namespace::AbstractBuffer::setAppended(), Data_Namespace::AbstractBuffer::setDirty(), Data_Namespace::AbstractBuffer::setUpdated(), Data_Namespace::AbstractBuffer::size_, File_Namespace::FileInfo::write(), and writeHeader().

538  {
539  CHECK(srcBufferType == CPU_LEVEL) << "Unsupported Buffer type";
540 
541  bool tempIsAppended = false;
542  setDirty();
543  if (offset < size_) {
544  setUpdated();
545  }
546  if (offset + numBytes > size_) {
547  tempIsAppended = true; // because is_appended_ could have already been true - to
548  // avoid rewriting header
549  setAppended();
550  size_ = offset + numBytes;
551  }
552 
553  size_t startPage = offset / pageDataSize_;
554  size_t startPageOffset = offset % pageDataSize_;
555  size_t numPagesToWrite =
556  (numBytes + startPageOffset + pageDataSize_ - 1) / pageDataSize_;
557  size_t bytesLeft = numBytes;
558  int8_t* curPtr = src; // a pointer to the current location in dst being written to
559  size_t initialNumPages = multiPages_.size();
560  auto epoch = getFileMgrEpoch();
561 
562  if (startPage >
563  initialNumPages) { // means there is a gap we need to allocate pages for
564  for (size_t pageNum = initialNumPages; pageNum < startPage; ++pageNum) {
565  Page page = addNewMultiPage(epoch);
566  writeHeader(page, pageNum, epoch);
567  }
568  }
569  for (size_t pageNum = startPage; pageNum < startPage + numPagesToWrite; ++pageNum) {
570  Page page;
571  if (pageNum >= initialNumPages) {
572  page = addNewMultiPage(epoch);
573  writeHeader(page, pageNum, epoch);
574  } else if (multiPages_[pageNum].current().epoch <
575  epoch) { // need to create new page b/c this current one lags epoch and we
576  // can't overwrite it also need to copy if we are on first or
577  // last page
578  Page lastPage = multiPages_[pageNum].current().page;
579  page = fm_->requestFreePage(pageSize_, false);
580  multiPages_[pageNum].push(page, epoch);
581  if (pageNum == startPage && startPageOffset > 0) {
582  // copyPage takes care of header offset so don't worry
583  // about it
584  copyPage(lastPage, page, startPageOffset, 0);
585  }
586  if (pageNum == (startPage + numPagesToWrite - 1) &&
587  bytesLeft > 0) { // bytesLeft should always > 0
588  copyPage(lastPage,
589  page,
590  pageDataSize_ - bytesLeft,
591  bytesLeft); // these would be empty if we're appending but we won't
592  // worry about it right now
593  }
594  writeHeader(page, pageNum, epoch);
595  } else {
596  // we already have a new page at current
597  // epoch for this page - just grab this page
598  page = multiPages_[pageNum].current().page;
599  }
600  CHECK(page.fileId >= 0); // make sure page was initialized
601  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
602  size_t bytesWritten;
603  if (pageNum == startPage) {
604  bytesWritten = fileInfo->write(
605  page.pageNum * pageSize_ + startPageOffset + reservedHeaderSize_,
606  min(pageDataSize_ - startPageOffset, bytesLeft),
607  curPtr);
608  } else {
609  bytesWritten = fileInfo->write(page.pageNum * pageSize_ + reservedHeaderSize_,
610  min(pageDataSize_, bytesLeft),
611  curPtr);
612  }
613  curPtr += bytesWritten;
614  bytesLeft -= bytesWritten;
615  if (tempIsAppended && pageNum == startPage + numPagesToWrite - 1) { // if last page
616  //@todo below can lead to undefined - we're overwriting num
617  // bytes valid at checkpoint
618  writeHeader(page, 0, multiPages_[0].current().epoch, true);
619  }
620  }
621  CHECK(bytesLeft == 0);
622 }
size_t write(const size_t offset, const size_t size, const int8_t *buf)
Definition: FileInfo.cpp:60
virtual Page requestFreePage(size_t pagesize, const bool isMetadata)
Definition: FileMgr.cpp:851
Page addNewMultiPage(const int32_t epoch)
Definition: FileBuffer.cpp:406
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:194
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:414
#define CHECK(condition)
Definition: Logger.h:209
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:218
void copyPage(Page &srcPage, Page &destPage, const size_t numBytes, const size_t offset=0)
Definition: FileBuffer.cpp:386

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::writeHeader ( Page page,
const int32_t  pageId,
const int32_t  epoch,
const bool  writeMetadata = false 
)
private

Write header writes header at top of page in format.

Definition at line 414 of file FileBuffer.cpp.

References chunkKey_, gpu_enabled::copy(), File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), METADATA_PAGE_SIZE, File_Namespace::Page::pageNum, pageSize(), pageSize_, and File_Namespace::FileInfo::write().

Referenced by append(), reserve(), write(), and writeMetadata().

417  {
418  int32_t intHeaderSize = chunkKey_.size() + 3; // does not include chunkSize
419  vector<int32_t> header(intHeaderSize);
420  // in addition to chunkkey we need size of header, pageId, version
421  header[0] =
422  (intHeaderSize - 1) * sizeof(int32_t); // don't need to include size of headerSize
423  // value - sizeof(size_t) is for chunkSize
424  std::copy(chunkKey_.begin(), chunkKey_.end(), header.begin() + 1);
425  header[intHeaderSize - 2] = pageId;
426  header[intHeaderSize - 1] = epoch;
427  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
429  fileInfo->write(
430  page.pageNum * pageSize, (intHeaderSize) * sizeof(int32_t), (int8_t*)&header[0]);
431 }
#define METADATA_PAGE_SIZE
Definition: FileBuffer.h:37
void writeMetadata(const int32_t epoch)
Definition: FileBuffer.cpp:459
DEVICE auto copy(ARGS &&...args)
Definition: gpu_enabled.h:51
size_t pageSize() const override
Returns the size in bytes of each page in the FileBuffer.
Definition: FileBuffer.h:138
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:218

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::writeMetadata ( const int32_t  epoch)
private

Definition at line 459 of file FileBuffer.cpp.

References Data_Namespace::AbstractBuffer::encoder_, f, File_Namespace::Page::fileId, fm_, SQLTypeInfo::get_comp_param(), SQLTypeInfo::get_compression(), SQLTypeInfo::get_dimension(), SQLTypeInfo::get_notnull(), SQLTypeInfo::get_scale(), SQLTypeInfo::get_size(), SQLTypeInfo::get_subtype(), SQLTypeInfo::get_type(), File_Namespace::FileMgr::getFileForFileId(), Data_Namespace::AbstractBuffer::hasEncoder(), METADATA_PAGE_SIZE, METADATA_VERSION, metadataPages_, NUM_METADATA, File_Namespace::Page::pageNum, pageSize_, File_Namespace::MultiPage::push(), File_Namespace::FileMgr::requestFreePage(), reservedHeaderSize_, Data_Namespace::AbstractBuffer::size_, Data_Namespace::AbstractBuffer::sql_type_, and writeHeader().

459  {
460  // Right now stats page is size_ (in bytes), bufferType, encodingType,
461  // encodingDataType, numElements
462  Page page = fm_->requestFreePage(METADATA_PAGE_SIZE, true);
463  writeHeader(page, -1, epoch, true);
464  FILE* f = fm_->getFileForFileId(page.fileId);
465  fseek(f, page.pageNum * METADATA_PAGE_SIZE + reservedHeaderSize_, SEEK_SET);
466  fwrite((int8_t*)&pageSize_, sizeof(size_t), 1, f);
467  fwrite((int8_t*)&size_, sizeof(size_t), 1, f);
468  vector<int32_t> typeData(
469  NUM_METADATA); // assumes we will encode hasEncoder, bufferType,
470  // encodingType, encodingBits all as int32_t
471  typeData[0] = METADATA_VERSION;
472  typeData[1] = static_cast<int32_t>(hasEncoder());
473  if (hasEncoder()) {
474  typeData[2] = static_cast<int32_t>(sql_type_.get_type());
475  typeData[3] = static_cast<int32_t>(sql_type_.get_subtype());
476  typeData[4] = sql_type_.get_dimension();
477  typeData[5] = sql_type_.get_scale();
478  typeData[6] = static_cast<int32_t>(sql_type_.get_notnull());
479  typeData[7] = static_cast<int32_t>(sql_type_.get_compression());
480  typeData[8] = sql_type_.get_comp_param();
481  typeData[9] = sql_type_.get_size();
482  }
483  fwrite((int8_t*)&(typeData[0]), sizeof(int32_t), typeData.size(), f);
484  if (hasEncoder()) { // redundant
485  encoder_->writeMetadata(f);
486  }
487  metadataPages_.push(page, epoch);
488 }
HOST DEVICE SQLTypes get_subtype() const
Definition: sqltypes.h:330
#define METADATA_PAGE_SIZE
Definition: FileBuffer.h:37
HOST DEVICE int get_size() const
Definition: sqltypes.h:339
virtual Page requestFreePage(size_t pagesize, const bool isMetadata)
Definition: FileMgr.cpp:851
HOST DEVICE int get_scale() const
Definition: sqltypes.h:334
HOST DEVICE SQLTypes get_type() const
Definition: sqltypes.h:329
FILE * getFileForFileId(const int32_t fileId)
Returns FILE pointer associated with requested fileId.
Definition: FileMgr.cpp:964
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:414
HOST DEVICE EncodingType get_compression() const
Definition: sqltypes.h:337
#define NUM_METADATA
Definition: FileBuffer.h:35
void push(const Page &page, const int epoch)
Pushes a new page with epoch value.
Definition: Page.h:102
HOST DEVICE int get_dimension() const
Definition: sqltypes.h:331
std::unique_ptr< Encoder > encoder_
HOST DEVICE int get_comp_param() const
Definition: sqltypes.h:338
char * f
HOST DEVICE bool get_notnull() const
Definition: sqltypes.h:336
#define METADATA_VERSION
Definition: FileBuffer.h:36

+ Here is the call graph for this function:

Friends And Related Function Documentation

friend class CachingFileMgr
friend

Definition at line 60 of file FileBuffer.h.

friend class FileMgr
friend

Definition at line 59 of file FileBuffer.h.

Member Data Documentation

ChunkKey File_Namespace::FileBuffer::chunkKey_
private
constexpr size_t File_Namespace::FileBuffer::headerBufferOffset_ = 32
static

Definition at line 166 of file FileBuffer.h.

Referenced by calcHeaderBuffer().

MultiPage File_Namespace::FileBuffer::metadataPages_
private
std::vector<MultiPage> File_Namespace::FileBuffer::multiPages_
private
size_t File_Namespace::FileBuffer::pageDataSize_
private
size_t File_Namespace::FileBuffer::pageSize_
private
size_t File_Namespace::FileBuffer::reservedHeaderSize_
private

The documentation for this class was generated from the following files: