OmniSciDB  c1a53651b2
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
File_Namespace::FileBuffer Class Reference

Represents/provides access to contiguous data stored in the file system. More...

#include <FileBuffer.h>

+ Inheritance diagram for File_Namespace::FileBuffer:
+ Collaboration diagram for File_Namespace::FileBuffer:

Public Member Functions

 FileBuffer (FileMgr *fm, const size_t pageSize, const ChunkKey &chunkKey, const size_t initialSize=0)
 Constructs a FileBuffer object. More...
 
 FileBuffer (FileMgr *fm, const size_t pageSize, const ChunkKey &chunkKey, const SQLTypeInfo sqlType, const size_t initialSize=0)
 
 FileBuffer (FileMgr *fm, const ChunkKey &chunkKey, const std::vector< HeaderInfo >::const_iterator &headerStartIt, const std::vector< HeaderInfo >::const_iterator &headerEndIt)
 
 ~FileBuffer () override
 Destructor. More...
 
Page addNewMultiPage (const int32_t epoch)
 
void reserve (const size_t numBytes) override
 
size_t freeMetadataPages ()
 
size_t freeChunkPages ()
 
size_t freePages ()
 
void freePagesBeforeEpoch (const int32_t targetEpoch)
 
void read (int8_t *const dst, const size_t numBytes=0, const size_t offset=0, const MemoryLevel dstMemoryLevel=CPU_LEVEL, const int32_t deviceId=-1) override
 
void write (int8_t *src, const size_t numBytes, const size_t offset=0, const MemoryLevel srcMemoryLevel=CPU_LEVEL, const int32_t deviceId=-1) override
 Writes the contents of source (src) into new versions of the affected logical pages. More...
 
void append (int8_t *src, const size_t numBytes, const MemoryLevel srcMemoryLevel=CPU_LEVEL, const int32_t deviceId=-1) override
 
void copyPage (Page &srcPage, Page &destPage, const size_t numBytes, const size_t offset=0)
 
Data_Namespace::MemoryLevel getType () const override
 
int8_t * getMemoryPtr () override
 Not implemented for FileMgr – throws a runtime_error. More...
 
size_t pageCount () const override
 Returns the number of pages in the FileBuffer. More...
 
bool hasDataPages () const
 
size_t pageSize () const override
 Returns the size in bytes of each page in the FileBuffer. More...
 
virtual size_t pageDataSize () const
 Returns the size in bytes of the data portion of each page in the FileBuffer. More...
 
virtual size_t reservedHeaderSize () const
 
virtual std::vector< MultiPagegetMultiPage () const
 Returns vector of MultiPages in the FileBuffer. More...
 
MultiPage getMetadataPage () const
 
size_t reservedSize () const override
 Returns the total number of bytes allocated for the FileBuffer. More...
 
size_t numMetadataPages () const
 Returns the total number of used bytes in the FileBuffer. More...
 
bool isMissingPages () const
 
size_t numChunkPages () const
 
std::string dump () const
 
void freePage (const Page &page)
 
- Public Member Functions inherited from Data_Namespace::AbstractBuffer
 AbstractBuffer (const int device_id)
 
 AbstractBuffer (const int device_id, const SQLTypeInfo sql_type)
 
virtual ~AbstractBuffer ()
 
virtual void read (int8_t *const dst, const size_t num_bytes, const size_t offset=0, const MemoryLevel dst_buffer_type=CPU_LEVEL, const int dst_device_id=-1)=0
 
virtual void write (int8_t *src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int src_device_id=-1)=0
 
virtual void append (int8_t *src, const size_t num_bytes, const MemoryLevel src_buffer_type=CPU_LEVEL, const int device_id=-1)=0
 
virtual void setMemoryPtr (int8_t *new_ptr)
 
virtual int pin ()
 
virtual int unPin ()
 
virtual int getPinCount ()
 
size_t size () const
 
int getDeviceId () const
 
bool isDirty () const
 
bool isAppended () const
 
bool isUpdated () const
 
bool hasEncoder () const
 
SQLTypeInfo getSqlType () const
 
void setSqlType (const SQLTypeInfo &sql_type)
 
EncodergetEncoder () const
 
void setDirty ()
 
void setUpdated ()
 
void setAppended ()
 
void setSize (const size_t size)
 
void clearDirtyBits ()
 
void initEncoder (const SQLTypeInfo &tmp_sql_type)
 
void syncEncoder (const AbstractBuffer *src_buffer)
 
void copyTo (AbstractBuffer *destination_buffer, const size_t num_bytes=0)
 
void resetToEmpty ()
 

Static Public Attributes

static constexpr size_t headerBufferOffset_ = 32
 

Private Member Functions

void writeHeader (Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
 Write header writes header at top of page in format. More...
 
void writeMetadata (const int32_t epoch)
 
void readMetadata (const Page &page)
 
void calcHeaderBuffer ()
 
void freePage (const Page &page, const bool isRolloff)
 
void freePagesBeforeEpochForMultiPage (MultiPage &multiPage, const int32_t targetEpoch, const int32_t currentEpoch)
 
void initMetadataAndPageDataSize ()
 
int32_t getFileMgrEpoch ()
 

Private Attributes

FileMgrfm_
 
const size_t metadataPageSize_
 
MultiPage metadataPages_
 
std::vector< MultiPagemultiPages_
 
size_t pageSize_
 
size_t pageDataSize_
 
size_t reservedHeaderSize_
 
ChunkKey chunkKey_
 

Friends

class FileMgr
 
class CachingFileMgr
 

Additional Inherited Members

- Protected Attributes inherited from Data_Namespace::AbstractBuffer
std::unique_ptr< Encoderencoder_
 
SQLTypeInfo sql_type_
 
size_t size_
 
int device_id_
 

Detailed Description

Represents/provides access to contiguous data stored in the file system.

The FileBuffer consists of logical pages, which can map to any identically-sized page in any file of the underlying file system. A page's metadata (file and page number) are stored in MultiPage objects, and each MultiPage includes page metadata for multiple versions of the same page.

Note that a "Chunk" is brought into a FileBuffer by the FileMgr.

Note(s): Forbid Copying Idiom 4.1

Definition at line 57 of file FileBuffer.h.

Constructor & Destructor Documentation

File_Namespace::FileBuffer::FileBuffer ( FileMgr fm,
const size_t  pageSize,
const ChunkKey chunkKey,
const size_t  initialSize = 0 
)

Constructs a FileBuffer object.

Definition at line 38 of file FileBuffer.cpp.

References calcHeaderBuffer(), CHECK, CHECK_GT, fm_, pageDataSize_, pageSize_, and reservedHeaderSize_.

42  : AbstractBuffer(fm->getDeviceId())
43  , fm_(fm)
47  , chunkKey_(chunkKey) {
48  // Create a new FileBuffer
49  CHECK(fm_);
53  //@todo reintroduce initialSize - need to develop easy way of
54  // differentiating these pre-allocated pages from "written-to" pages
55  /*
56  if (initalSize > 0) {
57  // should expand to initialSize bytes
58  size_t initialNumPages = (initalSize + pageSize_ -1) / pageSize_;
59  int32_t epoch = fm_->epoch();
60  for (size_t pageNum = 0; pageNum < initialNumPages; ++pageNum) {
61  Page page = addNewMultiPage(epoch);
62  writeHeader(page,pageNum,epoch);
63  }
64  }
65  */
66 }
const size_t metadataPageSize_
Definition: FileBuffer.h:196
#define CHECK_GT(x, y)
Definition: Logger.h:305
AbstractBuffer(const int device_id)
size_t pageSize() const override
Returns the size in bytes of each page in the FileBuffer.
Definition: FileBuffer.h:137
#define CHECK(condition)
Definition: Logger.h:291
size_t getMetadataPageSize() const
Definition: FileMgr.h:366

+ Here is the call graph for this function:

File_Namespace::FileBuffer::FileBuffer ( FileMgr fm,
const size_t  pageSize,
const ChunkKey chunkKey,
const SQLTypeInfo  sqlType,
const size_t  initialSize = 0 
)

Definition at line 68 of file FileBuffer.cpp.

References calcHeaderBuffer(), CHECK, fm_, pageDataSize_, pageSize_, and reservedHeaderSize_.

73  : AbstractBuffer(fm->getDeviceId(), sqlType)
74  , fm_(fm)
75  , metadataPageSize_(fm->getMetadataPageSize())
78  , chunkKey_(chunkKey) {
79  CHECK(fm_);
82 }
const size_t metadataPageSize_
Definition: FileBuffer.h:196
AbstractBuffer(const int device_id)
size_t pageSize() const override
Returns the size in bytes of each page in the FileBuffer.
Definition: FileBuffer.h:137
#define CHECK(condition)
Definition: Logger.h:291

+ Here is the call graph for this function:

File_Namespace::FileBuffer::FileBuffer ( FileMgr fm,
const ChunkKey chunkKey,
const std::vector< HeaderInfo >::const_iterator &  headerStartIt,
const std::vector< HeaderInfo >::const_iterator &  headerEndIt 
)

Definition at line 84 of file FileBuffer.cpp.

References calcHeaderBuffer(), CHECK, File_Namespace::FileMgr::failOnReadError(), logger::FATAL, fm_, initMetadataAndPageDataSize(), LOG, metadataPages_, multiPages_, pageSize_, File_Namespace::MultiPage::push(), and show_chunk().

88  : AbstractBuffer(fm->getDeviceId())
89  , fm_(fm)
90  , metadataPageSize_(fm->getMetadataPageSize())
92  , pageSize_(0)
93  , chunkKey_(chunkKey) {
94  // We are being assigned an existing FileBuffer on disk
95 
96  CHECK(fm_);
98  int32_t lastPageId = -1;
99  int32_t curPageId = 0;
100  for (auto vecIt = headerStartIt; vecIt != headerEndIt; ++vecIt) {
101  curPageId = vecIt->pageId;
102 
103  // We only want to read last metadata page
104  if (curPageId == -1) { // stats page
105  metadataPages_.push(vecIt->page, vecIt->versionEpoch);
106  } else {
107  if (curPageId != lastPageId) {
108  // protect from bad data on disk, and give diagnostics
109  if (fm->failOnReadError()) {
110  if (curPageId != lastPageId + 1) {
111  LOG(FATAL) << "Failure reading DB file " << show_chunk(chunkKey)
112  << " Current page " << curPageId << " last page " << lastPageId
113  << " epoch " << vecIt->versionEpoch;
114  }
115  }
116  if (lastPageId == -1) { // If we are on first real page
118  }
119  MultiPage multiPage(pageSize_);
120  multiPages_.push_back(multiPage);
121  lastPageId = curPageId;
122  }
123  multiPages_.back().push(vecIt->page, vecIt->versionEpoch);
124  }
125  }
126  if (curPageId == -1) { // meaning there was only a metadata page
128  }
129 }
#define LOG(tag)
Definition: Logger.h:285
const size_t metadataPageSize_
Definition: FileBuffer.h:196
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
std::string show_chunk(const ChunkKey &key)
Definition: types.h:98
AbstractBuffer(const int device_id)
void push(const Page &page, const int epoch)
Pushes a new page with epoch value.
Definition: Page.h:102
#define CHECK(condition)
Definition: Logger.h:291

+ Here is the call graph for this function:

File_Namespace::FileBuffer::~FileBuffer ( )
override

Destructor.

Definition at line 131 of file FileBuffer.cpp.

131  {
132  // need to free pages
133  // NOP
134 }

Member Function Documentation

Page File_Namespace::FileBuffer::addNewMultiPage ( const int32_t  epoch)

Definition at line 409 of file FileBuffer.cpp.

References fm_, multiPages_, pageSize_, File_Namespace::MultiPage::push(), and File_Namespace::FileMgr::requestFreePage().

Referenced by append(), reserve(), and write().

409  {
410  Page page = fm_->requestFreePage(pageSize_, false);
411  MultiPage multiPage(pageSize_);
412  multiPage.push(page, epoch);
413  multiPages_.emplace_back(multiPage);
414  return page;
415 }
virtual Page requestFreePage(size_t pagesize, const bool isMetadata)
Definition: FileMgr.cpp:875
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::append ( int8_t *  src,
const size_t  numBytes,
const MemoryLevel  srcMemoryLevel = CPU_LEVEL,
const int32_t  deviceId = -1 
)
override

Definition at line 493 of file FileBuffer.cpp.

References addNewMultiPage(), CHECK, File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), getFileMgrEpoch(), multiPages_, pageDataSize_, File_Namespace::Page::pageNum, pageSize_, reservedHeaderSize_, Data_Namespace::AbstractBuffer::setAppended(), Data_Namespace::AbstractBuffer::size_, File_Namespace::FileInfo::write(), and writeHeader().

496  {
497  setAppended();
498 
499  size_t startPage = size_ / pageDataSize_;
500  size_t startPageOffset = size_ % pageDataSize_;
501  size_t numPagesToWrite =
502  (numBytes + startPageOffset + pageDataSize_ - 1) / pageDataSize_;
503  size_t bytesLeft = numBytes;
504  int8_t* curPtr = src; // a pointer to the current location in dst being written to
505  size_t initialNumPages = multiPages_.size();
506  size_ = size_ + numBytes;
507  auto epoch = getFileMgrEpoch();
508  for (size_t pageNum = startPage; pageNum < startPage + numPagesToWrite; ++pageNum) {
509  Page page;
510  if (pageNum >= initialNumPages) {
511  page = addNewMultiPage(epoch);
512  writeHeader(page, pageNum, epoch);
513  } else {
514  // we already have a new page at current
515  // epoch for this page - just grab this page
516  page = multiPages_[pageNum].current().page;
517  }
518  CHECK(page.fileId >= 0); // make sure page was initialized
519  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
520  size_t bytesWritten;
521  if (pageNum == startPage) {
522  bytesWritten = fileInfo->write(
523  page.pageNum * pageSize_ + startPageOffset + reservedHeaderSize_,
524  min(pageDataSize_ - startPageOffset, bytesLeft),
525  curPtr);
526  } else {
527  bytesWritten = fileInfo->write(page.pageNum * pageSize_ + reservedHeaderSize_,
528  min(pageDataSize_, bytesLeft),
529  curPtr);
530  }
531  curPtr += bytesWritten;
532  bytesLeft -= bytesWritten;
533  }
534  CHECK(bytesLeft == 0);
535 }
size_t write(const size_t offset, const size_t size, const int8_t *buf)
Definition: FileInfo.cpp:69
Page addNewMultiPage(const int32_t epoch)
Definition: FileBuffer.cpp:409
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:417
#define CHECK(condition)
Definition: Logger.h:291
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:222

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::calcHeaderBuffer ( )
private

Definition at line 146 of file FileBuffer.cpp.

References chunkKey_, headerBufferOffset_, and reservedHeaderSize_.

Referenced by FileBuffer().

146  {
147  // 3 * sizeof(int32_t) is for headerSize, for pageId and versionEpoch
148  // sizeof(size_t) is for chunkSize
149  reservedHeaderSize_ = (chunkKey_.size() + 3) * sizeof(int32_t);
150  size_t headerMod = reservedHeaderSize_ % headerBufferOffset_;
151  if (headerMod > 0) {
153  }
154 }
static constexpr size_t headerBufferOffset_
Definition: FileBuffer.h:165

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::copyPage ( Page srcPage,
Page destPage,
const size_t  numBytes,
const size_t  offset = 0 
)

Definition at line 389 of file FileBuffer.cpp.

References CHECK, CHECK_LE, checked_malloc(), File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), pageDataSize_, File_Namespace::Page::pageNum, pageSize_, File_Namespace::FileInfo::read(), reservedHeaderSize_, and File_Namespace::FileInfo::write().

Referenced by write().

392  {
393  // FILE *srcFile = fm_->files_[srcPage.fileId]->f;
394  // FILE *destFile = fm_->files_[destPage.fileId]->f;
395  CHECK_LE(offset + numBytes, pageDataSize_);
396  FileInfo* srcFileInfo = fm_->getFileInfoForFileId(srcPage.fileId);
397  FileInfo* destFileInfo = fm_->getFileInfoForFileId(destPage.fileId);
398 
399  int8_t* buffer = reinterpret_cast<int8_t*>(checked_malloc(numBytes));
400  size_t bytesRead = srcFileInfo->read(
401  srcPage.pageNum * pageSize_ + offset + reservedHeaderSize_, numBytes, buffer);
402  CHECK(bytesRead == numBytes);
403  size_t bytesWritten = destFileInfo->write(
404  destPage.pageNum * pageSize_ + offset + reservedHeaderSize_, numBytes, buffer);
405  CHECK(bytesWritten == numBytes);
406  free(buffer);
407 }
void * checked_malloc(const size_t size)
Definition: checked_alloc.h:45
#define CHECK_LE(x, y)
Definition: Logger.h:304
size_t read(const size_t offset, const size_t size, int8_t *buf)
Definition: FileInfo.cpp:75
#define CHECK(condition)
Definition: Logger.h:291
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:222

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string File_Namespace::FileBuffer::dump ( ) const

Definition at line 632 of file FileBuffer.cpp.

References chunkKey_, Data_Namespace::AbstractBuffer::hasEncoder(), show_chunk(), and Data_Namespace::AbstractBuffer::size_.

632  {
633  std::stringstream ss;
634  ss << "chunk_key = " << show_chunk(chunkKey_) << "\n";
635  ss << "has_encoder = " << (hasEncoder() ? "true\n" : "false\n");
636  ss << "size_ = " << size_ << "\n";
637  return ss.str();
638 }
std::string show_chunk(const ChunkKey &key)
Definition: types.h:98

+ Here is the call graph for this function:

size_t File_Namespace::FileBuffer::freeChunkPages ( )

Definition at line 178 of file FileBuffer.cpp.

References freePage(), and multiPages_.

Referenced by freePages(), and freePagesBeforeEpoch().

178  {
179  size_t num_pages_freed = multiPages_.size();
180  for (auto multiPageIt = multiPages_.begin(); multiPageIt != multiPages_.end();
181  ++multiPageIt) {
182  for (auto pageIt = multiPageIt->pageVersions.begin();
183  pageIt != multiPageIt->pageVersions.end();
184  ++pageIt) {
185  freePage(pageIt->page, false /* isRolloff */);
186  }
187  }
188  multiPages_.clear();
189  return num_pages_freed;
190 }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
void freePage(const Page &page)
Definition: FileBuffer.cpp:156

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::FileBuffer::freeMetadataPages ( )

Definition at line 165 of file FileBuffer.cpp.

References freePage(), metadataPages_, File_Namespace::MultiPage::pageVersions, and File_Namespace::MultiPage::pop().

Referenced by freePages().

165  {
166  size_t num_pages_freed = metadataPages_.pageVersions.size();
167  for (auto metaPageIt = metadataPages_.pageVersions.begin();
168  metaPageIt != metadataPages_.pageVersions.end();
169  ++metaPageIt) {
170  freePage(metaPageIt->page, false /* isRolloff */);
171  }
172  while (metadataPages_.pageVersions.size() > 0) {
174  }
175  return num_pages_freed;
176 }
void pop()
Purges the oldest Page.
Definition: Page.h:110
std::deque< EpochedPage > pageVersions
Definition: Page.h:81
void freePage(const Page &page)
Definition: FileBuffer.cpp:156

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::freePage ( const Page page)

Definition at line 156 of file FileBuffer.cpp.

Referenced by freeChunkPages(), freeMetadataPages(), and freePagesBeforeEpochForMultiPage().

156  {
157  freePage(page, false);
158 }
void freePage(const Page &page)
Definition: FileBuffer.cpp:156

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::freePage ( const Page page,
const bool  isRolloff 
)
private

Definition at line 160 of file FileBuffer.cpp.

References File_Namespace::Page::fileId, fm_, File_Namespace::FileInfo::freePage(), File_Namespace::FileMgr::getFileInfoForFileId(), getFileMgrEpoch(), and File_Namespace::Page::pageNum.

160  {
161  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
162  fileInfo->freePage(page.pageNum, isRolloff, getFileMgrEpoch());
163 }
void freePage(int32_t pageId, const bool isRolloff, int32_t epoch)
Definition: FileInfo.cpp:191
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:222

+ Here is the call graph for this function:

size_t File_Namespace::FileBuffer::freePages ( )

Definition at line 192 of file FileBuffer.cpp.

References freeChunkPages(), and freeMetadataPages().

192  {
193  return freeMetadataPages() + freeChunkPages();
194 }

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::freePagesBeforeEpoch ( const int32_t  targetEpoch)

Definition at line 206 of file FileBuffer.cpp.

References CHECK_LE, chunkKey_, fm_, freeChunkPages(), freePagesBeforeEpochForMultiPage(), getFileMgrEpoch(), metadataPages_, multiPages_, pageSize_, File_Namespace::MultiPage::pageVersions, and Data_Namespace::AbstractBuffer::size_.

206  {
207  // This method is only safe to be called within a checkpoint, after the sync and epoch
208  // increment where a failure at any point32_t in the process would lead to a safe
209  // rollback
210  auto currentEpoch = getFileMgrEpoch();
211  CHECK_LE(targetEpoch, currentEpoch);
212  freePagesBeforeEpochForMultiPage(metadataPages_, targetEpoch, currentEpoch);
213  for (auto& multiPage : multiPages_) {
214  freePagesBeforeEpochForMultiPage(multiPage, targetEpoch, currentEpoch);
215  }
216 
217  // Check if all buffer pages can be freed
218  if (size_ == 0) {
219  size_t max_historical_buffer_size{0};
220  for (auto& epoch_page : metadataPages_.pageVersions) {
221  // Create buffer that is used to get the buffer size at the epoch version
222  FileBuffer buffer{fm_, pageSize_, chunkKey_};
223  buffer.readMetadata(epoch_page.page);
224  max_historical_buffer_size = std::max(max_historical_buffer_size, buffer.size());
225  }
226 
227  // Free all chunk pages, if none of the old chunk versions has any data
228  if (max_historical_buffer_size == 0) {
229  freeChunkPages();
230  }
231  }
232 }
void freePagesBeforeEpochForMultiPage(MultiPage &multiPage, const int32_t targetEpoch, const int32_t currentEpoch)
Definition: FileBuffer.cpp:196
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
std::deque< EpochedPage > pageVersions
Definition: Page.h:81
#define CHECK_LE(x, y)
Definition: Logger.h:304
FileBuffer(FileMgr *fm, const size_t pageSize, const ChunkKey &chunkKey, const size_t initialSize=0)
Constructs a FileBuffer object.
Definition: FileBuffer.cpp:38

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::freePagesBeforeEpochForMultiPage ( MultiPage multiPage,
const int32_t  targetEpoch,
const int32_t  currentEpoch 
)
private

Definition at line 196 of file FileBuffer.cpp.

References freePage(), and File_Namespace::MultiPage::freePagesBeforeEpoch().

Referenced by freePagesBeforeEpoch().

198  {
199  std::vector<EpochedPage> epochedPagesToFree =
200  multiPage.freePagesBeforeEpoch(targetEpoch, currentEpoch);
201  for (const auto& epochedPageToFree : epochedPagesToFree) {
202  freePage(epochedPageToFree.page, true /* isRolloff */);
203  }
204 }
void freePage(const Page &page)
Definition: FileBuffer.cpp:156

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

int32_t File_Namespace::FileBuffer::getFileMgrEpoch ( )
private

Definition at line 627 of file FileBuffer.cpp.

References chunkKey_, File_Namespace::FileMgr::epoch(), fm_, and get_table_prefix().

Referenced by append(), freePage(), freePagesBeforeEpoch(), reserve(), and write().

627  {
628  auto [db_id, tb_id] = get_table_prefix(chunkKey_);
629  return fm_->epoch(db_id, tb_id);
630 }
virtual int32_t epoch(int32_t db_id, int32_t tb_id) const
Returns current value of epoch - should be one greater than recorded at last checkpoint. Because FileMgr only contains buffers from one table we can just return the FileMgr&#39;s epoch instead of finding a table-specific epoch.
Definition: FileMgr.h:277
std::pair< int, int > get_table_prefix(const ChunkKey &key)
Definition: types.h:62

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

int8_t* File_Namespace::FileBuffer::getMemoryPtr ( )
inlineoverridevirtual

Not implemented for FileMgr – throws a runtime_error.

Implements Data_Namespace::AbstractBuffer.

Definition at line 124 of file FileBuffer.h.

References logger::FATAL, and LOG.

124  {
125  LOG(FATAL) << "Operation not supported.";
126  return nullptr; // satisfy return-type warning
127  }
#define LOG(tag)
Definition: Logger.h:285
MultiPage File_Namespace::FileBuffer::getMetadataPage ( ) const
inline

Definition at line 148 of file FileBuffer.h.

148 { return metadataPages_; }
virtual std::vector<MultiPage> File_Namespace::FileBuffer::getMultiPage ( ) const
inlinevirtual

Returns vector of MultiPages in the FileBuffer.

Definition at line 147 of file FileBuffer.h.

Referenced by read().

147 { return multiPages_; }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198

+ Here is the caller graph for this function:

Data_Namespace::MemoryLevel File_Namespace::FileBuffer::getType ( ) const
inlineoverridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 121 of file FileBuffer.h.

References Data_Namespace::DISK_LEVEL.

bool File_Namespace::FileBuffer::hasDataPages ( ) const
inline

Returns whether or not a buffer has data pages. It is possible for a buffer to represent metadata (have a size and encode) but not contain actual data.

Definition at line 134 of file FileBuffer.h.

134 { return pageCount() > 0; }
size_t pageCount() const override
Returns the number of pages in the FileBuffer.
Definition: FileBuffer.h:130
void File_Namespace::FileBuffer::initMetadataAndPageDataSize ( )
private

Definition at line 640 of file FileBuffer.cpp.

References CHECK, File_Namespace::MultiPage::current(), File_Namespace::Page::fileId, metadataPages_, File_Namespace::EpochedPage::page, pageDataSize_, pageSize_, readMetadata(), and reservedHeaderSize_.

Referenced by FileBuffer().

640  {
641  CHECK(metadataPages_.current().page.fileId != -1); // was initialized
644 }
int32_t fileId
Definition: Page.h:47
#define CHECK(condition)
Definition: Logger.h:291
void readMetadata(const Page &page)
Definition: FileBuffer.cpp:436
EpochedPage current() const
Returns a reference to the most recent version of the page.
Definition: Page.h:94

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

bool File_Namespace::FileBuffer::isMissingPages ( ) const

Definition at line 646 of file FileBuffer.cpp.

References multiPages_, pageDataSize_, and Data_Namespace::AbstractBuffer::size().

646  {
647  // Detect the case where a page is missing by comparing the amount of pages read
648  // with the metadata size.
649  return ((size() + pageDataSize_ - 1) / pageDataSize_ != multiPages_.size());
650 }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198

+ Here is the call graph for this function:

size_t File_Namespace::FileBuffer::numChunkPages ( ) const

Definition at line 652 of file FileBuffer.cpp.

References multiPages_.

652  {
653  size_t total_size = 0;
654  for (const auto& multi_page : multiPages_) {
655  total_size += multi_page.pageVersions.size();
656  }
657  return total_size;
658 }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
size_t File_Namespace::FileBuffer::numMetadataPages ( ) const
inline

Returns the total number of used bytes in the FileBuffer.

Definition at line 156 of file FileBuffer.h.

156 { return metadataPages_.pageVersions.size(); };
std::deque< EpochedPage > pageVersions
Definition: Page.h:81
size_t File_Namespace::FileBuffer::pageCount ( ) const
inlineoverridevirtual

Returns the number of pages in the FileBuffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 130 of file FileBuffer.h.

130 { return multiPages_.size(); }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
virtual size_t File_Namespace::FileBuffer::pageDataSize ( ) const
inlinevirtual

Returns the size in bytes of the data portion of each page in the FileBuffer.

Definition at line 140 of file FileBuffer.h.

Referenced by File_Namespace::readForThread().

140 { return pageDataSize_; }

+ Here is the caller graph for this function:

size_t File_Namespace::FileBuffer::pageSize ( ) const
inlineoverridevirtual

Returns the size in bytes of each page in the FileBuffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 137 of file FileBuffer.h.

Referenced by File_Namespace::readForThread(), and writeHeader().

137 { return pageSize_; }

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::read ( int8_t *const  dst,
const size_t  numBytes = 0,
const size_t  offset = 0,
const MemoryLevel  dstMemoryLevel = CPU_LEVEL,
const int32_t  deviceId = -1 
)
override

Definition at line 286 of file FileBuffer.cpp.

References threading_serial::async(), CHECK, Data_Namespace::CPU_LEVEL, logger::FATAL, fm_, getMultiPage(), File_Namespace::FileMgr::getNumReaderThreads(), LOG, File_Namespace::readThreadDS::multiPages, multiPages_, pageDataSize_, File_Namespace::readForThread(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.

290  {
291  if (dstBufferType != CPU_LEVEL) {
292  LOG(FATAL) << "Unsupported Buffer type";
293  }
294 
295  // variable declarations
296  size_t startPage = offset / pageDataSize_;
297  size_t startPageOffset = offset % pageDataSize_;
298  size_t numPagesToRead =
299  (numBytes + startPageOffset + pageDataSize_ - 1) / pageDataSize_;
300  /*
301  if (startPage + numPagesToRead > multiPages_.size()) {
302  cout << "Start page: " << startPage << endl;
303  cout << "Num pages to read: " << numPagesToRead << endl;
304  cout << "Num multipages: " << multiPages_.size() << endl;
305  cout << "Offset: " << offset << endl;
306  cout << "Num bytes: " << numBytes << endl;
307  }
308  */
309 
310  CHECK(startPage + numPagesToRead <= multiPages_.size());
311 
312  size_t numPagesPerThread = 0;
313  size_t numBytesCurrent = numBytes; // total number of bytes still to be read
314  size_t bytesRead = 0; // total number of bytes already being read
315  size_t bytesLeftForThread = 0; // number of bytes to be read in the thread
316  size_t numExtraPages = 0; // extra pages to be assigned one per thread as needed
317  size_t numThreads = fm_->getNumReaderThreads();
318  std::vector<readThreadDS>
319  threadDSArr; // array of threadDS, needed to avoid racing conditions
320 
321  if (numPagesToRead > numThreads) {
322  numPagesPerThread = numPagesToRead / numThreads;
323  numExtraPages = numPagesToRead - (numThreads * numPagesPerThread);
324  } else {
325  numThreads = numPagesToRead;
326  numPagesPerThread = 1;
327  }
328 
329  /* set threadDS for the first thread */
330  readThreadDS threadDS;
331  threadDS.t_fm = fm_;
332  threadDS.t_startPage = offset / pageDataSize_;
333  if (numExtraPages > 0) {
334  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread + 1;
335  numExtraPages--;
336  } else {
337  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread;
338  }
339  threadDS.t_curPtr = dst;
340  threadDS.t_startPageOffset = offset % pageDataSize_;
341  threadDS.t_isFirstPage = true;
342 
343  bytesLeftForThread = min(((threadDS.t_endPage - threadDS.t_startPage) * pageDataSize_ -
344  threadDS.t_startPageOffset),
345  numBytesCurrent);
346  threadDS.t_bytesLeft = bytesLeftForThread;
347  threadDS.multiPages = getMultiPage();
348 
349  if (numThreads == 1) {
350  bytesRead += readForThread(this, threadDS);
351  } else {
352  std::vector<std::future<size_t>> threads;
353 
354  for (size_t i = 0; i < numThreads; i++) {
355  threadDSArr.push_back(threadDS);
356  threads.push_back(
357  std::async(std::launch::async, readForThread, this, threadDSArr[i]));
358 
359  // calculate elements of threadDS
360  threadDS.t_fm = fm_;
361  threadDS.t_isFirstPage = false;
362  threadDS.t_curPtr += bytesLeftForThread;
363  threadDS.t_startPage +=
364  threadDS.t_endPage -
365  threadDS.t_startPage; // based on # of pages read on previous iteration
366  if (numExtraPages > 0) {
367  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread + 1;
368  numExtraPages--;
369  } else {
370  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread;
371  }
372  numBytesCurrent -= bytesLeftForThread;
373  bytesLeftForThread = min(
374  ((threadDS.t_endPage - threadDS.t_startPage) * pageDataSize_), numBytesCurrent);
375  threadDS.t_bytesLeft = bytesLeftForThread;
376  threadDS.multiPages = getMultiPage();
377  }
378 
379  for (auto& p : threads) {
380  p.wait();
381  }
382  for (auto& p : threads) {
383  bytesRead += p.get();
384  }
385  }
386  CHECK(bytesRead == numBytes);
387 }
virtual std::vector< MultiPage > getMultiPage() const
Returns vector of MultiPages in the FileBuffer.
Definition: FileBuffer.h:147
static size_t readForThread(FileBuffer *fileBuffer, const readThreadDS threadDS)
Definition: FileBuffer.cpp:245
#define LOG(tag)
Definition: Logger.h:285
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
future< Result > async(Fn &&fn, Args &&...args)
#define CHECK(condition)
Definition: Logger.h:291
size_t getNumReaderThreads()
Returns number of threads defined by parameter num-reader-threads which should be used during initial...
Definition: FileMgr.h:312

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::readMetadata ( const Page page)
private

Definition at line 436 of file FileBuffer.cpp.

References CHECK, Data_Namespace::AbstractBuffer::encoder_, anonymous_namespace{Utm.h}::f, File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileForFileId(), Data_Namespace::AbstractBuffer::initEncoder(), METADATA_VERSION, metadataPageSize_, NUM_METADATA, File_Namespace::Page::pageNum, pageSize_, reservedHeaderSize_, SQLTypeInfo::set_comp_param(), SQLTypeInfo::set_compression(), SQLTypeInfo::set_dimension(), SQLTypeInfo::set_notnull(), SQLTypeInfo::set_scale(), SQLTypeInfo::set_size(), SQLTypeInfo::set_subtype(), SQLTypeInfo::set_type(), Data_Namespace::AbstractBuffer::size_, Data_Namespace::AbstractBuffer::sql_type_, and setup::version.

Referenced by initMetadataAndPageDataSize().

436  {
437  FILE* f = fm_->getFileForFileId(page.fileId);
438  fseek(f, page.pageNum * metadataPageSize_ + reservedHeaderSize_, SEEK_SET);
439  fread((int8_t*)&pageSize_, sizeof(size_t), 1, f);
440  fread((int8_t*)&size_, sizeof(size_t), 1, f);
441  vector<int32_t> typeData(
442  NUM_METADATA); // assumes we will encode hasEncoder, bufferType,
443  // encodingType, encodingBits all as int
444  fread((int8_t*)&(typeData[0]), sizeof(int32_t), typeData.size(), f);
445  int32_t version = typeData[0];
446  CHECK(version == METADATA_VERSION); // add backward compatibility code here
447  bool has_encoder = static_cast<bool>(typeData[1]);
448  if (has_encoder) {
449  sql_type_.set_type(static_cast<SQLTypes>(typeData[2]));
450  sql_type_.set_subtype(static_cast<SQLTypes>(typeData[3]));
451  sql_type_.set_dimension(typeData[4]);
452  sql_type_.set_scale(typeData[5]);
453  sql_type_.set_notnull(static_cast<bool>(typeData[6]));
454  sql_type_.set_compression(static_cast<EncodingType>(typeData[7]));
455  sql_type_.set_comp_param(typeData[8]);
456  sql_type_.set_size(typeData[9]);
458  encoder_->readMetadata(f);
459  }
460 }
void set_compression(EncodingType c)
Definition: sqltypes.h:504
void set_size(int s)
Definition: sqltypes.h:501
HOST DEVICE void set_subtype(SQLTypes st)
Definition: sqltypes.h:494
const size_t metadataPageSize_
Definition: FileBuffer.h:196
void initEncoder(const SQLTypeInfo &tmp_sql_type)
constexpr double f
Definition: Utm.h:31
string version
Definition: setup.in.py:73
void set_scale(int s)
Definition: sqltypes.h:498
FILE * getFileForFileId(const int32_t fileId)
Returns FILE pointer associated with requested fileId.
Definition: FileMgr.cpp:988
void set_comp_param(int p)
Definition: sqltypes.h:505
#define NUM_METADATA
Definition: FileBuffer.h:35
void set_dimension(int d)
Definition: sqltypes.h:495
std::unique_ptr< Encoder > encoder_
void set_notnull(bool n)
Definition: sqltypes.h:500
#define CHECK(condition)
Definition: Logger.h:291
#define METADATA_VERSION
Definition: FileBuffer.h:36
HOST DEVICE void set_type(SQLTypes t)
Definition: sqltypes.h:493

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::reserve ( const size_t  numBytes)
overridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 136 of file FileBuffer.cpp.

References addNewMultiPage(), getFileMgrEpoch(), multiPages_, pageSize_, and writeHeader().

136  {
137  size_t numPagesRequested = (numBytes + pageSize_ - 1) / pageSize_;
138  size_t numCurrentPages = multiPages_.size();
139  auto epoch = getFileMgrEpoch();
140  for (size_t pageNum = numCurrentPages; pageNum < numPagesRequested; ++pageNum) {
141  Page page = addNewMultiPage(epoch);
142  writeHeader(page, pageNum, epoch);
143  }
144 }
Page addNewMultiPage(const int32_t epoch)
Definition: FileBuffer.cpp:409
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:417

+ Here is the call graph for this function:

virtual size_t File_Namespace::FileBuffer::reservedHeaderSize ( ) const
inlinevirtual

Returns the size in bytes of the reserved header portion of each page in the FileBuffer.

Definition at line 144 of file FileBuffer.h.

Referenced by File_Namespace::readForThread().

144 { return reservedHeaderSize_; }

+ Here is the caller graph for this function:

size_t File_Namespace::FileBuffer::reservedSize ( ) const
inlineoverridevirtual

Returns the total number of bytes allocated for the FileBuffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 151 of file FileBuffer.h.

151 { return multiPages_.size() * pageSize_; }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
void File_Namespace::FileBuffer::write ( int8_t *  src,
const size_t  numBytes,
const size_t  offset = 0,
const MemoryLevel  srcMemoryLevel = CPU_LEVEL,
const int32_t  deviceId = -1 
)
override

Writes the contents of source (src) into new versions of the affected logical pages.

This method will write the contents of source (src) into new version of the affected logical pages. New pages are only appended if the value of epoch (in FileMgr)

Definition at line 537 of file FileBuffer.cpp.

References addNewMultiPage(), CHECK, copyPage(), Data_Namespace::CPU_LEVEL, File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), getFileMgrEpoch(), multiPages_, pageDataSize_, File_Namespace::Page::pageNum, pageSize_, File_Namespace::FileMgr::requestFreePage(), reservedHeaderSize_, Data_Namespace::AbstractBuffer::setAppended(), Data_Namespace::AbstractBuffer::setDirty(), Data_Namespace::AbstractBuffer::setUpdated(), Data_Namespace::AbstractBuffer::size_, File_Namespace::FileInfo::write(), and writeHeader().

541  {
542  CHECK(srcBufferType == CPU_LEVEL) << "Unsupported Buffer type";
543 
544  bool tempIsAppended = false;
545  setDirty();
546  if (offset < size_) {
547  setUpdated();
548  }
549  if (offset + numBytes > size_) {
550  tempIsAppended = true; // because is_appended_ could have already been true - to
551  // avoid rewriting header
552  setAppended();
553  size_ = offset + numBytes;
554  }
555 
556  size_t startPage = offset / pageDataSize_;
557  size_t startPageOffset = offset % pageDataSize_;
558  size_t numPagesToWrite =
559  (numBytes + startPageOffset + pageDataSize_ - 1) / pageDataSize_;
560  size_t bytesLeft = numBytes;
561  int8_t* curPtr = src; // a pointer to the current location in dst being written to
562  size_t initialNumPages = multiPages_.size();
563  auto epoch = getFileMgrEpoch();
564 
565  if (startPage >
566  initialNumPages) { // means there is a gap we need to allocate pages for
567  for (size_t pageNum = initialNumPages; pageNum < startPage; ++pageNum) {
568  Page page = addNewMultiPage(epoch);
569  writeHeader(page, pageNum, epoch);
570  }
571  }
572  for (size_t pageNum = startPage; pageNum < startPage + numPagesToWrite; ++pageNum) {
573  Page page;
574  if (pageNum >= initialNumPages) {
575  page = addNewMultiPage(epoch);
576  writeHeader(page, pageNum, epoch);
577  } else if (multiPages_[pageNum].current().epoch <
578  epoch) { // need to create new page b/c this current one lags epoch and we
579  // can't overwrite it also need to copy if we are on first or
580  // last page
581  Page lastPage = multiPages_[pageNum].current().page;
582  page = fm_->requestFreePage(pageSize_, false);
583  multiPages_[pageNum].push(page, epoch);
584  if (pageNum == startPage && startPageOffset > 0) {
585  // copyPage takes care of header offset so don't worry
586  // about it
587  copyPage(lastPage, page, startPageOffset, 0);
588  }
589  if (pageNum == (startPage + numPagesToWrite - 1) &&
590  bytesLeft > 0) { // bytesLeft should always > 0
591  copyPage(lastPage,
592  page,
593  pageDataSize_ - bytesLeft,
594  bytesLeft); // these would be empty if we're appending but we won't
595  // worry about it right now
596  }
597  writeHeader(page, pageNum, epoch);
598  } else {
599  // we already have a new page at current
600  // epoch for this page - just grab this page
601  page = multiPages_[pageNum].current().page;
602  }
603  CHECK(page.fileId >= 0); // make sure page was initialized
604  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
605  size_t bytesWritten;
606  if (pageNum == startPage) {
607  bytesWritten = fileInfo->write(
608  page.pageNum * pageSize_ + startPageOffset + reservedHeaderSize_,
609  min(pageDataSize_ - startPageOffset, bytesLeft),
610  curPtr);
611  } else {
612  bytesWritten = fileInfo->write(page.pageNum * pageSize_ + reservedHeaderSize_,
613  min(pageDataSize_, bytesLeft),
614  curPtr);
615  }
616  curPtr += bytesWritten;
617  bytesLeft -= bytesWritten;
618  if (tempIsAppended && pageNum == startPage + numPagesToWrite - 1) { // if last page
619  //@todo below can lead to undefined - we're overwriting num
620  // bytes valid at checkpoint
621  writeHeader(page, 0, multiPages_[0].current().epoch, true);
622  }
623  }
624  CHECK(bytesLeft == 0);
625 }
size_t write(const size_t offset, const size_t size, const int8_t *buf)
Definition: FileInfo.cpp:69
virtual Page requestFreePage(size_t pagesize, const bool isMetadata)
Definition: FileMgr.cpp:875
Page addNewMultiPage(const int32_t epoch)
Definition: FileBuffer.cpp:409
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:417
#define CHECK(condition)
Definition: Logger.h:291
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:222
void copyPage(Page &srcPage, Page &destPage, const size_t numBytes, const size_t offset=0)
Definition: FileBuffer.cpp:389

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::writeHeader ( Page page,
const int32_t  pageId,
const int32_t  epoch,
const bool  writeMetadata = false 
)
private

Write header writes header at top of page in format.

Definition at line 417 of file FileBuffer.cpp.

References chunkKey_, gpu_enabled::copy(), File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), metadataPageSize_, File_Namespace::Page::pageNum, pageSize(), pageSize_, and File_Namespace::FileInfo::write().

Referenced by append(), reserve(), write(), and writeMetadata().

420  {
421  int32_t intHeaderSize = chunkKey_.size() + 3; // does not include chunkSize
422  vector<int32_t> header(intHeaderSize);
423  // in addition to chunkkey we need size of header, pageId, version
424  header[0] =
425  (intHeaderSize - 1) * sizeof(int32_t); // don't need to include size of headerSize
426  // value - sizeof(size_t) is for chunkSize
427  std::copy(chunkKey_.begin(), chunkKey_.end(), header.begin() + 1);
428  header[intHeaderSize - 2] = pageId;
429  header[intHeaderSize - 1] = epoch;
430  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
432  fileInfo->write(
433  page.pageNum * pageSize, (intHeaderSize) * sizeof(int32_t), (int8_t*)&header[0]);
434 }
void writeMetadata(const int32_t epoch)
Definition: FileBuffer.cpp:462
const size_t metadataPageSize_
Definition: FileBuffer.h:196
DEVICE auto copy(ARGS &&...args)
Definition: gpu_enabled.h:51
size_t pageSize() const override
Returns the size in bytes of each page in the FileBuffer.
Definition: FileBuffer.h:137
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:222

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::writeMetadata ( const int32_t  epoch)
private

Definition at line 462 of file FileBuffer.cpp.

References Data_Namespace::AbstractBuffer::encoder_, anonymous_namespace{Utm.h}::f, File_Namespace::Page::fileId, fm_, SQLTypeInfo::get_comp_param(), SQLTypeInfo::get_compression(), SQLTypeInfo::get_dimension(), SQLTypeInfo::get_notnull(), SQLTypeInfo::get_scale(), SQLTypeInfo::get_size(), SQLTypeInfo::get_subtype(), SQLTypeInfo::get_type(), File_Namespace::FileMgr::getFileForFileId(), Data_Namespace::AbstractBuffer::hasEncoder(), METADATA_VERSION, metadataPages_, metadataPageSize_, NUM_METADATA, File_Namespace::Page::pageNum, pageSize_, File_Namespace::MultiPage::push(), File_Namespace::FileMgr::requestFreePage(), reservedHeaderSize_, Data_Namespace::AbstractBuffer::size_, Data_Namespace::AbstractBuffer::sql_type_, and writeHeader().

462  {
463  // Right now stats page is size_ (in bytes), bufferType, encodingType,
464  // encodingDataType, numElements
465  Page page = fm_->requestFreePage(metadataPageSize_, true);
466  writeHeader(page, -1, epoch, true);
467  FILE* f = fm_->getFileForFileId(page.fileId);
468  fseek(f, page.pageNum * metadataPageSize_ + reservedHeaderSize_, SEEK_SET);
469  fwrite((int8_t*)&pageSize_, sizeof(size_t), 1, f);
470  fwrite((int8_t*)&size_, sizeof(size_t), 1, f);
471  vector<int32_t> typeData(
472  NUM_METADATA); // assumes we will encode hasEncoder, bufferType,
473  // encodingType, encodingBits all as int32_t
474  typeData[0] = METADATA_VERSION;
475  typeData[1] = static_cast<int32_t>(hasEncoder());
476  if (hasEncoder()) {
477  typeData[2] = static_cast<int32_t>(sql_type_.get_type());
478  typeData[3] = static_cast<int32_t>(sql_type_.get_subtype());
479  typeData[4] = sql_type_.get_dimension();
480  typeData[5] = sql_type_.get_scale();
481  typeData[6] = static_cast<int32_t>(sql_type_.get_notnull());
482  typeData[7] = static_cast<int32_t>(sql_type_.get_compression());
483  typeData[8] = sql_type_.get_comp_param();
484  typeData[9] = sql_type_.get_size();
485  }
486  fwrite((int8_t*)&(typeData[0]), sizeof(int32_t), typeData.size(), f);
487  if (hasEncoder()) { // redundant
488  encoder_->writeMetadata(f);
489  }
490  metadataPages_.push(page, epoch);
491 }
HOST DEVICE SQLTypes get_subtype() const
Definition: sqltypes.h:382
HOST DEVICE int get_size() const
Definition: sqltypes.h:393
virtual Page requestFreePage(size_t pagesize, const bool isMetadata)
Definition: FileMgr.cpp:875
HOST DEVICE int get_scale() const
Definition: sqltypes.h:386
const size_t metadataPageSize_
Definition: FileBuffer.h:196
HOST DEVICE SQLTypes get_type() const
Definition: sqltypes.h:381
constexpr double f
Definition: Utm.h:31
FILE * getFileForFileId(const int32_t fileId)
Returns FILE pointer associated with requested fileId.
Definition: FileMgr.cpp:988
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:417
HOST DEVICE EncodingType get_compression() const
Definition: sqltypes.h:389
#define NUM_METADATA
Definition: FileBuffer.h:35
void push(const Page &page, const int epoch)
Pushes a new page with epoch value.
Definition: Page.h:102
HOST DEVICE int get_dimension() const
Definition: sqltypes.h:383
std::unique_ptr< Encoder > encoder_
HOST DEVICE int get_comp_param() const
Definition: sqltypes.h:392
HOST DEVICE bool get_notnull() const
Definition: sqltypes.h:388
#define METADATA_VERSION
Definition: FileBuffer.h:36

+ Here is the call graph for this function:

Friends And Related Function Documentation

friend class CachingFileMgr
friend

Definition at line 59 of file FileBuffer.h.

friend class FileMgr
friend

Definition at line 58 of file FileBuffer.h.

Member Data Documentation

ChunkKey File_Namespace::FileBuffer::chunkKey_
private
constexpr size_t File_Namespace::FileBuffer::headerBufferOffset_ = 32
static

Definition at line 165 of file FileBuffer.h.

Referenced by calcHeaderBuffer().

MultiPage File_Namespace::FileBuffer::metadataPages_
private
const size_t File_Namespace::FileBuffer::metadataPageSize_
private

Definition at line 196 of file FileBuffer.h.

Referenced by readMetadata(), writeHeader(), and writeMetadata().

std::vector<MultiPage> File_Namespace::FileBuffer::multiPages_
private
size_t File_Namespace::FileBuffer::pageDataSize_
private
size_t File_Namespace::FileBuffer::pageSize_
private
size_t File_Namespace::FileBuffer::reservedHeaderSize_
private

The documentation for this class was generated from the following files: