OmniSciDB  72c90bc290
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
File_Namespace::FileBuffer Class Reference

Represents/provides access to contiguous data stored in the file system. More...

#include <FileBuffer.h>

+ Inheritance diagram for File_Namespace::FileBuffer:
+ Collaboration diagram for File_Namespace::FileBuffer:

Public Member Functions

 FileBuffer (FileMgr *fm, const size_t pageSize, const ChunkKey &chunkKey, const size_t initialSize=0)
 Constructs a FileBuffer object. More...
 
 FileBuffer (FileMgr *fm, const size_t pageSize, const ChunkKey &chunkKey, const SQLTypeInfo sqlType, const size_t initialSize=0)
 
 FileBuffer (FileMgr *fm, const ChunkKey &chunkKey, const std::vector< HeaderInfo >::const_iterator &headerStartIt, const std::vector< HeaderInfo >::const_iterator &headerEndIt)
 
 ~FileBuffer () override
 Destructor. More...
 
Page addNewMultiPage (const int32_t epoch)
 
void reserve (const size_t numBytes) override
 
size_t freeMetadataPages ()
 
size_t freeChunkPages ()
 
size_t freePages ()
 
void freePagesBeforeEpoch (const int32_t targetEpoch)
 
void read (int8_t *const dst, const size_t numBytes=0, const size_t offset=0, const MemoryLevel dstMemoryLevel=CPU_LEVEL, const int32_t deviceId=-1) override
 
void write (int8_t *src, const size_t numBytes, const size_t offset=0, const MemoryLevel srcMemoryLevel=CPU_LEVEL, const int32_t deviceId=-1) override
 Writes the contents of source (src) into new versions of the affected logical pages. More...
 
void append (int8_t *src, const size_t numBytes, const MemoryLevel srcMemoryLevel=CPU_LEVEL, const int32_t deviceId=-1) override
 
void copyPage (Page &srcPage, Page &destPage, const size_t numBytes, const size_t offset=0)
 
Data_Namespace::MemoryLevel getType () const override
 
int8_t * getMemoryPtr () override
 Not implemented for FileMgr – throws a runtime_error. More...
 
size_t pageCount () const override
 Returns the number of pages in the FileBuffer. More...
 
bool hasDataPages () const
 
size_t pageSize () const override
 Returns the size in bytes of each page in the FileBuffer. More...
 
virtual size_t pageDataSize () const
 Returns the size in bytes of the data portion of each page in the FileBuffer. More...
 
virtual size_t reservedHeaderSize () const
 
virtual std::vector< MultiPagegetMultiPage () const
 Returns vector of MultiPages in the FileBuffer. More...
 
MultiPage getMetadataPage () const
 
size_t reservedSize () const override
 Returns the total number of bytes allocated for the FileBuffer. More...
 
size_t numMetadataPages () const
 Returns the total number of used bytes in the FileBuffer. More...
 
bool isMissingPages () const
 
size_t numChunkPages () const
 
std::string dump () const
 
void freePage (const Page &page)
 
- Public Member Functions inherited from Data_Namespace::AbstractBuffer
 AbstractBuffer (const int device_id)
 
 AbstractBuffer (const int device_id, const SQLTypeInfo sql_type)
 
virtual ~AbstractBuffer ()
 
virtual void read (int8_t *const dst, const size_t num_bytes, const size_t offset=0, const MemoryLevel dst_buffer_type=CPU_LEVEL, const int dst_device_id=-1)=0
 
virtual void write (int8_t *src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int src_device_id=-1)=0
 
virtual void append (int8_t *src, const size_t num_bytes, const MemoryLevel src_buffer_type=CPU_LEVEL, const int device_id=-1)=0
 
virtual void setMemoryPtr (int8_t *new_ptr)
 
virtual int pin ()
 
virtual int unPin ()
 
virtual int getPinCount ()
 
size_t size () const
 
int getDeviceId () const
 
bool isDirty () const
 
bool isAppended () const
 
bool isUpdated () const
 
bool hasEncoder () const
 
SQLTypeInfo getSqlType () const
 
void setSqlType (const SQLTypeInfo &sql_type)
 
EncodergetEncoder () const
 
void setDirty ()
 
void setUpdated ()
 
void setAppended ()
 
void setSize (const size_t size)
 
void clearDirtyBits ()
 
void initEncoder (const SQLTypeInfo &tmp_sql_type)
 
void syncEncoder (const AbstractBuffer *src_buffer)
 
void copyTo (AbstractBuffer *destination_buffer, const size_t num_bytes=0)
 
void resetToEmpty ()
 

Static Public Attributes

static constexpr size_t headerBufferOffset_ = 32
 

Private Member Functions

void writeHeader (Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
 Write header writes header at top of page in format. More...
 
void writeMetadata (const int32_t epoch)
 
void readMetadata (const Page &page)
 
void calcHeaderBuffer ()
 
void freePage (const Page &page, const bool isRolloff)
 
void freePagesBeforeEpochForMultiPage (MultiPage &multiPage, const int32_t targetEpoch, const int32_t currentEpoch)
 
void initMetadataAndPageDataSize ()
 
int32_t getFileMgrEpoch ()
 

Private Attributes

FileMgrfm_
 
const size_t metadataPageSize_
 
MultiPage metadataPages_
 
std::vector< MultiPagemultiPages_
 
size_t pageSize_
 
size_t pageDataSize_
 
size_t reservedHeaderSize_
 
ChunkKey chunkKey_
 

Friends

class FileMgr
 
class CachingFileMgr
 

Additional Inherited Members

- Protected Attributes inherited from Data_Namespace::AbstractBuffer
std::unique_ptr< Encoderencoder_
 
SQLTypeInfo sql_type_
 
size_t size_
 
int device_id_
 

Detailed Description

Represents/provides access to contiguous data stored in the file system.

The FileBuffer consists of logical pages, which can map to any identically-sized page in any file of the underlying file system. A page's metadata (file and page number) are stored in MultiPage objects, and each MultiPage includes page metadata for multiple versions of the same page.

Note that a "Chunk" is brought into a FileBuffer by the FileMgr.

Note(s): Forbid Copying Idiom 4.1

Definition at line 57 of file FileBuffer.h.

Constructor & Destructor Documentation

File_Namespace::FileBuffer::FileBuffer ( FileMgr fm,
const size_t  pageSize,
const ChunkKey chunkKey,
const size_t  initialSize = 0 
)

Constructs a FileBuffer object.

Definition at line 37 of file FileBuffer.cpp.

References calcHeaderBuffer(), CHECK, CHECK_GT, fm_, pageDataSize_, pageSize_, and reservedHeaderSize_.

41  : AbstractBuffer(fm->getDeviceId())
42  , fm_(fm)
46  , chunkKey_(chunkKey) {
47  // Create a new FileBuffer
48  CHECK(fm_);
52  //@todo reintroduce initialSize - need to develop easy way of
53  // differentiating these pre-allocated pages from "written-to" pages
54  /*
55  if (initalSize > 0) {
56  // should expand to initialSize bytes
57  size_t initialNumPages = (initalSize + pageSize_ -1) / pageSize_;
58  int32_t epoch = fm_->epoch();
59  for (size_t pageNum = 0; pageNum < initialNumPages; ++pageNum) {
60  Page page = addNewMultiPage(epoch);
61  writeHeader(page,pageNum,epoch);
62  }
63  }
64  */
65 }
const size_t metadataPageSize_
Definition: FileBuffer.h:196
#define CHECK_GT(x, y)
Definition: Logger.h:305
AbstractBuffer(const int device_id)
size_t pageSize() const override
Returns the size in bytes of each page in the FileBuffer.
Definition: FileBuffer.h:137
#define CHECK(condition)
Definition: Logger.h:291
size_t getMetadataPageSize() const
Definition: FileMgr.h:369

+ Here is the call graph for this function:

File_Namespace::FileBuffer::FileBuffer ( FileMgr fm,
const size_t  pageSize,
const ChunkKey chunkKey,
const SQLTypeInfo  sqlType,
const size_t  initialSize = 0 
)

Definition at line 67 of file FileBuffer.cpp.

References calcHeaderBuffer(), CHECK, fm_, pageDataSize_, pageSize_, and reservedHeaderSize_.

72  : AbstractBuffer(fm->getDeviceId(), sqlType)
73  , fm_(fm)
74  , metadataPageSize_(fm->getMetadataPageSize())
77  , chunkKey_(chunkKey) {
78  CHECK(fm_);
81 }
const size_t metadataPageSize_
Definition: FileBuffer.h:196
AbstractBuffer(const int device_id)
size_t pageSize() const override
Returns the size in bytes of each page in the FileBuffer.
Definition: FileBuffer.h:137
#define CHECK(condition)
Definition: Logger.h:291

+ Here is the call graph for this function:

File_Namespace::FileBuffer::FileBuffer ( FileMgr fm,
const ChunkKey chunkKey,
const std::vector< HeaderInfo >::const_iterator &  headerStartIt,
const std::vector< HeaderInfo >::const_iterator &  headerEndIt 
)

Definition at line 83 of file FileBuffer.cpp.

References calcHeaderBuffer(), CHECK, File_Namespace::FileMgr::failOnReadError(), logger::FATAL, fm_, initMetadataAndPageDataSize(), LOG, metadataPages_, multiPages_, pageSize_, File_Namespace::MultiPage::push(), and show_chunk().

87  : AbstractBuffer(fm->getDeviceId())
88  , fm_(fm)
89  , metadataPageSize_(fm->getMetadataPageSize())
91  , pageSize_(0)
92  , chunkKey_(chunkKey) {
93  // We are being assigned an existing FileBuffer on disk
94 
95  CHECK(fm_);
97  int32_t lastPageId = -1;
98  int32_t curPageId = 0;
99  for (auto vecIt = headerStartIt; vecIt != headerEndIt; ++vecIt) {
100  curPageId = vecIt->pageId;
101 
102  // We only want to read last metadata page
103  if (curPageId == -1) { // stats page
104  metadataPages_.push(vecIt->page, vecIt->versionEpoch);
105  } else {
106  if (curPageId != lastPageId) {
107  // protect from bad data on disk, and give diagnostics
108  if (fm->failOnReadError()) {
109  if (curPageId != lastPageId + 1) {
110  LOG(FATAL) << "Failure reading DB file " << show_chunk(chunkKey)
111  << " Current page " << curPageId << " last page " << lastPageId
112  << " epoch " << vecIt->versionEpoch;
113  }
114  }
115  if (lastPageId == -1) { // If we are on first real page
117  }
118  MultiPage multiPage(pageSize_);
119  multiPages_.push_back(multiPage);
120  lastPageId = curPageId;
121  }
122  multiPages_.back().push(vecIt->page, vecIt->versionEpoch);
123  }
124  }
125  if (curPageId == -1) { // meaning there was only a metadata page
127  }
128 }
#define LOG(tag)
Definition: Logger.h:285
const size_t metadataPageSize_
Definition: FileBuffer.h:196
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
std::string show_chunk(const ChunkKey &key)
Definition: types.h:98
AbstractBuffer(const int device_id)
void push(const Page &page, const int epoch)
Pushes a new page with epoch value.
Definition: Page.h:102
#define CHECK(condition)
Definition: Logger.h:291

+ Here is the call graph for this function:

File_Namespace::FileBuffer::~FileBuffer ( )
override

Destructor.

Definition at line 130 of file FileBuffer.cpp.

130  {
131  // need to free pages
132  // NOP
133 }

Member Function Documentation

Page File_Namespace::FileBuffer::addNewMultiPage ( const int32_t  epoch)

Definition at line 408 of file FileBuffer.cpp.

References fm_, multiPages_, pageSize_, File_Namespace::MultiPage::push(), and File_Namespace::FileMgr::requestFreePage().

Referenced by append(), reserve(), and write().

408  {
409  Page page = fm_->requestFreePage(pageSize_, false);
410  MultiPage multiPage(pageSize_);
411  multiPage.push(page, epoch);
412  multiPages_.emplace_back(multiPage);
413  return page;
414 }
virtual Page requestFreePage(size_t pagesize, const bool isMetadata)
Definition: FileMgr.cpp:877
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::append ( int8_t *  src,
const size_t  numBytes,
const MemoryLevel  srcMemoryLevel = CPU_LEVEL,
const int32_t  deviceId = -1 
)
override

Definition at line 492 of file FileBuffer.cpp.

References addNewMultiPage(), CHECK, File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), getFileMgrEpoch(), multiPages_, pageDataSize_, File_Namespace::Page::pageNum, pageSize_, reservedHeaderSize_, Data_Namespace::AbstractBuffer::setAppended(), Data_Namespace::AbstractBuffer::size_, File_Namespace::FileInfo::write(), and writeHeader().

495  {
496  setAppended();
497 
498  size_t startPage = size_ / pageDataSize_;
499  size_t startPageOffset = size_ % pageDataSize_;
500  size_t numPagesToWrite =
501  (numBytes + startPageOffset + pageDataSize_ - 1) / pageDataSize_;
502  size_t bytesLeft = numBytes;
503  int8_t* curPtr = src; // a pointer to the current location in dst being written to
504  size_t initialNumPages = multiPages_.size();
505  size_ = size_ + numBytes;
506  auto epoch = getFileMgrEpoch();
507  for (size_t pageNum = startPage; pageNum < startPage + numPagesToWrite; ++pageNum) {
508  Page page;
509  if (pageNum >= initialNumPages) {
510  page = addNewMultiPage(epoch);
511  writeHeader(page, pageNum, epoch);
512  } else {
513  // we already have a new page at current
514  // epoch for this page - just grab this page
515  page = multiPages_[pageNum].current().page;
516  }
517  CHECK(page.fileId >= 0); // make sure page was initialized
518  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
519  size_t bytesWritten;
520  if (pageNum == startPage) {
521  bytesWritten = fileInfo->write(
522  page.pageNum * pageSize_ + startPageOffset + reservedHeaderSize_,
523  min(pageDataSize_ - startPageOffset, bytesLeft),
524  curPtr);
525  } else {
526  bytesWritten = fileInfo->write(page.pageNum * pageSize_ + reservedHeaderSize_,
527  min(pageDataSize_, bytesLeft),
528  curPtr);
529  }
530  curPtr += bytesWritten;
531  bytesLeft -= bytesWritten;
532  }
533  CHECK(bytesLeft == 0);
534 }
size_t write(const size_t offset, const size_t size, const int8_t *buf)
Definition: FileInfo.cpp:64
Page addNewMultiPage(const int32_t epoch)
Definition: FileBuffer.cpp:408
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:416
#define CHECK(condition)
Definition: Logger.h:291
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:229

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::calcHeaderBuffer ( )
private

Definition at line 145 of file FileBuffer.cpp.

References chunkKey_, headerBufferOffset_, and reservedHeaderSize_.

Referenced by FileBuffer().

145  {
146  // 3 * sizeof(int32_t) is for headerSize, for pageId and versionEpoch
147  // sizeof(size_t) is for chunkSize
148  reservedHeaderSize_ = (chunkKey_.size() + 3) * sizeof(int32_t);
149  size_t headerMod = reservedHeaderSize_ % headerBufferOffset_;
150  if (headerMod > 0) {
152  }
153 }
static constexpr size_t headerBufferOffset_
Definition: FileBuffer.h:165

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::copyPage ( Page srcPage,
Page destPage,
const size_t  numBytes,
const size_t  offset = 0 
)

Definition at line 390 of file FileBuffer.cpp.

References CHECK, CHECK_LE, checked_malloc(), File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), pageDataSize_, File_Namespace::Page::pageNum, pageSize_, File_Namespace::FileInfo::read(), reservedHeaderSize_, and File_Namespace::FileInfo::write().

Referenced by write().

393  {
394  CHECK_LE(offset + numBytes, pageDataSize_);
395  FileInfo* srcFileInfo = fm_->getFileInfoForFileId(srcPage.fileId);
396  FileInfo* destFileInfo = fm_->getFileInfoForFileId(destPage.fileId);
397 
398  int8_t* buffer = reinterpret_cast<int8_t*>(checked_malloc(numBytes));
399  ScopeGuard guard = [&buffer] { free(buffer); };
400  size_t bytesRead = srcFileInfo->read(
401  srcPage.pageNum * pageSize_ + offset + reservedHeaderSize_, numBytes, buffer);
402  CHECK(bytesRead == numBytes);
403  size_t bytesWritten = destFileInfo->write(
404  destPage.pageNum * pageSize_ + offset + reservedHeaderSize_, numBytes, buffer);
405  CHECK(bytesWritten == numBytes);
406 }
void * checked_malloc(const size_t size)
Definition: checked_alloc.h:45
#define CHECK_LE(x, y)
Definition: Logger.h:304
#define CHECK(condition)
Definition: Logger.h:291
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:229

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string File_Namespace::FileBuffer::dump ( ) const

Definition at line 631 of file FileBuffer.cpp.

References chunkKey_, Data_Namespace::AbstractBuffer::hasEncoder(), show_chunk(), and Data_Namespace::AbstractBuffer::size_.

631  {
632  std::stringstream ss;
633  ss << "chunk_key = " << show_chunk(chunkKey_) << "\n";
634  ss << "has_encoder = " << (hasEncoder() ? "true\n" : "false\n");
635  ss << "size_ = " << size_ << "\n";
636  return ss.str();
637 }
std::string show_chunk(const ChunkKey &key)
Definition: types.h:98

+ Here is the call graph for this function:

size_t File_Namespace::FileBuffer::freeChunkPages ( )

Definition at line 178 of file FileBuffer.cpp.

References freePage(), and multiPages_.

Referenced by freePages(), and freePagesBeforeEpoch().

178  {
179  size_t num_pages_freed = multiPages_.size();
180  for (auto multiPageIt = multiPages_.begin(); multiPageIt != multiPages_.end();
181  ++multiPageIt) {
182  for (auto pageIt = multiPageIt->pageVersions.begin();
183  pageIt != multiPageIt->pageVersions.end();
184  ++pageIt) {
185  freePage(pageIt->page, false /* isRolloff */);
186  }
187  }
188  multiPages_.clear();
189  return num_pages_freed;
190 }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
void freePage(const Page &page)
Definition: FileBuffer.cpp:155

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t File_Namespace::FileBuffer::freeMetadataPages ( )

Definition at line 165 of file FileBuffer.cpp.

References freePage(), metadataPages_, File_Namespace::MultiPage::pageVersions, and File_Namespace::MultiPage::pop().

Referenced by freePages().

165  {
166  size_t num_pages_freed = metadataPages_.pageVersions.size();
167  for (auto metaPageIt = metadataPages_.pageVersions.begin();
168  metaPageIt != metadataPages_.pageVersions.end();
169  ++metaPageIt) {
170  freePage(metaPageIt->page, false /* isRolloff */);
171  }
172  while (metadataPages_.pageVersions.size() > 0) {
174  }
175  return num_pages_freed;
176 }
void pop()
Purges the oldest Page.
Definition: Page.h:110
std::deque< EpochedPage > pageVersions
Definition: Page.h:81
void freePage(const Page &page)
Definition: FileBuffer.cpp:155

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::freePage ( const Page page)

Definition at line 155 of file FileBuffer.cpp.

Referenced by freeChunkPages(), freeMetadataPages(), and freePagesBeforeEpochForMultiPage().

155  {
156  freePage(page, false);
157 }
void freePage(const Page &page)
Definition: FileBuffer.cpp:155

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::freePage ( const Page page,
const bool  isRolloff 
)
private

Definition at line 159 of file FileBuffer.cpp.

References CHECK, File_Namespace::Page::fileId, fm_, File_Namespace::FileInfo::freePage(), File_Namespace::FileMgr::getFileInfoForFileId(), getFileMgrEpoch(), and File_Namespace::Page::pageNum.

159  {
160  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
161  CHECK(fileInfo);
162  fileInfo->freePage(page.pageNum, isRolloff, getFileMgrEpoch());
163 }
#define CHECK(condition)
Definition: Logger.h:291
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:229

+ Here is the call graph for this function:

size_t File_Namespace::FileBuffer::freePages ( )

Definition at line 192 of file FileBuffer.cpp.

References freeChunkPages(), and freeMetadataPages().

192  {
193  return freeMetadataPages() + freeChunkPages();
194 }

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::freePagesBeforeEpoch ( const int32_t  targetEpoch)

Definition at line 206 of file FileBuffer.cpp.

References CHECK_LE, chunkKey_, fm_, freeChunkPages(), freePagesBeforeEpochForMultiPage(), getFileMgrEpoch(), metadataPages_, multiPages_, pageSize_, File_Namespace::MultiPage::pageVersions, and Data_Namespace::AbstractBuffer::size_.

206  {
207  // This method is only safe to be called within a checkpoint, after the sync and epoch
208  // increment where a failure at any point32_t in the process would lead to a safe
209  // rollback
210  auto currentEpoch = getFileMgrEpoch();
211  CHECK_LE(targetEpoch, currentEpoch);
212  freePagesBeforeEpochForMultiPage(metadataPages_, targetEpoch, currentEpoch);
213  for (auto& multiPage : multiPages_) {
214  freePagesBeforeEpochForMultiPage(multiPage, targetEpoch, currentEpoch);
215  }
216 
217  // Check if all buffer pages can be freed
218  if (size_ == 0) {
219  size_t max_historical_buffer_size{0};
220  for (auto& epoch_page : metadataPages_.pageVersions) {
221  // Create buffer that is used to get the buffer size at the epoch version
222  FileBuffer buffer{fm_, pageSize_, chunkKey_};
223  buffer.readMetadata(epoch_page.page);
224  max_historical_buffer_size = std::max(max_historical_buffer_size, buffer.size());
225  }
226 
227  // Free all chunk pages, if none of the old chunk versions has any data
228  if (max_historical_buffer_size == 0) {
229  freeChunkPages();
230  }
231  }
232 }
void freePagesBeforeEpochForMultiPage(MultiPage &multiPage, const int32_t targetEpoch, const int32_t currentEpoch)
Definition: FileBuffer.cpp:196
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
std::deque< EpochedPage > pageVersions
Definition: Page.h:81
#define CHECK_LE(x, y)
Definition: Logger.h:304
FileBuffer(FileMgr *fm, const size_t pageSize, const ChunkKey &chunkKey, const size_t initialSize=0)
Constructs a FileBuffer object.
Definition: FileBuffer.cpp:37

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::freePagesBeforeEpochForMultiPage ( MultiPage multiPage,
const int32_t  targetEpoch,
const int32_t  currentEpoch 
)
private

Definition at line 196 of file FileBuffer.cpp.

References freePage(), and File_Namespace::MultiPage::freePagesBeforeEpoch().

Referenced by freePagesBeforeEpoch().

198  {
199  std::vector<EpochedPage> epochedPagesToFree =
200  multiPage.freePagesBeforeEpoch(targetEpoch, currentEpoch);
201  for (const auto& epochedPageToFree : epochedPagesToFree) {
202  freePage(epochedPageToFree.page, true /* isRolloff */);
203  }
204 }
void freePage(const Page &page)
Definition: FileBuffer.cpp:155

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

int32_t File_Namespace::FileBuffer::getFileMgrEpoch ( )
private

Definition at line 626 of file FileBuffer.cpp.

References chunkKey_, File_Namespace::FileMgr::epoch(), fm_, and get_table_prefix().

Referenced by append(), freePage(), freePagesBeforeEpoch(), reserve(), and write().

626  {
627  auto [db_id, tb_id] = get_table_prefix(chunkKey_);
628  return fm_->epoch(db_id, tb_id);
629 }
virtual int32_t epoch(int32_t db_id, int32_t tb_id) const
Returns current value of epoch - should be one greater than recorded at last checkpoint. Because FileMgr only contains buffers from one table we can just return the FileMgr&#39;s epoch instead of finding a table-specific epoch.
Definition: FileMgr.h:281
std::pair< int, int > get_table_prefix(const ChunkKey &key)
Definition: types.h:62

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

int8_t* File_Namespace::FileBuffer::getMemoryPtr ( )
inlineoverridevirtual

Not implemented for FileMgr – throws a runtime_error.

Implements Data_Namespace::AbstractBuffer.

Definition at line 124 of file FileBuffer.h.

References logger::FATAL, and LOG.

124  {
125  LOG(FATAL) << "Operation not supported.";
126  return nullptr; // satisfy return-type warning
127  }
#define LOG(tag)
Definition: Logger.h:285
MultiPage File_Namespace::FileBuffer::getMetadataPage ( ) const
inline

Definition at line 148 of file FileBuffer.h.

148 { return metadataPages_; }
virtual std::vector<MultiPage> File_Namespace::FileBuffer::getMultiPage ( ) const
inlinevirtual

Returns vector of MultiPages in the FileBuffer.

Definition at line 147 of file FileBuffer.h.

Referenced by read().

147 { return multiPages_; }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198

+ Here is the caller graph for this function:

Data_Namespace::MemoryLevel File_Namespace::FileBuffer::getType ( ) const
inlineoverridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 121 of file FileBuffer.h.

References Data_Namespace::DISK_LEVEL.

bool File_Namespace::FileBuffer::hasDataPages ( ) const
inline

Returns whether or not a buffer has data pages. It is possible for a buffer to represent metadata (have a size and encode) but not contain actual data.

Definition at line 134 of file FileBuffer.h.

134 { return pageCount() > 0; }
size_t pageCount() const override
Returns the number of pages in the FileBuffer.
Definition: FileBuffer.h:130
void File_Namespace::FileBuffer::initMetadataAndPageDataSize ( )
private

Definition at line 639 of file FileBuffer.cpp.

References CHECK, File_Namespace::MultiPage::current(), File_Namespace::Page::fileId, metadataPages_, File_Namespace::EpochedPage::page, pageDataSize_, pageSize_, readMetadata(), and reservedHeaderSize_.

Referenced by FileBuffer().

639  {
640  CHECK(metadataPages_.current().page.fileId != -1); // was initialized
643 }
int32_t fileId
Definition: Page.h:47
#define CHECK(condition)
Definition: Logger.h:291
void readMetadata(const Page &page)
Definition: FileBuffer.cpp:435
EpochedPage current() const
Returns a reference to the most recent version of the page.
Definition: Page.h:94

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

bool File_Namespace::FileBuffer::isMissingPages ( ) const

Definition at line 645 of file FileBuffer.cpp.

References multiPages_, pageDataSize_, and Data_Namespace::AbstractBuffer::size().

645  {
646  // Detect the case where a page is missing by comparing the amount of pages read
647  // with the metadata size.
648  return ((size() + pageDataSize_ - 1) / pageDataSize_ != multiPages_.size());
649 }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198

+ Here is the call graph for this function:

size_t File_Namespace::FileBuffer::numChunkPages ( ) const

Definition at line 651 of file FileBuffer.cpp.

References multiPages_.

651  {
652  size_t total_size = 0;
653  for (const auto& multi_page : multiPages_) {
654  total_size += multi_page.pageVersions.size();
655  }
656  return total_size;
657 }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
size_t File_Namespace::FileBuffer::numMetadataPages ( ) const
inline

Returns the total number of used bytes in the FileBuffer.

Definition at line 156 of file FileBuffer.h.

156 { return metadataPages_.pageVersions.size(); };
std::deque< EpochedPage > pageVersions
Definition: Page.h:81
size_t File_Namespace::FileBuffer::pageCount ( ) const
inlineoverridevirtual

Returns the number of pages in the FileBuffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 130 of file FileBuffer.h.

130 { return multiPages_.size(); }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
virtual size_t File_Namespace::FileBuffer::pageDataSize ( ) const
inlinevirtual

Returns the size in bytes of the data portion of each page in the FileBuffer.

Definition at line 140 of file FileBuffer.h.

Referenced by File_Namespace::readForThread().

140 { return pageDataSize_; }

+ Here is the caller graph for this function:

size_t File_Namespace::FileBuffer::pageSize ( ) const
inlineoverridevirtual

Returns the size in bytes of each page in the FileBuffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 137 of file FileBuffer.h.

Referenced by File_Namespace::readForThread(), and writeHeader().

137 { return pageSize_; }

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::read ( int8_t *const  dst,
const size_t  numBytes = 0,
const size_t  offset = 0,
const MemoryLevel  dstMemoryLevel = CPU_LEVEL,
const int32_t  deviceId = -1 
)
override

Definition at line 286 of file FileBuffer.cpp.

References threading_serial::async(), CHECK, Data_Namespace::CPU_LEVEL, logger::FATAL, fm_, getMultiPage(), File_Namespace::FileMgr::getNumReaderThreads(), LOG, File_Namespace::readThreadDS::multiPages, multiPages_, pageDataSize_, File_Namespace::readForThread(), File_Namespace::readThreadDS::t_bytesLeft, File_Namespace::readThreadDS::t_curPtr, File_Namespace::readThreadDS::t_endPage, File_Namespace::readThreadDS::t_fm, File_Namespace::readThreadDS::t_isFirstPage, File_Namespace::readThreadDS::t_startPage, and File_Namespace::readThreadDS::t_startPageOffset.

290  {
291  if (dstBufferType != CPU_LEVEL) {
292  LOG(FATAL) << "Unsupported Buffer type";
293  }
294 
295  // variable declarations
296  size_t startPage = offset / pageDataSize_;
297  size_t startPageOffset = offset % pageDataSize_;
298  size_t numPagesToRead =
299  (numBytes + startPageOffset + pageDataSize_ - 1) / pageDataSize_;
300  /*
301  if (startPage + numPagesToRead > multiPages_.size()) {
302  cout << "Start page: " << startPage << endl;
303  cout << "Num pages to read: " << numPagesToRead << endl;
304  cout << "Num multipages: " << multiPages_.size() << endl;
305  cout << "Offset: " << offset << endl;
306  cout << "Num bytes: " << numBytes << endl;
307  }
308  */
309 
310  CHECK(startPage + numPagesToRead <= multiPages_.size())
311  << "Requested page out of bounds";
312 
313  size_t numPagesPerThread = 0;
314  size_t numBytesCurrent = numBytes; // total number of bytes still to be read
315  size_t bytesRead = 0; // total number of bytes already being read
316  size_t bytesLeftForThread = 0; // number of bytes to be read in the thread
317  size_t numExtraPages = 0; // extra pages to be assigned one per thread as needed
318  size_t numThreads = fm_->getNumReaderThreads();
319  std::vector<readThreadDS>
320  threadDSArr; // array of threadDS, needed to avoid racing conditions
321 
322  if (numPagesToRead > numThreads) {
323  numPagesPerThread = numPagesToRead / numThreads;
324  numExtraPages = numPagesToRead - (numThreads * numPagesPerThread);
325  } else {
326  numThreads = numPagesToRead;
327  numPagesPerThread = 1;
328  }
329 
330  /* set threadDS for the first thread */
331  readThreadDS threadDS;
332  threadDS.t_fm = fm_;
333  threadDS.t_startPage = offset / pageDataSize_;
334  if (numExtraPages > 0) {
335  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread + 1;
336  numExtraPages--;
337  } else {
338  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread;
339  }
340  threadDS.t_curPtr = dst;
341  threadDS.t_startPageOffset = offset % pageDataSize_;
342  threadDS.t_isFirstPage = true;
343 
344  bytesLeftForThread = min(((threadDS.t_endPage - threadDS.t_startPage) * pageDataSize_ -
345  threadDS.t_startPageOffset),
346  numBytesCurrent);
347  threadDS.t_bytesLeft = bytesLeftForThread;
348  threadDS.multiPages = getMultiPage();
349 
350  if (numThreads == 1) {
351  bytesRead += readForThread(this, threadDS);
352  } else {
353  std::vector<std::future<size_t>> threads;
354 
355  for (size_t i = 0; i < numThreads; i++) {
356  threadDSArr.push_back(threadDS);
357  threads.push_back(
358  std::async(std::launch::async, readForThread, this, threadDSArr[i]));
359 
360  // calculate elements of threadDS
361  threadDS.t_fm = fm_;
362  threadDS.t_isFirstPage = false;
363  threadDS.t_curPtr += bytesLeftForThread;
364  threadDS.t_startPage +=
365  threadDS.t_endPage -
366  threadDS.t_startPage; // based on # of pages read on previous iteration
367  if (numExtraPages > 0) {
368  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread + 1;
369  numExtraPages--;
370  } else {
371  threadDS.t_endPage = threadDS.t_startPage + numPagesPerThread;
372  }
373  numBytesCurrent -= bytesLeftForThread;
374  bytesLeftForThread = min(
375  ((threadDS.t_endPage - threadDS.t_startPage) * pageDataSize_), numBytesCurrent);
376  threadDS.t_bytesLeft = bytesLeftForThread;
377  threadDS.multiPages = getMultiPage();
378  }
379 
380  for (auto& p : threads) {
381  p.wait();
382  }
383  for (auto& p : threads) {
384  bytesRead += p.get();
385  }
386  }
387  CHECK(bytesRead == numBytes);
388 }
virtual std::vector< MultiPage > getMultiPage() const
Returns vector of MultiPages in the FileBuffer.
Definition: FileBuffer.h:147
static size_t readForThread(FileBuffer *fileBuffer, const readThreadDS threadDS)
Definition: FileBuffer.cpp:245
#define LOG(tag)
Definition: Logger.h:285
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
future< Result > async(Fn &&fn, Args &&...args)
#define CHECK(condition)
Definition: Logger.h:291
size_t getNumReaderThreads()
Returns number of threads defined by parameter num-reader-threads which should be used during initial...
Definition: FileMgr.h:316

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::readMetadata ( const Page page)
private

Definition at line 435 of file FileBuffer.cpp.

References CHECK, Data_Namespace::AbstractBuffer::encoder_, f(), File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileForFileId(), Data_Namespace::AbstractBuffer::initEncoder(), METADATA_VERSION, metadataPageSize_, NUM_METADATA, File_Namespace::Page::pageNum, pageSize_, reservedHeaderSize_, SQLTypeInfo::set_comp_param(), SQLTypeInfo::set_compression(), SQLTypeInfo::set_dimension(), SQLTypeInfo::set_notnull(), SQLTypeInfo::set_scale(), SQLTypeInfo::set_size(), SQLTypeInfo::set_subtype(), SQLTypeInfo::set_type(), Data_Namespace::AbstractBuffer::size_, Data_Namespace::AbstractBuffer::sql_type_, and setup::version.

Referenced by initMetadataAndPageDataSize().

435  {
436  FILE* f = fm_->getFileForFileId(page.fileId);
437  fseek(f, page.pageNum * metadataPageSize_ + reservedHeaderSize_, SEEK_SET);
438  fread((int8_t*)&pageSize_, sizeof(size_t), 1, f);
439  fread((int8_t*)&size_, sizeof(size_t), 1, f);
440  vector<int32_t> typeData(
441  NUM_METADATA); // assumes we will encode hasEncoder, bufferType,
442  // encodingType, encodingBits all as int
443  fread((int8_t*)&(typeData[0]), sizeof(int32_t), typeData.size(), f);
444  int32_t version = typeData[0];
445  CHECK(version == METADATA_VERSION); // add backward compatibility code here
446  bool has_encoder = static_cast<bool>(typeData[1]);
447  if (has_encoder) {
448  sql_type_.set_type(static_cast<SQLTypes>(typeData[2]));
449  sql_type_.set_subtype(static_cast<SQLTypes>(typeData[3]));
450  sql_type_.set_dimension(typeData[4]);
451  sql_type_.set_scale(typeData[5]);
452  sql_type_.set_notnull(static_cast<bool>(typeData[6]));
453  sql_type_.set_compression(static_cast<EncodingType>(typeData[7]));
454  sql_type_.set_comp_param(typeData[8]);
455  sql_type_.set_size(typeData[9]);
457  encoder_->readMetadata(f);
458  }
459 }
void set_compression(EncodingType c)
Definition: sqltypes.h:479
void set_size(int s)
Definition: sqltypes.h:476
HOST DEVICE void set_subtype(SQLTypes st)
Definition: sqltypes.h:469
const size_t metadataPageSize_
Definition: FileBuffer.h:196
void initEncoder(const SQLTypeInfo &tmp_sql_type)
string version
Definition: setup.in.py:73
void set_scale(int s)
Definition: sqltypes.h:473
FILE * getFileForFileId(const int32_t fileId)
Returns FILE pointer associated with requested fileId.
Definition: FileMgr.cpp:1003
void set_comp_param(int p)
Definition: sqltypes.h:480
#define NUM_METADATA
Definition: FileBuffer.h:35
void set_dimension(int d)
Definition: sqltypes.h:470
torch::Tensor f(torch::Tensor x, torch::Tensor W_target, torch::Tensor b_target)
std::unique_ptr< Encoder > encoder_
void set_notnull(bool n)
Definition: sqltypes.h:475
#define CHECK(condition)
Definition: Logger.h:291
#define METADATA_VERSION
Definition: FileBuffer.h:36
HOST DEVICE void set_type(SQLTypes t)
Definition: sqltypes.h:468

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::reserve ( const size_t  numBytes)
overridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 135 of file FileBuffer.cpp.

References addNewMultiPage(), getFileMgrEpoch(), multiPages_, pageSize_, and writeHeader().

135  {
136  size_t numPagesRequested = (numBytes + pageSize_ - 1) / pageSize_;
137  size_t numCurrentPages = multiPages_.size();
138  auto epoch = getFileMgrEpoch();
139  for (size_t pageNum = numCurrentPages; pageNum < numPagesRequested; ++pageNum) {
140  Page page = addNewMultiPage(epoch);
141  writeHeader(page, pageNum, epoch);
142  }
143 }
Page addNewMultiPage(const int32_t epoch)
Definition: FileBuffer.cpp:408
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:416

+ Here is the call graph for this function:

virtual size_t File_Namespace::FileBuffer::reservedHeaderSize ( ) const
inlinevirtual

Returns the size in bytes of the reserved header portion of each page in the FileBuffer.

Definition at line 144 of file FileBuffer.h.

Referenced by File_Namespace::readForThread().

144 { return reservedHeaderSize_; }

+ Here is the caller graph for this function:

size_t File_Namespace::FileBuffer::reservedSize ( ) const
inlineoverridevirtual

Returns the total number of bytes allocated for the FileBuffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 151 of file FileBuffer.h.

151 { return multiPages_.size() * pageSize_; }
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
void File_Namespace::FileBuffer::write ( int8_t *  src,
const size_t  numBytes,
const size_t  offset = 0,
const MemoryLevel  srcMemoryLevel = CPU_LEVEL,
const int32_t  deviceId = -1 
)
override

Writes the contents of source (src) into new versions of the affected logical pages.

This method will write the contents of source (src) into new version of the affected logical pages. New pages are only appended if the value of epoch (in FileMgr)

Definition at line 536 of file FileBuffer.cpp.

References addNewMultiPage(), CHECK, copyPage(), Data_Namespace::CPU_LEVEL, File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), getFileMgrEpoch(), multiPages_, pageDataSize_, File_Namespace::Page::pageNum, pageSize_, File_Namespace::FileMgr::requestFreePage(), reservedHeaderSize_, Data_Namespace::AbstractBuffer::setAppended(), Data_Namespace::AbstractBuffer::setDirty(), Data_Namespace::AbstractBuffer::setUpdated(), Data_Namespace::AbstractBuffer::size_, File_Namespace::FileInfo::write(), and writeHeader().

540  {
541  CHECK(srcBufferType == CPU_LEVEL) << "Unsupported Buffer type";
542 
543  bool tempIsAppended = false;
544  setDirty();
545  if (offset < size_) {
546  setUpdated();
547  }
548  if (offset + numBytes > size_) {
549  tempIsAppended = true; // because is_appended_ could have already been true - to
550  // avoid rewriting header
551  setAppended();
552  size_ = offset + numBytes;
553  }
554 
555  size_t startPage = offset / pageDataSize_;
556  size_t startPageOffset = offset % pageDataSize_;
557  size_t numPagesToWrite =
558  (numBytes + startPageOffset + pageDataSize_ - 1) / pageDataSize_;
559  size_t bytesLeft = numBytes;
560  int8_t* curPtr = src; // a pointer to the current location in dst being written to
561  size_t initialNumPages = multiPages_.size();
562  auto epoch = getFileMgrEpoch();
563 
564  if (startPage >
565  initialNumPages) { // means there is a gap we need to allocate pages for
566  for (size_t pageNum = initialNumPages; pageNum < startPage; ++pageNum) {
567  Page page = addNewMultiPage(epoch);
568  writeHeader(page, pageNum, epoch);
569  }
570  }
571  for (size_t pageNum = startPage; pageNum < startPage + numPagesToWrite; ++pageNum) {
572  Page page;
573  if (pageNum >= initialNumPages) {
574  page = addNewMultiPage(epoch);
575  writeHeader(page, pageNum, epoch);
576  } else if (multiPages_[pageNum].current().epoch <
577  epoch) { // need to create new page b/c this current one lags epoch and we
578  // can't overwrite it also need to copy if we are on first or
579  // last page
580  Page lastPage = multiPages_[pageNum].current().page;
581  page = fm_->requestFreePage(pageSize_, false);
582  multiPages_[pageNum].push(page, epoch);
583  if (pageNum == startPage && startPageOffset > 0) {
584  // copyPage takes care of header offset so don't worry
585  // about it
586  copyPage(lastPage, page, startPageOffset, 0);
587  }
588  if (pageNum == (startPage + numPagesToWrite - 1) &&
589  bytesLeft > 0) { // bytesLeft should always > 0
590  copyPage(lastPage,
591  page,
592  pageDataSize_ - bytesLeft,
593  bytesLeft); // these would be empty if we're appending but we won't
594  // worry about it right now
595  }
596  writeHeader(page, pageNum, epoch);
597  } else {
598  // we already have a new page at current
599  // epoch for this page - just grab this page
600  page = multiPages_[pageNum].current().page;
601  }
602  CHECK(page.fileId >= 0); // make sure page was initialized
603  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
604  size_t bytesWritten;
605  if (pageNum == startPage) {
606  bytesWritten = fileInfo->write(
607  page.pageNum * pageSize_ + startPageOffset + reservedHeaderSize_,
608  min(pageDataSize_ - startPageOffset, bytesLeft),
609  curPtr);
610  } else {
611  bytesWritten = fileInfo->write(page.pageNum * pageSize_ + reservedHeaderSize_,
612  min(pageDataSize_, bytesLeft),
613  curPtr);
614  }
615  curPtr += bytesWritten;
616  bytesLeft -= bytesWritten;
617  if (tempIsAppended && pageNum == startPage + numPagesToWrite - 1) { // if last page
618  //@todo below can lead to undefined - we're overwriting num
619  // bytes valid at checkpoint
620  writeHeader(page, 0, multiPages_[0].current().epoch, true);
621  }
622  }
623  CHECK(bytesLeft == 0);
624 }
size_t write(const size_t offset, const size_t size, const int8_t *buf)
Definition: FileInfo.cpp:64
virtual Page requestFreePage(size_t pagesize, const bool isMetadata)
Definition: FileMgr.cpp:877
Page addNewMultiPage(const int32_t epoch)
Definition: FileBuffer.cpp:408
std::vector< MultiPage > multiPages_
Definition: FileBuffer.h:198
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:416
#define CHECK(condition)
Definition: Logger.h:291
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:229
void copyPage(Page &srcPage, Page &destPage, const size_t numBytes, const size_t offset=0)
Definition: FileBuffer.cpp:390

+ Here is the call graph for this function:

void File_Namespace::FileBuffer::writeHeader ( Page page,
const int32_t  pageId,
const int32_t  epoch,
const bool  writeMetadata = false 
)
private

Write header writes header at top of page in format.

Definition at line 416 of file FileBuffer.cpp.

References chunkKey_, gpu_enabled::copy(), File_Namespace::Page::fileId, fm_, File_Namespace::FileMgr::getFileInfoForFileId(), report::header, metadataPageSize_, File_Namespace::Page::pageNum, pageSize(), pageSize_, and File_Namespace::FileInfo::write().

Referenced by append(), reserve(), write(), and writeMetadata().

419  {
420  int32_t intHeaderSize = chunkKey_.size() + 3; // does not include chunkSize
421  vector<int32_t> header(intHeaderSize);
422  // in addition to chunkkey we need size of header, pageId, version
423  header[0] =
424  (intHeaderSize - 1) * sizeof(int32_t); // don't need to include size of headerSize
425  // value - sizeof(size_t) is for chunkSize
426  std::copy(chunkKey_.begin(), chunkKey_.end(), header.begin() + 1);
427  header[intHeaderSize - 2] = pageId;
428  header[intHeaderSize - 1] = epoch;
429  FileInfo* fileInfo = fm_->getFileInfoForFileId(page.fileId);
431  fileInfo->write(
432  page.pageNum * pageSize, (intHeaderSize) * sizeof(int32_t), (int8_t*)&header[0]);
433 }
void writeMetadata(const int32_t epoch)
Definition: FileBuffer.cpp:461
const size_t metadataPageSize_
Definition: FileBuffer.h:196
DEVICE auto copy(ARGS &&...args)
Definition: gpu_enabled.h:51
size_t pageSize() const override
Returns the size in bytes of each page in the FileBuffer.
Definition: FileBuffer.h:137
list header
Definition: report.py:113
FileInfo * getFileInfoForFileId(const int32_t fileId) const
Definition: FileMgr.h:229

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void File_Namespace::FileBuffer::writeMetadata ( const int32_t  epoch)
private

Definition at line 461 of file FileBuffer.cpp.

References Data_Namespace::AbstractBuffer::encoder_, f(), File_Namespace::Page::fileId, fm_, SQLTypeInfo::get_comp_param(), SQLTypeInfo::get_compression(), SQLTypeInfo::get_dimension(), SQLTypeInfo::get_notnull(), SQLTypeInfo::get_scale(), SQLTypeInfo::get_size(), SQLTypeInfo::get_subtype(), SQLTypeInfo::get_type(), File_Namespace::FileMgr::getFileForFileId(), Data_Namespace::AbstractBuffer::hasEncoder(), METADATA_VERSION, metadataPages_, metadataPageSize_, NUM_METADATA, File_Namespace::Page::pageNum, pageSize_, File_Namespace::MultiPage::push(), File_Namespace::FileMgr::requestFreePage(), reservedHeaderSize_, Data_Namespace::AbstractBuffer::size_, Data_Namespace::AbstractBuffer::sql_type_, and writeHeader().

461  {
462  // Right now stats page is size_ (in bytes), bufferType, encodingType,
463  // encodingDataType, numElements
464  Page page = fm_->requestFreePage(metadataPageSize_, true);
465  writeHeader(page, -1, epoch, true);
466  FILE* f = fm_->getFileForFileId(page.fileId);
467  fseek(f, page.pageNum * metadataPageSize_ + reservedHeaderSize_, SEEK_SET);
468  fwrite((int8_t*)&pageSize_, sizeof(size_t), 1, f);
469  fwrite((int8_t*)&size_, sizeof(size_t), 1, f);
470  vector<int32_t> typeData(
471  NUM_METADATA); // assumes we will encode hasEncoder, bufferType,
472  // encodingType, encodingBits all as int32_t
473  typeData[0] = METADATA_VERSION;
474  typeData[1] = static_cast<int32_t>(hasEncoder());
475  if (hasEncoder()) {
476  typeData[2] = static_cast<int32_t>(sql_type_.get_type());
477  typeData[3] = static_cast<int32_t>(sql_type_.get_subtype());
478  typeData[4] = sql_type_.get_dimension();
479  typeData[5] = sql_type_.get_scale();
480  typeData[6] = static_cast<int32_t>(sql_type_.get_notnull());
481  typeData[7] = static_cast<int32_t>(sql_type_.get_compression());
482  typeData[8] = sql_type_.get_comp_param();
483  typeData[9] = sql_type_.get_size();
484  }
485  fwrite((int8_t*)&(typeData[0]), sizeof(int32_t), typeData.size(), f);
486  if (hasEncoder()) { // redundant
487  encoder_->writeMetadata(f);
488  }
489  metadataPages_.push(page, epoch);
490 }
HOST DEVICE SQLTypes get_subtype() const
Definition: sqltypes.h:392
HOST DEVICE int get_size() const
Definition: sqltypes.h:403
virtual Page requestFreePage(size_t pagesize, const bool isMetadata)
Definition: FileMgr.cpp:877
HOST DEVICE int get_scale() const
Definition: sqltypes.h:396
const size_t metadataPageSize_
Definition: FileBuffer.h:196
HOST DEVICE SQLTypes get_type() const
Definition: sqltypes.h:391
FILE * getFileForFileId(const int32_t fileId)
Returns FILE pointer associated with requested fileId.
Definition: FileMgr.cpp:1003
void writeHeader(Page &page, const int32_t pageId, const int32_t epoch, const bool writeMetadata=false)
Write header writes header at top of page in format.
Definition: FileBuffer.cpp:416
HOST DEVICE EncodingType get_compression() const
Definition: sqltypes.h:399
#define NUM_METADATA
Definition: FileBuffer.h:35
void push(const Page &page, const int epoch)
Pushes a new page with epoch value.
Definition: Page.h:102
HOST DEVICE int get_dimension() const
Definition: sqltypes.h:393
torch::Tensor f(torch::Tensor x, torch::Tensor W_target, torch::Tensor b_target)
std::unique_ptr< Encoder > encoder_
HOST DEVICE int get_comp_param() const
Definition: sqltypes.h:402
HOST DEVICE bool get_notnull() const
Definition: sqltypes.h:398
#define METADATA_VERSION
Definition: FileBuffer.h:36

+ Here is the call graph for this function:

Friends And Related Function Documentation

friend class CachingFileMgr
friend

Definition at line 59 of file FileBuffer.h.

friend class FileMgr
friend

Definition at line 58 of file FileBuffer.h.

Member Data Documentation

ChunkKey File_Namespace::FileBuffer::chunkKey_
private
constexpr size_t File_Namespace::FileBuffer::headerBufferOffset_ = 32
static

Definition at line 165 of file FileBuffer.h.

Referenced by calcHeaderBuffer().

MultiPage File_Namespace::FileBuffer::metadataPages_
private
const size_t File_Namespace::FileBuffer::metadataPageSize_
private

Definition at line 196 of file FileBuffer.h.

Referenced by readMetadata(), writeHeader(), and writeMetadata().

std::vector<MultiPage> File_Namespace::FileBuffer::multiPages_
private
size_t File_Namespace::FileBuffer::pageDataSize_
private
size_t File_Namespace::FileBuffer::pageSize_
private
size_t File_Namespace::FileBuffer::reservedHeaderSize_
private

The documentation for this class was generated from the following files: