OmniSciDB  c07336695a
Buffer_Namespace::Buffer Class Referenceabstract

Note(s): Forbid Copying Idiom 4.1. More...

#include <Buffer.h>

+ Inheritance diagram for Buffer_Namespace::Buffer:
+ Collaboration diagram for Buffer_Namespace::Buffer:

Public Member Functions

 Buffer (BufferMgr *bm, BufferList::iterator segIt, const int deviceId, const size_t pageSize=512, const size_t numBytes=0)
 Constructs a Buffer object. The constructor requires a memory address (provided by BufferMgr), number of pages, and the size in bytes of each page. Additionally, the Buffer can be initialized with an epoch. More...
 
 ~Buffer () override
 Destructor. More...
 
void read (int8_t *const dst, const size_t numBytes, const size_t offset=0, const MemoryLevel dstBufferType=CPU_LEVEL, const int deviceId=-1) override
 Reads (copies) data from the buffer to the destination (dst) memory location. Reads (copies) nbytes of data from the buffer, beginning at the specified byte offset, into the destination (dst) memory location. More...
 
void reserve (const size_t numBytes) override
 
void write (int8_t *src, const size_t numBytes, const size_t offset=0, const MemoryLevel srcBufferType=CPU_LEVEL, const int deviceId=-1) override
 Writes (copies) data from src into the buffer. Writes (copies) nbytes of data into the buffer at the specified byte offset, from the source (src) memory location. More...
 
void append (int8_t *src, const size_t numBytes, const MemoryLevel srcBufferType=CPU_LEVEL, const int deviceId=-1) override
 
int8_t * getMemoryPtr () override
 Returns a raw, constant (read-only) pointer to the underlying buffer. More...
 
size_t size () const override
 
size_t reservedSize () const override
 Returns the total number of bytes allocated for the buffer. More...
 
size_t pageCount () const override
 Returns the number of pages in the buffer. More...
 
size_t pageSize () const override
 Returns the size in bytes of each page in the buffer. More...
 
bool isDirty () const override
 Returns whether or not the buffer has been modified since the last flush/checkpoint. More...
 
int pin () override
 
int unPin () override
 
int getPinCount () override
 
- Public Member Functions inherited from Data_Namespace::AbstractBuffer
 AbstractBuffer (const int deviceId)
 
 AbstractBuffer (const int deviceId, const SQLTypeInfo sqlType)
 
virtual ~AbstractBuffer ()
 
virtual int getDeviceId () const
 
virtual MemoryLevel getType () const =0
 
virtual bool isAppended () const
 
virtual bool isUpdated () const
 
virtual void setDirty ()
 
virtual void setUpdated ()
 
virtual void setAppended ()
 
void setSize (const size_t size)
 
void clearDirtyBits ()
 
void initEncoder (const SQLTypeInfo tmpSqlType)
 
void syncEncoder (const AbstractBuffer *srcBuffer)
 

Protected Attributes

int8_t * mem_
 
- Protected Attributes inherited from Data_Namespace::AbstractBuffer
size_t size_
 
bool isDirty_
 
bool isAppended_
 
bool isUpdated_
 
int deviceId_
 

Private Member Functions

 Buffer (const Buffer &)
 pointer to beginning of buffer's memory More...
 
Bufferoperator= (const Buffer &)
 
virtual void readData (int8_t *const dst, const size_t numBytes, const size_t offset=0, const MemoryLevel dstBufferType=CPU_LEVEL, const int dstDeviceId=-1)=0
 
virtual void writeData (int8_t *const src, const size_t numBytes, const size_t offset=0, const MemoryLevel srcBufferType=CPU_LEVEL, const int srcDeviceId=-1)=0
 

Private Attributes

BufferMgrbm_
 
BufferList::iterator segIt_
 
size_t pageSize_
 
size_t numPages_
 the size of each page in the buffer More...
 
int epoch_
 
std::vector< bool > pageDirtyFlags_
 indicates when the buffer was last flushed More...
 
int pinCount_
 
std::mutex pinMutex_
 

Friends

class BufferMgr
 
class FileMgr
 

Additional Inherited Members

- Public Attributes inherited from Data_Namespace::AbstractBuffer
std::unique_ptr< Encoderencoder
 
bool hasEncoder
 
SQLTypeInfo sqlType
 

Detailed Description

Note(s): Forbid Copying Idiom 4.1.

Definition at line 45 of file Buffer.h.

Constructor & Destructor Documentation

◆ Buffer() [1/2]

Buffer_Namespace::Buffer::Buffer ( BufferMgr bm,
BufferList::iterator  segIt,
const int  deviceId,
const size_t  pageSize = 512,
const size_t  numBytes = 0 
)

Constructs a Buffer object. The constructor requires a memory address (provided by BufferMgr), number of pages, and the size in bytes of each page. Additionally, the Buffer can be initialized with an epoch.

Parameters
memThe beginning memory address of the buffer.
numPagesThe number of pages into which the buffer's memory space is divided.
pageSizeThe size in bytes of each page that composes the buffer.
epochA temporal reference implying the buffer is up-to-date up to the epoch.

Definition at line 35 of file Buffer.cpp.

References pin(), reserve(), and segIt_.

40  : AbstractBuffer(deviceId)
41  , mem_(0)
42  , bm_(bm)
43  , segIt_(segIt)
45  , numPages_(0)
46  , pinCount_(0) {
47  pin();
48  // so that the pointer value of this Buffer is stored
49  segIt_->buffer = this;
50  if (numBytes > 0) {
51  reserve(numBytes);
52  }
53 }
size_t numPages_
the size of each page in the buffer
Definition: Buffer.h:169
void reserve(const size_t numBytes) override
Definition: Buffer.cpp:69
AbstractBuffer(const int deviceId)
BufferList::iterator segIt_
Definition: Buffer.h:166
size_t pageSize() const override
Returns the size in bytes of each page in the buffer.
Definition: Buffer.h:129
int pin() override
Definition: Buffer.h:134
+ Here is the call graph for this function:

◆ ~Buffer()

Buffer_Namespace::Buffer::~Buffer ( )
override

Destructor.

Definition at line 67 of file Buffer.cpp.

67 {}

◆ Buffer() [2/2]

Buffer_Namespace::Buffer::Buffer ( const Buffer )
private

pointer to beginning of buffer's memory

Member Function Documentation

◆ append()

void Buffer_Namespace::Buffer::append ( int8_t *  src,
const size_t  numBytes,
const MemoryLevel  srcBufferType = CPU_LEVEL,
const int  deviceId = -1 
)
overridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 139 of file Buffer.cpp.

References Data_Namespace::AbstractBuffer::isAppended_, Data_Namespace::AbstractBuffer::isDirty_, reserve(), reservedSize(), Data_Namespace::AbstractBuffer::size_, and writeData().

142  {
143 #ifdef BUFFER_MUTEX
144  boost::shared_lock<boost::shared_mutex> readLock(
145  readWriteMutex_); // keep another thread from getting a write lock
146  boost::unique_lock<boost::shared_mutex> appendLock(
147  appendMutex_); // keep another thread from getting an append lock
148 #endif
149 
150  isDirty_ = true;
151  isAppended_ = true;
152 
153  if (numBytes + size_ > reservedSize()) {
154  reserve(numBytes + size_);
155  }
156 
157  writeData(src, numBytes, size_, srcBufferType, srcDeviceId);
158  size_ += numBytes;
159  // Do we worry about dirty flags here or does append avoid them
160 }
int64_t * src
void reserve(const size_t numBytes) override
Definition: Buffer.cpp:69
virtual void writeData(int8_t *const src, const size_t numBytes, const size_t offset=0, const MemoryLevel srcBufferType=CPU_LEVEL, const int srcDeviceId=-1)=0
size_t reservedSize() const override
Returns the total number of bytes allocated for the buffer.
Definition: Buffer.h:122
+ Here is the call graph for this function:

◆ getMemoryPtr()

int8_t * Buffer_Namespace::Buffer::getMemoryPtr ( )
overridevirtual

Returns a raw, constant (read-only) pointer to the underlying buffer.

Returns
A constant memory pointer for read-only access.

Implements Data_Namespace::AbstractBuffer.

Definition at line 162 of file Buffer.cpp.

References mem_.

162  {
163  return mem_;
164 }

◆ getPinCount()

int Buffer_Namespace::Buffer::getPinCount ( )
inlineoverridevirtual

Reimplemented from Data_Namespace::AbstractBuffer.

Definition at line 143 of file Buffer.h.

143  {
144  std::lock_guard<std::mutex> pinLock(pinMutex_);
145  return (pinCount_);
146  }
std::mutex pinMutex_
Definition: Buffer.h:175

◆ isDirty()

bool Buffer_Namespace::Buffer::isDirty ( ) const
inlineoverridevirtual

Returns whether or not the buffer has been modified since the last flush/checkpoint.

Reimplemented from Data_Namespace::AbstractBuffer.

Definition at line 132 of file Buffer.h.

◆ operator=()

Buffer& Buffer_Namespace::Buffer::operator= ( const Buffer )
private

◆ pageCount()

size_t Buffer_Namespace::Buffer::pageCount ( ) const
inlineoverridevirtual

Returns the number of pages in the buffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 125 of file Buffer.h.

125 { return numPages_; }
size_t numPages_
the size of each page in the buffer
Definition: Buffer.h:169

◆ pageSize()

size_t Buffer_Namespace::Buffer::pageSize ( ) const
inlineoverridevirtual

Returns the size in bytes of each page in the buffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 129 of file Buffer.h.

129 { return pageSize_; }

◆ pin()

int Buffer_Namespace::Buffer::pin ( )
inlineoverridevirtual

Reimplemented from Data_Namespace::AbstractBuffer.

Definition at line 134 of file Buffer.h.

Referenced by Buffer().

134  {
135  std::lock_guard<std::mutex> pinLock(pinMutex_);
136  return (++pinCount_);
137  }
std::mutex pinMutex_
Definition: Buffer.h:175
+ Here is the caller graph for this function:

◆ read()

void Buffer_Namespace::Buffer::read ( int8_t *const  dst,
const size_t  numBytes,
const size_t  offset = 0,
const MemoryLevel  dstBufferType = CPU_LEVEL,
const int  deviceId = -1 
)
overridevirtual

Reads (copies) data from the buffer to the destination (dst) memory location. Reads (copies) nbytes of data from the buffer, beginning at the specified byte offset, into the destination (dst) memory location.

Parameters
dstThe destination address to where the buffer's data is being copied.
offsetThe byte offset into the buffer from where reading (copying) begins.
nbytesThe number of bytes being read (copied) into the destination (dst).

Implements Data_Namespace::AbstractBuffer.

Definition at line 87 of file Buffer.cpp.

References logger::FATAL, LOG, mem_, readData(), and Data_Namespace::AbstractBuffer::size_.

91  {
92  if (numBytes == 0) {
93  return;
94  }
95  assert(dst && mem_);
96 #ifdef BUFFER_MUTEX
97  boost::shared_lock<boost::shared_mutex> readLock(readWriteMutex_);
98 #endif
99 
100  if (numBytes + offset > size_) {
101  LOG(FATAL) << "Buffer: Out of bounds read error";
102  }
103  readData(dst, numBytes, offset, dstBufferType, dstDeviceId);
104 }
virtual void readData(int8_t *const dst, const size_t numBytes, const size_t offset=0, const MemoryLevel dstBufferType=CPU_LEVEL, const int dstDeviceId=-1)=0
#define LOG(tag)
Definition: Logger.h:182
+ Here is the call graph for this function:

◆ readData()

virtual void Buffer_Namespace::Buffer::readData ( int8_t *const  dst,
const size_t  numBytes,
const size_t  offset = 0,
const MemoryLevel  dstBufferType = CPU_LEVEL,
const int  dstDeviceId = -1 
)
privatepure virtual

Implemented in Buffer_Namespace::CpuBuffer, and Buffer_Namespace::GpuCudaBuffer.

Referenced by read().

+ Here is the caller graph for this function:

◆ reserve()

void Buffer_Namespace::Buffer::reserve ( const size_t  numBytes)
overridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 69 of file Buffer.cpp.

References bm_, numPages_, pageDirtyFlags_, pageSize_, Buffer_Namespace::BufferMgr::reserveBuffer(), and segIt_.

Referenced by append(), Buffer(), and write().

69  {
70 #ifdef BUFFER_MUTEX
71  boost::unique_lock<boost::shared_mutex> writeLock(readWriteMutex_);
72 #endif
73  size_t numPages = (numBytes + pageSize_ - 1) / pageSize_;
74  // std::cout << "NumPages reserved: " << numPages << std::endl;
75  if (numPages > numPages_) {
76  // When running out of cpu buffers, reserveBuffer() will fail and
77  // trigger a SlabTooBig exception, so pageDirtyFlags_ and numPages_
78  // MUST NOT be set until reserveBuffer() returns; otherwise, this
79  // buffer is not properly resized, so any call to FileMgr::fetchBuffer()
80  // will proceed to read(), corrupt heap memory and cause core dump later.
81  segIt_ = bm_->reserveBuffer(segIt_, pageSize_ * numPages);
82  pageDirtyFlags_.resize(numPages);
83  numPages_ = numPages;
84  }
85 }
size_t numPages_
the size of each page in the buffer
Definition: Buffer.h:169
BufferList::iterator segIt_
Definition: Buffer.h:166
std::vector< bool > pageDirtyFlags_
indicates when the buffer was last flushed
Definition: Buffer.h:173
BufferList::iterator reserveBuffer(BufferList::iterator &segIt, const size_t numBytes)
Definition: BufferMgr.cpp:179
+ Here is the call graph for this function:
+ Here is the caller graph for this function:

◆ reservedSize()

size_t Buffer_Namespace::Buffer::reservedSize ( ) const
inlineoverridevirtual

Returns the total number of bytes allocated for the buffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 122 of file Buffer.h.

Referenced by append(), and write().

122 { return pageSize_ * numPages_; }
size_t numPages_
the size of each page in the buffer
Definition: Buffer.h:169
+ Here is the caller graph for this function:

◆ size()

size_t Buffer_Namespace::Buffer::size ( ) const
inlineoverridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 119 of file Buffer.h.

119 { return size_; }

◆ unPin()

int Buffer_Namespace::Buffer::unPin ( )
inlineoverridevirtual

Reimplemented from Data_Namespace::AbstractBuffer.

Definition at line 139 of file Buffer.h.

139  {
140  std::lock_guard<std::mutex> pinLock(pinMutex_);
141  return (--pinCount_);
142  }
std::mutex pinMutex_
Definition: Buffer.h:175

◆ write()

void Buffer_Namespace::Buffer::write ( int8_t *  src,
const size_t  numBytes,
const size_t  offset = 0,
const MemoryLevel  srcBufferType = CPU_LEVEL,
const int  deviceId = -1 
)
overridevirtual

Writes (copies) data from src into the buffer. Writes (copies) nbytes of data into the buffer at the specified byte offset, from the source (src) memory location.

Parameters
srcThe source address from where data is being copied to the buffer.
offsetThe byte offset into the buffer to where writing begins.
nbytesThe number of bytes being written (copied) into the buffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 106 of file Buffer.cpp.

References Data_Namespace::AbstractBuffer::isAppended_, Data_Namespace::AbstractBuffer::isDirty_, Data_Namespace::AbstractBuffer::isUpdated_, pageDirtyFlags_, pageSize_, reserve(), reservedSize(), Data_Namespace::AbstractBuffer::size_, and writeData().

110  {
111  assert(numBytes > 0); // cannot write 0 bytes
112 #ifdef BUFFER_MUTEX
113  boost::unique_lock<boost::shared_mutex> writeLock(readWriteMutex_);
114 #endif
115  if (numBytes + offset > reservedSize()) {
116  reserve(numBytes + offset);
117  }
118  // write source contents to buffer
119  writeData(src, numBytes, offset, srcBufferType, srcDeviceId);
120 
121  // update dirty flags for buffer and each affected page
122  isDirty_ = true;
123  if (offset < size_) {
124  isUpdated_ = true;
125  }
126  if (offset + numBytes > size_) {
127  isAppended_ = true;
128  size_ = offset + numBytes;
129  }
130  // std::cout << "Size after write: " << size_ << std::endl;
131 
132  size_t firstDirtyPage = offset / pageSize_;
133  size_t lastDirtyPage = (offset + numBytes - 1) / pageSize_;
134  for (size_t i = firstDirtyPage; i <= lastDirtyPage; ++i) {
135  pageDirtyFlags_[i] = true;
136  }
137 }
int64_t * src
void reserve(const size_t numBytes) override
Definition: Buffer.cpp:69
virtual void writeData(int8_t *const src, const size_t numBytes, const size_t offset=0, const MemoryLevel srcBufferType=CPU_LEVEL, const int srcDeviceId=-1)=0
std::vector< bool > pageDirtyFlags_
indicates when the buffer was last flushed
Definition: Buffer.h:173
size_t reservedSize() const override
Returns the total number of bytes allocated for the buffer.
Definition: Buffer.h:122
+ Here is the call graph for this function:

◆ writeData()

virtual void Buffer_Namespace::Buffer::writeData ( int8_t *const  src,
const size_t  numBytes,
const size_t  offset = 0,
const MemoryLevel  srcBufferType = CPU_LEVEL,
const int  srcDeviceId = -1 
)
privatepure virtual

Implemented in Buffer_Namespace::CpuBuffer, and Buffer_Namespace::GpuCudaBuffer.

Referenced by append(), and write().

+ Here is the caller graph for this function:

Friends And Related Function Documentation

◆ BufferMgr

friend class BufferMgr
friend

Definition at line 46 of file Buffer.h.

◆ FileMgr

friend class FileMgr
friend

Definition at line 47 of file Buffer.h.

Member Data Documentation

◆ bm_

BufferMgr* Buffer_Namespace::Buffer::bm_
private

Definition at line 165 of file Buffer.h.

Referenced by reserve().

◆ epoch_

int Buffer_Namespace::Buffer::epoch_
private

Definition at line 170 of file Buffer.h.

◆ mem_

◆ numPages_

size_t Buffer_Namespace::Buffer::numPages_
private

the size of each page in the buffer

Definition at line 169 of file Buffer.h.

Referenced by reserve().

◆ pageDirtyFlags_

std::vector<bool> Buffer_Namespace::Buffer::pageDirtyFlags_
private

indicates when the buffer was last flushed

Definition at line 173 of file Buffer.h.

Referenced by reserve(), and write().

◆ pageSize_

size_t Buffer_Namespace::Buffer::pageSize_
private

Definition at line 168 of file Buffer.h.

Referenced by reserve(), and write().

◆ pinCount_

int Buffer_Namespace::Buffer::pinCount_
private

Definition at line 174 of file Buffer.h.

◆ pinMutex_

std::mutex Buffer_Namespace::Buffer::pinMutex_
private

Definition at line 175 of file Buffer.h.

◆ segIt_

BufferList::iterator Buffer_Namespace::Buffer::segIt_
private

Definition at line 166 of file Buffer.h.

Referenced by Buffer(), Buffer_Namespace::BufferMgr::free(), and reserve().


The documentation for this class was generated from the following files: