OmniSciDB  340b00dbf6
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
Buffer_Namespace::Buffer Class Referenceabstract

Note(s): Forbid Copying Idiom 4.1. More...

#include <Buffer.h>

+ Inheritance diagram for Buffer_Namespace::Buffer:
+ Collaboration diagram for Buffer_Namespace::Buffer:

Public Member Functions

 Buffer (BufferMgr *bm, BufferList::iterator seg_it, const int device_id, const size_t page_size=512, const size_t num_bytes=0)
 Constructs a Buffer object. The constructor requires a memory address (provided by BufferMgr), number of pages, and the size in bytes of each page. Additionally, the Buffer can be initialized with an epoch. More...
 
 ~Buffer () override
 Destructor. More...
 
void read (int8_t *const dst, const size_t num_bytes, const size_t offset=0, const MemoryLevel dst_buffer_type=CPU_LEVEL, const int device_id=-1) override
 Reads (copies) data from the buffer to the destination (dst) memory location. Reads (copies) nbytes of data from the buffer, beginning at the specified byte offset, into the destination (dst) memory location. More...
 
void reserve (const size_t num_bytes) override
 
void write (int8_t *src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int device_id=-1) override
 Writes (copies) data from src into the buffer. Writes (copies) nbytes of data into the buffer at the specified byte offset, from the source (src) memory location. More...
 
void append (int8_t *src, const size_t num_bytes, const MemoryLevel src_buffer_type=CPU_LEVEL, const int deviceId=-1) override
 
int8_t * getMemoryPtr () override
 Returns a raw, constant (read-only) pointer to the underlying buffer. More...
 
size_t reservedSize () const override
 Returns the total number of bytes allocated for the buffer. More...
 
size_t pageCount () const override
 Returns the number of pages in the buffer. More...
 
size_t pageSize () const override
 Returns the size in bytes of each page in the buffer. More...
 
int pin () override
 
int unPin () override
 
int getPinCount () override
 
- Public Member Functions inherited from Data_Namespace::AbstractBuffer
 AbstractBuffer (const int device_id)
 
 AbstractBuffer (const int device_id, const SQLTypeInfo sql_type)
 
virtual ~AbstractBuffer ()
 
virtual MemoryLevel getType () const =0
 
size_t size () const
 
int getDeviceId () const
 
bool isDirty () const
 
bool isAppended () const
 
bool isUpdated () const
 
bool hasEncoder () const
 
SQLTypeInfo getSqlType () const
 
void setSqlType (const SQLTypeInfo &sql_type)
 
EncodergetEncoder () const
 
void setDirty ()
 
void setUpdated ()
 
void setAppended ()
 
void setSize (const size_t size)
 
void clearDirtyBits ()
 
void initEncoder (const SQLTypeInfo &tmp_sql_type)
 
void syncEncoder (const AbstractBuffer *src_buffer)
 
void copyTo (AbstractBuffer *destination_buffer, const size_t num_bytes=0)
 
void resetToEmpty ()
 

Protected Attributes

int8_t * mem_
 
- Protected Attributes inherited from Data_Namespace::AbstractBuffer
std::unique_ptr< Encoderencoder_
 
SQLTypeInfo sql_type_
 
size_t size_
 
int device_id_
 

Private Member Functions

 Buffer (const Buffer &)
 pointer to beginning of buffer's memory More...
 
Bufferoperator= (const Buffer &)
 
virtual void readData (int8_t *const dst, const size_t num_bytes, const size_t offset=0, const MemoryLevel dst_buffer_type=CPU_LEVEL, const int dst_device_id=-1)=0
 
virtual void writeData (int8_t *const src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int src_device_id=-1)=0
 

Private Attributes

BufferMgrbm_
 
BufferList::iterator seg_it_
 
size_t page_size_
 
size_t num_pages_
 the size of each page in the buffer More...
 
int epoch_
 
std::vector< bool > page_dirty_flags_
 indicates when the buffer was last flushed More...
 
int pin_count_
 
std::mutex pin_mutex_
 

Friends

class BufferMgr
 
class FileMgr
 

Detailed Description

Note(s): Forbid Copying Idiom 4.1.

Definition at line 42 of file Buffer.h.

Constructor & Destructor Documentation

Buffer_Namespace::Buffer::Buffer ( BufferMgr bm,
BufferList::iterator  seg_it,
const int  device_id,
const size_t  page_size = 512,
const size_t  num_bytes = 0 
)

Constructs a Buffer object. The constructor requires a memory address (provided by BufferMgr), number of pages, and the size in bytes of each page. Additionally, the Buffer can be initialized with an epoch.

Parameters
memThe beginning memory address of the buffer.
numPagesThe number of pages into which the buffer's memory space is divided.
pageSizeThe size in bytes of each page that composes the buffer.
epochA temporal reference implying the buffer is up-to-date up to the epoch.

Definition at line 27 of file Buffer.cpp.

References pin(), reserve(), and seg_it_.

32  : AbstractBuffer(device_id)
33  , mem_(nullptr)
34  , bm_(bm)
35  , seg_it_(seg_it)
36  , page_size_(page_size)
37  , num_pages_(0)
38  , pin_count_(0) {
39  pin();
40  // so that the pointer value of this Buffer is stored
41  seg_it_->buffer = this;
42  if (num_bytes > 0) {
43  reserve(num_bytes);
44  }
45 }
size_t num_pages_
the size of each page in the buffer
Definition: Buffer.h:159
BufferList::iterator seg_it_
Definition: Buffer.h:157
AbstractBuffer(const int device_id)
void reserve(const size_t num_bytes) override
Definition: Buffer.cpp:49
int pin() override
Definition: Buffer.h:125

+ Here is the call graph for this function:

Buffer_Namespace::Buffer::~Buffer ( )
override

Destructor.

Definition at line 47 of file Buffer.cpp.

47 {}
Buffer_Namespace::Buffer::Buffer ( const Buffer )
private

pointer to beginning of buffer's memory

Member Function Documentation

void Buffer_Namespace::Buffer::append ( int8_t *  src,
const size_t  num_bytes,
const MemoryLevel  src_buffer_type = CPU_LEVEL,
const int  deviceId = -1 
)
overridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 120 of file Buffer.cpp.

References reserve(), reservedSize(), Data_Namespace::AbstractBuffer::setAppended(), Data_Namespace::AbstractBuffer::size_, and writeData().

123  {
124 #ifdef BUFFER_MUTEX
125  boost::shared_lock<boost::shared_mutex> read_lock(
126  read_write_mutex_); // keep another thread from getting a write lock
127  boost::unique_lock<boost::shared_mutex> append_lock(
128  append_mutex_); // keep another thread from getting an append lock
129 #endif
130 
131  setAppended();
132 
133  if (num_bytes + size_ > reservedSize()) {
134  reserve(num_bytes + size_);
135  }
136 
137  writeData(src, num_bytes, size_, src_buffer_type, src_device_id);
138  size_ += num_bytes;
139  // Do we worry about dirty flags here or does append avoid them
140 }
virtual void writeData(int8_t *const src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int src_device_id=-1)=0
mapd_shared_lock< mapd_shared_mutex > read_lock
size_t reservedSize() const override
Returns the total number of bytes allocated for the buffer.
Definition: Buffer.h:117
void reserve(const size_t num_bytes) override
Definition: Buffer.cpp:49

+ Here is the call graph for this function:

int8_t * Buffer_Namespace::Buffer::getMemoryPtr ( )
overridevirtual

Returns a raw, constant (read-only) pointer to the underlying buffer.

Returns
A constant memory pointer for read-only access.

Implements Data_Namespace::AbstractBuffer.

Definition at line 142 of file Buffer.cpp.

References mem_.

142  {
143  return mem_;
144 }
int Buffer_Namespace::Buffer::getPinCount ( )
inlineoverridevirtual

Reimplemented from Data_Namespace::AbstractBuffer.

Definition at line 134 of file Buffer.h.

134  {
135  std::lock_guard<std::mutex> pin_lock(pin_mutex_);
136  return (pin_count_);
137  }
std::mutex pin_mutex_
Definition: Buffer.h:163
Buffer& Buffer_Namespace::Buffer::operator= ( const Buffer )
private
size_t Buffer_Namespace::Buffer::pageCount ( ) const
inlineoverridevirtual

Returns the number of pages in the buffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 120 of file Buffer.h.

120 { return num_pages_; }
size_t num_pages_
the size of each page in the buffer
Definition: Buffer.h:159
size_t Buffer_Namespace::Buffer::pageSize ( ) const
inlineoverridevirtual

Returns the size in bytes of each page in the buffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 123 of file Buffer.h.

123 { return page_size_; }
int Buffer_Namespace::Buffer::pin ( )
inlineoverridevirtual

Reimplemented from Data_Namespace::AbstractBuffer.

Definition at line 125 of file Buffer.h.

Referenced by Buffer().

125  {
126  std::lock_guard<std::mutex> pin_lock(pin_mutex_);
127  return (++pin_count_);
128  }
std::mutex pin_mutex_
Definition: Buffer.h:163

+ Here is the caller graph for this function:

void Buffer_Namespace::Buffer::read ( int8_t *const  dst,
const size_t  num_bytes,
const size_t  offset = 0,
const MemoryLevel  dst_buffer_type = CPU_LEVEL,
const int  device_id = -1 
)
overridevirtual

Reads (copies) data from the buffer to the destination (dst) memory location. Reads (copies) nbytes of data from the buffer, beginning at the specified byte offset, into the destination (dst) memory location.

Parameters
dstThe destination address to where the buffer's data is being copied.
offsetThe byte offset into the buffer from where reading (copying) begins.
nbytesThe number of bytes being read (copied) into the destination (dst).

Implements Data_Namespace::AbstractBuffer.

Definition at line 67 of file Buffer.cpp.

References CHECK, logger::FATAL, LOG, mem_, readData(), and Data_Namespace::AbstractBuffer::size_.

71  {
72  if (num_bytes == 0) {
73  return;
74  }
75  CHECK(dst && mem_);
76 #ifdef BUFFER_MUTEX
77  boost::shared_lock<boost::shared_mutex> read_lock(read_write_mutex_);
78 #endif
79 
80  if (num_bytes + offset > size_) {
81  LOG(FATAL) << "Buffer: Out of bounds read error";
82  }
83  readData(dst, num_bytes, offset, dst_buffer_type, dst_device_id);
84 }
#define LOG(tag)
Definition: Logger.h:188
virtual void readData(int8_t *const dst, const size_t num_bytes, const size_t offset=0, const MemoryLevel dst_buffer_type=CPU_LEVEL, const int dst_device_id=-1)=0
mapd_shared_lock< mapd_shared_mutex > read_lock
#define CHECK(condition)
Definition: Logger.h:197

+ Here is the call graph for this function:

virtual void Buffer_Namespace::Buffer::readData ( int8_t *const  dst,
const size_t  num_bytes,
const size_t  offset = 0,
const MemoryLevel  dst_buffer_type = CPU_LEVEL,
const int  dst_device_id = -1 
)
privatepure virtual

Implemented in Buffer_Namespace::CpuBuffer, and Buffer_Namespace::GpuCudaBuffer.

Referenced by read().

+ Here is the caller graph for this function:

void Buffer_Namespace::Buffer::reserve ( const size_t  num_bytes)
overridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 49 of file Buffer.cpp.

References bm_, num_pages_, page_dirty_flags_, page_size_, Buffer_Namespace::BufferMgr::reserveBuffer(), and seg_it_.

Referenced by append(), Buffer(), and write().

49  {
50 #ifdef BUFFER_MUTEX
51  boost::unique_lock<boost::shared_mutex> write_lock(read_write_mutex_);
52 #endif
53  size_t num_pages = (num_bytes + page_size_ - 1) / page_size_;
54  // std::cout << "NumPages reserved: " << numPages << std::endl;
55  if (num_pages > num_pages_) {
56  // When running out of cpu buffers, reserveBuffer() will fail and
57  // trigger a SlabTooBig exception, so pageDirtyFlags_ and numPages_
58  // MUST NOT be set until reserveBuffer() returns; otherwise, this
59  // buffer is not properly resized, so any call to FileMgr::fetchBuffer()
60  // will proceed to read(), corrupt heap memory and cause core dump later.
61  seg_it_ = bm_->reserveBuffer(seg_it_, page_size_ * num_pages);
62  page_dirty_flags_.resize(num_pages);
63  num_pages_ = num_pages;
64  }
65 }
size_t num_pages_
the size of each page in the buffer
Definition: Buffer.h:159
BufferList::iterator seg_it_
Definition: Buffer.h:157
BufferList::iterator reserveBuffer(BufferList::iterator &seg_it, const size_t num_bytes)
Definition: BufferMgr.cpp:192
std::vector< bool > page_dirty_flags_
indicates when the buffer was last flushed
Definition: Buffer.h:161
mapd_unique_lock< mapd_shared_mutex > write_lock

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t Buffer_Namespace::Buffer::reservedSize ( ) const
inlineoverridevirtual

Returns the total number of bytes allocated for the buffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 117 of file Buffer.h.

Referenced by append(), and write().

117 { return page_size_ * num_pages_; }
size_t num_pages_
the size of each page in the buffer
Definition: Buffer.h:159

+ Here is the caller graph for this function:

int Buffer_Namespace::Buffer::unPin ( )
inlineoverridevirtual

Reimplemented from Data_Namespace::AbstractBuffer.

Definition at line 130 of file Buffer.h.

130  {
131  std::lock_guard<std::mutex> pin_lock(pin_mutex_);
132  return (--pin_count_);
133  }
std::mutex pin_mutex_
Definition: Buffer.h:163
void Buffer_Namespace::Buffer::write ( int8_t *  src,
const size_t  num_bytes,
const size_t  offset = 0,
const MemoryLevel  src_buffer_type = CPU_LEVEL,
const int  device_id = -1 
)
overridevirtual

Writes (copies) data from src into the buffer. Writes (copies) nbytes of data into the buffer at the specified byte offset, from the source (src) memory location.

Parameters
srcThe source address from where data is being copied to the buffer.
num_bytesThe number of bytes being written (copied) into the buffer.
offsetThe byte offset into the buffer to where writing begins.

Implements Data_Namespace::AbstractBuffer.

Definition at line 86 of file Buffer.cpp.

References CHECK_GT, page_dirty_flags_, page_size_, reserve(), reservedSize(), Data_Namespace::AbstractBuffer::setAppended(), Data_Namespace::AbstractBuffer::setDirty(), Data_Namespace::AbstractBuffer::setUpdated(), Data_Namespace::AbstractBuffer::size_, and writeData().

90  {
91  CHECK_GT(num_bytes, size_t(0)); // cannot write 0 bytes
92 #ifdef BUFFER_MUTEX
93  boost::unique_lock<boost::shared_mutex> write_lock(read_write_mutex_);
94 #endif
95  if (num_bytes + offset > reservedSize()) {
96  reserve(num_bytes + offset);
97  }
98  // write source contents to buffer
99  writeData(src, num_bytes, offset, src_buffer_type, src_device_id);
100 
101  // update dirty flags for buffer and each affected page
102  setDirty();
103  if (offset < size_) {
104  setUpdated();
105  }
106  if (offset + num_bytes > size_) {
107  setAppended();
108  size_ = offset + num_bytes;
109  }
110 
111  // std::cout << "Size after write: " << size_ << std::endl;
112 
113  size_t first_dirty_page = offset / page_size_;
114  size_t last_dirty_page = (offset + num_bytes - 1) / page_size_;
115  for (size_t i = first_dirty_page; i <= last_dirty_page; ++i) {
116  page_dirty_flags_[i] = true;
117  }
118 }
virtual void writeData(int8_t *const src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int src_device_id=-1)=0
#define CHECK_GT(x, y)
Definition: Logger.h:209
std::vector< bool > page_dirty_flags_
indicates when the buffer was last flushed
Definition: Buffer.h:161
mapd_unique_lock< mapd_shared_mutex > write_lock
size_t reservedSize() const override
Returns the total number of bytes allocated for the buffer.
Definition: Buffer.h:117
void reserve(const size_t num_bytes) override
Definition: Buffer.cpp:49

+ Here is the call graph for this function:

virtual void Buffer_Namespace::Buffer::writeData ( int8_t *const  src,
const size_t  num_bytes,
const size_t  offset = 0,
const MemoryLevel  src_buffer_type = CPU_LEVEL,
const int  src_device_id = -1 
)
privatepure virtual

Implemented in Buffer_Namespace::CpuBuffer, and Buffer_Namespace::GpuCudaBuffer.

Referenced by append(), and write().

+ Here is the caller graph for this function:

Friends And Related Function Documentation

friend class BufferMgr
friend

Definition at line 43 of file Buffer.h.

friend class FileMgr
friend

Definition at line 44 of file Buffer.h.

Member Data Documentation

BufferMgr* Buffer_Namespace::Buffer::bm_
private

Definition at line 156 of file Buffer.h.

Referenced by reserve().

int Buffer_Namespace::Buffer::epoch_
private

Definition at line 160 of file Buffer.h.

size_t Buffer_Namespace::Buffer::num_pages_
private

the size of each page in the buffer

Definition at line 159 of file Buffer.h.

Referenced by reserve().

std::vector<bool> Buffer_Namespace::Buffer::page_dirty_flags_
private

indicates when the buffer was last flushed

Definition at line 161 of file Buffer.h.

Referenced by reserve(), and write().

size_t Buffer_Namespace::Buffer::page_size_
private

Definition at line 158 of file Buffer.h.

Referenced by reserve(), and write().

int Buffer_Namespace::Buffer::pin_count_
private

Definition at line 162 of file Buffer.h.

std::mutex Buffer_Namespace::Buffer::pin_mutex_
private

Definition at line 163 of file Buffer.h.

BufferList::iterator Buffer_Namespace::Buffer::seg_it_
private

Definition at line 157 of file Buffer.h.

Referenced by Buffer(), Buffer_Namespace::BufferMgr::free(), and reserve().


The documentation for this class was generated from the following files: