OmniSciDB  1dac507f6e
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
Buffer_Namespace::Buffer Class Referenceabstract

Note(s): Forbid Copying Idiom 4.1. More...

#include <Buffer.h>

+ Inheritance diagram for Buffer_Namespace::Buffer:
+ Collaboration diagram for Buffer_Namespace::Buffer:

Public Member Functions

 Buffer (BufferMgr *bm, BufferList::iterator seg_it, const int device_id, const size_t page_size=512, const size_t num_bytes=0)
 Constructs a Buffer object. The constructor requires a memory address (provided by BufferMgr), number of pages, and the size in bytes of each page. Additionally, the Buffer can be initialized with an epoch. More...
 
 ~Buffer () override
 Destructor. More...
 
void read (int8_t *const dst, const size_t num_bytes, const size_t offset=0, const MemoryLevel dst_buffer_type=CPU_LEVEL, const int device_id=-1) override
 Reads (copies) data from the buffer to the destination (dst) memory location. Reads (copies) nbytes of data from the buffer, beginning at the specified byte offset, into the destination (dst) memory location. More...
 
void reserve (const size_t num_bytes) override
 
void write (int8_t *src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int device_id=-1) override
 Writes (copies) data from src into the buffer. Writes (copies) nbytes of data into the buffer at the specified byte offset, from the source (src) memory location. More...
 
void append (int8_t *src, const size_t num_bytes, const MemoryLevel src_buffer_type=CPU_LEVEL, const int deviceId=-1) override
 
int8_t * getMemoryPtr () override
 Returns a raw, constant (read-only) pointer to the underlying buffer. More...
 
size_t size () const override
 
size_t reservedSize () const override
 Returns the total number of bytes allocated for the buffer. More...
 
size_t pageCount () const override
 Returns the number of pages in the buffer. More...
 
size_t pageSize () const override
 Returns the size in bytes of each page in the buffer. More...
 
bool isDirty () const override
 Returns whether or not the buffer has been modified since the last flush/checkpoint. More...
 
int pin () override
 
int unPin () override
 
int getPinCount () override
 
- Public Member Functions inherited from Data_Namespace::AbstractBuffer
 AbstractBuffer (const int device_id)
 
 AbstractBuffer (const int device_id, const SQLTypeInfo sql_type)
 
virtual ~AbstractBuffer ()
 
virtual int getDeviceId () const
 
virtual MemoryLevel getType () const =0
 
virtual bool isAppended () const
 
virtual bool isUpdated () const
 
virtual void setDirty ()
 
virtual void setUpdated ()
 
virtual void setAppended ()
 
void setSize (const size_t size)
 
void clearDirtyBits ()
 
void initEncoder (const SQLTypeInfo tmp_sql_type)
 
void syncEncoder (const AbstractBuffer *src_buffer)
 

Protected Attributes

int8_t * mem_
 
- Protected Attributes inherited from Data_Namespace::AbstractBuffer
size_t size_
 
bool is_dirty_
 
bool is_appended_
 
bool is_updated_
 
int device_id_
 

Private Member Functions

 Buffer (const Buffer &)
 pointer to beginning of buffer's memory More...
 
Bufferoperator= (const Buffer &)
 
virtual void readData (int8_t *const dst, const size_t num_bytes, const size_t offset=0, const MemoryLevel dst_buffer_type=CPU_LEVEL, const int dst_device_id=-1)=0
 
virtual void writeData (int8_t *const src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int src_device_id=-1)=0
 

Private Attributes

BufferMgrbm_
 
BufferList::iterator seg_it_
 
size_t page_size_
 
size_t num_pages_
 the size of each page in the buffer More...
 
int epoch_
 
std::vector< bool > page_dirty_flags_
 indicates when the buffer was last flushed More...
 
int pin_count_
 
std::mutex pin_mutex_
 

Friends

class BufferMgr
 
class FileMgr
 

Additional Inherited Members

- Public Attributes inherited from Data_Namespace::AbstractBuffer
std::unique_ptr< Encoderencoder
 
bool has_encoder
 
SQLTypeInfo sql_type
 

Detailed Description

Note(s): Forbid Copying Idiom 4.1.

Definition at line 42 of file Buffer.h.

Constructor & Destructor Documentation

Buffer_Namespace::Buffer::Buffer ( BufferMgr bm,
BufferList::iterator  seg_it,
const int  device_id,
const size_t  page_size = 512,
const size_t  num_bytes = 0 
)

Constructs a Buffer object. The constructor requires a memory address (provided by BufferMgr), number of pages, and the size in bytes of each page. Additionally, the Buffer can be initialized with an epoch.

Parameters
memThe beginning memory address of the buffer.
numPagesThe number of pages into which the buffer's memory space is divided.
pageSizeThe size in bytes of each page that composes the buffer.
epochA temporal reference implying the buffer is up-to-date up to the epoch.

Definition at line 36 of file Buffer.cpp.

References pin(), reserve(), and seg_it_.

41  : AbstractBuffer(device_id)
42  , mem_(0)
43  , bm_(bm)
44  , seg_it_(seg_it)
45  , page_size_(page_size)
46  , num_pages_(0)
47  , pin_count_(0) {
48  pin();
49  // so that the pointer value of this Buffer is stored
50  seg_it_->buffer = this;
51  if (num_bytes > 0) {
52  reserve(num_bytes);
53  }
54 }
size_t num_pages_
the size of each page in the buffer
Definition: Buffer.h:164
BufferList::iterator seg_it_
Definition: Buffer.h:162
AbstractBuffer(const int device_id)
void reserve(const size_t num_bytes) override
Definition: Buffer.cpp:58
int pin() override
Definition: Buffer.h:130

+ Here is the call graph for this function:

Buffer_Namespace::Buffer::~Buffer ( )
override

Destructor.

Definition at line 56 of file Buffer.cpp.

56 {}
Buffer_Namespace::Buffer::Buffer ( const Buffer )
private

pointer to beginning of buffer's memory

Member Function Documentation

void Buffer_Namespace::Buffer::append ( int8_t *  src,
const size_t  num_bytes,
const MemoryLevel  src_buffer_type = CPU_LEVEL,
const int  deviceId = -1 
)
overridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 128 of file Buffer.cpp.

References Data_Namespace::AbstractBuffer::is_appended_, Data_Namespace::AbstractBuffer::is_dirty_, reserve(), reservedSize(), Data_Namespace::AbstractBuffer::size_, and writeData().

131  {
132 #ifdef BUFFER_MUTEX
133  boost::shared_lock<boost::shared_mutex> read_lock(
134  read_write_mutex_); // keep another thread from getting a write lock
135  boost::unique_lock<boost::shared_mutex> append_lock(
136  append_mutex_); // keep another thread from getting an append lock
137 #endif
138 
139  is_dirty_ = true;
140  is_appended_ = true;
141 
142  if (num_bytes + size_ > reservedSize()) {
143  reserve(num_bytes + size_);
144  }
145 
146  writeData(src, num_bytes, size_, src_buffer_type, src_device_id);
147  size_ += num_bytes;
148  // Do we worry about dirty flags here or does append avoid them
149 }
virtual void writeData(int8_t *const src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int src_device_id=-1)=0
int64_t * src
size_t reservedSize() const override
Returns the total number of bytes allocated for the buffer.
Definition: Buffer.h:119
void reserve(const size_t num_bytes) override
Definition: Buffer.cpp:58

+ Here is the call graph for this function:

int8_t * Buffer_Namespace::Buffer::getMemoryPtr ( )
overridevirtual

Returns a raw, constant (read-only) pointer to the underlying buffer.

Returns
A constant memory pointer for read-only access.

Implements Data_Namespace::AbstractBuffer.

Definition at line 151 of file Buffer.cpp.

References mem_.

151  {
152  return mem_;
153 }
int Buffer_Namespace::Buffer::getPinCount ( )
inlineoverridevirtual

Reimplemented from Data_Namespace::AbstractBuffer.

Definition at line 139 of file Buffer.h.

139  {
140  std::lock_guard<std::mutex> pin_lock(pin_mutex_);
141  return (pin_count_);
142  }
std::mutex pin_mutex_
Definition: Buffer.h:168
bool Buffer_Namespace::Buffer::isDirty ( ) const
inlineoverridevirtual

Returns whether or not the buffer has been modified since the last flush/checkpoint.

Reimplemented from Data_Namespace::AbstractBuffer.

Definition at line 128 of file Buffer.h.

Buffer& Buffer_Namespace::Buffer::operator= ( const Buffer )
private
size_t Buffer_Namespace::Buffer::pageCount ( ) const
inlineoverridevirtual

Returns the number of pages in the buffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 122 of file Buffer.h.

122 { return num_pages_; }
size_t num_pages_
the size of each page in the buffer
Definition: Buffer.h:164
size_t Buffer_Namespace::Buffer::pageSize ( ) const
inlineoverridevirtual

Returns the size in bytes of each page in the buffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 125 of file Buffer.h.

125 { return page_size_; }
int Buffer_Namespace::Buffer::pin ( )
inlineoverridevirtual

Reimplemented from Data_Namespace::AbstractBuffer.

Definition at line 130 of file Buffer.h.

Referenced by Buffer().

130  {
131  std::lock_guard<std::mutex> pin_lock(pin_mutex_);
132  return (++pin_count_);
133  }
std::mutex pin_mutex_
Definition: Buffer.h:168

+ Here is the caller graph for this function:

void Buffer_Namespace::Buffer::read ( int8_t *const  dst,
const size_t  num_bytes,
const size_t  offset = 0,
const MemoryLevel  dst_buffer_type = CPU_LEVEL,
const int  device_id = -1 
)
overridevirtual

Reads (copies) data from the buffer to the destination (dst) memory location. Reads (copies) nbytes of data from the buffer, beginning at the specified byte offset, into the destination (dst) memory location.

Parameters
dstThe destination address to where the buffer's data is being copied.
offsetThe byte offset into the buffer from where reading (copying) begins.
nbytesThe number of bytes being read (copied) into the destination (dst).

Implements Data_Namespace::AbstractBuffer.

Definition at line 76 of file Buffer.cpp.

References CHECK(), logger::FATAL, LOG, mem_, readData(), and Data_Namespace::AbstractBuffer::size_.

80  {
81  if (num_bytes == 0) {
82  return;
83  }
84  CHECK(dst && mem_);
85 #ifdef BUFFER_MUTEX
86  boost::shared_lock<boost::shared_mutex> read_lock(read_write_mutex_);
87 #endif
88 
89  if (num_bytes + offset > size_) {
90  LOG(FATAL) << "Buffer: Out of bounds read error";
91  }
92  readData(dst, num_bytes, offset, dst_buffer_type, dst_device_id);
93 }
#define LOG(tag)
Definition: Logger.h:185
CHECK(cgen_state)
virtual void readData(int8_t *const dst, const size_t num_bytes, const size_t offset=0, const MemoryLevel dst_buffer_type=CPU_LEVEL, const int dst_device_id=-1)=0

+ Here is the call graph for this function:

virtual void Buffer_Namespace::Buffer::readData ( int8_t *const  dst,
const size_t  num_bytes,
const size_t  offset = 0,
const MemoryLevel  dst_buffer_type = CPU_LEVEL,
const int  dst_device_id = -1 
)
privatepure virtual

Implemented in Buffer_Namespace::CpuBuffer, and Buffer_Namespace::GpuCudaBuffer.

Referenced by read().

+ Here is the caller graph for this function:

void Buffer_Namespace::Buffer::reserve ( const size_t  num_bytes)
overridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 58 of file Buffer.cpp.

References bm_, num_pages_, page_dirty_flags_, page_size_, Buffer_Namespace::BufferMgr::reserveBuffer(), and seg_it_.

Referenced by append(), Buffer(), and write().

58  {
59 #ifdef BUFFER_MUTEX
60  boost::unique_lock<boost::shared_mutex> write_lock(read_write_mutex_);
61 #endif
62  size_t num_pages = (num_bytes + page_size_ - 1) / page_size_;
63  // std::cout << "NumPages reserved: " << numPages << std::endl;
64  if (num_pages > num_pages_) {
65  // When running out of cpu buffers, reserveBuffer() will fail and
66  // trigger a SlabTooBig exception, so pageDirtyFlags_ and numPages_
67  // MUST NOT be set until reserveBuffer() returns; otherwise, this
68  // buffer is not properly resized, so any call to FileMgr::fetchBuffer()
69  // will proceed to read(), corrupt heap memory and cause core dump later.
70  seg_it_ = bm_->reserveBuffer(seg_it_, page_size_ * num_pages);
71  page_dirty_flags_.resize(num_pages);
72  num_pages_ = num_pages;
73  }
74 }
size_t num_pages_
the size of each page in the buffer
Definition: Buffer.h:164
BufferList::iterator seg_it_
Definition: Buffer.h:162
BufferList::iterator reserveBuffer(BufferList::iterator &seg_it, const size_t num_bytes)
Definition: BufferMgr.cpp:182
std::vector< bool > page_dirty_flags_
indicates when the buffer was last flushed
Definition: Buffer.h:166

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

size_t Buffer_Namespace::Buffer::reservedSize ( ) const
inlineoverridevirtual

Returns the total number of bytes allocated for the buffer.

Implements Data_Namespace::AbstractBuffer.

Definition at line 119 of file Buffer.h.

Referenced by append(), and write().

119 { return page_size_ * num_pages_; }
size_t num_pages_
the size of each page in the buffer
Definition: Buffer.h:164

+ Here is the caller graph for this function:

size_t Buffer_Namespace::Buffer::size ( ) const
inlineoverridevirtual

Implements Data_Namespace::AbstractBuffer.

Definition at line 116 of file Buffer.h.

116 { return size_; }
int Buffer_Namespace::Buffer::unPin ( )
inlineoverridevirtual

Reimplemented from Data_Namespace::AbstractBuffer.

Definition at line 135 of file Buffer.h.

135  {
136  std::lock_guard<std::mutex> pin_lock(pin_mutex_);
137  return (--pin_count_);
138  }
std::mutex pin_mutex_
Definition: Buffer.h:168
void Buffer_Namespace::Buffer::write ( int8_t *  src,
const size_t  num_bytes,
const size_t  offset = 0,
const MemoryLevel  src_buffer_type = CPU_LEVEL,
const int  device_id = -1 
)
overridevirtual

Writes (copies) data from src into the buffer. Writes (copies) nbytes of data into the buffer at the specified byte offset, from the source (src) memory location.

Parameters
srcThe source address from where data is being copied to the buffer.
num_bytesThe number of bytes being written (copied) into the buffer.
offsetThe byte offset into the buffer to where writing begins.

Implements Data_Namespace::AbstractBuffer.

Definition at line 95 of file Buffer.cpp.

References CHECK_GT, Data_Namespace::AbstractBuffer::is_appended_, Data_Namespace::AbstractBuffer::is_dirty_, Data_Namespace::AbstractBuffer::is_updated_, page_dirty_flags_, page_size_, reserve(), reservedSize(), Data_Namespace::AbstractBuffer::size_, and writeData().

99  {
100  CHECK_GT(num_bytes, size_t(0)); // cannot write 0 bytes
101 #ifdef BUFFER_MUTEX
102  boost::unique_lock<boost::shared_mutex> write_lock(read_write_mutex_);
103 #endif
104  if (num_bytes + offset > reservedSize()) {
105  reserve(num_bytes + offset);
106  }
107  // write source contents to buffer
108  writeData(src, num_bytes, offset, src_buffer_type, src_device_id);
109 
110  // update dirty flags for buffer and each affected page
111  is_dirty_ = true;
112  if (offset < size_) {
113  is_updated_ = true;
114  }
115  if (offset + num_bytes > size_) {
116  is_appended_ = true;
117  size_ = offset + num_bytes;
118  }
119  // std::cout << "Size after write: " << size_ << std::endl;
120 
121  size_t first_dirty_page = offset / page_size_;
122  size_t last_dirty_page = (offset + num_bytes - 1) / page_size_;
123  for (size_t i = first_dirty_page; i <= last_dirty_page; ++i) {
124  page_dirty_flags_[i] = true;
125  }
126 }
virtual void writeData(int8_t *const src, const size_t num_bytes, const size_t offset=0, const MemoryLevel src_buffer_type=CPU_LEVEL, const int src_device_id=-1)=0
int64_t * src
#define CHECK_GT(x, y)
Definition: Logger.h:202
std::vector< bool > page_dirty_flags_
indicates when the buffer was last flushed
Definition: Buffer.h:166
size_t reservedSize() const override
Returns the total number of bytes allocated for the buffer.
Definition: Buffer.h:119
void reserve(const size_t num_bytes) override
Definition: Buffer.cpp:58

+ Here is the call graph for this function:

virtual void Buffer_Namespace::Buffer::writeData ( int8_t *const  src,
const size_t  num_bytes,
const size_t  offset = 0,
const MemoryLevel  src_buffer_type = CPU_LEVEL,
const int  src_device_id = -1 
)
privatepure virtual

Implemented in Buffer_Namespace::CpuBuffer, and Buffer_Namespace::GpuCudaBuffer.

Referenced by append(), and write().

+ Here is the caller graph for this function:

Friends And Related Function Documentation

friend class BufferMgr
friend

Definition at line 43 of file Buffer.h.

friend class FileMgr
friend

Definition at line 44 of file Buffer.h.

Member Data Documentation

BufferMgr* Buffer_Namespace::Buffer::bm_
private

Definition at line 161 of file Buffer.h.

Referenced by reserve().

int Buffer_Namespace::Buffer::epoch_
private

Definition at line 165 of file Buffer.h.

size_t Buffer_Namespace::Buffer::num_pages_
private

the size of each page in the buffer

Definition at line 164 of file Buffer.h.

Referenced by reserve().

std::vector<bool> Buffer_Namespace::Buffer::page_dirty_flags_
private

indicates when the buffer was last flushed

Definition at line 166 of file Buffer.h.

Referenced by reserve(), and write().

size_t Buffer_Namespace::Buffer::page_size_
private

Definition at line 163 of file Buffer.h.

Referenced by reserve(), and write().

int Buffer_Namespace::Buffer::pin_count_
private

Definition at line 167 of file Buffer.h.

std::mutex Buffer_Namespace::Buffer::pin_mutex_
private

Definition at line 168 of file Buffer.h.

BufferList::iterator Buffer_Namespace::Buffer::seg_it_
private

Definition at line 162 of file Buffer.h.

Referenced by Buffer(), Buffer_Namespace::BufferMgr::free(), and reserve().


The documentation for this class was generated from the following files: