OmniSciDB  1dac507f6e
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
Fragmenter_Namespace::InsertDataLoader Struct Reference

#include <InsertDataLoader.h>

+ Collaboration diagram for Fragmenter_Namespace::InsertDataLoader:

Classes

struct  DistributedConnector
 

Public Member Functions

 InsertDataLoader (DistributedConnector &connector)
 
void insertData (const Catalog_Namespace::SessionInfo &session_info, InsertData &insert_data)
 

Private Member Functions

void moveToNextLeaf ()
 

Private Attributes

size_t leaf_count_
 
size_t current_leaf_index_
 
DistributedConnectorconnector_
 

Detailed Description

Definition at line 25 of file InsertDataLoader.h.

Constructor & Destructor Documentation

Fragmenter_Namespace::InsertDataLoader::InsertDataLoader ( DistributedConnector connector)
inline

Definition at line 35 of file InsertDataLoader.h.

36  : leaf_count_(connector.leafCount())
38  , connector_(connector){};

Member Function Documentation

void Fragmenter_Namespace::InsertDataLoader::insertData ( const Catalog_Namespace::SessionInfo session_info,
InsertData insert_data 
)

Definition at line 284 of file InsertDataLoader.cpp.

References CHECK(), Fragmenter_Namespace::computeRowIndicesOfShards(), connector_, Fragmenter_Namespace::copyDataOfShard(), current_leaf_index_, Catalog_Namespace::SessionInfo::getCatalog(), Catalog_Namespace::Catalog::getMetadataForTable(), Fragmenter_Namespace::InsertDataLoader::DistributedConnector::insertDataToLeaf(), Fragmenter_Namespace::InsertDataLoader::DistributedConnector::leafCount(), moveToNextLeaf(), and Fragmenter_Namespace::InsertData::tableId.

Referenced by Parser::InsertIntoTableAsSelectStmt::populateData().

285  {
286  const auto& cat = session_info.getCatalog();
287  const auto* td = cat.getMetadataForTable(insert_data.tableId);
288 
289  CHECK(td);
290  if (td->nShards == 0) {
291  connector_.insertDataToLeaf(session_info, current_leaf_index_, insert_data);
292  } else {
293  // we have a sharded target table, start spreading to physical tables
294  auto rowIndicesOfShards =
295  computeRowIndicesOfShards(cat, connector_.leafCount(), insert_data);
296 
297  auto insertShardData =
298  [this, &session_info, &insert_data, &cat, &td, &rowIndicesOfShards](
299  size_t shardId) {
300  const auto shard_tables = cat.getPhysicalTablesDescriptors(td);
301  auto stardTableIdx = shardId % td->nShards;
302  auto shardLeafIdx = shardId / td->nShards;
303 
304  const auto& rowIndicesOfShard = rowIndicesOfShards[shardId];
305  ShardDataOwner shardDataOwner;
306 
307  InsertData shardData = copyDataOfShard(
308  cat, shardDataOwner, insert_data, stardTableIdx, rowIndicesOfShard);
309  connector_.insertDataToLeaf(session_info, shardLeafIdx, shardData);
310  };
311 
312  std::vector<std::future<void>> worker_threads;
313  for (size_t shardId = 0; shardId < rowIndicesOfShards.size(); shardId++) {
314  if (rowIndicesOfShards[shardId].size() > 0) {
315  worker_threads.push_back(
316  std::async(std::launch::async, insertShardData, shardId));
317  }
318  }
319  for (auto& child : worker_threads) {
320  child.wait();
321  }
322  for (auto& child : worker_threads) {
323  child.get();
324  }
325  }
326 
327  moveToNextLeaf();
328 }
CHECK(cgen_state)
virtual void insertDataToLeaf(const Catalog_Namespace::SessionInfo &parent_session_info, const size_t leaf_idx, Fragmenter_Namespace::InsertData &insert_data)=0
Catalog & getCatalog() const
Definition: SessionInfo.h:90
std::vector< std::vector< size_t > > computeRowIndicesOfShards(size_t shardCount, size_t leafCount, size_t rowCount, SRC *src)
InsertData copyDataOfShard(const Catalog_Namespace::Catalog &cat, ShardDataOwner &dataOwner, InsertData &insert_data, int shardTableIndex, const std::vector< size_t > &rowIndices)
const TableDescriptor * getMetadataForTable(const std::string &tableName, const bool populateFragmenter=true) const
Returns a pointer to a const TableDescriptor struct matching the provided tableName.

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void Fragmenter_Namespace::InsertDataLoader::moveToNextLeaf ( )
inlineprivate

Definition at line 44 of file InsertDataLoader.h.

References current_leaf_index_, and leaf_count_.

Referenced by insertData().

+ Here is the caller graph for this function:

Member Data Documentation

DistributedConnector& Fragmenter_Namespace::InsertDataLoader::connector_
private

Definition at line 53 of file InsertDataLoader.h.

Referenced by insertData().

size_t Fragmenter_Namespace::InsertDataLoader::current_leaf_index_
private

Definition at line 52 of file InsertDataLoader.h.

Referenced by insertData(), and moveToNextLeaf().

size_t Fragmenter_Namespace::InsertDataLoader::leaf_count_
private

Definition at line 51 of file InsertDataLoader.h.

Referenced by moveToNextLeaf().


The documentation for this struct was generated from the following files: