PerfectJoinHashTableBuilder Class Reference

#include <PerfectHashTableBuilder.h>

Collaboration diagram for PerfectJoinHashTableBuilder:

Public Member Functions
	PerfectJoinHashTableBuilder (const Catalog_Namespace::Catalog *catalog)
void	allocateDeviceMemory (const JoinColumn &join_column, const HashType layout, HashEntryInfo &hash_entry_info, const size_t shard_count, const int device_id, const int device_count)
void	initOneToOneHashTableOnCpu (const JoinColumn &join_column, const ExpressionRange &col_range, const bool is_bitwise_eq, const InnerOuter &cols, const HashEntryInfo hash_entry_info, const int32_t hash_join_invalid_val, const Executor *executor)
void	initOneToManyHashTableOnCpu (const JoinColumn &join_column, const ExpressionRange &col_range, const bool is_bitwise_eq, const std::pair< const Analyzer::ColumnVar *, const Analyzer::Expr * > &cols, const HashEntryInfo hash_entry_info, const int32_t hash_join_invalid_val, const Executor *executor)
std::unique_ptr< PerfectHashTable >	getHashTable ()

Static Public Member Functions
static size_t	get_entries_per_shard (const size_t total_entry_count, const size_t shard_count)

Private Attributes
const Catalog_Namespace::Catalog *	catalog_
std::unique_ptr< PerfectHashTable >	hash_table_

Detailed Description

Definition at line 23 of file PerfectHashTableBuilder.h.

Constructor & Destructor Documentation

PerfectJoinHashTableBuilder::PerfectJoinHashTableBuilder ( const Catalog_Namespace::Catalog *  catalog )

inline

Definition at line 25 of file PerfectHashTableBuilder.h.

      : catalog_(catalog) {}

Member Function Documentation

void PerfectJoinHashTableBuilder::allocateDeviceMemory ( const JoinColumn &  join_column, const HashType  layout, HashEntryInfo &  hash_entry_info, const size_t  shard_count, const int  device_id, const int  device_count  )

inline

Definition at line 28 of file PerfectHashTableBuilder.h.

References catalog_, CHECK, CHECK_GT, get_entries_per_shard(), HashEntryInfo::getNormalizedHashEntryCount(), GPU, HashEntryInfo::hash_entry_count, hash_table_, JoinColumn::num_elems, OneToOne, and UNREACHABLE.

Referenced by PerfectJoinHashTable::initHashTableForDevice().

                                                    {
#ifdef HAVE_CUDA
    if (shard_count) {
      const auto shards_per_device = (shard_count + device_count - 1) / device_count;
      CHECK_GT(shards_per_device, 0u);
      const size_t entries_per_shard =
          get_entries_per_shard(hash_entry_info.hash_entry_count, shard_count);
      hash_entry_info.hash_entry_count = entries_per_shard * shards_per_device;
    }
    const size_t total_count =
        layout == HashType::OneToOne
            ? hash_entry_info.getNormalizedHashEntryCount()
            : 2 * hash_entry_info.getNormalizedHashEntryCount() + join_column.num_elems;
    CHECK(!hash_table_);
    hash_table_ =
        std::make_unique<PerfectHashTable>(catalog_,
                                           layout,
                                           ExecutorDeviceType::GPU,
                                           hash_entry_info.getNormalizedHashEntryCount(),
                                           join_column.num_elems);
    hash_table_->allocateGpuMemory(total_count, device_id);
#else
    UNREACHABLE();
#endif  // HAVE_CUDA
  }

Here is the call graph for this function:

Here is the caller graph for this function:

static size_t PerfectJoinHashTableBuilder::get_entries_per_shard ( const size_t  total_entry_count, const size_t  shard_count  )

inlinestatic

Definition at line 358 of file PerfectHashTableBuilder.h.

References CHECK_NE.

Referenced by allocateDeviceMemory().

                                                                {
    CHECK_NE(size_t(0), shard_count);
    return (total_entry_count + shard_count - 1) / shard_count;
  }

Here is the caller graph for this function:

std::unique_ptr<PerfectHashTable> PerfectJoinHashTableBuilder::getHashTable ( )

inline

Definition at line 356 of file PerfectHashTableBuilder.h.

References hash_table_.

Referenced by PerfectJoinHashTable::initHashTableForDevice().

{ return std::move(hash_table_); }

Here is the caller graph for this function:

void PerfectJoinHashTableBuilder::initOneToManyHashTableOnCpu ( const JoinColumn &  join_column, const ExpressionRange &  col_range, const bool  is_bitwise_eq, const std::pair< const Analyzer::ColumnVar *, const Analyzer::Expr * > &  cols, const HashEntryInfo  hash_entry_info, const int32_t  hash_join_invalid_val, const Executor *  executor  )

inline

Definition at line 270 of file PerfectHashTableBuilder.h.

References catalog_, CHECK, CHECK_EQ, CPU, cpu_threads(), DEBUG_TIMER, fill_one_to_many_hash_table(), fill_one_to_many_hash_table_bucketized(), get_join_column_type_kind(), ExpressionRange::getIntMax(), ExpressionRange::getIntMin(), HashEntryInfo::getNormalizedHashEntryCount(), Executor::getStringDictionaryProxy(), hash_table_, init_hash_join_buff(), inline_fixed_encoding_null_val(), kDATE, kENCODING_DICT, JoinColumn::num_elems, and OneToMany.

Referenced by PerfectJoinHashTable::initHashTableForDevice().

                                {
    auto timer = DEBUG_TIMER(__func__);
    const auto inner_col = cols.first;
    CHECK(inner_col);
    const auto& ti = inner_col->get_type_info();

    CHECK(!hash_table_);
    hash_table_ =
        std::make_unique<PerfectHashTable>(catalog_,
                                           HashType::OneToMany,
                                           ExecutorDeviceType::CPU,
                                           hash_entry_info.getNormalizedHashEntryCount(),
                                           join_column.num_elems);

    auto cpu_hash_table_buff = reinterpret_cast<int32_t*>(hash_table_->getCpuBuffer());
    const StringDictionaryProxy* sd_inner_proxy{nullptr};
    const StringDictionaryProxy* sd_outer_proxy{nullptr};
    if (ti.is_string()) {
      CHECK_EQ(kENCODING_DICT, ti.get_compression());
      sd_inner_proxy =
          executor->getStringDictionaryProxy(inner_col->get_comp_param(), true);
      CHECK(sd_inner_proxy);
      const auto outer_col = dynamic_cast<const Analyzer::ColumnVar*>(cols.second);
      CHECK(outer_col);
      sd_outer_proxy =
          executor->getStringDictionaryProxy(outer_col->get_comp_param(), true);
      CHECK(sd_outer_proxy);
    }
    int thread_count = cpu_threads();
    std::vector<std::future<void>> init_threads;
    for (int thread_idx = 0; thread_idx < thread_count; ++thread_idx) {
      init_threads.emplace_back(std::async(std::launch::async,
                                           init_hash_join_buff,
                                           cpu_hash_table_buff,
                                           hash_entry_info.getNormalizedHashEntryCount(),
                                           hash_join_invalid_val,
                                           thread_idx,
                                           thread_count));
    }
    for (auto& child : init_threads) {
      child.wait();
    }
    for (auto& child : init_threads) {
      child.get();
    }

    if (ti.get_type() == kDATE) {
      fill_one_to_many_hash_table_bucketized(cpu_hash_table_buff,
                                             hash_entry_info,
                                             hash_join_invalid_val,
                                             join_column,
                                             {static_cast<size_t>(ti.get_size()),
                                              col_range.getIntMin(),
                                              col_range.getIntMax(),
                                              inline_fixed_encoding_null_val(ti),
                                              is_bitwise_eq,
                                              col_range.getIntMax() + 1,
                                              get_join_column_type_kind(ti)},
                                             sd_inner_proxy,
                                             sd_outer_proxy,
                                             thread_count);
    } else {
      fill_one_to_many_hash_table(cpu_hash_table_buff,
                                  hash_entry_info,
                                  hash_join_invalid_val,
                                  join_column,
                                  {static_cast<size_t>(ti.get_size()),
                                   col_range.getIntMin(),
                                   col_range.getIntMax(),
                                   inline_fixed_encoding_null_val(ti),
                                   is_bitwise_eq,
                                   col_range.getIntMax() + 1,
                                   get_join_column_type_kind(ti)},
                                  sd_inner_proxy,
                                  sd_outer_proxy,
                                  thread_count);
    }
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void PerfectJoinHashTableBuilder::initOneToOneHashTableOnCpu ( const JoinColumn &  join_column, const ExpressionRange &  col_range, const bool  is_bitwise_eq, const InnerOuter &  cols, const HashEntryInfo  hash_entry_info, const int32_t  hash_join_invalid_val, const Executor *  executor  )

inline

Definition at line 172 of file PerfectHashTableBuilder.h.

References HashEntryInfo::bucket_normalization, catalog_, CHECK, CHECK_EQ, CPU, cpu_threads(), DEBUG_TIMER, fill_hash_join_buff_bucketized(), get_join_column_type_kind(), ExpressionRange::getIntMax(), ExpressionRange::getIntMin(), HashEntryInfo::getNormalizedHashEntryCount(), Executor::getStringDictionaryProxy(), hash_table_, init_hash_join_buff(), inline_fixed_encoding_null_val(), kENCODING_DICT, OneToOne, and generate_TableFunctionsFactory_init::t.

Referenced by PerfectJoinHashTable::initHashTableForDevice().

                                                            {
    auto timer = DEBUG_TIMER(__func__);
    const auto inner_col = cols.first;
    CHECK(inner_col);
    const auto& ti = inner_col->get_type_info();

    CHECK(!hash_table_);
    hash_table_ =
        std::make_unique<PerfectHashTable>(catalog_,
                                           HashType::OneToOne,
                                           ExecutorDeviceType::CPU,
                                           hash_entry_info.getNormalizedHashEntryCount(),
                                           0);

    auto cpu_hash_table_buff = reinterpret_cast<int32_t*>(hash_table_->getCpuBuffer());
    const StringDictionaryProxy* sd_inner_proxy{nullptr};
    const StringDictionaryProxy* sd_outer_proxy{nullptr};
    const auto outer_col = dynamic_cast<const Analyzer::ColumnVar*>(cols.second);
    if (ti.is_string() &&
        (outer_col && !(inner_col->get_comp_param() == outer_col->get_comp_param()))) {
      CHECK_EQ(kENCODING_DICT, ti.get_compression());
      sd_inner_proxy =
          executor->getStringDictionaryProxy(inner_col->get_comp_param(), true);
      CHECK(sd_inner_proxy);
      CHECK(outer_col);
      sd_outer_proxy =
          executor->getStringDictionaryProxy(outer_col->get_comp_param(), true);
      CHECK(sd_outer_proxy);
    }
    int thread_count = cpu_threads();
    std::vector<std::thread> init_cpu_buff_threads;
    for (int thread_idx = 0; thread_idx < thread_count; ++thread_idx) {
      init_cpu_buff_threads.emplace_back([hash_entry_info,
                                          hash_join_invalid_val,
                                          thread_idx,
                                          thread_count,
                                          cpu_hash_table_buff] {
        init_hash_join_buff(cpu_hash_table_buff,
                            hash_entry_info.getNormalizedHashEntryCount(),
                            hash_join_invalid_val,
                            thread_idx,
                            thread_count);
      });
    }
    for (auto& t : init_cpu_buff_threads) {
      t.join();
    }
    init_cpu_buff_threads.clear();
    std::atomic<int> err{0};
    for (int thread_idx = 0; thread_idx < thread_count; ++thread_idx) {
      init_cpu_buff_threads.emplace_back([hash_join_invalid_val,
                                          &join_column,
                                          sd_inner_proxy,
                                          sd_outer_proxy,
                                          thread_idx,
                                          thread_count,
                                          &ti,
                                          &err,
                                          &col_range,
                                          &is_bitwise_eq,
                                          cpu_hash_table_buff,
                                          hash_entry_info] {
        int partial_err =
            fill_hash_join_buff_bucketized(cpu_hash_table_buff,
                                           hash_join_invalid_val,
                                           join_column,
                                           {static_cast<size_t>(ti.get_size()),
                                            col_range.getIntMin(),
                                            col_range.getIntMax(),
                                            inline_fixed_encoding_null_val(ti),
                                            is_bitwise_eq,
                                            col_range.getIntMax() + 1,
                                            get_join_column_type_kind(ti)},
                                           sd_inner_proxy,
                                           sd_outer_proxy,
                                           thread_idx,
                                           thread_count,
                                           hash_entry_info.bucket_normalization);
        int zero{0};
        err.compare_exchange_strong(zero, partial_err);
      });
    }
    for (auto& t : init_cpu_buff_threads) {
      t.join();
    }
    if (err) {
      // Too many hash entries, need to retry with a 1:many table
      hash_table_ = nullptr;  // clear the hash table buffer
      throw NeedsOneToManyHash();
    }
  }

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

const Catalog_Namespace::Catalog* PerfectJoinHashTableBuilder::catalog_

private

Definition at line 365 of file PerfectHashTableBuilder.h.

Referenced by allocateDeviceMemory(), initOneToManyHashTableOnCpu(), and initOneToOneHashTableOnCpu().

std::unique_ptr<PerfectHashTable> PerfectJoinHashTableBuilder::hash_table_

private

Definition at line 367 of file PerfectHashTableBuilder.h.

Referenced by allocateDeviceMemory(), getHashTable(), initOneToManyHashTableOnCpu(), and initOneToOneHashTableOnCpu().

---

The documentation for this class was generated from the following file:

• /home/jenkins-slave/workspace/core-os-doxygen/QueryEngine/JoinHashTable/Builders/PerfectHashTableBuilder.h