StringDictionaryProxy Class Reference

#include <StringDictionaryProxy.h>

Public Member Functions
	StringDictionaryProxy (std::shared_ptr< StringDictionary > sd, const int32_t string_dict_id, const int64_t generation)
int32_t	getDictId () const noexcept
int32_t	getOrAdd (const std::string &str) noexcept
StringDictionary *	getDictionary () const noexcept
int64_t	getGeneration () const noexcept
std::vector< int32_t >	getTransientBulk (const std::vector< std::string > &strings) const
	Executes read-only lookup of a vector of strings and returns a vector of their integer ids. More...
int32_t	getOrAddTransient (const std::string &str)
std::vector< int32_t >	getOrAddTransientBulk (const std::vector< std::string > &strings)
int32_t	getIdOfString (const std::string &str) const
int32_t	getIdOfStringNoGeneration (const std::string &str) const
std::string	getString (int32_t string_id) const
std::vector< std::string >	getStrings (const std::vector< int32_t > &string_ids) const
std::pair< const char *, size_t >	getStringBytes (int32_t string_id) const noexcept
StringDictionaryProxyRange	getRange () const
StringDictionaryProxyRange	getRangeUnlocked () const
std::shared_ptr < StringDictionaryProxyTranslationMap >	buildTranslationMapToOtherProxy (const StringDictionaryProxy *dest_proxy) const
	Builds a vectorized string_id translation map from this proxy to dest_proxy. More...
size_t	storageEntryCount () const
	Returns the number of string entries in the underlying string dictionary, at this proxy's generation_ if it is set/valid, otherwise just the current size of the dictionary. More...
size_t	transientEntryCount () const
	Returns the number of transient string entries for this proxy,. More...
size_t	entryCount () const
	Returns the number of total string entries for this proxy, both stored in the underlying dictionary and in the transient map. Equal to storageEntryCount() + transientEntryCount() More...
void	updateGeneration (const int64_t generation) noexcept
std::vector< int32_t >	getLike (const std::string &pattern, const bool icase, const bool is_simple, const char escape) const
std::vector< int32_t >	getCompare (const std::string &pattern, const std::string &comp_operator) const
std::vector< int32_t >	getRegexpLike (const std::string &pattern, const char escape) const
const std::map< int32_t, std::string >	getTransientMapping () const
template<>
int32_t	lookupTransientStringUnlocked (const std::string &lookup_string) const
template<>
int32_t	lookupTransientStringUnlocked (const std::string_view &lookup_string) const

Private Member Functions
size_t	transientEntryCountUnlocked () const
size_t	entryCountUnlocked () const
int32_t	transientLookupAndAddUnlocked (const std::string &str)
template<typename String >
int32_t	lookupTransientStringUnlocked (const String &lookup_string) const
template<typename String >
void	transientLookupBulk (const std::vector< String > &lookup_strings, int32_t *string_ids) const
template<typename String >
void	transientLookupBulkUnlocked (const std::vector< String > &lookup_strings, int32_t *string_ids) const
template<typename String >
void	transientLookupBulkParallelUnlocked (const std::vector< String > &lookup_strings, int32_t *string_ids) const

Private Attributes
std::shared_ptr< StringDictionary >	string_dict_
const int32_t	string_dict_id_
std::map< int32_t, std::string >	transient_int_to_str_
std::map< std::string, int32_t >	transient_str_to_int_
int64_t	generation_
mapd_shared_mutex	rw_mutex_

Friends
bool	operator== (const StringDictionaryProxy &sdp1, const StringDictionaryProxy &sdp2)
bool	operator!= (const StringDictionaryProxy &sdp1, const StringDictionaryProxy &sdp2)

Detailed Description

Definition at line 63 of file StringDictionaryProxy.h.

Constructor & Destructor Documentation

StringDictionaryProxy::StringDictionaryProxy	(	std::shared_ptr< StringDictionary >	sd,
		const int32_t	string_dict_id,
		const int64_t	generation
	)

Definition at line 37 of file StringDictionaryProxy.cpp.

    : string_dict_(sd), string_dict_id_(string_dict_id), generation_(generation) {}

Member Function Documentation

std::shared_ptr< StringDictionaryProxyTranslationMap > StringDictionaryProxy::buildTranslationMapToOtherProxy

(

const StringDictionaryProxy *

dest_proxy

)

const

Builds a vectorized string_id translation map from this proxy to dest_proxy.

Parameters

dest_proxy

StringDictionaryProxy that we are to map this proxy's string ids to

Returns

A shared_ptr to a StringDictionaryProxyTranslationMap, which contains both the source domain range (i.e. the min/max string ids of this proxy), and a vector representing a lnear dense vector map of source proxy ids to destination proxy ids, where index 0 corresponds to the lowest (negative) transient id in this proxy, and with each increasing index corresponding to the next string_id I.e. if there are 3 transient entries in this proxy, and 20 in the underlying string dictionary, there will be 25 total entries, mapping transient id -5 (as -1 and -0 are reserved, transients start at -2 (transient_id_ceil) and descend downward). Entries corresponding to -1 and 0 may contain garbage, it is expected that these entries are never accessed. The payload of the vector map are the string ids in the dest_proxy corresponding to the indexed string ids from this proxy

Definition at line 205 of file StringDictionaryProxy.cpp.

References CHECK, CHECK_GE, CHECK_GT, CHECK_LE, gpu_enabled::copy(), DEBUG_TIMER, entryCountUnlocked(), generation_, getRangeUnlocked(), getTransientBulk(), StringDictionary::INVALID_STR_ID, lookupTransientStringUnlocked(), rw_mutex_, string_dict_, transient_id_ceil, transient_int_to_str_, transientEntryCountUnlocked(), and VLOG.

                                                   {
  auto timer = DEBUG_TIMER(__func__);
  mapd_shared_lock<mapd_shared_mutex> read_lock(rw_mutex_);
  auto str_proxy_translation_map =
      std::make_shared<StringDictionaryProxyTranslationMap>(getRangeUnlocked());
  CHECK(str_proxy_translation_map);

  if (str_proxy_translation_map->isEmpty()) {
    return str_proxy_translation_map;
  }

  // First map transient strings, store at front of vector map
  const size_t num_transient_entries = str_proxy_translation_map->numTransientEntries();
  if (num_transient_entries) {
    std::vector<std::string> transient_lookup_strings(num_transient_entries);

    for (const auto& transient_entry : transient_int_to_str_) {
      const auto transient_id = transient_entry.first;
      CHECK_LE(transient_id, transient_id_ceil);
      CHECK_GT(transient_id,
               transient_id_ceil - static_cast<int32_t>(num_transient_entries));
      const size_t map_idx = transient_entry.first + num_transient_entries + 1;
      transient_lookup_strings[map_idx] = transient_entry.second;
    }
    // This lookup may have a different snapshot of
    // dest_proxy transients and dictionary than what happends under
    // the below dest_proxy_read_lock. We may need an unlocked version of
    // getTransientBulk to ensure consistency (I don't believe
    // current behavior would cause crashes/races, verify this though)

    // Todo(todd): Consider implementing a getTransientBulk call that takes
    // an allocated pointer to avoid extra copy of return vector into
    // the already allocated translation map

    const auto transient_str_to_id_vec_map =
        dest_proxy->getTransientBulk(transient_lookup_strings);
    CHECK_GE(str_proxy_translation_map->size(), transient_str_to_id_vec_map.size());
    std::copy(transient_str_to_id_vec_map.begin(),
              transient_str_to_id_vec_map.end(),
              str_proxy_translation_map->translation_map_.begin());
  }

  // Now map strings in dictionary
  // We start non transient strings after the transient strings
  // if they exist, otherwise at 0
  auto translation_map_stored_entries_ptr =
      str_proxy_translation_map->storageEntriesPtr();

  auto dest_transient_lookup_callback = [dest_proxy, translation_map_stored_entries_ptr](
                                            const std::string_view& source_string,
                                            const int32_t source_string_id) {
    translation_map_stored_entries_ptr[source_string_id] =
        dest_proxy->lookupTransientStringUnlocked(source_string);
    return translation_map_stored_entries_ptr[source_string_id] ==
           StringDictionary::INVALID_STR_ID;
  };

  mapd_lock_guard<mapd_shared_mutex> dest_proxy_read_lock(dest_proxy->rw_mutex_);
  const size_t num_dest_transients = dest_proxy->transientEntryCountUnlocked();

  const size_t num_strings_not_translated =
      string_dict_->buildDictionaryTranslationMap(dest_proxy->string_dict_.get(),
                                                  translation_map_stored_entries_ptr,
                                                  generation_,
                                                  dest_proxy->generation_,
                                                  num_dest_transients > 0UL,
                                                  dest_transient_lookup_callback);
  const size_t num_dest_entries = dest_proxy->entryCountUnlocked();
  const size_t num_total_entries = str_proxy_translation_map->size();
  CHECK_GT(num_total_entries, 0UL);
  CHECK_LE(num_strings_not_translated, str_proxy_translation_map->size());
  const size_t num_entries_translated = num_total_entries - num_strings_not_translated;
  const float match_pct =
      100.0 * static_cast<float>(num_entries_translated) / num_total_entries;
  VLOG(1) << std::fixed << std::setprecision(2) << match_pct << "% ("
          << num_entries_translated << " entries) from dictionary ("
          << string_dict_->getDbId() << ", " << string_dict_->getDictId() << ") with "
          << num_total_entries << " total entries ( " << num_transient_entries
          << " literals)"
          << " translated to dictionary (" << dest_proxy->string_dict_->getDbId() << ", "
          << dest_proxy->string_dict_->getDictId() << ") with " << num_dest_entries
          << " total entries (" << dest_proxy->transientEntryCountUnlocked()
          << " literals).";

  return str_proxy_translation_map;
}

Here is the call graph for this function:

size_t StringDictionaryProxy::entryCount

(

)

const

Returns the number of total string entries for this proxy, both stored in the underlying dictionary and in the transient map. Equal to storageEntryCount() + transientEntryCount()

Returns

size_t Number of total string entries for this proxy

Definition at line 435 of file StringDictionaryProxy.cpp.

References entryCountUnlocked(), and rw_mutex_.

                                               {
  mapd_shared_lock<mapd_shared_mutex> read_lock(rw_mutex_);
  return entryCountUnlocked();
}

Here is the call graph for this function:

size_t StringDictionaryProxy::entryCountUnlocked ( ) const

private

Definition at line 431 of file StringDictionaryProxy.cpp.

References storageEntryCount(), and transientEntryCountUnlocked().

Referenced by buildTranslationMapToOtherProxy(), and entryCount().

                                                       {
  return storageEntryCount() + transientEntryCountUnlocked();
}

Here is the call graph for this function:

Here is the caller graph for this function:

std::vector< int32_t > StringDictionaryProxy::getCompare	(	const std::string &	pattern,
		const std::string &	comp_operator
	)		const

Definition at line 358 of file StringDictionaryProxy.cpp.

References CHECK_GE, anonymous_namespace{StringDictionaryProxy.cpp}::do_compare(), generation_, getString(), run_benchmark_import::result, string_dict_, and transient_int_to_str_.

Referenced by anonymous_namespace{StringOpsIR.cpp}::get_compared_ids().

                                          {
  CHECK_GE(generation_, 0);
  auto result = string_dict_->getCompare(pattern, comp_operator, generation_);
  for (const auto& kv : transient_int_to_str_) {
    const auto str = getString(kv.first);
    if (do_compare(str, pattern, comp_operator)) {
      result.push_back(kv.first);
    }
  }
  return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

int32_t StringDictionaryProxy::getDictId ( ) const

inlinenoexcept

Definition at line 74 of file StringDictionaryProxy.h.

References string_dict_id_.

{ return string_dict_id_; };

StringDictionary * StringDictionaryProxy::getDictionary ( ) const

noexcept

Definition at line 562 of file StringDictionaryProxy.cpp.

References string_dict_.

                                                                      {
  return string_dict_.get();
}

int64_t StringDictionaryProxy::getGeneration ( ) const

noexcept

Definition at line 566 of file StringDictionaryProxy.cpp.

References generation_.

                                                            {
  return generation_;
}

int32_t StringDictionaryProxy::getIdOfString

(

const std::string &

str

)

const

Definition at line 110 of file StringDictionaryProxy.cpp.

References CHECK_GE, generation_, StringDictionary::INVALID_STR_ID, rw_mutex_, string_dict_, transient_str_to_int_, and truncate_to_generation().

Referenced by anonymous_namespace{RelAlgTranslator.cpp}::fill_dictionary_encoded_in_vals(), and string_compress().

                                                                       {
  mapd_shared_lock<mapd_shared_mutex> read_lock(rw_mutex_);
  CHECK_GE(generation_, 0);
  auto str_id = truncate_to_generation(string_dict_->getIdOfString(str), generation_);
  if (str_id != StringDictionary::INVALID_STR_ID || transient_str_to_int_.empty()) {
    return str_id;
  }
  auto it = transient_str_to_int_.find(str);
  return it != transient_str_to_int_.end() ? it->second
                                           : StringDictionary::INVALID_STR_ID;
}

Here is the call graph for this function:

Here is the caller graph for this function:

int32_t StringDictionaryProxy::getIdOfStringNoGeneration

(

const std::string &

str

)

const

Definition at line 122 of file StringDictionaryProxy.cpp.

References StringDictionary::INVALID_STR_ID, rw_mutex_, string_dict_, and transient_str_to_int_.

                                                                                   {
  mapd_shared_lock<mapd_shared_mutex> read_lock(rw_mutex_);
  auto str_id = string_dict_->getIdOfString(str);
  if (str_id != StringDictionary::INVALID_STR_ID || transient_str_to_int_.empty()) {
    return str_id;
  }
  auto it = transient_str_to_int_.find(str);
  return it != transient_str_to_int_.end() ? it->second
                                           : StringDictionary::INVALID_STR_ID;
}

std::vector< int32_t > StringDictionaryProxy::getLike	(	const std::string &	pattern,
		const bool	icase,
		const bool	is_simple,
		const char	escape
	)		const

Definition at line 319 of file StringDictionaryProxy.cpp.

References CHECK_GE, generation_, getString(), anonymous_namespace{StringDictionary.cpp}::is_like(), run_benchmark_import::result, string_dict_, and transient_int_to_str_.

                                                                             {
  CHECK_GE(generation_, 0);
  auto result = string_dict_->getLike(pattern, icase, is_simple, escape, generation_);
  for (const auto& kv : transient_int_to_str_) {
    const auto str = getString(kv.first);
    if (is_like(str, pattern, icase, is_simple, escape)) {
      result.push_back(kv.first);
    }
  }
  return result;
}

Here is the call graph for this function:

int32_t StringDictionaryProxy::getOrAdd ( const std::string &  str )

noexcept

Definition at line 395 of file StringDictionaryProxy.cpp.

Referenced by DictionaryValueConverter< TARGET_TYPE >::convertTransientStringIdToPermanentId().

                                                                     {
  return string_dict_->getOrAdd(str);
}

Here is the caller graph for this function:

int32_t StringDictionaryProxy::getOrAddTransient

(

const std::string &

str

)

Definition at line 70 of file StringDictionaryProxy.cpp.

References CHECK_GE, generation_, StringDictionary::INVALID_STR_ID, rw_mutex_, string_dict_, transientLookupAndAddUnlocked(), and truncate_to_generation().

Referenced by lower_encoded(), and TransientStringLiteralsVisitor::visitConstant().

                                                                     {
  CHECK_GE(generation_, 0);
  auto transient_id =
      truncate_to_generation(string_dict_->getIdOfString(str), generation_);
  if (transient_id != StringDictionary::INVALID_STR_ID) {
    return transient_id;
  }
  mapd_lock_guard<mapd_shared_mutex> write_lock(rw_mutex_);
  return transientLookupAndAddUnlocked(str);
}

Here is the call graph for this function:

Here is the caller graph for this function:

std::vector< int32_t > StringDictionaryProxy::getOrAddTransientBulk

(

const std::vector< std::string > &

strings

)

Definition at line 81 of file StringDictionaryProxy.cpp.

References CHECK_GE, generation_, StringDictionary::INVALID_STR_ID, rw_mutex_, string_dict_, transientLookupAndAddUnlocked(), and truncate_to_generation().

                                         {
  CHECK_GE(generation_, 0);
  const size_t num_strings = strings.size();
  std::vector<int32_t> string_ids(num_strings);
  if (num_strings == 0) {
    return string_ids;
  }
  // Since new strings added to a StringDictionaryProxy are not materialized in the
  // proxy's underlying StringDictionary, we can use the fast parallel
  // StringDictionary::getBulk method to fetch ids from the underlying dictionary (which
  // will return StringDictionary::INVALID_STR_ID for strings that don't exist)

  // Don't need to be under lock here as the string ids for strings in the underlying
  // materialized dictionary are immutable
  const size_t num_strings_not_found = string_dict_->getBulk(strings, string_ids.data());
  if (num_strings_not_found > 0) {
    mapd_lock_guard<mapd_shared_mutex> write_lock(rw_mutex_);
    for (size_t string_idx = 0; string_idx < num_strings; ++string_idx) {
      const auto transient_id =
          truncate_to_generation(string_ids[string_idx], generation_);
      if (transient_id == StringDictionary::INVALID_STR_ID) {
        string_ids[string_idx] = transientLookupAndAddUnlocked(strings[string_idx]);
      }
    }
  }
  return string_ids;
}

Here is the call graph for this function:

std::pair< int32_t, int32_t > StringDictionaryProxy::getRange

(

)

const

Definition at line 170 of file StringDictionaryProxy.cpp.

References getRangeUnlocked(), and rw_mutex_.

                                                                {
  mapd_shared_lock<mapd_shared_mutex> read_lock(rw_mutex_);
  return getRangeUnlocked();
}

Here is the call graph for this function:

std::pair< int32_t, int32_t > StringDictionaryProxy::getRangeUnlocked

(

)

const

Definition at line 175 of file StringDictionaryProxy.cpp.

References storageEntryCount(), transient_id_ceil, and transientEntryCountUnlocked().

Referenced by buildTranslationMapToOtherProxy(), and getRange().

                                                                        {
  const int32_t storage_entry_count = storageEntryCount();
  const int32_t transient_entry_count = transientEntryCountUnlocked();
  const int32_t min_elem =
      transient_entry_count > 0 ? transient_id_ceil + 1 - transient_entry_count : 0;
  const int32_t max_elem = storage_entry_count;
  return std::make_pair(min_elem, max_elem);
}

Here is the call graph for this function:

Here is the caller graph for this function:

std::vector< int32_t > StringDictionaryProxy::getRegexpLike	(	const std::string &	pattern,
		const char	escape
	)		const

Definition at line 382 of file StringDictionaryProxy.cpp.

References CHECK_GE, generation_, getString(), anonymous_namespace{StringDictionary.cpp}::is_regexp_like(), run_benchmark_import::result, string_dict_, and transient_int_to_str_.

                                                                                   {
  CHECK_GE(generation_, 0);
  auto result = string_dict_->getRegexpLike(pattern, escape, generation_);
  for (const auto& kv : transient_int_to_str_) {
    const auto str = getString(kv.first);
    if (is_regexp_like(str, pattern, escape)) {
      result.push_back(kv.first);
    }
  }
  return result;
}

Here is the call graph for this function:

std::string StringDictionaryProxy::getString

(

int32_t

string_id

)

const

Definition at line 133 of file StringDictionaryProxy.cpp.

References CHECK, CHECK_NE, StringDictionary::INVALID_STR_ID, rw_mutex_, storageEntryCount(), string_dict_, and transient_int_to_str_.

Referenced by anonymous_namespace{ResultSetIteration.cpp}::build_string_array_target_value(), StringValueConverter::convertToColumnarFormatFromDict(), DictionaryValueConverter< TARGET_TYPE >::convertTransientStringIdToPermanentId(), anonymous_namespace{RelAlgTranslator.cpp}::fill_dictionary_encoded_in_vals(), getCompare(), getLike(), getRegexpLike(), lower_encoded(), GenericKeyHandler::operator()(), and anonymous_namespace{HashJoinRuntime.cpp}::translate_str_id_to_outer_dict().

                                                                  {
  if (inline_int_null_value<int32_t>() == string_id) {
    return "";
  }
  mapd_shared_lock<mapd_shared_mutex> read_lock(rw_mutex_);
  if (string_id >= 0 && storageEntryCount() > 0) {
    return string_dict_->getString(string_id);
  }
  CHECK_NE(StringDictionary::INVALID_STR_ID, string_id);
  auto it = transient_int_to_str_.find(string_id);
  CHECK(it != transient_int_to_str_.end());
  return it->second;
}

Here is the call graph for this function:

Here is the caller graph for this function:

std::pair< const char *, size_t > StringDictionaryProxy::getStringBytes ( int32_t  string_id ) const

noexcept

Definition at line 399 of file StringDictionaryProxy.cpp.

References CHECK, CHECK_NE, and StringDictionary::INVALID_STR_ID.

Referenced by anonymous_namespace{ExternalExecutor.cpp}::decode_string(), and string_decompress().

                                      {
  if (string_id >= 0) {
    return string_dict_.get()->getStringBytes(string_id);
  }
  CHECK_NE(StringDictionary::INVALID_STR_ID, string_id);
  auto it = transient_int_to_str_.find(string_id);
  CHECK(it != transient_int_to_str_.end());
  return std::make_pair(it->second.c_str(), it->second.size());
}

Here is the caller graph for this function:

std::vector< std::string > StringDictionaryProxy::getStrings

(

const std::vector< int32_t > &

string_ids

)

const

Definition at line 147 of file StringDictionaryProxy.cpp.

References rw_mutex_, string_dict_, and transient_int_to_str_.

                                                {
  const size_t num_string_ids = string_ids.size();
  std::vector<std::string> strings;
  if (num_string_ids == size_t(0)) {
    return strings;
  }
  strings.reserve(num_string_ids);
  mapd_shared_lock<mapd_shared_mutex> read_lock(rw_mutex_);
  for (const auto& string_id : string_ids) {
    if (inline_int_null_value<int32_t>() == string_id) {
      strings.emplace_back("");
      continue;
    }
    if (string_id >= 0) {
      strings.emplace_back(string_dict_->getString(string_id));
      continue;
    }
    auto it = transient_int_to_str_.find(string_id);
    strings.emplace_back(it->second);
  }
  return strings;
}

std::vector< int32_t > StringDictionaryProxy::getTransientBulk

(

const std::vector< std::string > &

strings

)

const

Executes read-only lookup of a vector of strings and returns a vector of their integer ids.

This function, unlike getOrAddTransientBulk, will not add strings to the dictionary. Use this function if strings that don't currently exist in the StringDictionaryProxy should not be added to the proxy as transient entries. This method also has performance advantages over getOrAddTransientBulk for read-only use cases, in that it can: 1) Take a read lock instead of a write lock for the transient lookups 2) Use a tbb::parallel_for implementation of the transient string lookups as we are guaranteed that the underlying map of strings to int ids cannot change

Parameters

strings

- Vector of strings to perform string id lookups on

Returns

A vector of string_ids of the same length as strings, containing the id of any strings for which were found in the underlying StringDictionary instance or in the proxy's tranient map, otherwise StringDictionary::INVALID_STRING_ID for strings not found.

Definition at line 50 of file StringDictionaryProxy.cpp.

References CHECK_GE, generation_, string_dict_, and transientLookupBulk().

Referenced by buildTranslationMapToOtherProxy().

                                               {
  CHECK_GE(generation_, 0);
  const size_t num_strings = strings.size();
  std::vector<int32_t> string_ids(num_strings);
  if (num_strings == 0) {
    return string_ids;
  }
  // Use fast parallel String::Dictionary getBulk method
  // Todo: Evaluate getBulk method that takes callback to do transient lookup
  // to avoid a second rescan of the data
  const size_t num_strings_not_found = string_dict_->getBulk(strings, string_ids.data());
  if (num_strings_not_found > 0) {
    // Dictionary could not find at least 1 target string, now look these up
    // in the transient dictionary
    transientLookupBulk(strings, string_ids.data());
  }
  return string_ids;
}

Here is the call graph for this function:

Here is the caller graph for this function:

const std::map<int32_t, std::string> StringDictionaryProxy::getTransientMapping ( ) const

inline

Definition at line 175 of file StringDictionaryProxy.h.

References transient_int_to_str_.

Referenced by DictionaryValueConverter< TARGET_TYPE >::DictionaryValueConverter(), and DictionaryValueConverter< TARGET_TYPE >::processBuffer().

                                                               {
    return transient_int_to_str_;
  }

Here is the caller graph for this function:

template<typename String >

int32_t StringDictionaryProxy::lookupTransientStringUnlocked ( const String &  lookup_string ) const

private

Referenced by buildTranslationMapToOtherProxy(), transientLookupBulkParallelUnlocked(), and transientLookupBulkUnlocked().

Here is the caller graph for this function:

template<>

int32_t StringDictionaryProxy::lookupTransientStringUnlocked

(

const std::string &

lookup_string

)

const

Definition at line 185 of file StringDictionaryProxy.cpp.

References StringDictionary::INVALID_STR_ID, and transient_str_to_int_.

                                          {
  const auto it = transient_str_to_int_.find(lookup_string);
  if (it != transient_str_to_int_.end()) {
    return it->second;
  }
  return StringDictionary::INVALID_STR_ID;
}

template<>

int32_t StringDictionaryProxy::lookupTransientStringUnlocked

(

const std::string_view &

lookup_string

)

const

Definition at line 195 of file StringDictionaryProxy.cpp.

References StringDictionary::INVALID_STR_ID, and transient_str_to_int_.

                                               {
  const auto it = transient_str_to_int_.find(std::string(lookup_string));
  if (it != transient_str_to_int_.end()) {
    return it->second;
  }
  return StringDictionary::INVALID_STR_ID;
}

size_t StringDictionaryProxy::storageEntryCount

(

)

const

Returns the number of string entries in the underlying string dictionary, at this proxy's generation_ if it is set/valid, otherwise just the current size of the dictionary.

Returns

size_t Number of entries in the string dictionary (at this proxy's generation if set)

Definition at line 410 of file StringDictionaryProxy.cpp.

References CHECK_LE, generation_, and string_dict_.

Referenced by entryCountUnlocked(), getRangeUnlocked(), and getString().

                                                      {
  const size_t num_storage_entries{generation_ == -1 ? string_dict_->storageEntryCount()
                                                     : generation_};
  CHECK_LE(num_storage_entries, static_cast<size_t>(std::numeric_limits<int32_t>::max()));
  return num_storage_entries;
}

Here is the caller graph for this function:

size_t StringDictionaryProxy::transientEntryCount

(

)

const

Returns the number of transient string entries for this proxy,.

Returns

size_t Number of transient string entries for this proxy

Definition at line 426 of file StringDictionaryProxy.cpp.

References rw_mutex_, and transientEntryCountUnlocked().

                                                        {
  mapd_shared_lock<mapd_shared_mutex> read_lock(rw_mutex_);
  return transientEntryCountUnlocked();
}

Here is the call graph for this function:

size_t StringDictionaryProxy::transientEntryCountUnlocked ( ) const

private

Definition at line 417 of file StringDictionaryProxy.cpp.

References CHECK_LE, and transient_str_to_int_.

Referenced by buildTranslationMapToOtherProxy(), entryCountUnlocked(), getRangeUnlocked(), and transientEntryCount().

                                                                {
  // CHECK_LE(num_storage_entries,
  // static_cast<size_t>(std::numeric_limits<int32_t>::max()));
  const size_t num_transient_entries{transient_str_to_int_.size()};
  CHECK_LE(num_transient_entries,
           static_cast<size_t>(std::numeric_limits<int32_t>::max()) - 1);
  return num_transient_entries;
}

Here is the caller graph for this function:

int32_t StringDictionaryProxy::transientLookupAndAddUnlocked ( const std::string &  str )

private

Definition at line 451 of file StringDictionaryProxy.cpp.

References CHECK, transient_int_to_str_, and transient_str_to_int_.

Referenced by getOrAddTransient(), and getOrAddTransientBulk().

                                                                                 {
  const auto it = transient_str_to_int_.find(str);
  if (it != transient_str_to_int_.end()) {
    return it->second;
  }
  int32_t transient_id =
      -(transient_str_to_int_.size() + 2);  // make sure it's not INVALID_STR_ID
  {
    auto it_ok = transient_str_to_int_.insert(std::make_pair(str, transient_id));
    CHECK(it_ok.second);
  }
  {
    auto it_ok = transient_int_to_str_.insert(std::make_pair(transient_id, str));
    CHECK(it_ok.second);
  }
  return transient_id;
}

Here is the caller graph for this function:

template<typename String >

template void StringDictionaryProxy::transientLookupBulk ( const std::vector< String > &  lookup_strings, int32_t *  string_ids  ) const

private

Definition at line 470 of file StringDictionaryProxy.cpp.

References rw_mutex_, transient_str_to_int_, transientLookupBulkParallelUnlocked(), and transientLookupBulkUnlocked().

Referenced by getTransientBulk().

                                                                           {
  // std::vector<int32_t>& string_ids) {
  const size_t num_strings = lookup_strings.size();
  mapd_shared_lock<mapd_shared_mutex> read_lock(rw_mutex_);

  if (num_strings == static_cast<size_t>(0) || transient_str_to_int_.empty()) {
    return;
  }
  constexpr size_t tbb_parallel_threshold{20000};
  if (num_strings < tbb_parallel_threshold) {
    transientLookupBulkUnlocked(lookup_strings, string_ids);
  } else {
    transientLookupBulkParallelUnlocked(lookup_strings, string_ids);
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

template<typename String >

template void StringDictionaryProxy::transientLookupBulkParallelUnlocked ( const std::vector< String > &  lookup_strings, int32_t *  string_ids  ) const

private

Definition at line 519 of file StringDictionaryProxy.cpp.

References StringDictionary::INVALID_STR_ID, lookupTransientStringUnlocked(), max_inputs_per_thread, and threading_serial::parallel_for().

Referenced by transientLookupBulk().

                               {
  const size_t num_strings = lookup_strings.size();
  const size_t max_thread_count = std::thread::hardware_concurrency();
  const size_t max_inputs_per_thread = 20000;
  const size_t min_grain_size = max_inputs_per_thread / 2;
  const size_t num_threads =
      std::min(max_thread_count,
               ((num_strings + max_inputs_per_thread - 1) / max_inputs_per_thread));

  tbb::task_arena limited_arena(num_threads);
  tbb::task_group tg;
  limited_arena.execute([&] {
    tg.run([&] {
      tbb::parallel_for(
          tbb::blocked_range<size_t>(0, num_strings, min_grain_size),
          [&](const tbb::blocked_range<size_t>& r) {
            const size_t start_idx = r.begin();
            const size_t end_idx = r.end();
            for (size_t string_idx = start_idx; string_idx < end_idx; ++string_idx) {
              if (string_ids[string_idx] != StringDictionary::INVALID_STR_ID) {
                continue;
              }
              string_ids[string_idx] =
                  lookupTransientStringUnlocked(lookup_strings[string_idx]);
            }
          },
          tbb::simple_partitioner());
    });
  });

  limited_arena.execute([&] { tg.wait(); });
}

Here is the call graph for this function:

Here is the caller graph for this function:

template<typename String >

template void StringDictionaryProxy::transientLookupBulkUnlocked ( const std::vector< String > &  lookup_strings, int32_t *  string_ids  ) const

private

Definition at line 496 of file StringDictionaryProxy.cpp.

References StringDictionary::INVALID_STR_ID, and lookupTransientStringUnlocked().

Referenced by transientLookupBulk().

                               {
  const size_t num_strings = lookup_strings.size();
  for (size_t string_idx = 0; string_idx < num_strings; ++string_idx) {
    if (string_ids[string_idx] != StringDictionary::INVALID_STR_ID) {
      continue;
    }
    // If we're here it means we need to look up this string as we don't
    // have a valid id for it
    string_ids[string_idx] = lookupTransientStringUnlocked(lookup_strings[string_idx]);
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

void StringDictionaryProxy::updateGeneration ( const int64_t  generation )

noexcept

Definition at line 440 of file StringDictionaryProxy.cpp.

References CHECK_EQ.

                                                                              {
  if (generation == -1) {
    return;
  }
  if (generation_ != -1) {
    CHECK_EQ(generation_, generation);
    return;
  }
  generation_ = generation;
}

Friends And Related Function Documentation

bool operator!= ( const StringDictionaryProxy &  sdp1, const StringDictionaryProxy &  sdp2  )

friend

Definition at line 582 of file StringDictionaryProxy.cpp.

                                                                                      {
  return !(sdp1 == sdp2);
}

bool operator== ( const StringDictionaryProxy &  sdp1, const StringDictionaryProxy &  sdp2  )

friend

Definition at line 570 of file StringDictionaryProxy.cpp.

                                                                                      {
  if (sdp1.string_dict_id_ != sdp2.string_dict_id_) {
    return false;
  }
  if (sdp1.transient_int_to_str_.size() != sdp2.transient_int_to_str_.size()) {
    return false;
  }
  return std::equal(sdp1.transient_int_to_str_.begin(),
                    sdp1.transient_int_to_str_.end(),
                    sdp2.transient_int_to_str_.begin());
}

Member Data Documentation

int64_t StringDictionaryProxy::generation_

private

Definition at line 198 of file StringDictionaryProxy.h.

Referenced by buildTranslationMapToOtherProxy(), getCompare(), getGeneration(), getIdOfString(), getLike(), getOrAddTransient(), getOrAddTransientBulk(), getRegexpLike(), getTransientBulk(), and storageEntryCount().

mapd_shared_mutex StringDictionaryProxy::rw_mutex_

mutableprivate

Definition at line 199 of file StringDictionaryProxy.h.

Referenced by buildTranslationMapToOtherProxy(), entryCount(), getIdOfString(), getIdOfStringNoGeneration(), getOrAddTransient(), getOrAddTransientBulk(), getRange(), getString(), getStrings(), transientEntryCount(), and transientLookupBulk().

std::shared_ptr<StringDictionary> StringDictionaryProxy::string_dict_

private

Definition at line 194 of file StringDictionaryProxy.h.

Referenced by buildTranslationMapToOtherProxy(), getCompare(), getDictionary(), getIdOfString(), getIdOfStringNoGeneration(), getLike(), getOrAddTransient(), getOrAddTransientBulk(), getRegexpLike(), getString(), getStrings(), getTransientBulk(), and storageEntryCount().

const int32_t StringDictionaryProxy::string_dict_id_

private

Definition at line 195 of file StringDictionaryProxy.h.

Referenced by getDictId(), and operator==().

std::map<int32_t, std::string> StringDictionaryProxy::transient_int_to_str_

private

Definition at line 196 of file StringDictionaryProxy.h.

Referenced by buildTranslationMapToOtherProxy(), getCompare(), getLike(), getRegexpLike(), getString(), getStrings(), getTransientMapping(), operator==(), and transientLookupAndAddUnlocked().

std::map<std::string, int32_t> StringDictionaryProxy::transient_str_to_int_

private

Definition at line 197 of file StringDictionaryProxy.h.

Referenced by getIdOfString(), getIdOfStringNoGeneration(), lookupTransientStringUnlocked(), transientEntryCountUnlocked(), transientLookupAndAddUnlocked(), and transientLookupBulk().

---

The documentation for this class was generated from the following files:

• /home/jenkins-slave/workspace/core-os-doxygen/StringDictionary/StringDictionaryProxy.h
• /home/jenkins-slave/workspace/core-os-doxygen/StringDictionary/StringDictionaryProxy.cpp