OmniSciDB  fe05a0c208
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
generate_TableFunctionsFactory_init Namespace Reference

Functions

def type_parse
 
def find_comma
 

Variables

string ExtArgumentTypes
 
string OutputBufferSizeTypes
 
tuple translate_map
 
tuple _is_int = re.compile(r'\d+')
 
list add_stmts = []
 
 last_line = None
 
tuple line = line.replace(' ', '')
 
tuple i
 
list name = line[:i]
 
list args_line = line[i+1:j]
 
list outputs = line[j+1:]
 
list args = []
 
tuple j = args_line.find('<', 0, i)
 
tuple k = args_line.rfind('>', 0, i)
 
list input_types = []
 
list output_types = []
 
list sql_types = []
 
 sizer = None
 
int consumed_nargs = 0
 
tuple r = type_parse(a)
 
tuple t = str(consumed_nargs + 1)
 
string add = 'TableFunctionsFactory::add("%s", %s, %s, %s, %s);'
 
string content
 
list output_filename = sys.argv[-1]
 
tuple dirname = os.path.dirname(output_filename)
 
tuple f = open(output_filename, 'w')
 

Detailed Description

Given a list of input files, scan for lines containing UDTF
specification statements in the following form:

  UDTF: function_name(<arguments>) -> <output column types>

where <arguments> is a comma-separated list of argument types. The
argument types specifications are:

- scalar types:
    Int8, Int16, Int32, Int64, Float, Double, Bool, etc
- column types:
    ColumnInt8, ColumnInt16, ColumnInt32, ColumnInt64, ColumnFloat, ColumnDouble, ColumnBool, etc
- column list types:
    ColumnListInt8, ColumnListInt16, ColumnListInt32, ColumnListInt64, ColumnListFloat, ColumnListDouble, ColumnListBool, etc
- cursor type:
    Cursor<t0, t1, ...>
  where t0, t1 are column or column list types
- output buffer size parameter type:
    RowMultiplier<i>, ConstantParameter<i>, Constant<i>
  where i is literal integer

The output column types is a comma-separated list of column types, see above.

In addition, the following equivalents are suppored:
  Column<T> == ColumnT
  ColumnList<T> == ColumnListT
  Cursor<T, V, ...> == Cursor<ColumnT, ColumnV, ...>
  int8 == int8_t == Int8, etc
  float == Float, double == Double, bool == Bool
  T == ColumnT for output column types
  RowMultiplier == RowMultiplier<i> where i is the one-based position of the sizer argument
  when no sizer argument is provided, Constant<1> is assumed

Function Documentation

def generate_TableFunctionsFactory_init.find_comma (   line)

Definition at line 104 of file generate_TableFunctionsFactory_init.py.

105 def find_comma(line):
106  d = 0
107  for i, c in enumerate(line):
108  if c in '<([{':
109  d += 1
110  elif c in '>)]{':
111  d -= 1
112  elif d == 0 and c == ',':
113  return i
114  return -1
def generate_TableFunctionsFactory_init.type_parse (   a)

Definition at line 72 of file generate_TableFunctionsFactory_init.py.

References _is_int, and split().

72 
73 def type_parse(a):
74  i = a.find('<')
75  if i >= 0:
76  assert a.endswith('>'), a
77  n = a[:i]
78  n = translate_map.get(n, n)
79  if n in OutputBufferSizeTypes:
80  v = a[i+1:-1]
81  assert _is_int(v)
82  return n, v
83  if n == 'Cursor':
84  lst = []
85  # map Cursor<T> to Cursor<ColumnT>
86  for t in map(type_parse, a[i+1:-1].split(',')):
87  if 'Column' + t in ExtArgumentTypes:
88  lst.append('Column' + t)
89  else:
90  lst.append(t)
91  return n, tuple(lst)
92  if n == 'Column':
93  return n + type_parse(a[i+1:-1])
94  if n == 'ColumnList':
95  return n + type_parse(a[i+1:-1])
96  else:
97  a = translate_map.get(a, a)
98  if a in ExtArgumentTypes:
99  return a
100  if a in OutputBufferSizeTypes:
101  return a, None
102  raise ValueError('Cannot parse `%s` to ExtArgumentTypes or OutputBufferSizeTypes' % (a,))
103 
std::vector< std::string > split(std::string_view str, std::string_view delim, std::optional< size_t > maxsplit)
split apart a string into a vector of substrings

+ Here is the call graph for this function:

Variable Documentation

tuple generate_TableFunctionsFactory_init._is_int = re.compile(r'\d+')

Definition at line 70 of file generate_TableFunctionsFactory_init.py.

Referenced by type_parse().

string generate_TableFunctionsFactory_init.add = 'TableFunctionsFactory::add("%s", %s, %s, %s, %s);'

Definition at line 213 of file generate_TableFunctionsFactory_init.py.

Referenced by com.omnisci.jdbc.OmniSciDatabaseMetaData.getColumns(), com.omnisci.jdbc.OmniSciDatabaseMetaData.getTables(), com.omnisci.jdbc.OmniSciDatabaseMetaData.getTableTypes(), com.mapd.calcite.parser.MapDSqlOperatorTable.RampFunction.inferReturnType(), com.mapd.calcite.parser.MapDSqlOperatorTable.DedupFunction.inferReturnType(), org.apache.calcite.prepare.SqlIdentifierCapturer.scan(), com.omnisci.jdbc.OmniSciDatabaseMetaData.tablePermProcess(), com.omnisci.jdbc.OmniSciDatabaseMetaDataTest.tst09_user_table(), com.omnisci.jdbc.OmniSciDatabaseMetaDataTest.tst10_omnisci_table(), and com.omnisci.jdbc.OmniSciDatabaseMetaDataTest.tst11_user2_table().

list generate_TableFunctionsFactory_init.add_stmts = []

Definition at line 115 of file generate_TableFunctionsFactory_init.py.

list generate_TableFunctionsFactory_init.args = []

Definition at line 142 of file generate_TableFunctionsFactory_init.py.

list generate_TableFunctionsFactory_init.args_line = line[i+1:j]

Definition at line 137 of file generate_TableFunctionsFactory_init.py.

int generate_TableFunctionsFactory_init.consumed_nargs = 0

Definition at line 163 of file generate_TableFunctionsFactory_init.py.

string generate_TableFunctionsFactory_init.content
Initial value:
1 = '''
2 /*
3  This file is generated by %s. Do no edit!
4 */
5 
6 #include "QueryEngine/TableFunctions/TableFunctionsFactory.h"
7 
8 extern bool g_enable_table_functions;
9 
10 namespace table_functions {
11 
12 std::once_flag init_flag;
13 
14 void TableFunctionsFactory::init() {
15  if (!g_enable_table_functions) {
16  return;
17  }
18  std::call_once(init_flag, []() {
19  %s
20  });
21 }
22 
23 } // namespace table_functions
24 '''

Definition at line 216 of file generate_TableFunctionsFactory_init.py.

tuple generate_TableFunctionsFactory_init.dirname = os.path.dirname(output_filename)

Definition at line 242 of file generate_TableFunctionsFactory_init.py.

string generate_TableFunctionsFactory_init.ExtArgumentTypes
Initial value:
1 = ''' Int8, Int16, Int32, Int64, Float, Double, Void, PInt8, PInt16,
2 PInt32, PInt64, PFloat, PDouble, PBool, Bool, ArrayInt8, ArrayInt16,
3 ArrayInt32, ArrayInt64, ArrayFloat, ArrayDouble, ArrayBool, GeoPoint,
4 GeoLineString, Cursor, GeoPolygon, GeoMultiPolygon, ColumnInt8,
5 ColumnInt16, ColumnInt32, ColumnInt64, ColumnFloat, ColumnDouble,
6 ColumnBool, TextEncodingNone, TextEncodingDict8, TextEncodingDict16,
7 TextEncodingDict32, ColumnListInt8, ColumnListInt16, ColumnListInt32, ColumnListInt64,
8 ColumnListFloat, ColumnListDouble, ColumnListBool '''

Definition at line 41 of file generate_TableFunctionsFactory_init.py.

tuple generate_TableFunctionsFactory_init.f = open(output_filename, 'w')

Definition at line 246 of file generate_TableFunctionsFactory_init.py.

tuple generate_TableFunctionsFactory_init.i
Initial value:
1 = line.find('(')
2  j = line.find(')')

Definition at line 131 of file generate_TableFunctionsFactory_init.py.

string generate_TableFunctionsFactory_init.input_types = []

Definition at line 159 of file generate_TableFunctionsFactory_init.py.

tuple generate_TableFunctionsFactory_init.j = args_line.find('<', 0, i)

Definition at line 149 of file generate_TableFunctionsFactory_init.py.

Referenced by import_export::TypedImportBuffer.add_values(), foreign_storage::ParquetArrayEncoder.appendData(), foreign_storage::ParquetStringNoneEncoder.appendData(), foreign_storage::ParquetGeospatialEncoder.appendData(), ResultSetLogicalValuesBuilder.build(), QueryFragmentDescriptor.buildFragmentPerKernelForTable(), QueryFragmentDescriptor.buildFragmentPerKernelMapForUnion(), QueryFragmentDescriptor.buildMultifragKernelMap(), CodeGenerator.codegenFunctionOper(), CodeGenerator.codegenFunctionOperCastArgs(), CodeGenerator.codegenFunctionOperNullArg(), anonymous_namespace{DBHandler.cpp}.column_ids_by_names(), compress_reference_path(), compute_bucket_sizes_impl(), concat(), concat_with(), create_dev_group_by_buffers(), ArrowForeignStorageBase.createDictionaryEncodedColumn(), org.apache.calcite.sql2rel.SqlToRelConverter.createInsertBlackboard(), foreign_storage::ParquetInPlaceEncoder.decodeNullsAndEncodeData(), org.apache.calcite.sql2rel.SqlToRelConverter.deriveAlias(), distance_ring_linestring(), distance_ring_ring(), get_candidate_rows(), import_export::delimited_parser.get_row(), ArrowResultSetConverter.getArrowBatch(), init_group_by_buffer_gpu(), ResultSetStorage.initializeBaselineValueSlots(), ResultSetStorage.initializeRowWise(), anonymous_namespace{HashTable.cpp}.inner_to_string(), insert_sorted(), OverlapsKeyHandler.operator()(), QueryMemoryInitializer.QueryMemoryInitializer(), ResultSetReductionJIT.reduceOneEntryBaseline(), ResultSetStorage.reduceOneEntrySlotsBaseline(), ResultSetStorage.rewriteAggregateBufferOffsets(), com.mapd.logrunner.LogRunner.myThread.run(), ArrowForeignStorageBase.setNullValues(), split(), string_ilike_simple(), string_like_simple(), strip(), anonymous_namespace{RelAlgOptimizer.cpp}.sweep_dead_columns(), anonymous_namespace{HashTable.cpp}::perfect_hash.to_set_one_to_many(), anonymous_namespace{HashTable.cpp}::keyed_hash.to_set_one_to_many(), anonymous_namespace{HashTable.cpp}::keyed_hash.to_set_one_to_one(), and import_export.trim_space().

tuple generate_TableFunctionsFactory_init.k = args_line.rfind('>', 0, i)

Definition at line 150 of file generate_TableFunctionsFactory_init.py.

Referenced by anonymous_namespace{WindowContext.cpp}.apply_lag_to_partition(), CodeGenerator.codegenFunctionOperCastArgs(), org.apache.calcite.sql2rel.SqlToRelConverter.createAggImpl(), distance_point_line(), distance_point_line_squared(), get_bin_from_k_heap_impl(), hll_size_for_rate(), MurmurHash1Impl(), MurmurHash64AImpl(), lockmgr.operator<<(), operator<<(), ArrowForeignStorageBase.parseArrowTable(), Catalog_Namespace::Catalog.renameTable(), org.apache.calcite.sql2rel.SqlToRelConverter.rewriteAggregateWithGroupId(), and org.apache.calcite.sql2rel.SqlToRelConverter.translateIn().

generate_TableFunctionsFactory_init.last_line = None

Definition at line 119 of file generate_TableFunctionsFactory_init.py.

list generate_TableFunctionsFactory_init.line = line.replace(' ', '')

Definition at line 121 of file generate_TableFunctionsFactory_init.py.

Referenced by Geospatial::GeoLineString.GeoLineString(), import_export::Detector.importDelimited(), logger.operator>>(), com.mapd.parser.server.ExtensionFunctionSignatureParser.parse(), com.mapd.parser.server.ExtensionFunctionSignatureParser.parseFromString(), com.mapd.parser.server.ExtensionFunctionSignatureParser.parseUdfAst(), Data_Namespace::ProcMeminfoParser.ProcMeminfoParser(), com.mapd.parser.server.CalciteServerCaller.readPropertyFile(), com.mapd.utility.SQLImporter.run_init(), anonymous_namespace{CommandLineOptions.cpp}.sanitize_config_file(), and foreign_storage::SingleFileReader.SingleFileReader().

list generate_TableFunctionsFactory_init.name = line[:i]

Definition at line 136 of file generate_TableFunctionsFactory_init.py.

list generate_TableFunctionsFactory_init.output_filename = sys.argv[-1]

Definition at line 241 of file generate_TableFunctionsFactory_init.py.

string generate_TableFunctionsFactory_init.output_types = []

Definition at line 160 of file generate_TableFunctionsFactory_init.py.

string generate_TableFunctionsFactory_init.OutputBufferSizeTypes
Initial value:
1 = '''
2 kConstant, kUserSpecifiedConstantParameter, kUserSpecifiedRowMultiplier
3 '''

Definition at line 50 of file generate_TableFunctionsFactory_init.py.

tuple generate_TableFunctionsFactory_init.outputs = line[j+1:]

Definition at line 138 of file generate_TableFunctionsFactory_init.py.

Referenced by anonymous_namespace{RelAlgDagBuilder.cpp}.n_outputs().

string generate_TableFunctionsFactory_init.r = type_parse(a)

Definition at line 166 of file generate_TableFunctionsFactory_init.py.

string generate_TableFunctionsFactory_init.sizer = None

Definition at line 162 of file generate_TableFunctionsFactory_init.py.

Referenced by bind_function(), and table_functions::TableFunction.getSqlOutputRowSizeParameter().

string generate_TableFunctionsFactory_init.sql_types = []

Definition at line 161 of file generate_TableFunctionsFactory_init.py.

Referenced by Parser::CreateTableStmt.CreateTableStmt(), and Parser::AlterTableStmt.delegateExecute().

tuple generate_TableFunctionsFactory_init.t = str(consumed_nargs + 1)

Definition at line 185 of file generate_TableFunctionsFactory_init.py.

tuple generate_TableFunctionsFactory_init.translate_map
Initial value:
1 = dict(
2  Constant = 'kConstant',
3  ConstantParameter = 'kUserSpecifiedConstantParameter',
4  RowMultiplier = 'kUserSpecifiedRowMultiplier',
5  UserSpecifiedConstantParameter = 'kUserSpecifiedConstantParameter',
6  UserSpecifiedRowMultiplier = 'kUserSpecifiedRowMultiplier',
7  short = 'Int16',
8  int = 'Int32',
9  long = 'Int64',
10 )

Definition at line 54 of file generate_TableFunctionsFactory_init.py.