OmniSciDB  b24e664e58
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
RowToColumnLoader.cpp File Reference

Based on StreamInsert code but using binary columnar format for inserting a stream of rows with optional transformations from stdin to a MapD table. More...

+ Include dependency graph for RowToColumnLoader.cpp:

Go to the source code of this file.

Functions

SQLTypes get_sql_types (const TColumnType &ct)
 
SQLTypeInfo create_sql_type_info_from_col_type (const TColumnType &ct)
 
SQLTypeInfo create_array_sql_type_info_from_col_type (const TColumnType &ct)
 
void remove_partial_row (size_t failed_column, std::vector< SQLTypeInfo > column_type_info_vector, std::vector< TColumn > &input_col_vec)
 
void populate_TColumn (TStringValue ts, SQLTypeInfo column_type_info, TColumn &input_col, const Importer_NS::CopyParams &copy_params)
 

Detailed Description

Based on StreamInsert code but using binary columnar format for inserting a stream of rows with optional transformations from stdin to a MapD table.

Author
Michael micha.nosp@m.el@m.nosp@m.apd.c.nosp@m.om Copyright (c) 2017 MapD Technologies, Inc. All rights reserved.

Definition in file RowToColumnLoader.cpp.

Function Documentation

SQLTypeInfo create_array_sql_type_info_from_col_type ( const TColumnType &  ct)

Definition at line 101 of file RowToColumnLoader.cpp.

References get_sql_types(), kENCODING_NONE, and kNULLT.

Referenced by RowToColumnLoader::RowToColumnLoader().

101  {
102  return SQLTypeInfo(get_sql_types(ct),
103  ct.col_type.precision,
104  ct.col_type.scale,
105  ct.col_type.nullable,
107  0,
109 }
SQLTypeInfoCore< ArrayContextTypeSizer, ExecutorTypePackaging, DateTimeFacilities > SQLTypeInfo
Definition: sqltypes.h:852
SQLTypes get_sql_types(const TColumnType &ct)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

SQLTypeInfo create_sql_type_info_from_col_type ( const TColumnType &  ct)

Definition at line 75 of file RowToColumnLoader.cpp.

References get_sql_types(), kARRAY, kENCODING_NONE, and kNULLT.

Referenced by RowToColumnLoader::RowToColumnLoader().

75  {
76  if (ct.col_type.is_array) {
78  ct.col_type.precision,
79  ct.col_type.scale,
80  ct.col_type.nullable,
82  0,
83  get_sql_types(ct));
84  } else {
85  // normal column
86  // NOTE(se)
87  // for geo types, the values inserted for the other fields
88  // may not be valid, but only the type field is ever used
89  return SQLTypeInfo(get_sql_types(ct),
90  ct.col_type.precision,
91  ct.col_type.scale,
92  ct.col_type.nullable,
94  0,
96  }
97 }
SQLTypeInfoCore< ArrayContextTypeSizer, ExecutorTypePackaging, DateTimeFacilities > SQLTypeInfo
Definition: sqltypes.h:852
SQLTypes get_sql_types(const TColumnType &ct)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

SQLTypes get_sql_types ( const TColumnType &  ct)

Definition at line 32 of file RowToColumnLoader.cpp.

References logger::FATAL, kBIGINT, kBOOLEAN, kDATE, kDECIMAL, kDOUBLE, kFLOAT, kINT, kLINESTRING, kMULTIPOLYGON, kNULLT, kPOINT, kPOLYGON, kSMALLINT, kTEXT, kTIME, kTIMESTAMP, kVARCHAR, and LOG.

Referenced by create_array_sql_type_info_from_col_type(), and create_sql_type_info_from_col_type().

32  {
33  switch (ct.col_type.type) {
34  case TDatumType::BIGINT:
35  return SQLTypes::kBIGINT;
36  case TDatumType::BOOL:
37  return SQLTypes::kBOOLEAN;
38  case TDatumType::DATE:
39  return SQLTypes::kDATE;
40  case TDatumType::DECIMAL:
41  return SQLTypes::kDECIMAL;
42  case TDatumType::DOUBLE:
43  return SQLTypes::kDOUBLE;
44  case TDatumType::FLOAT:
45  return SQLTypes::kFLOAT;
46  case TDatumType::INT:
47  return SQLTypes::kINT;
48  case TDatumType::STR:
49  // Tdataum is lossy here so need to look at precision to see if it was defined
50  if (ct.col_type.precision == 0) {
51  return SQLTypes::kTEXT;
52  } else {
53  return SQLTypes::kVARCHAR;
54  }
55  case TDatumType::TIME:
56  return SQLTypes::kTIME;
57  case TDatumType::TIMESTAMP:
58  return SQLTypes::kTIMESTAMP;
59  case TDatumType::SMALLINT:
60  return SQLTypes::kSMALLINT;
61  case TDatumType::POINT:
62  return SQLTypes::kPOINT;
63  case TDatumType::LINESTRING:
64  return SQLTypes::kLINESTRING;
65  case TDatumType::POLYGON:
66  return SQLTypes::kPOLYGON;
67  case TDatumType::MULTIPOLYGON:
69  default:
70  LOG(FATAL) << "Unsupported TColumnType found, should not be possible";
71  return SQLTypes::kNULLT; // satisfy return-type warning
72  }
73 }
Definition: sqltypes.h:52
#define LOG(tag)
Definition: Logger.h:185
Definition: sqltypes.h:55
Definition: sqltypes.h:56
Definition: sqltypes.h:48

+ Here is the caller graph for this function:

void populate_TColumn ( TStringValue  ts,
SQLTypeInfo  column_type_info,
TColumn &  input_col,
const Importer_NS::CopyParams copy_params 
)

Definition at line 174 of file RowToColumnLoader.cpp.

References Datum::bigintval, Datum::doubleval, logger::FATAL, Datum::floatval, SQLTypeInfoCore< TYPE_FACET_PACK >::get_precision(), SQLTypeInfoCore< TYPE_FACET_PACK >::get_type(), Datum::intval, kARRAY, kBIGINT, kBOOLEAN, kCHAR, kDATE, kDECIMAL, kDOUBLE, kFLOAT, kINT, kLINESTRING, kMULTIPOLYGON, kNUMERIC, kPOINT, kPOLYGON, kSMALLINT, kTEXT, kTIME, kTIMESTAMP, kVARCHAR, LOG, Datum::smallintval, and StringToDatum().

Referenced by RowToColumnLoader::convert_string_to_column().

177  {
178  // create datum and push data to column structure from row data
179  switch (column_type_info.get_type()) {
180  case SQLTypes::kARRAY:
181  LOG(FATAL) << "Trying to process ARRAY at item level something is wrong";
182  break;
183  case SQLTypes::kTEXT:
184  case SQLTypes::kCHAR:
185  case SQLTypes::kVARCHAR:
186  case SQLTypes::kPOINT:
188  case SQLTypes::kPOLYGON:
190  if (ts.is_null) {
191  input_col.nulls.push_back(true);
192  input_col.data.str_col.emplace_back("");
193 
194  } else {
195  input_col.nulls.push_back(false);
196  switch (column_type_info.get_type()) {
197  case SQLTypes::kCHAR:
198  case SQLTypes::kVARCHAR:
199  input_col.data.str_col.push_back(
200  ts.str_val.substr(0, column_type_info.get_precision()));
201  break;
202  case SQLTypes::kTEXT:
203  case SQLTypes::kPOINT:
205  case SQLTypes::kPOLYGON:
207  input_col.data.str_col.push_back(ts.str_val);
208  break;
209  default:
210  LOG(FATAL) << " trying to process a STRING transport type not handled "
211  << column_type_info.get_type();
212  }
213  }
214  break;
215  case SQLTypes::kINT:
216  case SQLTypes::kBIGINT:
217  case SQLTypes::kSMALLINT:
218  case SQLTypes::kDATE:
219  case SQLTypes::kTIME:
221  case SQLTypes::kNUMERIC:
222  case SQLTypes::kDECIMAL:
223  case SQLTypes::kBOOLEAN:
224  if (ts.is_null) {
225  input_col.nulls.push_back(true);
226  input_col.data.int_col.push_back(0);
227  } else {
228  input_col.nulls.push_back(false);
229  Datum d = StringToDatum(ts.str_val, column_type_info);
230  switch (column_type_info.get_type()) {
231  case SQLTypes::kINT:
232  case SQLTypes::kBOOLEAN:
233  input_col.data.int_col.push_back(d.intval);
234  break;
235  case SQLTypes::kBIGINT:
236  case SQLTypes::kNUMERIC:
237  case SQLTypes::kDECIMAL:
238  input_col.data.int_col.push_back(d.bigintval);
239  break;
240  case SQLTypes::kSMALLINT:
241  input_col.data.int_col.push_back(d.smallintval);
242  break;
243  case SQLTypes::kDATE:
244  case SQLTypes::kTIME:
246  input_col.data.int_col.push_back(d.bigintval);
247  break;
248  default:
249  LOG(FATAL) << " trying to process an INT transport type not handled "
250  << column_type_info.get_type();
251  }
252  }
253  break;
254  case SQLTypes::kFLOAT:
255  case SQLTypes::kDOUBLE:
256  if (ts.is_null) {
257  input_col.nulls.push_back(true);
258  input_col.data.real_col.push_back(0);
259 
260  } else {
261  input_col.nulls.push_back(false);
262  Datum d = StringToDatum(ts.str_val, column_type_info);
263  switch (column_type_info.get_type()) {
264  case SQLTypes::kFLOAT:
265  input_col.data.real_col.push_back(d.floatval);
266  break;
267  case SQLTypes::kDOUBLE:
268  input_col.data.real_col.push_back(d.doubleval);
269  break;
270  default:
271  LOG(FATAL) << " trying to process a REAL transport type not handled "
272  << column_type_info.get_type();
273  }
274  }
275  break;
276  default:
277  LOG(FATAL) << "Trying to process an unsupported datatype, should be impossible";
278  }
279 }
Definition: sqltypes.h:52
#define LOG(tag)
Definition: Logger.h:185
int32_t intval
Definition: sqltypes.h:128
float floatval
Definition: sqltypes.h:130
HOST DEVICE SQLTypes get_type() const
Definition: sqltypes.h:326
int64_t bigintval
Definition: sqltypes.h:129
int16_t smallintval
Definition: sqltypes.h:127
Definition: sqltypes.h:55
Definition: sqltypes.h:56
Datum StringToDatum(const std::string &s, SQLTypeInfo &ti)
Definition: Datum.cpp:90
Definition: sqltypes.h:44
int get_precision() const
Definition: sqltypes.h:329
Definition: sqltypes.h:48
double doubleval
Definition: sqltypes.h:131

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void remove_partial_row ( size_t  failed_column,
std::vector< SQLTypeInfo column_type_info_vector,
std::vector< TColumn > &  input_col_vec 
)

Definition at line 129 of file RowToColumnLoader.cpp.

References logger::FATAL, kARRAY, kBIGINT, kBOOLEAN, kCHAR, kDATE, kDECIMAL, kDOUBLE, kFLOAT, kINT, kLINESTRING, kMULTIPOLYGON, kNUMERIC, kPOINT, kPOLYGON, kSMALLINT, kTEXT, kTIME, kTIMESTAMP, kVARCHAR, and LOG.

Referenced by RowToColumnLoader::convert_string_to_column().

131  {
132  for (size_t idx = 0; idx < failed_column; idx++) {
133  switch (column_type_info_vector[idx].get_type()) {
134  case SQLTypes::kARRAY:
135  input_col_vec[idx].nulls.pop_back();
136  input_col_vec[idx].data.arr_col.pop_back();
137  break;
138  case SQLTypes::kTEXT:
139  case SQLTypes::kCHAR:
140  case SQLTypes::kVARCHAR:
141  input_col_vec[idx].nulls.pop_back();
142  input_col_vec[idx].data.str_col.pop_back();
143  break;
144  case SQLTypes::kINT:
145  case SQLTypes::kBIGINT:
146  case SQLTypes::kSMALLINT:
147  case SQLTypes::kDATE:
148  case SQLTypes::kTIME:
150  case SQLTypes::kNUMERIC:
151  case SQLTypes::kDECIMAL:
152  case SQLTypes::kBOOLEAN:
153  input_col_vec[idx].nulls.pop_back();
154  input_col_vec[idx].data.int_col.pop_back();
155  break;
156  case SQLTypes::kFLOAT:
157  case SQLTypes::kDOUBLE:
158  input_col_vec[idx].nulls.pop_back();
159  input_col_vec[idx].data.real_col.pop_back();
160  break;
161  case SQLTypes::kPOINT:
163  case SQLTypes::kPOLYGON:
165  input_col_vec[idx].nulls.pop_back();
166  input_col_vec[idx].data.str_col.pop_back();
167  break;
168  default:
169  LOG(FATAL) << "Trying to process an unsupported datatype, should be impossible";
170  }
171  }
172 }
Definition: sqltypes.h:52
#define LOG(tag)
Definition: Logger.h:185
Definition: sqltypes.h:55
Definition: sqltypes.h:56
Definition: sqltypes.h:44
Definition: sqltypes.h:48

+ Here is the caller graph for this function: