OmniSciDB  c0231cc57d
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
UdfCompiler Class Reference

#include <UdfCompiler.h>

+ Collaboration diagram for UdfCompiler:

Public Member Functions

 UdfCompiler (CudaMgr_Namespace::NvidiaDeviceArch target_arch, const std::string &clang_path_override="")
 
 UdfCompiler (CudaMgr_Namespace::NvidiaDeviceArch target_arch, const std::string &clang_path_override, const std::vector< std::string > clang_options)
 
std::pair< std::string,
std::string > 
compileUdf (const std::string &udf_file_name) const
 

Static Public Member Functions

static std::string getAstFileName (const std::string &udf_file_name)
 

Private Member Functions

void generateAST (const std::string &file_name) const
 
std::string compileToLLVMIR (const std::string &udf_file_name) const
 
int compileFromCommandLine (const std::vector< std::string > &command_line) const
 

Static Private Member Functions

static std::string genLLVMIRFilename (const std::string &udf_file_name)
 
static std::string genNVVMIRFilename (const std::string &udf_file_name)
 

Private Attributes

std::string clang_path_
 
std::vector< std::string > clang_options_
 

Detailed Description

Driver for calling clang/clang++ to compile C++ programs to LLVM IR for use as a UDF. Default initialization will find Clang using the clang library invocations. An optional clang override and additional arguments to the clang binary can be added. Once initialized the class holds the state for calling clang until destruction.

Definition at line 31 of file UdfCompiler.h.

Constructor & Destructor Documentation

UdfCompiler::UdfCompiler ( CudaMgr_Namespace::NvidiaDeviceArch  target_arch,
const std::string &  clang_path_override = "" 
)

Definition at line 293 of file UdfCompiler.cpp.

References clang_options_.

295  : clang_path_(get_clang_path(clang_path_override))
296 #ifdef HAVE_CUDA
297  , target_arch_(target_arch)
298 #endif
299 {
300  // To ifguard unsupported code included in heavydbTypes.h (i.e. <shared_mutex>)
301  clang_options_.emplace_back(std::string("-D UDF_COMPILED"));
302 }
std::vector< std::string > clang_options_
Definition: UdfCompiler.h:73
std::string get_clang_path(const std::string &clang_path_override)
std::string clang_path_
Definition: UdfCompiler.h:72
UdfCompiler::UdfCompiler ( CudaMgr_Namespace::NvidiaDeviceArch  target_arch,
const std::string &  clang_path_override,
const std::vector< std::string >  clang_options 
)

Definition at line 304 of file UdfCompiler.cpp.

References clang_options_.

307  : clang_path_(get_clang_path(clang_path_override))
308  , clang_options_(clang_options)
309 #ifdef HAVE_CUDA
310  , target_arch_(target_arch)
311 #endif
312 {
313  // To ifguard unsupported code included in heavydbTypes.h (i.e. <shared_mutex>)
314  clang_options_.emplace_back(std::string("-D UDF_COMPILED"));
315 }
std::vector< std::string > clang_options_
Definition: UdfCompiler.h:73
std::string get_clang_path(const std::string &clang_path_override)
std::string clang_path_
Definition: UdfCompiler.h:72

Member Function Documentation

int UdfCompiler::compileFromCommandLine ( const std::vector< std::string > &  command_line) const
private

Formulate the full compile command and call the compiler.

Definition at line 387 of file UdfCompiler.cpp.

References CHECK_EQ, clang_options_, clang_path_, anonymous_namespace{UdfCompiler.cpp}::exec_output(), logger::init(), run_benchmark_import::res, and shared::transform().

Referenced by compileToLLVMIR().

388  {
389  UdfClangDriver compiler_driver = UdfClangDriver::init(clang_path_);
390  auto the_driver(compiler_driver.getClangDriver());
391 
392  std::vector<const char*> clang_command_opts;
393  clang_command_opts.reserve(command_line.size() + clang_options_.size());
394  // add required options first
395  std::transform(std::begin(command_line),
396  std::end(command_line),
397  std::back_inserter(clang_command_opts),
398  [&](const std::string& str) { return str.c_str(); });
399 
400  // If there were additional clang options passed to the system, append them here
401  if (!clang_options_.empty()) {
402  std::transform(std::begin(clang_options_),
403  std::end(clang_options_),
404  std::back_inserter(clang_command_opts),
405  [&](const std::string& str) { return str.c_str(); });
406  }
407 
408  std::unique_ptr<driver::Compilation> compilation(
409  the_driver->BuildCompilation(clang_command_opts));
410  if (!compilation) {
411  throw std::runtime_error("failed to build compilation object!");
412  }
413  auto [clang_version_major, clang_version_minor, clang_version_patchlevel] =
414  compiler_driver.getClangVersion();
415  if (clang_version_major != CLANG_VERSION_MAJOR
416  // mismatch of clang driver and compulier versions requires
417  // modified workflow that removes incompatible driver flags for
418  // compiler.
419  || CLANG_VERSION_MAJOR == 9
420  // clang driver 9 requires cudatoolkit 8 that we don't support,
421  // hence switching to modified clang compiler 9 workflow that is
422  // able to produce bytecode to GPU when using cudatoolkit 11.
423  ) {
424  /* Fix incompatibilities when driver and clang versions differ.
425  */
426  auto& jobs = compilation->getJobs();
427  CHECK_EQ(jobs.size(), size_t(1));
428  auto& job = *jobs.begin();
429 
430  std::string cmd = job.getExecutable();
431  int skip = 0;
432  std::string last = "";
433 
434  for (auto& arg : job.getArguments()) {
435  const std::string& s = arg;
436  if (skip > 0) {
437  skip--;
438  last = s;
439  continue;
440  }
441 
442  // inclusion of __clang_cuda_runtime_wrapper.h leads to either
443  // clang >9 compilation failure or clang 9 failure for using
444  // cuda >8 (unsupported CUDA version).
445  if (s == "-include") {
446  last = s;
447  continue;
448  }
449  if (last == "-include") {
450  if (s != "__clang_cuda_runtime_wrapper.h") {
451  cmd += " -include " + s;
452  }
453  last = s;
454  continue;
455  }
456 
457  // Using -ffcuda-is-device flag produces empty gpu module
458  if (s == "-fcuda-is-device") {
459  last = s;
460  continue;
461  }
462 
463  if constexpr (CLANG_VERSION_MAJOR == 9) {
464  if (clang_version_major > 9) {
465  // The following clang 9 flags are unknown to clang >9:
466  if (s == "-masm-verbose" || s == "-fuse-init-array" ||
467  s == "-dwarf-column-info" || s == "-momit-leaf-frame-pointer" ||
468  s == "-fdiagnostics-show-option" || s == "-mdisable-fp-elim") {
469  last = s;
470  continue;
471  }
472  if (s == "-fmessage-length") {
473  skip = 1;
474  last = s;
475  continue;
476  }
477  }
478  }
479 
480  if constexpr (CLANG_VERSION_MAJOR == 10) {
481  if (clang_version_major > 10) {
482  // The following clang 10 flags are unknown to clang >10:
483  if (s == "-masm-verbose" || s == "-dwarf-column-info" ||
484  s == "-fdiagnostics-show-option") {
485  last = s;
486  continue;
487  }
488  if (s == "-fmessage-length") {
489  skip = 1;
490  last = s;
491  continue;
492  }
493  }
494  }
495 
496  if constexpr (CLANG_VERSION_MAJOR >= 10) {
497  if (clang_version_major < 10) {
498  // The following clang >10 flags are unknown to clang <10:
499  if (s == "-fno-rounding-math" || s.rfind("-mframe-pointer=", 0) == 0 ||
500  s.rfind("-fgnuc-version=", 0) == 0) {
501  last = s;
502  continue;
503  }
504  }
505  }
506 
507  if constexpr (CLANG_VERSION_MAJOR == 11) {
508  if (clang_version_major < 11) {
509  // The following clang 11 flags are unknown to clang <11:
510  if (s == "-fno-verbose-asm") {
511  last = s;
512  continue;
513  }
514  if (s == "-aux-target-cpu") {
515  last = s;
516  skip = 1;
517  continue;
518  }
519  }
520  }
521 
522  cmd += " " + s;
523  last = s;
524  }
525  // TODO: Here we don't use the_driver->ExecuteCompilation because
526  // could not find a better way to modify driver arguments. As a
527  // workaround, we run clang compiler via pipe (it could be run
528  // also via shell).
529  exec_output(cmd);
530  return 0;
531  }
532 
533  llvm::SmallVector<std::pair<int, const driver::Command*>, 10> failing_commands;
534  int res = the_driver->ExecuteCompilation(*compilation, failing_commands);
535  if (res < 0) {
536  for (const std::pair<int, const driver::Command*>& p : failing_commands) {
537  if (p.first) {
538  the_driver->generateCompilationDiagnostics(*compilation, *p.second);
539  }
540  }
541  }
542  return res;
543 }
#define CHECK_EQ(x, y)
Definition: Logger.h:230
std::vector< std::string > clang_options_
Definition: UdfCompiler.h:73
std::string exec_output(std::string cmd)
void init(LogOptions const &log_opts)
Definition: Logger.cpp:308
OUTPUT transform(INPUT const &input, FUNC const &func)
Definition: misc.h:296
std::string clang_path_
Definition: UdfCompiler.h:72

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string UdfCompiler::compileToLLVMIR ( const std::string &  udf_file_name) const
private

Formulate Clang command line command and call clang binary to generate LLVM IR for the C/C++ file.

Definition at line 584 of file UdfCompiler.cpp.

References clang_path_, compileFromCommandLine(), genLLVMIRFilename(), run_benchmark_import::res, and to_string().

Referenced by compileUdf().

584  {
585  std::string cpu_out_filename = genLLVMIRFilename(udf_file_name);
586 
587  std::vector<std::string> command_line{clang_path_,
588  "-c",
589  "-O2",
590  "-emit-llvm",
591  "-o",
592  cpu_out_filename,
593  "-std=c++17",
594  "-DNO_BOOST",
595  udf_file_name};
596  auto res = compileFromCommandLine(command_line);
597  if (res != 0) {
598  throw std::runtime_error("Failed to compile CPU UDF (status code " +
599  std::to_string(res) + ")");
600  }
601  if (!boost::filesystem::exists(cpu_out_filename)) {
602  throw std::runtime_error("udf compile did not produce output file " +
603  cpu_out_filename);
604  }
605  return cpu_out_filename;
606 }
int compileFromCommandLine(const std::vector< std::string > &command_line) const
std::string to_string(char const *&&v)
static std::string genLLVMIRFilename(const std::string &udf_file_name)
std::string clang_path_
Definition: UdfCompiler.h:72

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::pair< std::string, std::string > UdfCompiler::compileUdf ( const std::string &  udf_file_name) const

Compile a C++ file to LLVM IR, and generate an AST file. Both artifacts exist as files on disk. Three artifacts will be generated; the AST file, the CPU LLVM IR, and GPU LLVM IR (if CUDA is enabled and compilation succeeds). These LLVM IR files can be loaded by the Executor. The AST will be processed by Calcite.

Definition at line 317 of file UdfCompiler.cpp.

References compileToLLVMIR(), generateAST(), logger::INFO, LOG, and logger::WARNING.

Referenced by DBHandler::initialize().

318  {
319  LOG(INFO) << "UDFCompiler filename to compile: " << udf_file_name;
320  if (!boost::filesystem::exists(udf_file_name)) {
321  throw std::runtime_error("User defined function file " + udf_file_name +
322  " does not exist.");
323  }
324 
325  // create the AST file for the input function
326  generateAST(udf_file_name);
327 
328  // Compile udf file to generate cpu and gpu bytecode files
329  std::string cpu_file_name = "";
330  std::string cuda_file_name = "";
331 
332  cpu_file_name = compileToLLVMIR(udf_file_name);
333 
334 #ifdef HAVE_CUDA
335  try {
336  cuda_file_name = compileToNVVMIR(udf_file_name);
337  } catch (const std::exception& e) {
338  LOG(WARNING)
339  << "Failed to generate GPU IR for UDF " + udf_file_name +
340  ", attempting to use CPU compiled IR for GPU.\nUDF Compiler exception: " +
341  e.what();
342  }
343 #endif
344  return std::make_pair(cpu_file_name, cuda_file_name);
345 }
#define LOG(tag)
Definition: Logger.h:216
void generateAST(const std::string &file_name) const
std::string compileToLLVMIR(const std::string &udf_file_name) const

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void UdfCompiler::generateAST ( const std::string &  file_name) const
private

Call clang binary to generate abstract syntax tree file for registration in Calcite.

Definition at line 608 of file UdfCompiler.cpp.

References clang_options_, clang_path_, anonymous_namespace{UdfCompiler.cpp}::convert(), gpu_enabled::copy(), File_Namespace::create(), logger::init(), anonymous_namespace{UdfCompiler.cpp}::replace_extension(), run_benchmark_import::result, to_string(), ToolingSampleCategory(), and shared::transform().

Referenced by compileUdf().

608  {
609  UdfClangDriver the_driver = UdfClangDriver::init(clang_path_);
610  std::string resource_path = the_driver.getClangDriver()->ResourceDir;
611  std::string include_option =
612  std::string("-I") + resource_path + std::string("/include");
613 
614  std::vector<std::string> arg_vector;
615  arg_vector.emplace_back("astparser");
616  arg_vector.emplace_back(file_name);
617  arg_vector.emplace_back("--");
618  arg_vector.emplace_back("-DNO_BOOST");
619  arg_vector.emplace_back(include_option);
620  arg_vector.emplace_back("-std=c++17");
621 
622  if (clang_options_.size() > 0) {
623  std::copy(
624  clang_options_.begin(), clang_options_.end(), std::back_inserter(arg_vector));
625  }
626  std::vector<const char*> arg_vec2;
628  arg_vector.begin(), arg_vector.end(), std::back_inserter(arg_vec2), convert);
629 
630  int num_args = arg_vec2.size();
631 #if LLVM_VERSION_MAJOR > 12
632  auto op = CommonOptionsParser::create(num_args, &arg_vec2[0], ToolingSampleCategory);
633  ClangTool tool(op->getCompilations(), op->getSourcePathList());
634 #else
635  CommonOptionsParser op(num_args, &arg_vec2[0], ToolingSampleCategory);
636  ClangTool tool(op.getCompilations(), op.getSourcePathList());
637 #endif
638 
639  std::string out_name(file_name);
640  std::string file_ext("ast");
641  replace_extension(out_name, file_ext);
642 
643  std::error_code out_error_info;
644  llvm::raw_fd_ostream out_file(
645  llvm::StringRef(out_name), out_error_info, llvm::sys::fs::OF_None);
646 
647  auto factory = std::make_unique<ToolFactory>(out_file);
648  const auto result = tool.run(factory.get());
649  if (result != 0) {
650  throw std::runtime_error(
651  "Unable to create AST file for udf compilation (error code " +
652  std::to_string(result) + ")");
653  }
654 }
std::vector< std::string > clang_options_
Definition: UdfCompiler.h:73
std::string to_string(char const *&&v)
FILE * create(const std::string &basePath, const int fileId, const size_t pageSize, const size_t numPages)
Definition: File.cpp:57
void init(LogOptions const &log_opts)
Definition: Logger.cpp:308
DEVICE auto copy(ARGS &&...args)
Definition: gpu_enabled.h:51
OUTPUT transform(INPUT const &input, FUNC const &func)
Definition: misc.h:296
void replace_extension(std::string &s, const std::string &new_ext)
const char * convert(const std::string &s)
std::string clang_path_
Definition: UdfCompiler.h:72
static llvm::cl::OptionCategory ToolingSampleCategory("UDF Tooling")

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string UdfCompiler::genLLVMIRFilename ( const std::string &  udf_file_name)
staticprivate

Definition at line 383 of file UdfCompiler.cpp.

References anonymous_namespace{UdfCompiler.cpp}::remove_file_extension().

Referenced by compileToLLVMIR().

383  {
384  return remove_file_extension(udf_file_name) + "_cpu.bc";
385 }
std::string remove_file_extension(const std::string &path)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string UdfCompiler::genNVVMIRFilename ( const std::string &  udf_file_name)
staticprivate

Definition at line 379 of file UdfCompiler.cpp.

References anonymous_namespace{UdfCompiler.cpp}::remove_file_extension().

379  {
380  return remove_file_extension(udf_file_name) + "_gpu.bc";
381 }
std::string remove_file_extension(const std::string &path)

+ Here is the call graph for this function:

std::string UdfCompiler::getAstFileName ( const std::string &  udf_file_name)
static

Definition at line 656 of file UdfCompiler.cpp.

References anonymous_namespace{UdfCompiler.cpp}::replace_extension().

Referenced by DBHandler::initialize().

656  {
657  auto ast_file_name = udf_file_name;
658  replace_extension(ast_file_name, "ast");
659  return ast_file_name;
660 }
void replace_extension(std::string &s, const std::string &new_ext)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

Member Data Documentation

std::vector<std::string> UdfCompiler::clang_options_
private

Definition at line 73 of file UdfCompiler.h.

Referenced by compileFromCommandLine(), generateAST(), and UdfCompiler().

std::string UdfCompiler::clang_path_
private

Definition at line 72 of file UdfCompiler.h.

Referenced by compileFromCommandLine(), compileToLLVMIR(), and generateAST().


The documentation for this class was generated from the following files: