OmniSciDB  72c90bc290
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
UdfCompiler Class Reference

#include <UdfCompiler.h>

+ Collaboration diagram for UdfCompiler:

Public Member Functions

 UdfCompiler (CudaMgr_Namespace::NvidiaDeviceArch target_arch, const std::string &clang_path_override="")
 
 UdfCompiler (CudaMgr_Namespace::NvidiaDeviceArch target_arch, const std::string &clang_path_override, const std::vector< std::string > clang_options)
 
std::pair< std::string,
std::string > 
compileUdf (const std::string &udf_file_name) const
 

Static Public Member Functions

static std::string getAstFileName (const std::string &udf_file_name)
 

Private Member Functions

void generateAST (const std::string &file_name) const
 
std::string compileToLLVMIR (const std::string &udf_file_name) const
 
int compileFromCommandLine (const std::vector< std::string > &command_line) const
 

Static Private Member Functions

static std::string genLLVMIRFilename (const std::string &udf_file_name)
 
static std::string genNVVMIRFilename (const std::string &udf_file_name)
 

Private Attributes

std::string clang_path_
 
std::vector< std::string > clang_options_
 

Detailed Description

Driver for calling clang/clang++ to compile C++ programs to LLVM IR for use as a UDF. Default initialization will find Clang using the clang library invocations. An optional clang override and additional arguments to the clang binary can be added. Once initialized the class holds the state for calling clang until destruction.

Definition at line 31 of file UdfCompiler.h.

Constructor & Destructor Documentation

UdfCompiler::UdfCompiler ( CudaMgr_Namespace::NvidiaDeviceArch  target_arch,
const std::string &  clang_path_override = "" 
)

Definition at line 314 of file UdfCompiler.cpp.

References clang_options_.

316  : clang_path_(get_clang_path(clang_path_override))
317 #ifdef HAVE_CUDA
318  , target_arch_(target_arch)
319 #endif
320 {
321  // To ifguard unsupported code included in heavydbTypes.h (i.e. <shared_mutex>)
322  clang_options_.emplace_back(std::string("-D UDF_COMPILED"));
323 }
std::vector< std::string > clang_options_
Definition: UdfCompiler.h:73
std::string get_clang_path(const std::string &clang_path_override)
std::string clang_path_
Definition: UdfCompiler.h:72
UdfCompiler::UdfCompiler ( CudaMgr_Namespace::NvidiaDeviceArch  target_arch,
const std::string &  clang_path_override,
const std::vector< std::string >  clang_options 
)

Definition at line 325 of file UdfCompiler.cpp.

References clang_options_.

328  : clang_path_(get_clang_path(clang_path_override))
329  , clang_options_(clang_options)
330 #ifdef HAVE_CUDA
331  , target_arch_(target_arch)
332 #endif
333 {
334  // To ifguard unsupported code included in heavydbTypes.h (i.e. <shared_mutex>)
335  clang_options_.emplace_back(std::string("-D UDF_COMPILED"));
336 }
std::vector< std::string > clang_options_
Definition: UdfCompiler.h:73
std::string get_clang_path(const std::string &clang_path_override)
std::string clang_path_
Definition: UdfCompiler.h:72

Member Function Documentation

int UdfCompiler::compileFromCommandLine ( const std::vector< std::string > &  command_line) const
private

Formulate the full compile command and call the compiler.

Definition at line 408 of file UdfCompiler.cpp.

References CHECK_EQ, clang_options_, clang_path_, anonymous_namespace{UdfCompiler.cpp}::exec_output(), logger::init(), run_benchmark_import::res, and shared::transform().

Referenced by compileToLLVMIR().

409  {
410  UdfClangDriver compiler_driver = UdfClangDriver::init(clang_path_);
411  auto the_driver(compiler_driver.getClangDriver());
412 
413  std::vector<const char*> clang_command_opts;
414  clang_command_opts.reserve(command_line.size() + clang_options_.size());
415  // add required options first
416  std::transform(std::begin(command_line),
417  std::end(command_line),
418  std::back_inserter(clang_command_opts),
419  [&](const std::string& str) { return str.c_str(); });
420 
421  // If there were additional clang options passed to the system, append them here
422  if (!clang_options_.empty()) {
423  std::transform(std::begin(clang_options_),
424  std::end(clang_options_),
425  std::back_inserter(clang_command_opts),
426  [&](const std::string& str) { return str.c_str(); });
427  }
428 
429  std::unique_ptr<driver::Compilation> compilation(
430  the_driver->BuildCompilation(clang_command_opts));
431  if (!compilation) {
432  throw std::runtime_error("failed to build compilation object!");
433  }
434  auto [clang_version_major, clang_version_minor, clang_version_patchlevel] =
435  compiler_driver.getClangVersion();
436  if (clang_version_major != CLANG_VERSION_MAJOR
437  // mismatch of clang driver and compulier versions requires
438  // modified workflow that removes incompatible driver flags for
439  // compiler.
440  || CLANG_VERSION_MAJOR == 9
441  // clang driver 9 requires cudatoolkit 8 that we don't support,
442  // hence switching to modified clang compiler 9 workflow that is
443  // able to produce bytecode to GPU when using cudatoolkit 11.
444  ) {
445  /* Fix incompatibilities when driver and clang versions differ.
446  */
447  auto& jobs = compilation->getJobs();
448  CHECK_EQ(jobs.size(), size_t(1));
449  auto& job = *jobs.begin();
450 
451  std::string cmd = job.getExecutable();
452  int skip = 0;
453  std::string last = "";
454 
455  for (auto& arg : job.getArguments()) {
456  const std::string& s = arg;
457  if (skip > 0) {
458  skip--;
459  last = s;
460  continue;
461  }
462 
463  // inclusion of __clang_cuda_runtime_wrapper.h leads to either
464  // clang >9 compilation failure or clang 9 failure for using
465  // cuda >8 (unsupported CUDA version).
466  if (s == "-include") {
467  last = s;
468  continue;
469  }
470  if (last == "-include") {
471  if (s != "__clang_cuda_runtime_wrapper.h") {
472  cmd += " -include " + s;
473  }
474  last = s;
475  continue;
476  }
477 
478  // Using -ffcuda-is-device flag produces empty gpu module
479  if (s == "-fcuda-is-device") {
480  last = s;
481  continue;
482  }
483 
484  if constexpr (CLANG_VERSION_MAJOR == 9) {
485  if (clang_version_major > 9) {
486  // The following clang 9 flags are unknown to clang >9:
487  if (s == "-masm-verbose" || s == "-fuse-init-array" ||
488  s == "-dwarf-column-info" || s == "-momit-leaf-frame-pointer" ||
489  s == "-fdiagnostics-show-option" || s == "-mdisable-fp-elim") {
490  last = s;
491  continue;
492  }
493  if (s == "-fmessage-length") {
494  skip = 1;
495  last = s;
496  continue;
497  }
498  }
499  }
500 
501  if constexpr (CLANG_VERSION_MAJOR == 10) {
502  if (clang_version_major > 10) {
503  // The following clang 10 flags are unknown to clang >10:
504  if (s == "-masm-verbose" || s == "-dwarf-column-info" ||
505  s == "-fdiagnostics-show-option") {
506  last = s;
507  continue;
508  }
509  if (s == "-fmessage-length") {
510  skip = 1;
511  last = s;
512  continue;
513  }
514  }
515  }
516 
517  if constexpr (CLANG_VERSION_MAJOR >= 10) {
518  if (clang_version_major < 10) {
519  // The following clang >10 flags are unknown to clang <10:
520  if (s == "-fno-rounding-math" || s.rfind("-mframe-pointer=", 0) == 0 ||
521  s.rfind("-fgnuc-version=", 0) == 0) {
522  last = s;
523  continue;
524  }
525  }
526  }
527 
528  if constexpr (CLANG_VERSION_MAJOR == 11) {
529  if (clang_version_major < 11) {
530  // The following clang 11 flags are unknown to clang <11:
531  if (s == "-fno-verbose-asm") {
532  last = s;
533  continue;
534  }
535  if (s == "-aux-target-cpu") {
536  last = s;
537  skip = 1;
538  continue;
539  }
540  }
541  }
542 
543  cmd += " " + s;
544  last = s;
545  }
546  // TODO: Here we don't use the_driver->ExecuteCompilation because
547  // could not find a better way to modify driver arguments. As a
548  // workaround, we run clang compiler via pipe (it could be run
549  // also via shell).
550  exec_output(cmd);
551  return 0;
552  }
553 
554  llvm::SmallVector<std::pair<int, const driver::Command*>, 10> failing_commands;
555  int res = the_driver->ExecuteCompilation(*compilation, failing_commands);
556  if (res < 0) {
557  for (const std::pair<int, const driver::Command*>& p : failing_commands) {
558  if (p.first) {
559  the_driver->generateCompilationDiagnostics(*compilation, *p.second);
560  }
561  }
562  }
563  return res;
564 }
#define CHECK_EQ(x, y)
Definition: Logger.h:301
std::vector< std::string > clang_options_
Definition: UdfCompiler.h:73
std::string exec_output(std::string cmd)
void init(LogOptions const &log_opts)
Definition: Logger.cpp:364
OUTPUT transform(INPUT const &input, FUNC const &func)
Definition: misc.h:320
std::string clang_path_
Definition: UdfCompiler.h:72

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string UdfCompiler::compileToLLVMIR ( const std::string &  udf_file_name) const
private

Formulate Clang command line command and call clang binary to generate LLVM IR for the C/C++ file.

Definition at line 605 of file UdfCompiler.cpp.

References clang_path_, compileFromCommandLine(), genLLVMIRFilename(), run_benchmark_import::res, and to_string().

Referenced by compileUdf().

605  {
606  std::string cpu_out_filename = genLLVMIRFilename(udf_file_name);
607 
608  std::vector<std::string> command_line{clang_path_,
609  "-c",
610  "-O2",
611  "-emit-llvm",
612  "-o",
613  cpu_out_filename,
614  "-std=c++17",
615  "-DNO_BOOST",
616  udf_file_name};
617  auto res = compileFromCommandLine(command_line);
618  if (res != 0) {
619  throw std::runtime_error("Failed to compile CPU UDF (status code " +
620  std::to_string(res) + ")");
621  }
622  if (!boost::filesystem::exists(cpu_out_filename)) {
623  throw std::runtime_error("udf compile did not produce output file " +
624  cpu_out_filename);
625  }
626  return cpu_out_filename;
627 }
int compileFromCommandLine(const std::vector< std::string > &command_line) const
std::string to_string(char const *&&v)
static std::string genLLVMIRFilename(const std::string &udf_file_name)
std::string clang_path_
Definition: UdfCompiler.h:72

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::pair< std::string, std::string > UdfCompiler::compileUdf ( const std::string &  udf_file_name) const

Compile a C++ file to LLVM IR, and generate an AST file. Both artifacts exist as files on disk. Three artifacts will be generated; the AST file, the CPU LLVM IR, and GPU LLVM IR (if CUDA is enabled and compilation succeeds). These LLVM IR files can be loaded by the Executor. The AST will be processed by Calcite.

Definition at line 338 of file UdfCompiler.cpp.

References compileToLLVMIR(), generateAST(), logger::INFO, LOG, and logger::WARNING.

Referenced by DBHandler::initialize().

339  {
340  LOG(INFO) << "UDFCompiler filename to compile: " << udf_file_name;
341  if (!boost::filesystem::exists(udf_file_name)) {
342  throw std::runtime_error("User defined function file " + udf_file_name +
343  " does not exist.");
344  }
345 
346  // create the AST file for the input function
347  generateAST(udf_file_name);
348 
349  // Compile udf file to generate cpu and gpu bytecode files
350  std::string cpu_file_name = "";
351  std::string cuda_file_name = "";
352 
353  cpu_file_name = compileToLLVMIR(udf_file_name);
354 
355 #ifdef HAVE_CUDA
356  try {
357  cuda_file_name = compileToNVVMIR(udf_file_name);
358  } catch (const std::exception& e) {
359  LOG(WARNING)
360  << "Failed to generate GPU IR for UDF " + udf_file_name +
361  ", attempting to use CPU compiled IR for GPU.\nUDF Compiler exception: " +
362  e.what();
363  }
364 #endif
365  return std::make_pair(cpu_file_name, cuda_file_name);
366 }
#define LOG(tag)
Definition: Logger.h:285
void generateAST(const std::string &file_name) const
std::string compileToLLVMIR(const std::string &udf_file_name) const

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void UdfCompiler::generateAST ( const std::string &  file_name) const
private

Call clang binary to generate abstract syntax tree file for registration in Calcite.

Definition at line 629 of file UdfCompiler.cpp.

References clang_options_, clang_path_, anonymous_namespace{UdfCompiler.cpp}::convert(), gpu_enabled::copy(), File_Namespace::create(), report::error_code(), logger::init(), anonymous_namespace{UdfCompiler.cpp}::replace_extension(), run_benchmark_import::result, to_string(), ToolingSampleCategory(), and shared::transform().

Referenced by compileUdf().

629  {
630  UdfClangDriver the_driver = UdfClangDriver::init(clang_path_);
631  std::string resource_path = the_driver.getClangDriver()->ResourceDir;
632  std::string include_option =
633  std::string("-I") + resource_path + std::string("/include");
634 
635  std::vector<std::string> arg_vector;
636  arg_vector.emplace_back("astparser");
637  arg_vector.emplace_back(file_name);
638  arg_vector.emplace_back("--");
639  arg_vector.emplace_back("-DNO_BOOST");
640  arg_vector.emplace_back(include_option);
641  arg_vector.emplace_back("-std=c++17");
642 
643  if (clang_options_.size() > 0) {
644  std::copy(
645  clang_options_.begin(), clang_options_.end(), std::back_inserter(arg_vector));
646  }
647  std::vector<const char*> arg_vec2;
649  arg_vector.begin(), arg_vector.end(), std::back_inserter(arg_vec2), convert);
650 
651  int num_args = arg_vec2.size();
652 #if LLVM_VERSION_MAJOR > 12
653  auto op = CommonOptionsParser::create(num_args, &arg_vec2[0], ToolingSampleCategory);
654  ClangTool tool(op->getCompilations(), op->getSourcePathList());
655 #else
656  CommonOptionsParser op(num_args, &arg_vec2[0], ToolingSampleCategory);
657  ClangTool tool(op.getCompilations(), op.getSourcePathList());
658 #endif
659 
660  std::string out_name(file_name);
661  std::string file_ext("ast");
662  replace_extension(out_name, file_ext);
663 
664  std::error_code out_error_info;
665  llvm::raw_fd_ostream out_file(
666  llvm::StringRef(out_name), out_error_info, llvm::sys::fs::OF_None);
667 
668  auto factory = std::make_unique<ToolFactory>(out_file);
669  const auto result = tool.run(factory.get());
670  if (result != 0) {
671  throw std::runtime_error(
672  "Unable to create AST file for udf compilation (error code " +
673  std::to_string(result) + ")");
674  }
675 }
std::vector< std::string > clang_options_
Definition: UdfCompiler.h:73
std::pair< FILE *, std::string > create(const std::string &basePath, const int fileId, const size_t pageSize, const size_t numPages)
Definition: File.cpp:55
std::string to_string(char const *&&v)
void init(LogOptions const &log_opts)
Definition: Logger.cpp:364
DEVICE auto copy(ARGS &&...args)
Definition: gpu_enabled.h:51
OUTPUT transform(INPUT const &input, FUNC const &func)
Definition: misc.h:320
void replace_extension(std::string &s, const std::string &new_ext)
def error_code
Definition: report.py:244
const char * convert(const std::string &s)
std::string clang_path_
Definition: UdfCompiler.h:72
static llvm::cl::OptionCategory ToolingSampleCategory("UDF Tooling")

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string UdfCompiler::genLLVMIRFilename ( const std::string &  udf_file_name)
staticprivate

Definition at line 404 of file UdfCompiler.cpp.

References anonymous_namespace{UdfCompiler.cpp}::remove_file_extension().

Referenced by compileToLLVMIR().

404  {
405  return remove_file_extension(udf_file_name) + "_cpu.bc";
406 }
std::string remove_file_extension(const std::string &path)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string UdfCompiler::genNVVMIRFilename ( const std::string &  udf_file_name)
staticprivate

Definition at line 400 of file UdfCompiler.cpp.

References anonymous_namespace{UdfCompiler.cpp}::remove_file_extension().

400  {
401  return remove_file_extension(udf_file_name) + "_gpu.bc";
402 }
std::string remove_file_extension(const std::string &path)

+ Here is the call graph for this function:

std::string UdfCompiler::getAstFileName ( const std::string &  udf_file_name)
static

Definition at line 677 of file UdfCompiler.cpp.

References anonymous_namespace{UdfCompiler.cpp}::replace_extension().

Referenced by DBHandler::initialize().

677  {
678  auto ast_file_name = udf_file_name;
679  replace_extension(ast_file_name, "ast");
680  return ast_file_name;
681 }
void replace_extension(std::string &s, const std::string &new_ext)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

Member Data Documentation

std::vector<std::string> UdfCompiler::clang_options_
private

Definition at line 73 of file UdfCompiler.h.

Referenced by compileFromCommandLine(), generateAST(), and UdfCompiler().

std::string UdfCompiler::clang_path_
private

Definition at line 72 of file UdfCompiler.h.

Referenced by compileFromCommandLine(), compileToLLVMIR(), and generateAST().


The documentation for this class was generated from the following files: