OmniSciDB  16c4e035a1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
UdfCompiler Class Reference

#include <UdfCompiler.h>

+ Collaboration diagram for UdfCompiler:

Public Member Functions

 UdfCompiler (CudaMgr_Namespace::NvidiaDeviceArch target_arch, const std::string &clang_path_override="")
 
 UdfCompiler (CudaMgr_Namespace::NvidiaDeviceArch target_arch, const std::string &clang_path_override, const std::vector< std::string > clang_options)
 
std::pair< std::string,
std::string > 
compileUdf (const std::string &udf_file_name) const
 

Static Public Member Functions

static std::string getAstFileName (const std::string &udf_file_name)
 

Private Member Functions

void generateAST (const std::string &file_name) const
 
std::string compileToLLVMIR (const std::string &udf_file_name) const
 
int compileFromCommandLine (const std::vector< std::string > &command_line) const
 

Static Private Member Functions

static std::string genLLVMIRFilename (const std::string &udf_file_name)
 
static std::string genNVVMIRFilename (const std::string &udf_file_name)
 

Private Attributes

std::string clang_path_
 
std::vector< std::string > clang_options_
 

Detailed Description

Driver for calling clang/clang++ to compile C++ programs to LLVM IR for use as a UDF. Default initialization will find Clang using the clang library invocations. An optional clang override and additional arguments to the clang binary can be added. Once initialized the class holds the state for calling clang until destruction.

Definition at line 31 of file UdfCompiler.h.

Constructor & Destructor Documentation

UdfCompiler::UdfCompiler ( CudaMgr_Namespace::NvidiaDeviceArch  target_arch,
const std::string &  clang_path_override = "" 
)

Definition at line 292 of file UdfCompiler.cpp.

294  : clang_path_(get_clang_path(clang_path_override))
295 #ifdef HAVE_CUDA
296  , target_arch_(target_arch)
297 #endif
298 {
299 }
std::string get_clang_path(const std::string &clang_path_override)
std::string clang_path_
Definition: UdfCompiler.h:72
UdfCompiler::UdfCompiler ( CudaMgr_Namespace::NvidiaDeviceArch  target_arch,
const std::string &  clang_path_override,
const std::vector< std::string >  clang_options 
)

Definition at line 301 of file UdfCompiler.cpp.

304  : clang_path_(get_clang_path(clang_path_override))
305  , clang_options_(clang_options)
306 #ifdef HAVE_CUDA
307  , target_arch_(target_arch)
308 #endif
309 {
310 }
std::vector< std::string > clang_options_
Definition: UdfCompiler.h:73
std::string get_clang_path(const std::string &clang_path_override)
std::string clang_path_
Definition: UdfCompiler.h:72

Member Function Documentation

int UdfCompiler::compileFromCommandLine ( const std::vector< std::string > &  command_line) const
private

Formulate the full compile command and call the compiler.

Definition at line 382 of file UdfCompiler.cpp.

References CHECK_EQ, clang_options_, clang_path_, anonymous_namespace{UdfCompiler.cpp}::exec_output(), logger::init(), run_benchmark_import::res, and shared::transform().

Referenced by compileToLLVMIR().

383  {
384  UdfClangDriver compiler_driver = UdfClangDriver::init(clang_path_);
385  auto the_driver(compiler_driver.getClangDriver());
386 
387  std::vector<const char*> clang_command_opts;
388  clang_command_opts.reserve(command_line.size() + clang_options_.size());
389  // add required options first
390  std::transform(std::begin(command_line),
391  std::end(command_line),
392  std::back_inserter(clang_command_opts),
393  [&](const std::string& str) { return str.c_str(); });
394 
395  // If there were additional clang options passed to the system, append them here
396  if (!clang_options_.empty()) {
397  std::transform(std::begin(clang_options_),
398  std::end(clang_options_),
399  std::back_inserter(clang_command_opts),
400  [&](const std::string& str) { return str.c_str(); });
401  }
402 
403  std::unique_ptr<driver::Compilation> compilation(
404  the_driver->BuildCompilation(clang_command_opts));
405  if (!compilation) {
406  throw std::runtime_error("failed to build compilation object!");
407  }
408  auto [clang_version_major, clang_version_minor, clang_version_patchlevel] =
409  compiler_driver.getClangVersion();
410  if (clang_version_major != CLANG_VERSION_MAJOR
411  // mismatch of clang driver and compulier versions requires
412  // modified workflow that removes incompatible driver flags for
413  // compiler.
414  || CLANG_VERSION_MAJOR == 9
415  // clang driver 9 requires cudatoolkit 8 that we don't support,
416  // hence switching to modified clang compiler 9 workflow that is
417  // able to produce bytecode to GPU when using cudatoolkit 11.
418  ) {
419  /* Fix incompatibilities when driver and clang versions differ.
420  */
421  auto& jobs = compilation->getJobs();
422  CHECK_EQ(jobs.size(), size_t(1));
423  auto& job = *jobs.begin();
424 
425  std::string cmd = job.getExecutable();
426  int skip = 0;
427  std::string last = "";
428 
429  for (auto& arg : job.getArguments()) {
430  const std::string& s = arg;
431  if (skip > 0) {
432  skip--;
433  last = s;
434  continue;
435  }
436 
437  // inclusion of __clang_cuda_runtime_wrapper.h leads to either
438  // clang >9 compilation failure or clang 9 failure for using
439  // cuda >8 (unsupported CUDA version).
440  if (s == "-include") {
441  last = s;
442  continue;
443  }
444  if (last == "-include") {
445  if (s != "__clang_cuda_runtime_wrapper.h") {
446  cmd += " -include " + s;
447  }
448  last = s;
449  continue;
450  }
451 
452  // Using -ffcuda-is-device flag produces empty gpu module
453  if (s == "-fcuda-is-device") {
454  last = s;
455  continue;
456  }
457 
458  if constexpr (CLANG_VERSION_MAJOR == 9) {
459  if (clang_version_major > 9) {
460  // The following clang 9 flags are unknown to clang >9:
461  if (s == "-masm-verbose" || s == "-fuse-init-array" ||
462  s == "-dwarf-column-info" || s == "-momit-leaf-frame-pointer" ||
463  s == "-fdiagnostics-show-option" || s == "-mdisable-fp-elim") {
464  last = s;
465  continue;
466  }
467  if (s == "-fmessage-length") {
468  skip = 1;
469  last = s;
470  continue;
471  }
472  }
473  }
474 
475  if constexpr (CLANG_VERSION_MAJOR == 10) {
476  if (clang_version_major > 10) {
477  // The following clang 10 flags are unknown to clang >10:
478  if (s == "-masm-verbose" || s == "-dwarf-column-info" ||
479  s == "-fdiagnostics-show-option") {
480  last = s;
481  continue;
482  }
483  if (s == "-fmessage-length") {
484  skip = 1;
485  last = s;
486  continue;
487  }
488  }
489  }
490 
491  if constexpr (CLANG_VERSION_MAJOR >= 10) {
492  if (clang_version_major < 10) {
493  // The following clang >10 flags are unknown to clang <10:
494  if (s == "-fno-rounding-math" || s.rfind("-mframe-pointer=", 0) == 0 ||
495  s.rfind("-fgnuc-version=", 0) == 0) {
496  last = s;
497  continue;
498  }
499  }
500  }
501 
502  if constexpr (CLANG_VERSION_MAJOR == 11) {
503  if (clang_version_major < 11) {
504  // The following clang 11 flags are unknown to clang <11:
505  if (s == "-fno-verbose-asm") {
506  last = s;
507  continue;
508  }
509  if (s == "-aux-target-cpu") {
510  last = s;
511  skip = 1;
512  continue;
513  }
514  }
515  }
516 
517  cmd += " " + s;
518  last = s;
519  }
520  // TODO: Here we don't use the_driver->ExecuteCompilation because
521  // could not find a better way to modify driver arguments. As a
522  // workaround, we run clang compiler via pipe (it could be run
523  // also via shell).
524  exec_output(cmd);
525  return 0;
526  }
527 
528  llvm::SmallVector<std::pair<int, const driver::Command*>, 10> failing_commands;
529  int res = the_driver->ExecuteCompilation(*compilation, failing_commands);
530  if (res < 0) {
531  for (const std::pair<int, const driver::Command*>& p : failing_commands) {
532  if (p.first) {
533  the_driver->generateCompilationDiagnostics(*compilation, *p.second);
534  }
535  }
536  }
537  return res;
538 }
#define CHECK_EQ(x, y)
Definition: Logger.h:219
std::vector< std::string > clang_options_
Definition: UdfCompiler.h:73
std::string exec_output(std::string cmd)
void init(LogOptions const &log_opts)
Definition: Logger.cpp:305
OUTPUT transform(INPUT const &input, FUNC const &func)
Definition: misc.h:290
std::string clang_path_
Definition: UdfCompiler.h:72

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string UdfCompiler::compileToLLVMIR ( const std::string &  udf_file_name) const
private

Formulate Clang command line command and call clang binary to generate LLVM IR for the C/C++ file.

Definition at line 579 of file UdfCompiler.cpp.

References clang_path_, compileFromCommandLine(), genLLVMIRFilename(), run_benchmark_import::res, and to_string().

Referenced by compileUdf().

579  {
580  std::string cpu_out_filename = genLLVMIRFilename(udf_file_name);
581 
582  std::vector<std::string> command_line{clang_path_,
583  "-c",
584  "-O2",
585  "-emit-llvm",
586  "-o",
587  cpu_out_filename,
588  "-std=c++14",
589  "-DNO_BOOST",
590  udf_file_name};
591  auto res = compileFromCommandLine(command_line);
592  if (res != 0) {
593  throw std::runtime_error("Failed to compile CPU UDF (status code " +
594  std::to_string(res) + ")");
595  }
596  if (!boost::filesystem::exists(cpu_out_filename)) {
597  throw std::runtime_error("udf compile did not produce output file " +
598  cpu_out_filename);
599  }
600  return cpu_out_filename;
601 }
int compileFromCommandLine(const std::vector< std::string > &command_line) const
std::string to_string(char const *&&v)
static std::string genLLVMIRFilename(const std::string &udf_file_name)
std::string clang_path_
Definition: UdfCompiler.h:72

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::pair< std::string, std::string > UdfCompiler::compileUdf ( const std::string &  udf_file_name) const

Compile a C++ file to LLVM IR, and generate an AST file. Both artifacts exist as files on disk. Three artifacts will be generated; the AST file, the CPU LLVM IR, and GPU LLVM IR (if CUDA is enabled and compilation succeeds). These LLVM IR files can be loaded by the Executor. The AST will be processed by Calcite.

Definition at line 312 of file UdfCompiler.cpp.

References compileToLLVMIR(), generateAST(), logger::INFO, LOG, and logger::WARNING.

Referenced by DBHandler::initialize().

313  {
314  LOG(INFO) << "UDFCompiler filename to compile: " << udf_file_name;
315  if (!boost::filesystem::exists(udf_file_name)) {
316  throw std::runtime_error("User defined function file " + udf_file_name +
317  " does not exist.");
318  }
319 
320  // create the AST file for the input function
321  generateAST(udf_file_name);
322 
323  // Compile udf file to generate cpu and gpu bytecode files
324  std::string cpu_file_name = "";
325  std::string cuda_file_name = "";
326 
327  cpu_file_name = compileToLLVMIR(udf_file_name);
328 
329 #ifdef HAVE_CUDA
330  try {
331  cuda_file_name = compileToNVVMIR(udf_file_name);
332  } catch (const std::exception& e) {
333  LOG(WARNING)
334  << "Failed to generate GPU IR for UDF " + udf_file_name +
335  ", attempting to use CPU compiled IR for GPU.\nUDF Compiler exception: " +
336  e.what();
337  }
338 #endif
339  return std::make_pair(cpu_file_name, cuda_file_name);
340 }
#define LOG(tag)
Definition: Logger.h:205
void generateAST(const std::string &file_name) const
std::string compileToLLVMIR(const std::string &udf_file_name) const

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

void UdfCompiler::generateAST ( const std::string &  file_name) const
private

Call clang binary to generate abstract syntax tree file for registration in Calcite.

Definition at line 603 of file UdfCompiler.cpp.

References clang_options_, clang_path_, anonymous_namespace{UdfCompiler.cpp}::convert(), gpu_enabled::copy(), File_Namespace::create(), logger::init(), anonymous_namespace{UdfCompiler.cpp}::replace_extension(), run_benchmark_import::result, to_string(), ToolingSampleCategory(), and shared::transform().

Referenced by compileUdf().

603  {
604  UdfClangDriver the_driver = UdfClangDriver::init(clang_path_);
605  std::string resource_path = the_driver.getClangDriver()->ResourceDir;
606  std::string include_option =
607  std::string("-I") + resource_path + std::string("/include");
608 
609  std::vector<std::string> arg_vector;
610  arg_vector.emplace_back("astparser");
611  arg_vector.emplace_back(file_name);
612  arg_vector.emplace_back("--");
613  arg_vector.emplace_back("-DNO_BOOST");
614  arg_vector.emplace_back(include_option);
615 
616  if (clang_options_.size() > 0) {
617  std::copy(
618  clang_options_.begin(), clang_options_.end(), std::back_inserter(arg_vector));
619  }
620  std::vector<const char*> arg_vec2;
622  arg_vector.begin(), arg_vector.end(), std::back_inserter(arg_vec2), convert);
623 
624  int num_args = arg_vec2.size();
625 #if LLVM_VERSION_MAJOR > 12
626  auto op = CommonOptionsParser::create(num_args, &arg_vec2[0], ToolingSampleCategory);
627  ClangTool tool(op->getCompilations(), op->getSourcePathList());
628 #else
629  CommonOptionsParser op(num_args, &arg_vec2[0], ToolingSampleCategory);
630  ClangTool tool(op.getCompilations(), op.getSourcePathList());
631 #endif
632 
633  std::string out_name(file_name);
634  std::string file_ext("ast");
635  replace_extension(out_name, file_ext);
636 
637  std::error_code out_error_info;
638  llvm::raw_fd_ostream out_file(
639  llvm::StringRef(out_name), out_error_info, llvm::sys::fs::OF_None);
640 
641  auto factory = std::make_unique<ToolFactory>(out_file);
642  const auto result = tool.run(factory.get());
643  if (result != 0) {
644  throw std::runtime_error(
645  "Unable to create AST file for udf compilation (error code " +
646  std::to_string(result) + ")");
647  }
648 }
std::vector< std::string > clang_options_
Definition: UdfCompiler.h:73
std::string to_string(char const *&&v)
FILE * create(const std::string &basePath, const int fileId, const size_t pageSize, const size_t numPages)
Definition: File.cpp:51
void init(LogOptions const &log_opts)
Definition: Logger.cpp:305
DEVICE auto copy(ARGS &&...args)
Definition: gpu_enabled.h:51
OUTPUT transform(INPUT const &input, FUNC const &func)
Definition: misc.h:290
void replace_extension(std::string &s, const std::string &new_ext)
const char * convert(const std::string &s)
std::string clang_path_
Definition: UdfCompiler.h:72
static llvm::cl::OptionCategory ToolingSampleCategory("UDF Tooling")

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string UdfCompiler::genLLVMIRFilename ( const std::string &  udf_file_name)
staticprivate

Definition at line 378 of file UdfCompiler.cpp.

References anonymous_namespace{UdfCompiler.cpp}::remove_file_extension().

Referenced by compileToLLVMIR().

378  {
379  return remove_file_extension(udf_file_name) + "_cpu.bc";
380 }
std::string remove_file_extension(const std::string &path)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

std::string UdfCompiler::genNVVMIRFilename ( const std::string &  udf_file_name)
staticprivate

Definition at line 374 of file UdfCompiler.cpp.

References anonymous_namespace{UdfCompiler.cpp}::remove_file_extension().

374  {
375  return remove_file_extension(udf_file_name) + "_gpu.bc";
376 }
std::string remove_file_extension(const std::string &path)

+ Here is the call graph for this function:

std::string UdfCompiler::getAstFileName ( const std::string &  udf_file_name)
static

Definition at line 650 of file UdfCompiler.cpp.

References anonymous_namespace{UdfCompiler.cpp}::replace_extension().

Referenced by DBHandler::initialize().

650  {
651  auto ast_file_name = udf_file_name;
652  replace_extension(ast_file_name, "ast");
653  return ast_file_name;
654 }
void replace_extension(std::string &s, const std::string &new_ext)

+ Here is the call graph for this function:

+ Here is the caller graph for this function:

Member Data Documentation

std::vector<std::string> UdfCompiler::clang_options_
private

Definition at line 73 of file UdfCompiler.h.

Referenced by compileFromCommandLine(), and generateAST().

std::string UdfCompiler::clang_path_
private

Definition at line 72 of file UdfCompiler.h.

Referenced by compileFromCommandLine(), compileToLLVMIR(), and generateAST().


The documentation for this class was generated from the following files: