1
0
mirror of https://github.com/RPCS3/llvm-mirror.git synced 2024-10-19 19:12:56 +02:00
llvm-mirror/tools/llvm-cov/CoverageExporterJson.cpp
Alexandre Ganea ae05eb086d [Support] On Windows, ensure hardware_concurrency() extends to all CPU sockets and all NUMA groups
The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.

== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.

By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.

This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.

== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std:🧵:hardware_concurrency() -- which can only return processors from the current "processor group".

== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).

When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.

When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.

Differential Revision: https://reviews.llvm.org/D71775
2020-02-14 10:24:22 -05:00

238 lines
10 KiB
C++

//===- CoverageExporterJson.cpp - Code coverage export --------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
// This file implements export of code coverage data to JSON.
//
//===----------------------------------------------------------------------===//
//===----------------------------------------------------------------------===//
//
// The json code coverage export follows the following format
// Root: dict => Root Element containing metadata
// -- Data: array => Homogeneous array of one or more export objects
// -- Export: dict => Json representation of one CoverageMapping
// -- Files: array => List of objects describing coverage for files
// -- File: dict => Coverage for a single file
// -- Segments: array => List of Segments contained in the file
// -- Segment: dict => Describes a segment of the file with a counter
// -- Expansions: array => List of expansion records
// -- Expansion: dict => Object that descibes a single expansion
// -- CountedRegion: dict => The region to be expanded
// -- TargetRegions: array => List of Regions in the expansion
// -- CountedRegion: dict => Single Region in the expansion
// -- Summary: dict => Object summarizing the coverage for this file
// -- LineCoverage: dict => Object summarizing line coverage
// -- FunctionCoverage: dict => Object summarizing function coverage
// -- RegionCoverage: dict => Object summarizing region coverage
// -- Functions: array => List of objects describing coverage for functions
// -- Function: dict => Coverage info for a single function
// -- Filenames: array => List of filenames that the function relates to
// -- Summary: dict => Object summarizing the coverage for the entire binary
// -- LineCoverage: dict => Object summarizing line coverage
// -- FunctionCoverage: dict => Object summarizing function coverage
// -- InstantiationCoverage: dict => Object summarizing inst. coverage
// -- RegionCoverage: dict => Object summarizing region coverage
//
//===----------------------------------------------------------------------===//
#include "CoverageExporterJson.h"
#include "CoverageReport.h"
#include "llvm/ADT/Optional.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/Support/JSON.h"
#include "llvm/Support/ThreadPool.h"
#include "llvm/Support/Threading.h"
#include <algorithm>
#include <limits>
#include <mutex>
#include <utility>
/// The semantic version combined as a string.
#define LLVM_COVERAGE_EXPORT_JSON_STR "2.0.0"
/// Unique type identifier for JSON coverage export.
#define LLVM_COVERAGE_EXPORT_JSON_TYPE_STR "llvm.coverage.json.export"
using namespace llvm;
namespace {
// The JSON library accepts int64_t, but profiling counts are stored as uint64_t.
// Therefore we need to explicitly convert from unsigned to signed, since a naive
// cast is implementation-defined behavior when the unsigned value cannot be
// represented as a signed value. We choose to clamp the values to preserve the
// invariant that counts are always >= 0.
int64_t clamp_uint64_to_int64(uint64_t u) {
return std::min(u, static_cast<uint64_t>(std::numeric_limits<int64_t>::max()));
}
json::Array renderSegment(const coverage::CoverageSegment &Segment) {
return json::Array({Segment.Line, Segment.Col, clamp_uint64_to_int64(Segment.Count),
Segment.HasCount, Segment.IsRegionEntry});
}
json::Array renderRegion(const coverage::CountedRegion &Region) {
return json::Array({Region.LineStart, Region.ColumnStart, Region.LineEnd,
Region.ColumnEnd, clamp_uint64_to_int64(Region.ExecutionCount),
Region.FileID, Region.ExpandedFileID,
int64_t(Region.Kind)});
}
json::Array renderRegions(ArrayRef<coverage::CountedRegion> Regions) {
json::Array RegionArray;
for (const auto &Region : Regions)
RegionArray.push_back(renderRegion(Region));
return RegionArray;
}
json::Object renderExpansion(const coverage::ExpansionRecord &Expansion) {
return json::Object(
{{"filenames", json::Array(Expansion.Function.Filenames)},
// Mark the beginning and end of this expansion in the source file.
{"source_region", renderRegion(Expansion.Region)},
// Enumerate the coverage information for the expansion.
{"target_regions", renderRegions(Expansion.Function.CountedRegions)}});
}
json::Object renderSummary(const FileCoverageSummary &Summary) {
return json::Object(
{{"lines",
json::Object({{"count", int64_t(Summary.LineCoverage.getNumLines())},
{"covered", int64_t(Summary.LineCoverage.getCovered())},
{"percent", Summary.LineCoverage.getPercentCovered()}})},
{"functions",
json::Object(
{{"count", int64_t(Summary.FunctionCoverage.getNumFunctions())},
{"covered", int64_t(Summary.FunctionCoverage.getExecuted())},
{"percent", Summary.FunctionCoverage.getPercentCovered()}})},
{"instantiations",
json::Object(
{{"count",
int64_t(Summary.InstantiationCoverage.getNumFunctions())},
{"covered", int64_t(Summary.InstantiationCoverage.getExecuted())},
{"percent", Summary.InstantiationCoverage.getPercentCovered()}})},
{"regions",
json::Object(
{{"count", int64_t(Summary.RegionCoverage.getNumRegions())},
{"covered", int64_t(Summary.RegionCoverage.getCovered())},
{"notcovered", int64_t(Summary.RegionCoverage.getNumRegions() -
Summary.RegionCoverage.getCovered())},
{"percent", Summary.RegionCoverage.getPercentCovered()}})}});
}
json::Array renderFileExpansions(const coverage::CoverageData &FileCoverage,
const FileCoverageSummary &FileReport) {
json::Array ExpansionArray;
for (const auto &Expansion : FileCoverage.getExpansions())
ExpansionArray.push_back(renderExpansion(Expansion));
return ExpansionArray;
}
json::Array renderFileSegments(const coverage::CoverageData &FileCoverage,
const FileCoverageSummary &FileReport) {
json::Array SegmentArray;
for (const auto &Segment : FileCoverage)
SegmentArray.push_back(renderSegment(Segment));
return SegmentArray;
}
json::Object renderFile(const coverage::CoverageMapping &Coverage,
const std::string &Filename,
const FileCoverageSummary &FileReport,
const CoverageViewOptions &Options) {
json::Object File({{"filename", Filename}});
if (!Options.ExportSummaryOnly) {
// Calculate and render detailed coverage information for given file.
auto FileCoverage = Coverage.getCoverageForFile(Filename);
File["segments"] = renderFileSegments(FileCoverage, FileReport);
if (!Options.SkipExpansions) {
File["expansions"] = renderFileExpansions(FileCoverage, FileReport);
}
}
File["summary"] = renderSummary(FileReport);
return File;
}
json::Array renderFiles(const coverage::CoverageMapping &Coverage,
ArrayRef<std::string> SourceFiles,
ArrayRef<FileCoverageSummary> FileReports,
const CoverageViewOptions &Options) {
auto NumThreads = Options.NumThreads;
if (NumThreads == 0)
NumThreads = SourceFiles.size();
ThreadPool Pool(heavyweight_hardware_concurrency(NumThreads));
json::Array FileArray;
std::mutex FileArrayMutex;
for (unsigned I = 0, E = SourceFiles.size(); I < E; ++I) {
auto &SourceFile = SourceFiles[I];
auto &FileReport = FileReports[I];
Pool.async([&] {
auto File = renderFile(Coverage, SourceFile, FileReport, Options);
{
std::lock_guard<std::mutex> Lock(FileArrayMutex);
FileArray.push_back(std::move(File));
}
});
}
Pool.wait();
return FileArray;
}
json::Array renderFunctions(
const iterator_range<coverage::FunctionRecordIterator> &Functions) {
json::Array FunctionArray;
for (const auto &F : Functions)
FunctionArray.push_back(
json::Object({{"name", F.Name},
{"count", clamp_uint64_to_int64(F.ExecutionCount)},
{"regions", renderRegions(F.CountedRegions)},
{"filenames", json::Array(F.Filenames)}}));
return FunctionArray;
}
} // end anonymous namespace
void CoverageExporterJson::renderRoot(const CoverageFilters &IgnoreFilters) {
std::vector<std::string> SourceFiles;
for (StringRef SF : Coverage.getUniqueSourceFiles()) {
if (!IgnoreFilters.matchesFilename(SF))
SourceFiles.emplace_back(SF);
}
renderRoot(SourceFiles);
}
void CoverageExporterJson::renderRoot(ArrayRef<std::string> SourceFiles) {
FileCoverageSummary Totals = FileCoverageSummary("Totals");
auto FileReports = CoverageReport::prepareFileReports(Coverage, Totals,
SourceFiles, Options);
auto Files = renderFiles(Coverage, SourceFiles, FileReports, Options);
// Sort files in order of their names.
std::sort(Files.begin(), Files.end(),
[](const json::Value &A, const json::Value &B) {
const json::Object *ObjA = A.getAsObject();
const json::Object *ObjB = B.getAsObject();
assert(ObjA != nullptr && "Value A was not an Object");
assert(ObjB != nullptr && "Value B was not an Object");
const StringRef FilenameA = ObjA->getString("filename").getValue();
const StringRef FilenameB = ObjB->getString("filename").getValue();
return FilenameA.compare(FilenameB) < 0;
});
auto Export = json::Object(
{{"files", std::move(Files)}, {"totals", renderSummary(Totals)}});
// Skip functions-level information if necessary.
if (!Options.ExportSummaryOnly && !Options.SkipFunctions)
Export["functions"] = renderFunctions(Coverage.getCoveredFunctions());
auto ExportArray = json::Array({std::move(Export)});
OS << json::Object({{"version", LLVM_COVERAGE_EXPORT_JSON_STR},
{"type", LLVM_COVERAGE_EXPORT_JSON_TYPE_STR},
{"data", std::move(ExportArray)}});
}