Hardware Locality (hwloc) 1.11.9

Terms and Definitions

# Objects

Object

  Interesting kind of part of the system, such as a Core, a Cache, a NUMA memory node, etc. The different types detected by hwloc are detailed in the hwloc_obj_type_t enumeration.

  They are topologically sorted by CPU set into a tree (see Hierarchy, Tree and Levels).

Processing Unit (or Logical Processor)

  The smallest processing element that can be represented by a hwloc object. It may be a single-core processor, a core of a multicore processor, or a single thread in a SMT processor. hwloc's PU acronym stands for Processing Unit.

  "Logical processor" should not be confused with "Logical index of a processor".

Package

  A processor Package is the physical package that usually gets inserted into a socket on the motherboard. It is also often called a physical processor or a CPU even if these names bring confusion with respect to cores and processing units. A processor package usually contains multiple cores (and may also be composed of multiple dies). hwloc Package objects were called Sockets up to hwloc 1.10.

NUMA Node

  An object that contains memory that is directly and byte-accessible to the host processors. It is usually close to some cores (the corresponding objects are descendants of the NUMA node object in the hwloc tree, see Hierarchy, Tree and Levels) as specified by its CPU set.

# Indexes and Sets

OS or physical index

  The index that the operating system (OS) uses to identify the object. This may be completely arbitrary, non-unique, non-contiguous, not representative of logical proximity, and may depend on the BIOS configuration. That is why hwloc almost never uses them, only in the default lstopo output (`P#x`) and cpuset masks. See also Should I use logical or physical/OS indexes? and how?.

Logical index

  Index to uniquely identify objects of the same type and depth, automatically computed by hwloc according to the topology. It expresses logical proximity in a generic way, i.e. objects which have adjacent logical indexes are adjacent in the topology. That is why hwloc almost always uses it in its API, since it expresses logical proximity. They can be shown (as `L#x`) by `lstopo` thanks to the `-l` option. This index is always linear and in the range [0, num_objs_same_type_same_level-1]. Think of it as ``cousin rank.'' The ordering is based on topology first, and then on OS CPU numbers, so it is stable across everything except firmware CPU renumbering. "Logical index" should not be confused with "Logical processor". A "Logical processor" (which in hwloc we rather call "processing unit" to avoid the confusion) has both a physical index (as chosen arbitrarily by BIOS/OS) and a logical index (as computed according to logical proximity by hwloc). See also Should I use logical or physical/OS indexes? and how?.

CPU set

  The set of logical processors (or processing units) logically included in an object (if it makes sense). They are always expressed using physical logical processor numbers (as announced by the OS). They are implemented as the hwloc_bitmap_t opaque structure. hwloc CPU sets are just masks, they do not have any relation with an operating system actual binding notion like Linux' cpusets.

Node set

  The set of NUMA memory nodes logically included in an object (if it makes sense). They are always expressed using physical node numbers (as announced by the OS). They are implemented with the hwloc_bitmap_t opaque structure. as bitmaps.

Bitmap

  A possibly-infinite set of bits used for describing sets of objects such as CPUs (CPU sets) or memory nodes (Node sets). They are implemented with the hwloc_bitmap_t opaque structure.

# Hierarchy, Tree and Levels

Parent object

  The object logically containing the current object, for example because its CPU set includes the CPU set of the current object.

Ancestor object

  The parent object, or its own parent object, and so on.

Children object(s)

  The object (or objects) contained in the current object because their CPU set is included in the CPU set of the current object.

Arity

  The number of children of an object.

Sibling objects

  Objects which have the same parent. They usually have the same type (and hence are cousins, as well), but they may not if the topology is asymmetric.

Sibling rank

  Index to uniquely identify objects which have the same parent, and is always in the range [0, parent_arity).

Cousin objects

  Objects of the same type (and depth) as the current object, even if they do not have the same parent.

Level

  Set of objects of the same type and depth. All these objects are cousins.

Depth

  Nesting level in the object tree, starting from the root object. If the topology is symmetric, the depth of a child is equal to the parent depth plus one, and an object depth is also equal to the number of parent/child links between the root object and the given object. If the topology is asymmetric, the difference between some parent and child depths may be larger than one when some intermediate levels (for instance groups) are missing in only some parts of the machine.

The following diagram can help to understand the vocabulary of the relationships by showing the example of a machine with two dual core packages (with no hardware threads); thus, a topology with 5 levels. Each box with rounded corner corresponds to one hwloc_obj_t, containing the values of the different integer fields (depth, logical_index, etc.), and arrows show to which other hwloc_obj_t pointers point to (first_child, parent, etc.). The L2 cache of the last core is intentionally missing to show how asymmetric topologies are handled. See What happens if my topology is asymmetric? for more information about such strange topologies.

It should be noted that for PU objects, the logical index – as computed linearly by hwloc – is not the same as the OS index.