qemu

ppc-spapr-hotplug.rst (20127B)

---

 =============================
 sPAPR Dynamic Reconfiguration
 =============================
 
 sPAPR or pSeries guests make use of a facility called dynamic reconfiguration
 to handle hot plugging of dynamic "physical" resources like PCI cards, or
 "logical"/para-virtual resources like memory, CPUs, and "physical"
 host-bridges, which are generally managed by the host/hypervisor and provided
 to guests as virtualized resources. The specifics of dynamic reconfiguration
 are documented extensively in section 13 of the Linux on Power Architecture
 Reference document ([LoPAR]_). This document provides a summary of that
 information as it applies to the implementation within QEMU.
 
 Dynamic-reconfiguration Connectors
 ==================================
 
 To manage hot plug/unplug of these resources, a firmware abstraction known as
 a Dynamic Resource Connector (DRC) is used to assign a particular dynamic
 resource to the guest, and provide an interface for the guest to manage
 configuration/removal of the resource associated with it.
 
 Device tree description of DRCs
 ===============================
 
 A set of four Open Firmware device tree array properties are used to describe
 the name/index/power-domain/type of each DRC allocated to a guest at
 boot time. There may be multiple sets of these arrays, rooted at different
 paths in the device tree depending on the type of resource the DRCs manage.
 
 In some cases, the DRCs themselves may be provided by a dynamic resource,
 such as the DRCs managing PCI slots on a hot plugged PHB. In this case the
 arrays would be fetched as part of the device tree retrieval interfaces
 for hot plugged resources described under :ref:`guest-host-interface`.
 
 The array properties are described below. Each entry/element in an array
 describes the DRC identified by the element in the corresponding position
 of ``ibm,drc-indexes``:
 
 ``ibm,drc-names``
 -----------------
 
   First 4-bytes: big-endian (BE) encoded integer denoting the number of entries.
 
   Each entry: a NULL-terminated ``<name>`` string encoded as a byte array.
 
     ``<name>`` values for logical/virtual resources are defined in the Linux on
     Power Architecture Reference ([LoPAR]_) section 13.5.2.4, and basically
     consist of the type of the resource followed by a space and a numerical
     value that's unique across resources of that type.
 
     ``<name>`` values for "physical" resources such as PCI or VIO devices are
     defined as being "location codes", which are the "location labels" of each
     encapsulating device, starting from the chassis down to the individual slot
     for the device, concatenated by a hyphen. This provides a mapping of
     resources to a physical location in a chassis for debugging purposes. For
     QEMU, this mapping is less important, so we assign a location code that
     conforms to naming specifications, but is simply a location label for the
     slot by itself to simplify the implementation. The naming convention for
     location labels is documented in detail in the [LoPAR]_ section 12.3.1.5,
     and in our case amounts to using ``C<n>`` for PCI/VIO device slots, where
     ``<n>`` is unique across all PCI/VIO device slots.
 
 ``ibm,drc-indexes``
 -------------------
 
   First 4-bytes: BE-encoded integer denoting the number of entries.
 
   Each 4-byte entry: BE-encoded ``<index>`` integer that is unique across all
   DRCs in the machine.
 
     ``<index>`` is arbitrary, but in the case of QEMU we try to maintain the
     convention used to assign them to pSeries guests on pHyp (the hypervisor
     portion of PowerVM):
 
       ``bit[31:28]``: integer encoding of ``<type>``, where ``<type>`` is:
 
         ``1`` for CPU resource.
 
         ``2`` for PHB resource.
 
         ``3`` for VIO resource.
 
         ``4`` for PCI resource.
 
         ``8`` for memory resource.
 
       ``bit[27:0]``: integer encoding of ``<id>``, where ``<id>`` is unique
       across all resources of specified type.
 
 ``ibm,drc-power-domains``
 -------------------------
 
   First 4-bytes: BE-encoded integer denoting the number of entries.
 
   Each 4-byte entry: 32-bit, BE-encoded ``<index>`` integer that specifies the
   power domain the resource will be assigned to. In the case of QEMU we
   associated all resources with a "live insertion" domain, where the power is
   assumed to be managed automatically. The integer value for this domain is a
   special value of ``-1``.
 
 
 ``ibm,drc-types``
 -----------------
 
   First 4-bytes: BE-encoded integer denoting the number of entries.
 
   Each entry: a NULL-terminated ``<type>`` string encoded as a byte array.
   ``<type>`` is assigned as follows:
 
     "CPU" for a CPU.
 
     "PHB" for a physical host-bridge.
 
     "SLOT" for a VIO slot.
 
     "28" for a PCI slot.
 
     "MEM" for memory resource.
 
 .. _guest-host-interface:
 
 Guest->Host interface to manage dynamic resources
 =================================================
 
 Each DRC is given a globally unique DRC index, and resources associated with a
 particular DRC are configured/managed by the guest via a number of RTAS calls
 which reference individual DRCs based on the DRC index. This can be considered
 the guest->host interface.
 
 ``rtas-set-power-level``
 ------------------------
 
 Set the power level for a specified power domain.
 
   ``arg[0]``: integer identifying power domain.
 
   ``arg[1]``: new power level for the domain, ``0-100``.
 
   ``output[0]``: status, ``0`` on success.
 
   ``output[1]``: power level after command.
 
 ``rtas-get-power-level``
 ------------------------
 
 Get the power level for a specified power domain.
 
   ``arg[0]``: integer identifying power domain.
 
   ``output[0]``: status, ``0`` on success.
 
   ``output[1]``: current power level.
 
 ``rtas-set-indicator``
 ----------------------
 
 Set the state of an indicator or sensor.
 
   ``arg[0]``: integer identifying sensor/indicator type.
 
   ``arg[1]``: index of sensor, for DR-related sensors this is generally the DRC
   index.
 
   ``arg[2]``: desired sensor value.
 
   ``output[0]``: status, ``0`` on success.
 
 For the purpose of this document we focus on the indicator/sensor types
 associated with a DRC. The types are:
 
 * ``9001``: ``isolation-state``, controls/indicates whether a device has been
   made accessible to a guest. Supported sensor values:
 
     ``0``: ``isolate``, device is made inaccessible by guest OS.
 
     ``1``: ``unisolate``, device is made available to guest OS.
 
 * ``9002``: ``dr-indicator``, controls "visual" indicator associated with
   device. Supported sensor values:
 
     ``0``: ``inactive``, resource may be safely removed.
 
     ``1``: ``active``, resource is in use and cannot be safely removed.
 
     ``2``: ``identify``, used to visually identify slot for interactive hot plug.
 
     ``3``: ``action``, in most cases, used in the same manner as identify.
 
 * ``9003``: ``allocation-state``, generally only used for "logical" DR resources
   to request the allocation/deallocation of a resource prior to acquiring it via
   ``isolation-state->unisolate``, or after releasing it via
   ``isolation-state->isolate``, respectively. For "physical" DR (like PCI
   hot plug/unplug) the pre-allocation of the resource is implied and this sensor
   is unused. Supported sensor values:
 
     ``0``: ``unusable``, tell firmware/system the resource can be
     unallocated/reclaimed and added back to the system resource pool.
 
     ``1``: ``usable``, request the resource be allocated/reserved for use by
     guest OS.
 
     ``2``: ``exchange``, used to allocate a spare resource to use for fail-over
     in certain situations. Unused in QEMU.
 
     ``3``: ``recover``, used to reclaim a previously allocated resource that's
     not currently allocated to the guest OS. Unused in QEMU.
 
 ``rtas-get-sensor-state:``
 --------------------------
 
 Used to read an indicator or sensor value.
 
   ``arg[0]``: integer identifying sensor/indicator type.
 
   ``arg[1]``: index of sensor, for DR-related sensors this is generally the DRC
   index
 
   ``output[0]``: status, 0 on success
 
 For DR-related operations, the only noteworthy sensor is ``dr-entity-sense``,
 which has a type value of ``9003``, as ``allocation-state`` does in the case of
 ``rtas-set-indicator``. The semantics/encodings of the sensor values are
 distinct however.
 
 Supported sensor values for ``dr-entity-sense`` (``9003``) sensor:
 
   ``0``: empty.
 
     For physical resources: DRC/slot is empty.
 
     For logical resources: unused.
 
   ``1``: present.
 
     For physical resources: DRC/slot is populated with a device/resource.
 
     For logical resources: resource has been allocated to the DRC.
 
   ``2``: unusable.
 
     For physical resources: unused.
 
     For logical resources: DRC has no resource allocated to it.
 
   ``3``: exchange.
 
     For physical resources: unused.
 
     For logical resources: resource available for exchange (see
     ``allocation-state`` sensor semantics above).
 
   ``4``: recovery.
 
     For physical resources: unused.
 
     For logical resources: resource available for recovery (see
     ``allocation-state`` sensor semantics above).
 
 ``rtas-ibm-configure-connector``
 --------------------------------
 
 Used to fetch an OpenFirmware device tree description of the resource associated
 with a particular DRC.
 
   ``arg[0]``: guest physical address of 4096-byte work area buffer.
 
   ``arg[1]``: 0, or address of additional 4096-byte work area buffer; only
   non-zero if a prior RTAS response indicated a need for additional memory.
 
   ``output[0]``: status:
 
     ``0``: completed transmittal of device tree node.
 
     ``1``: instruct guest to prepare for next device tree sibling node.
 
     ``2``: instruct guest to prepare for next device tree child node.
 
     ``3``: instruct guest to prepare for next device tree property.
 
     ``4``: instruct guest to ascend to parent device tree node.
 
     ``5``: instruct guest to provide additional work-area buffer via ``arg[1]``.
 
     ``990x``: instruct guest that operation took too long and to try again
     later.
 
 The DRC index is encoded in the first 4-bytes of the first work area buffer.
 Work area (``wa``) layout, using 4-byte offsets:
 
   ``wa[0]``: DRC index of the DRC to fetch device tree nodes from.
 
   ``wa[1]``: ``0`` (hard-coded).
 
   ``wa[2]``:
 
     For next-sibling/next-child response:
 
       ``wa`` offset of null-terminated string denoting the new node's name.
 
     For next-property response:
 
       ``wa`` offset of null-terminated string denoting new property's name.
 
   ``wa[3]``: for next-property response (unused otherwise):
 
       Byte-length of new property's value.
 
   ``wa[4]``: for next-property response (unused otherwise):
 
       New property's value, encoded as an OFDT-compatible byte array.
 
 Hot plug/unplug events
 ======================
 
 For most DR operations, the hypervisor will issue host->guest add/remove events
 using the EPOW/check-exception notification framework, where the host issues a
 check-exception interrupt, then provides an RTAS event log via an
 rtas-check-exception call issued by the guest in response. This framework is
 documented by PAPR+ v2.7, and already use in by QEMU for generating powerdown
 requests via EPOW events.
 
 For DR, this framework has been extended to include hotplug events, which were
 previously unneeded due to direct manipulation of DR-related guest userspace
 tools by host-level management such as an HMC. This level of management is not
 applicable to KVM on Power, hence the reason for extending the notification
 framework to support hotplug events.
 
 The format for these EPOW-signalled events is described below under
 :ref:`hot-plug-unplug-event-structure`. Note that these events are not formally
 part of the PAPR+ specification, and have been superseded by a newer format,
 also described below under :ref:`hot-plug-unplug-event-structure`, and so are
 now deemed a "legacy" format. The formats are similar, but the "modern" format
 contains additional fields/flags, which are denoted for the purposes of this
 documentation with ``#ifdef GUEST_SUPPORTS_MODERN`` guards.
 
 QEMU should assume support only for "legacy" fields/flags unless the guest
 advertises support for the "modern" format via
 ``ibm,client-architecture-support`` hcall by setting byte 5, bit 6 of it's
 ``ibm,architecture-vec-5`` option vector structure (as described by [LoPAR]_,
 section B.5.2.3). As with "legacy" format events, "modern" format events are
 surfaced to the guest via check-exception RTAS calls, but use a dedicated event
 source to signal the guest. This event source is advertised to the guest by the
 addition of a ``hot-plug-events`` node under ``/event-sources`` node of the
 guest's device tree using the standard format described in [LoPAR]_,
 section B.5.12.2.
 
 .. _hot-plug-unplug-event-structure:
 
 Hot plug/unplug event structure
 ===============================
 
 The hot plug specific payload in QEMU is implemented as follows (with all values
 encoded in big-endian format):
 
 .. code-block:: c
 
    struct rtas_event_log_v6_hp {
    #define SECTION_ID_HOTPLUG              0x4850 /* HP */
        struct section_header {
            uint16_t section_id;            /* set to SECTION_ID_HOTPLUG */
            uint16_t section_length;        /* sizeof(rtas_event_log_v6_hp),
                                             * plus the length of the DRC name
                                             * if a DRC name identifier is
                                             * specified for hotplug_identifier
                                             */
            uint8_t section_version;        /* version 1 */
            uint8_t section_subtype;        /* unused */
            uint16_t creator_component_id;  /* unused */
        } hdr;
    #define RTAS_LOG_V6_HP_TYPE_CPU         1
    #define RTAS_LOG_V6_HP_TYPE_MEMORY      2
    #define RTAS_LOG_V6_HP_TYPE_SLOT        3
    #define RTAS_LOG_V6_HP_TYPE_PHB         4
    #define RTAS_LOG_V6_HP_TYPE_PCI         5
        uint8_t hotplug_type;               /* type of resource/device */
    #define RTAS_LOG_V6_HP_ACTION_ADD       1
    #define RTAS_LOG_V6_HP_ACTION_REMOVE    2
        uint8_t hotplug_action;             /* action (add/remove) */
    #define RTAS_LOG_V6_HP_ID_DRC_NAME          1
    #define RTAS_LOG_V6_HP_ID_DRC_INDEX         2
    #define RTAS_LOG_V6_HP_ID_DRC_COUNT         3
    #ifdef GUEST_SUPPORTS_MODERN
    #define RTAS_LOG_V6_HP_ID_DRC_COUNT_INDEXED 4
    #endif
        uint8_t hotplug_identifier;         /* type of the resource identifier,
                                             * which serves as the discriminator
                                             * for the 'drc' union field below
                                             */
    #ifdef GUEST_SUPPORTS_MODERN
        uint8_t capabilities;               /* capability flags, currently unused
                                             * by QEMU
                                             */
    #else
        uint8_t reserved;
    #endif
        union {
            uint32_t index;                 /* DRC index of resource to take action
                                             * on
                                             */
            uint32_t count;                 /* number of DR resources to take
                                             * action on (guest chooses which)
                                             */
    #ifdef GUEST_SUPPORTS_MODERN
            struct {
                uint32_t count;             /* number of DR resources to take
                                             * action on
                                             */
                uint32_t index;             /* DRC index of first resource to take
                                             * action on. guest will take action
                                             * on DRC index <index> through
                                             * DRC index <index + count - 1> in
                                             * sequential order
                                             */
            } count_indexed;
    #endif
            char name[1];                   /* string representing the name of the
                                             * DRC to take action on
                                             */
        } drc;
    } QEMU_PACKED;
 
 ``ibm,lrdr-capacity``
 =====================
 
 ``ibm,lrdr-capacity`` is a property in the /rtas device tree node that
 identifies the dynamic reconfiguration capabilities of the guest. It consists
 of a triple consisting of ``<phys>``, ``<size>`` and ``<maxcpus>``.
 
   ``<phys>``, encoded in BE format represents the maximum address in bytes and
   hence the maximum memory that can be allocated to the guest.
 
   ``<size>``, encoded in BE format represents the size increments in which
   memory can be hot-plugged to the guest.
 
   ``<maxcpus>``, a BE-encoded integer, represents the maximum number of
   processors that the guest can have.
 
 ``pseries`` guests use this property to note the maximum allowed CPUs for the
 guest.
 
 ``ibm,dynamic-reconfiguration-memory``
 ======================================
 
 ``ibm,dynamic-reconfiguration-memory`` is a device tree node that represents
 dynamically reconfigurable logical memory blocks (LMB). This node is generated
 only when the guest advertises the support for it via
 ``ibm,client-architecture-support`` call. Memory that is not dynamically
 reconfigurable is represented by ``/memory`` nodes. The properties of this node
 that are of interest to the sPAPR memory hotplug implementation in QEMU are
 described here.
 
 ``ibm,lmb-size``
 ----------------
 
 This 64-bit integer defines the size of each dynamically reconfigurable LMB.
 
 ``ibm,associativity-lookup-arrays``
 -----------------------------------
 
 This property defines a lookup array in which the NUMA associativity
 information for each LMB can be found. It is a property encoded array
 that begins with an integer M, the number of associativity lists followed
 by an integer N, the number of entries per associativity list and terminated
 by M associativity lists each of length N integers.
 
 This property provides the same information as given by ``ibm,associativity``
 property in a ``/memory`` node. Each assigned LMB has an index value between
 0 and M-1 which is used as an index into this table to select which
 associativity list to use for the LMB. This index value for each LMB is defined
 in ``ibm,dynamic-memory`` property.
 
 ``ibm,dynamic-memory``
 ----------------------
 
 This property describes the dynamically reconfigurable memory. It is a
 property encoded array that has an integer N, the number of LMBs followed
 by N LMB list entries.
 
 Each LMB list entry consists of the following elements:
 
 - Logical address of the start of the LMB encoded as a 64-bit integer. This
   corresponds to ``reg`` property in ``/memory`` node.
 - DRC index of the LMB that corresponds to ``ibm,my-drc-index`` property
   in a ``/memory`` node.
 - Four bytes reserved for expansion.
 - Associativity list index for the LMB that is used as an index into
   ``ibm,associativity-lookup-arrays`` property described earlier. This is used
   to retrieve the right associativity list to be used for this LMB.
 - A 32-bit flags word. The bit at bit position ``0x00000008`` defines whether
   the LMB is assigned to the partition as of boot time.
 
 ``ibm,dynamic-memory-v2``
 -------------------------
 
 This property describes the dynamically reconfigurable memory. This is
 an alternate and newer way to describe dynamically reconfigurable memory.
 It is a property encoded array that has an integer N (the number of
 LMB set entries) followed by N LMB set entries. There is an LMB set entry
 for each sequential group of LMBs that share common attributes.
 
 Each LMB set entry consists of the following elements:
 
 - Number of sequential LMBs in the entry represented by a 32-bit integer.
 - Logical address of the first LMB in the set encoded as a 64-bit integer.
 - DRC index of the first LMB in the set.
 - Associativity list index that is used as an index into
   ``ibm,associativity-lookup-arrays`` property described earlier. This
   is used to retrieve the right associativity list to be used for all
   the LMBs in this set.
 - A 32-bit flags word that applies to all the LMBs in the set.