File ucp_def.h

Defines

UCP_ENTITY_NAME_MAX

Maximum size of the UCP entity name in structure of entity attributes provided by a query method.

Typedefs

typedef struct ucp_tag_recv_info ucp_tag_recv_info_t

UCP receive information descriptor.

The UCP receive information descriptor is allocated by application and filled in with the information about the received message by ucp_tag_probe_nb or ucp_tag_recv_request_test routines or ucp_tag_recv_callback_t callback argument.

typedef struct ucp_am_recv_param ucp_am_recv_param_t

Operation parameters provided in ucp_am_recv_callback_t callback.

typedef struct ucp_context *ucp_context_h

UCP Application Context.

UCP application context (or just a context) is an opaque handle that holds a UCP communication instance’s global information. It represents a single UCP communication instance. The communication instance could be an OS process (an application) that uses UCP library. This global information includes communication resources, endpoints, memory, temporary file storage, and other communication information directly associated with a specific UCP instance. The context also acts as an isolation mechanism, allowing resources associated with the context to manage multiple concurrent communication instances. For example, users using both MPI and OpenSHMEM sessions simultaneously can isolate their communication by allocating and using separate contexts for each of them. Alternatively, users can share the communication resources (memory, network resource context, etc.) between them by using the same application context. A message sent or a RMA operation performed in one application context cannot be received in any other application context.

typedef struct ucp_config ucp_config_t

UCP configuration descriptor.

This descriptor defines the configuration for UCP application context. The configuration is loaded from the run-time environment (using configuration files of environment variables) using ucp_config_read routine and can be printed using ucp_config_print routine. In addition, application is responsible to release the descriptor using ucp_config_release routine.

typedef struct ucp_ep *ucp_ep_h

UCP Endpoint.

The endpoint handle is an opaque object that is used to address a remote worker. It typically provides a description of source, destination, or both. All UCP communication routines address a destination with the endpoint handle. The endpoint handle is associated with only one UCP context. UCP provides the endpoint create routine to create the endpoint handle and the destroy routine to destroy the endpoint handle.

typedef struct ucp_conn_request *ucp_conn_request_h

UCP connection request.

A server-side handle to incoming connection request. Can be used to create an endpoint which connects back to the client.

typedef struct ucp_address ucp_address_t

UCP worker address.

The address handle is an opaque object that is used as an identifier for a worker instance.

typedef struct ucp_rkey *ucp_rkey_h

UCP Remote memory handle.

Remote memory handle is an opaque object representing remote memory access information. Typically, the handle includes a memory access key and other network hardware specific information, which are input to remote memory access operations, such as PUT, GET, and ATOMIC. The object is communicated to remote peers to enable an access to the memory region.

typedef struct ucp_mem *ucp_mem_h

UCP Memory handle.

Memory handle is an opaque object representing a memory region allocated through UCP library, which is optimized for remote memory access operations (zero-copy operations). The memory handle is a self-contained object, which includes the information required to access the memory region locally, while remote key is used to access it remotely. The memory could be registered to one or multiple network resources that are supported by UCP, such as InfiniBand, Gemini, and others.

typedef struct ucp_listener *ucp_listener_h

UCP listen handle.

The listener handle is an opaque object that is used for listening on a specific address and accepting connections from clients.

typedef struct ucp_mem_attr ucp_mem_attr_t

Attributes of the UCP Memory handle, filled by ucp_mem_query function.

typedef struct ucp_worker *ucp_worker_h

UCP Worker.

UCP worker is an opaque object representing the communication context. The worker represents an instance of a local communication resource and the progress engine associated with it. The progress engine is a construct that is responsible for asynchronous and independent progress of communication directives. The progress engine could be implemented in hardware or software. The worker object abstracts an instance of network resources such as a host channel adapter port, network interface, or multiple resources such as multiple network interfaces or communication ports. It could also represent virtual communication resources that are defined across multiple devices. Although the worker can represent multiple network resources, it is associated with a single UCX application context. All communication functions require a context to perform the operation on the dedicated hardware resource(s) and an endpoint to address the destination.

Note

Worker are parallel “threading points” that an upper layer may use to optimize concurrent communications.

typedef uint64_t ucp_tag_t

UCP Tag Identifier.

UCP tag identifier is a 64bit object used for message identification. UCP tag send and receive operations use the object for an implementation tag matching semantics (derivative of MPI tag matching semantics).

typedef struct ucp_recv_desc *ucp_tag_message_h

UCP Message descriptor.

UCP Message descriptor is an opaque handle for a message returned by ucp_tag_probe_nb. This handle can be passed to ucp_tag_msg_recv_nb in order to receive the message data to a specific buffer.

typedef uint64_t ucp_datatype_t

UCP Datatype Identifier.

UCP datatype identifier is a 64bit object used for datatype identification. Predefined UCP identifiers are defined by ucp_dt_type.

typedef void (*ucp_request_init_callback_t)(void *request)

Request initialization callback.

This callback routine is responsible for the request initialization.

Parameters

request – [in] Request handle to initialize.

typedef void (*ucp_request_cleanup_callback_t)(void *request)

Request cleanup callback.

This callback routine is responsible for cleanup of the memory associated with the request.

Parameters

request – [in] Request handle to cleanup.

typedef void (*ucp_send_callback_t)(void *request, ucs_status_t status)

Completion callback for non-blocking sends.

This callback routine is invoked whenever the send operation is completed. It is important to note that the call-back is only invoked in a case when the operation cannot be completed in place.

Parameters

  • request – [in] The completed send request.

  • status – [in] Completion status. If the send operation was completed successfully UCS_OK is returned. If send operation was canceled UCS_ERR_CANCELED is returned. Otherwise, an error status is returned.

typedef void (*ucp_send_nbx_callback_t)(void *request, ucs_status_t status, void *user_data)

Completion callback for non-blocking sends.

This callback routine is invoked whenever the ucp_tag_send_nbx, ucp_am_send_nbx, ucp_stream_send_nbx, ucp_put_nbx, ucp_get_nbx, ucp_atomic_op_nbx or any other “send operation” is completed.

Parameters

  • request – [in] The completed send request.

  • status – [in] Completion status. If the send operation was completed successfully UCS_OK is returned. If send operation was canceled UCS_ERR_CANCELED is returned. Otherwise, an error status is returned.

  • user_data – [in] User data passed to “user_data” value, see ucp_request_param_t

typedef void (*ucp_err_handler_cb_t)(void *arg, ucp_ep_h ep, ucs_status_t status)

Callback to process peer failure.

This callback routine is invoked when transport level error detected.

Parameters

  • arg – [in] User argument to be passed to the callback.

  • ep – [in] Endpoint to handle transport level error. Upon return from the callback, this ep is no longer usable and all subsequent operations on this ep will fail with the error code passed in status.

  • status – [in] error status.

typedef struct ucp_err_handler ucp_err_handler_t

UCP endpoint error handling context.

This structure should be initialized in ucp_ep_params_t to handle peer failure

typedef void (*ucp_listener_accept_callback_t)(ucp_ep_h ep, void *arg)

A callback for accepting client/server connections on a listener ucp_listener_h.

This callback routine is invoked on the server side upon creating a connection to a remote client. The user can pass an argument to this callback. The user is responsible for releasing the ep handle using the ucp_ep_destroy() routine.

Parameters

  • ep – [in] Handle to a newly created endpoint which is connected to the remote peer which has initiated the connection.

  • arg – [in] User’s argument for the callback.

typedef void (*ucp_listener_conn_callback_t)(ucp_conn_request_h conn_request, void *arg)

A callback for handling of incoming connection request conn_request from a client.

This callback routine is invoked on the server side to handle incoming connections from remote clients. The user can pass an argument to this callback. The conn_request handle has to be released, either by ucp_ep_create or ucp_listener_reject routine.

Parameters

  • conn_request – [in] Connection request handle.

  • arg – [in] User’s argument for the callback.

typedef struct ucp_listener_conn_handler ucp_listener_conn_handler_t

UCP callback to handle the connection request in a client-server connection establishment flow.

This structure is used for handling an incoming connection request on the listener. Setting this type of handler allows creating an endpoint on any other worker and not limited to the worker on which the listener was created.

Note

  • Other than communication progress routines, it is allowed to call all other communication routines from the callback in the struct.

  • The callback is thread safe with respect to the worker it is invoked on.

  • It is the user’s responsibility to avoid potential dead lock accessing different worker.

typedef void (*ucp_stream_recv_callback_t)(void *request, ucs_status_t status, size_t length)

Completion callback for non-blocking stream oriented receives.

This callback routine is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Parameters

  • request – [in] The completed receive request.

  • status – [in] Completion status. If the send operation was completed successfully UCS_OK is returned. Otherwise, an error status is returned.

  • length – [in] The size of the received data in bytes, always boundary of base datatype size. The value is valid only if the status is UCS_OK.

typedef void (*ucp_stream_recv_nbx_callback_t)(void *request, ucs_status_t status, size_t length, void *user_data)

Completion callback for non-blocking stream receives ucp_stream_recv_nbx call.

This callback routine is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Parameters

  • request – [in] The completed receive request.

  • status – [in] Completion status. If the send operation was completed successfully UCS_OK is returned. Otherwise, an error status is returned.

  • length – [in] The size of the received data in bytes, always on the boundary of base datatype size. The value is valid only if the status is UCS_OK.

  • user_data – [in] User data passed to “user_data” value, see ucp_request_param_t.

typedef void (*ucp_tag_recv_callback_t)(void *request, ucs_status_t status, ucp_tag_recv_info_t *info)

Completion callback for non-blocking tag receives.

This callback routine is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Parameters

  • request – [in] The completed receive request.

  • status – [in] Completion status. If the send operation was completed successfully UCS_OK is returned. If send operation was canceled UCS_ERR_CANCELED is returned. If the data can not fit into the receive buffer the UCS_ERR_MESSAGE_TRUNCATED error code is returned. Otherwise, an error status is returned.

  • info – [in] Completion information The info descriptor is Valid only if the status is UCS_OK.

typedef void (*ucp_tag_recv_nbx_callback_t)(void *request, ucs_status_t status, const ucp_tag_recv_info_t *tag_info, void *user_data)

Completion callback for non-blocking tag receives ucp_tag_recv_nbx call.

This callback routine is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Parameters

  • request – [in] The completed receive request.

  • status – [in] Completion status. If the receive operation was completed successfully UCS_OK is returned. If send operation was canceled, UCS_ERR_CANCELED is returned. If the data can not fit into the receive buffer the UCS_ERR_MESSAGE_TRUNCATED error code is returned. Otherwise, an error status is returned.

  • tag_info – [in] Completion information The info descriptor is Valid only if the status is UCS_OK.

  • user_data – [in] User data passed to “user_data” value, see ucp_request_param_t

typedef void (*ucp_am_recv_data_nbx_callback_t)(void *request, ucs_status_t status, size_t length, void *user_data)

Completion callback for non-blocking Active Message receives.

This callback routine is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Parameters

  • request – [in] The completed receive request.

  • status – [in] Completion status. If the receive operation was completed successfully UCS_OK is returned. Otherwise, an error status is returned.

  • length – [in] The size of the received data in bytes, always boundary of base datatype size. The value is valid only if the status is UCS_OK.

  • user_data – [in] User data passed to “user_data” value, see ucp_request_param_t

typedef enum ucp_wakeup_event_types ucp_wakeup_event_t

UCP worker wakeup events mask.

The enumeration allows specifying which events are expected on wakeup. Empty events are possible for any type of event except for UCP_WAKEUP_TX and UCP_WAKEUP_RX.

Note

Send completions are reported by POLLIN-like events (see poll man page). Since outgoing operations can be initiated at any time, UCP does not generate POLLOUT-like events, although it must be noted that outgoing operations may be queued depending upon resource availability.

typedef ucs_status_t (*ucp_am_callback_t)(void *arg, void *data, size_t length, ucp_ep_h reply_ep, unsigned flags)

Callback to process incoming Active Message.

When the callback is called, flags indicates how data should be handled.

Parameters

  • arg – [in] User-defined argument.

  • data – [in] Points to the received data. This data may persist after the callback returns and needs to be freed with ucp_am_data_release.

  • length – [in] Length of data.

  • reply_ep – [in] If the Active Message is sent with the UCP_AM_SEND_FLAG_REPLY flag, the sending ep will be passed in. If not, NULL will be passed.

  • flags – [in] If this flag is set to UCP_CB_PARAM_FLAG_DATA, the callback can return UCS_INPROGRESS and data will persist after the callback returns.

Returns

UCS_OK data will not persist after the callback returns.

Returns

UCS_INPROGRESS Can only be returned if flags is set to UCP_CB_PARAM_FLAG_DATA. If UCP_INPROGRESS is returned, data will persist after the callback has returned. To free the memory, a pointer to the data must be passed into ucp_am_data_release.

Note

This callback should be set and released by ucp_worker_set_am_handler function.

typedef ucs_status_t (*ucp_am_recv_callback_t)(void *arg, const void *header, size_t header_length, void *data, size_t length, const ucp_am_recv_param_t *param)

Callback to process incoming Active Message sent by ucp_am_send_nbx routine.

The callback is always called from the progress context, therefore calling ucp_worker_progress() is not allowed. It is recommended to define callbacks with relatively short execution time to avoid blocking of communication progress.

Parameters

  • arg – [in] User-defined argument.

  • header – [in] User defined active message header. If header_length is 0, this value is undefined and must not be accessed.

  • header_length – [in] Active message header length in bytes.

  • data – [in] Points to the received data if UCP_AM_RECV_ATTR_FLAG_RNDV flag is not set in ucp_am_recv_param_t::recv_attr. Otherwise it points to the internal UCP descriptor which can further be used for initiating data receive by using ucp_am_recv_data_nbx routine.

  • length – [in] Length of data. If UCP_AM_RECV_ATTR_FLAG_RNDV flag is set in ucp_am_recv_param_t::recv_attr, it indicates the required receive buffer size for initiating rendezvous protocol.

  • param – [in] Data receive parameters.

Returns

UCS_OK data will not persist after the callback returns. If UCP_AM_RECV_ATTR_FLAG_RNDV flag is set in param->recv_attr and ucp_am_recv_data_nbx was not called for this data, the data descriptor will be dropped and the corresponding ucp_am_send_nbx call will complete with UCS_OK status.

Returns

UCS_INPROGRESS Can only be returned if param->recv_attr flags contains UCP_AM_RECV_ATTR_FLAG_DATA or UCP_AM_RECV_ATTR_FLAG_RNDV. The data will persist after the callback has returned. To free the memory, a pointer to the data must be passed into ucp_am_data_release or data receive is initiated by ucp_am_recv_data_nbx.

Returns

otherwise Can only be returned if param->recv_attr contains UCP_AM_RECV_ATTR_FLAG_RNDV. In this case data descriptor data will be dropped and the corresponding ucp_am_send_nbx call on the sender side will complete with the status returned from the callback.

Note

This callback should be set and released by ucp_worker_set_am_recv_handler function.

typedef struct ucp_ep_params ucp_ep_params_t

Tuning parameters for the UCP endpoint.

The structure defines the parameters that are used for the UCP endpoint tuning during the UCP ep creation.

Enums

enum ucp_err_handling_mode_t

Error handling mode for the UCP endpoint.

Specifies error handling mode for the UCP endpoint.

Values:

enumerator UCP_ERR_HANDLING_MODE_NONE

No guarantees about error reporting, imposes minimal overhead from a performance perspective.

Note

In this mode, any error reporting will not generate calls to ucp_ep_params_t::err_handler.

enumerator UCP_ERR_HANDLING_MODE_PEER

Guarantees that send requests are always completed (successfully or error) even in case of remote failure, disables protocols and APIs which may cause a hang or undefined behavior in case of peer failure, may affect performance and memory footprint

enum ucp_mem_attr_field

UCP Memory handle attributes field mask.

The enumeration allows specifying which fields in ucp_mem_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_MEM_ATTR_FIELD_ADDRESS

Virtual address

enumerator UCP_MEM_ATTR_FIELD_LENGTH

The size of memory region

enumerator UCP_MEM_ATTR_FIELD_MEM_TYPE

Type of allocated or registered memory

enum ucp_wakeup_event_types

UCP worker wakeup events mask.

The enumeration allows specifying which events are expected on wakeup. Empty events are possible for any type of event except for UCP_WAKEUP_TX and UCP_WAKEUP_RX.

Note

Send completions are reported by POLLIN-like events (see poll man page). Since outgoing operations can be initiated at any time, UCP does not generate POLLOUT-like events, although it must be noted that outgoing operations may be queued depending upon resource availability.

Values:

enumerator UCP_WAKEUP_RMA

Remote memory access send completion

enumerator UCP_WAKEUP_AMO

Atomic operation send completion

enumerator UCP_WAKEUP_TAG_SEND

Tag send completion

enumerator UCP_WAKEUP_TAG_RECV

Tag receive completion

enumerator UCP_WAKEUP_TX

This event type will generate an event on completion of any outgoing operation (complete or partial, according to the underlying protocol) for any type of transfer (send, atomic, or RMA).

enumerator UCP_WAKEUP_RX

This event type will generate an event on completion of any receive operation (complete or partial, according to the underlying protocol).

enumerator UCP_WAKEUP_EDGE

Use edge-triggered wakeup. The event file descriptor will be signaled only for new events, rather than existing ones.

struct ucp_mem_attr
#include <ucp_def.h>

Attributes of the UCP Memory handle, filled by ucp_mem_query function.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_mem_attr_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

void *address

Address of the memory segment.

size_t length

Size of the memory segment.

ucs_memory_type_t mem_type

Type of allocated or registered memory

struct ucp_err_handler
#include <ucp_def.h>

UCP endpoint error handling context.

This structure should be initialized in ucp_ep_params_t to handle peer failure

Public Members

ucp_err_handler_cb_t cb

Error handler callback, if NULL, will not be called.

void *arg

User defined argument associated with an endpoint, it will be overridden by ucp_ep_params_t::user_data if both are set.

struct ucp_listener_conn_handler
#include <ucp_def.h>

UCP callback to handle the connection request in a client-server connection establishment flow.

This structure is used for handling an incoming connection request on the listener. Setting this type of handler allows creating an endpoint on any other worker and not limited to the worker on which the listener was created.

Note

  • Other than communication progress routines, it is allowed to call all other communication routines from the callback in the struct.

  • The callback is thread safe with respect to the worker it is invoked on.

  • It is the user’s responsibility to avoid potential dead lock accessing different worker.

Public Members

ucp_listener_conn_callback_t cb

Connection request callback

void *arg

User defined argument for the callback

struct ucp_ep_params
#include <ucp_def.h>

Tuning parameters for the UCP endpoint.

The structure defines the parameters that are used for the UCP endpoint tuning during the UCP ep creation.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_ep_params_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

const ucp_address_t *address

Destination address; this field should be set along with its corresponding bit in the field_mask - UCP_EP_PARAM_FIELD_REMOTE_ADDRESS and must be obtained using ucp_worker_get_address.

ucp_err_handling_mode_t err_mode

Desired error handling mode, optional parameter. Default value is UCP_ERR_HANDLING_MODE_NONE.

ucp_err_handler_t err_handler

Handler to process transport level failure.

void *user_data

User data associated with an endpoint. See ucp_stream_poll_ep_t and ucp_err_handler_t

unsigned flags

Endpoint flags from ucp_ep_params_flags_field. This value is optional. If it’s not set (along with its corresponding bit in the field_mask - UCP_EP_PARAM_FIELD_FLAGS), the ucp_ep_create() routine will consider the flags as set to zero.

ucs_sock_addr_t sockaddr

Destination address in the form of a sockaddr; this field should be set along with its corresponding bit in the field_mask - UCP_EP_PARAM_FIELD_SOCK_ADDR and must be obtained from the user, it means that this type of the endpoint creation is possible only on client side in client-server connection establishment flow.

ucp_conn_request_h conn_request

Connection request from client; this field should be set along with its corresponding bit in the field_mask - UCP_EP_PARAM_FIELD_CONN_REQUEST and must be obtained from ucp_listener_conn_callback_t, it means that this type of the endpoint creation is possible only on server side in client-server connection establishment flow.

const char *name

Endpoint name. Tracing and analysis tools can identify the endpoint using this name. To retrieve the endpoint’s name, use ucp_ep_query, as the name you supply may be changed by UCX under some circumstances, e.g. a name conflict. This field is only assigned if you set UCP_EP_PARAM_FIELD_NAME in the field mask. If not, then a default unique name will be created for you.

ucs_sock_addr_t local_sockaddr

The sockaddr to bind locally. Specifies the associated network device to bind locally to establish new connections. To retrieve the endpoint’s local_sockaddr, use ucp_ep_query. This setting is optional. To enable it, the corresponding - UCP_EP_PARAM_FIELD_LOCAL_SOCK_ADDR bit in the field mask must be set.

struct ucp_transport_entry_t
#include <ucp_def.h>

The ucp_transports_t and ucp_transport_entry_t structures are used when ucp_ep_query is called to return an array of transport name and device name pairs that are used by an active endpoint.

The ucp_transport_t structure specifies the characteristics of the ucp_transport_entry_t array.

The caller is responsible for the allocation and de-allocation of the ucp_transport_entry_t array.

Example: Implementation of a function to query the set of transport and device name pairs used by the specified endpoint.

int query_transports(ucp_ep_h ep)
{
  ucs_status_t status;
  ucp_transport_entry_t *transport_entries;
  ucp_ep_attr_t ep_attrs;

  ep_attrs.field_mask = UCP_EP_ATTR_FIELD_TRANSPORTS;
  ep_attrs.transports.entries = (ucp_transport_entry_t *)
           malloc(10 * sizeof(ucp_transport_entry_t));
  ep_attrs.transports.num_entries = 10;
  ep_attrs.transports.entry_size = sizeof(ucp_transport_entry_t);
  status = ucp_ep_query(ep, &ep_attrs);
  if (status == UCS_OK) {
      // ep_attrs.transports.num_entries = number of returned entries 
      // ... process transport info ... 
  }
}

A transport name and device name pair used by this endpoint. The caller is responsible for the allocation and deallocation of an array of these structures large enough to contain the desired number of transport and device name pairs.

Any new fields must be added to the end of this structure.

Public Members

const char *transport_name

The name of a transport layer used by this endpoint. This ‘\0’-terminated string is valid until the endpoint is closed using a ucp_ep_close_nbx call.

const char *device_name

The name of the device used with this transport by this endpoint. This ‘\0’-terminated string is valid until the endpoint is closed using a ucp_ep_close_nbx call.

struct ucp_transports_t
#include <ucp_def.h>

Structure containing an array of transport layers and device names used by an endpoint.

The caller is responsible for allocation and deallocation of this structure.

Public Members

ucp_transport_entry_t *entries

Pointer to array of transport/device name pairs used by this endpoint. The caller is responsible for the allocation and deallocation of this array.

unsigned num_entries

Number of transport/device name pairs. The caller must set this to the maximum number of pairs the structure can contain. On return, this is set to the actual number of transport and device name pairs used by the endpoint.

size_t entry_size

Size of a single ucp_transport_entry_t object. The caller sets this to the size of the ucp_transport_entry_t they are using. UCP code must not set any fields in the ucp_transport_entry_t structure beyond this size.