API Documentation¶

The full API specification can be found here : HTML PDF
Code examples

Application Context routines
Worker routines
Endpoint routines
Data type routines
Memory routines
Configuration routines
Communication routines
Wake-up routines
Resources
Index

Application Context routines ¶

group UCP_CONTEXT

Application context is a primary concept of UCP design which provides an isolation mechanism, allowing resources associated with the context to separate or share network communication context across multiple instances of applications.

This section provides a detailed description of this concept and routines associated with it.

Defines

UCP_ENTITY_NAME_MAX¶: Maximum size of the UCP entity name in structure of entity attributes provided by a query method.

Typedefs

typedef struct ucp_lib_attr ucp_lib_attr_t¶

Lib attributes.

The structure defines the attributes that characterize the Library.

typedef struct ucp_context_attr ucp_context_attr_t¶

Context attributes.

The structure defines the attributes that characterize the particular context.

typedef struct ucp_tag_recv_info ucp_tag_recv_info_t¶

UCP receive information descriptor.

The UCP receive information descriptor is allocated by application and filled in with the information about the received message by ucp_tag_probe_nb or ucp_tag_recv_request_test routines or ucp_tag_recv_callback_t callback argument.

typedef struct ucp_context *ucp_context_h¶

UCP Application Context.

UCP application context (or just a context) is an opaque handle that holds a UCP communication instance’s global information. It represents a single UCP communication instance. The communication instance could be an OS process (an application) that uses UCP library. This global information includes communication resources, endpoints, memory, temporary file storage, and other communication information directly associated with a specific UCP instance. The context also acts as an isolation mechanism, allowing resources associated with the context to manage multiple concurrent communication instances. For example, users using both MPI and OpenSHMEM sessions simultaneously can isolate their communication by allocating and using separate contexts for each of them. Alternatively, users can share the communication resources (memory, network resource context, etc.) between them by using the same application context. A message sent or a RMA operation performed in one application context cannot be received in any other application context.

typedef void (*ucp_request_init_callback_t)(void *request)¶

Request initialization callback.

This callback routine is responsible for the request initialization.

Parameters: request – [in] Request handle to initialize.

typedef void (*ucp_request_cleanup_callback_t)(void *request)¶

Request cleanup callback.

This callback routine is responsible for cleanup of the memory associated with the request.

Parameters: request – [in] Request handle to cleanup.

Enums

enum ucp_params_field¶

UCP context parameters field mask.

The enumeration allows specifying which fields in ucp_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_PARAM_FIELD_FEATURES¶: features

enumerator UCP_PARAM_FIELD_REQUEST_SIZE¶: request_size

enumerator UCP_PARAM_FIELD_REQUEST_INIT¶: request_init

enumerator UCP_PARAM_FIELD_REQUEST_CLEANUP¶: request_cleanup

enumerator UCP_PARAM_FIELD_TAG_SENDER_MASK¶: tag_sender_mask

enumerator UCP_PARAM_FIELD_MT_WORKERS_SHARED¶: mt_workers_shared

enumerator UCP_PARAM_FIELD_ESTIMATED_NUM_EPS¶: estimated_num_eps

enumerator UCP_PARAM_FIELD_ESTIMATED_NUM_PPN¶: estimated_num_ppn

enumerator UCP_PARAM_FIELD_NAME¶: name

enum ucp_feature¶

UCP configuration features.

The enumeration list describes the features supported by UCP. An application can request the features using UCP parameters during UCP initialization process.

Values:

enumerator UCP_FEATURE_TAG¶: Request tag matching support

enumerator UCP_FEATURE_RMA¶: Request remote memory access support

enumerator UCP_FEATURE_AMO32¶: Request 32-bit atomic operations support

enumerator UCP_FEATURE_AMO64¶: Request 64-bit atomic operations support

enumerator UCP_FEATURE_WAKEUP¶: Request interrupt notification support

enumerator UCP_FEATURE_STREAM¶: Request stream support

enumerator UCP_FEATURE_AM¶: Request Active Message support

enumerator UCP_FEATURE_EXPORTED_MEMH¶: Request support mapping a peer’s memory handle that was created by ucp_mem_map and packed by ucp_memh_pack with the flag UCP_MEMH_PACK_FLAG_EXPORT and use it for local operations

enum ucp_lib_attr_field¶

UCP library attributes field mask.

The enumeration allows specifying which fields in ucp_lib_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_LIB_ATTR_FIELD_MAX_THREAD_LEVEL¶: UCP library maximum supported thread level flag

enum ucp_context_attr_field¶

UCP context attributes field mask.

The enumeration allows specifying which fields in ucp_context_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_ATTR_FIELD_REQUEST_SIZE¶: UCP request size

enumerator UCP_ATTR_FIELD_THREAD_MODE¶: UCP context thread flag

enumerator UCP_ATTR_FIELD_MEMORY_TYPES¶: UCP supported memory types

enumerator UCP_ATTR_FIELD_NAME¶: UCP context name

Functions

ucs_status_t ucp_lib_query(ucp_lib_attr_t *attr)¶

Get attributes of the UCP library.

This routine fetches information about the UCP library attributes.

Parameters: attr – [out] Filled with attributes of the UCP library.
Returns: Error code as defined by ucs_status_t

void ucp_get_version(unsigned *major_version, unsigned *minor_version, unsigned *release_number)¶

Get UCP library version.

This routine returns the UCP library version.

Parameters

major_version – [out] Filled with library major version.
minor_version – [out] Filled with library minor version.
release_number – [out] Filled with library release number.

const char *ucp_get_version_string(void)¶

Get UCP library version as a string.

This routine returns the UCP library version as a string which consists of: “major.minor.release”.

static inline ucs_status_t ucp_init(const ucp_params_t *params, const ucp_config_t *config, ucp_context_h *context_p)¶

UCP context initialization.

This routine creates and initializes a UCP application context.

This routine checks API version compatibility, then discovers the available network interfaces, and initializes the network resources required for discovering of the network and memory related devices. This routine is responsible for initialization all information required for a particular application scope, for example, MPI application, OpenSHMEM application, etc.

Parameters

config – [in] UCP configuration descriptor allocated through ucp_config_read() routine.
params – [in] User defined ucp_params_t configurations for the UCP application context.
context_p – [out] Initialized UCP application context.

Returns

Error code as defined by ucs_status_t

Warning

This routine must be called before any other UCP function call in the application.

Note

Higher level protocols can add additional communication isolation, as MPI does with its communicator object. A single communication context may be used to support multiple MPI communicators.
The context can be used to isolate the communication that corresponds to different protocols. For example, if MPI and OpenSHMEM are using UCP to isolate the MPI communication from the OpenSHMEM communication, users should use different application context for each of the communication libraries.

void ucp_cleanup(ucp_context_h context_p)¶

Release UCP application context.

This routine finalizes and releases the resources associated with a UCP application context.

The cleanup process releases and shuts down all resources associated with the application context. After calling this routine, calling any UCP routine without calling UCP initialization routine is invalid.

Parameters: context_p – [in] Handle to UCP application context.

Warning

An application cannot call any UCP routine once the UCP application context released.

ucs_status_t ucp_context_query(ucp_context_h context_p, ucp_context_attr_t *attr)¶

Get attributes specific to a particular context.

This routine fetches information about the context.

Parameters

context_p – [in] Handle to UCP application context.
attr – [out] Filled with attributes of context_p context.

Returns

Error code as defined by ucs_status_t

void ucp_context_print_info(const ucp_context_h context, FILE *stream)¶

Print context information.

This routine prints information about the context configuration: including memory domains, transport resources, and other useful information associated with the context.

Parameters

context – [in] Print this context object’s configuration.
stream – [in] Output stream on which to print the information.

struct ucp_lib_attr¶

#include <ucp.h>

Lib attributes.

The structure defines the attributes that characterize the Library.

struct ucp_context_attr¶

#include <ucp.h>

Context attributes.

The structure defines the attributes that characterize the particular context.

struct ucp_tag_recv_info¶

#include <ucp.h>

UCP receive information descriptor.

The UCP receive information descriptor is allocated by application and filled in with the information about the received message by ucp_tag_probe_nb or ucp_tag_recv_request_test routines or ucp_tag_recv_callback_t callback argument.

struct ucp_request_param_t¶

#include <ucp.h>

Operation parameters passed to ucp_tag_send_nbx, ucp_tag_send_sync_nbx, ucp_tag_recv_nbx, ucp_put_nbx, ucp_get_nbx, ucp_am_send_nbx and ucp_am_recv_data_nbx.

The structure ucp_request_param_t is used to specify datatype of operation, provide user request in case the external request is used, set completion callback and custom user data passed to this callback.

Example: implementation of function to send contiguous buffer to ep and invoke callback function at operation completion. If the operation completed immediately (status == UCS_OK) then callback is not called.

ucs_status_ptr_t send_data(ucp_ep_h ep, void *buffer, size_t length,
                           ucp_tag_t tag, void *request)
{
    ucp_request_param_t param = {
        .op_attr_mask               = UCP_OP_ATTR_FIELD_CALLBACK |
                                      UCP_OP_ATTR_FIELD_REQUEST,
        .request                    = request,
        .cb.send                    = custom_send_callback_f,
        .user_data                  = pointer_to_user_context_passed_to_cb
    };

    ucs_status_ptr_t status;

    status = ucp_tag_send_nbx(ep, buffer, length, tag, &param);
    if (UCS_PTR_IS_ERR(status)) {
        handle_error(status);
    } else if (status == UCS_OK) {
        // operation is completed
    }

    return status;
}

ucp_request_param_t.cb

Callback function that is invoked whenever the send or receive operation is completed.

Public Members

ucp_send_nbx_callback_t send¶

ucp_tag_recv_nbx_callback_t recv¶

ucp_stream_recv_nbx_callback_t recv_stream¶

ucp_am_recv_data_nbx_callback_t recv_am¶

ucp_request_param_t.recv_info

Pointer to the information where received data details are stored in case of an immediate completion of receive operation. The user has to provide a pointer to valid memory/variable which will be updated on function return.

Public Members

size_t *length¶

ucp_tag_recv_info_t *tag_info¶

Worker routines ¶

group UCP_WORKER

UCP Worker routines

Typedefs

typedef struct ucp_worker_attr ucp_worker_attr_t¶

UCP worker attributes.

The structure defines the attributes which characterize the particular worker.

typedef struct ucp_worker_params ucp_worker_params_t¶

Tuning parameters for the UCP worker.

The structure defines the parameters that are used for the UCP worker tuning during the UCP worker creation.

typedef struct ucp_worker_address_attr ucp_worker_address_attr_t¶

UCP worker address attributes.

The structure defines the attributes of the particular worker address.

typedef struct ucp_listener_attr ucp_listener_attr_t¶

UCP listener attributes.

The structure defines the attributes which characterize the particular listener.

typedef struct ucp_conn_request_attr ucp_conn_request_attr_t¶

UCP listener’s connection request attributes.

The structure defines the attributes that characterize the particular connection request received on the server side.

typedef struct ucp_listener_params ucp_listener_params_t¶

Parameters for a UCP listener object.

This structure defines parameters for ucp_listener_create, which is used to listen for incoming client/server connections.

typedef struct ucp_am_handler_param ucp_am_handler_param_t¶: Active Message handler parameters passed to ucp_worker_set_am_recv_handler routine.

typedef struct ucp_listener_accept_handler ucp_listener_accept_handler_t¶

Deprecated:: Replaced by ucp_listener_conn_handler_t.

typedef struct ucp_am_recv_param ucp_am_recv_param_t¶: Operation parameters provided in ucp_am_recv_callback_t callback.

typedef struct ucp_address ucp_address_t¶

UCP worker address.

The address handle is an opaque object that is used as an identifier for a worker instance.

typedef struct ucp_listener *ucp_listener_h¶

UCP listen handle.

The listener handle is an opaque object that is used for listening on a specific address and accepting connections from clients.

typedef struct ucp_worker *ucp_worker_h¶

UCP Worker.

UCP worker is an opaque object representing the communication context. The worker represents an instance of a local communication resource and the progress engine associated with it. The progress engine is a construct that is responsible for asynchronous and independent progress of communication directives. The progress engine could be implemented in hardware or software. The worker object abstracts an instance of network resources such as a host channel adapter port, network interface, or multiple resources such as multiple network interfaces or communication ports. It could also represent virtual communication resources that are defined across multiple devices. Although the worker can represent multiple network resources, it is associated with a single UCX application context. All communication functions require a context to perform the operation on the dedicated hardware resource(s) and an endpoint to address the destination.

Note

Worker are parallel “threading points” that an upper layer may use to optimize concurrent communications.

typedef void (*ucp_listener_accept_callback_t)(ucp_ep_h ep, void *arg)¶

A callback for accepting client/server connections on a listener ucp_listener_h.

This callback routine is invoked on the server side upon creating a connection to a remote client. The user can pass an argument to this callback. The user is responsible for releasing the ep handle using the ucp_ep_destroy() routine.

Parameters

ep – [in] Handle to a newly created endpoint which is connected to the remote peer which has initiated the connection.
arg – [in] User’s argument for the callback.

typedef void (*ucp_listener_conn_callback_t)(ucp_conn_request_h conn_request, void *arg)¶

A callback for handling of incoming connection request conn_request from a client.

This callback routine is invoked on the server side to handle incoming connections from remote clients. The user can pass an argument to this callback. The conn_request handle has to be released, either by ucp_ep_create or ucp_listener_reject routine.

Parameters

conn_request – [in] Connection request handle.
arg – [in] User’s argument for the callback.

typedef struct ucp_listener_conn_handler ucp_listener_conn_handler_t¶

UCP callback to handle the connection request in a client-server connection establishment flow.

This structure is used for handling an incoming connection request on the listener. Setting this type of handler allows creating an endpoint on any other worker and not limited to the worker on which the listener was created.

Note

Other than communication progress routines, it is allowed to call all other communication routines from the callback in the struct.
The callback is thread safe with respect to the worker it is invoked on.
It is the user’s responsibility to avoid potential dead lock accessing different worker.

typedef enum ucp_wakeup_event_types ucp_wakeup_event_t¶

UCP worker wakeup events mask.

The enumeration allows specifying which events are expected on wakeup. Empty events are possible for any type of event except for UCP_WAKEUP_TX and UCP_WAKEUP_RX.

Note

Send completions are reported by POLLIN-like events (see poll man page). Since outgoing operations can be initiated at any time, UCP does not generate POLLOUT-like events, although it must be noted that outgoing operations may be queued depending upon resource availability.

Enums

enum ucp_worker_params_field¶

UCP worker parameters field mask.

The enumeration allows specifying which fields in ucp_worker_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_WORKER_PARAM_FIELD_THREAD_MODE¶: UCP thread mode

enumerator UCP_WORKER_PARAM_FIELD_CPU_MASK¶: Worker’s CPU bitmap

enumerator UCP_WORKER_PARAM_FIELD_EVENTS¶: Worker’s events bitmap

enumerator UCP_WORKER_PARAM_FIELD_USER_DATA¶: User data

enumerator UCP_WORKER_PARAM_FIELD_EVENT_FD¶: External event file descriptor

enumerator UCP_WORKER_PARAM_FIELD_FLAGS¶: Worker flags

enumerator UCP_WORKER_PARAM_FIELD_NAME¶: Worker name

enumerator UCP_WORKER_PARAM_FIELD_AM_ALIGNMENT¶: Alignment of active messages on the receiver

enumerator UCP_WORKER_PARAM_FIELD_CLIENT_ID¶: Client id

enum ucp_worker_flags_t¶

UCP worker flags.

This enumeration allows specifying flags for ucp_worker_params_t::flags, which is used as parameter for ucp_worker_create.

Values:

enumerator UCP_WORKER_FLAG_IGNORE_REQUEST_LEAK¶: Do not print warnings about request leaks

enum ucp_listener_params_field¶

UCP listener parameters field mask.

The enumeration allows specifying which fields in ucp_listener_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_LISTENER_PARAM_FIELD_SOCK_ADDR¶: Sock address and length.

enumerator UCP_LISTENER_PARAM_FIELD_ACCEPT_HANDLER¶: User’s callback and argument for handling the creation of an endpoint. User’s callback and argument for handling the incoming connection request.

enumerator UCP_LISTENER_PARAM_FIELD_CONN_HANDLER¶

enum ucp_worker_address_flags_t¶

UCP worker address flags.

The enumeration list describes possible UCP worker address flags, indicating what needs to be included to the worker address returned by ucp_worker_query() routine.

Values:

enumerator UCP_WORKER_ADDRESS_FLAG_NET_ONLY¶: Pack addresses of network devices only. Using such shortened addresses for the remote node peers will reduce the amount of wireup data being exchanged during connection establishment phase.

enum ucp_worker_attr_field¶

UCP worker attributes field mask.

The enumeration allows specifying which fields in ucp_worker_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_WORKER_ATTR_FIELD_THREAD_MODE¶: UCP thread mode

enumerator UCP_WORKER_ATTR_FIELD_ADDRESS¶: UCP address

enumerator UCP_WORKER_ATTR_FIELD_ADDRESS_FLAGS¶: UCP address flags

enumerator UCP_WORKER_ATTR_FIELD_MAX_AM_HEADER¶: Maximum header size used by UCP AM API

enumerator UCP_WORKER_ATTR_FIELD_NAME¶: UCP worker name

enumerator UCP_WORKER_ATTR_FIELD_MAX_INFO_STRING¶: Maximum size of info string

enum ucp_worker_address_attr_field¶

UCP worker address attributes field mask.

The enumeration allows specifying which fields in ucp_worker_address_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_WORKER_ADDRESS_ATTR_FIELD_UID¶: Unique id of the worker

enum ucp_listener_attr_field¶

UCP listener attributes field mask.

The enumeration allows specifying which fields in ucp_listener_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_LISTENER_ATTR_FIELD_SOCKADDR¶: Sockaddr used for listening

enum ucp_conn_request_attr_field¶

UCP listener’s connection request attributes field mask.

The enumeration allows specifying which fields in ucp_conn_request_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_CONN_REQUEST_ATTR_FIELD_CLIENT_ADDR¶: Client’s address

enumerator UCP_CONN_REQUEST_ATTR_FIELD_CLIENT_ID¶: Remote client id

enum ucp_am_cb_flags¶

Flags for a UCP Active Message callback.

Flags that indicate how to handle UCP Active Messages.

Values:

enumerator UCP_AM_FLAG_WHOLE_MSG¶: Indicates that the entire message will be handled in one callback.

enumerator UCP_AM_FLAG_PERSISTENT_DATA¶: Guarantees that the specified ucp_am_recv_callback_t callback, will always be called with UCP_AM_RECV_ATTR_FLAG_DATA flag set, so the data will be accessible outside the callback, until ucp_am_data_release is called.

enum ucp_send_am_flags¶

Flags for sending a UCP Active Message.

Flags dictate the behavior of ucp_am_send_nb and ucp_am_send_nbx routines.

Values:

enumerator UCP_AM_SEND_FLAG_REPLY¶: Force relevant reply endpoint to be passed to the data callback on the receiver.

enumerator UCP_AM_SEND_FLAG_EAGER¶: Force UCP to use only eager protocol for AM sends.

enumerator UCP_AM_SEND_FLAG_RNDV¶: Force UCP to use only rendezvous protocol for AM sends.

enumerator UCP_AM_SEND_FLAG_COPY_HEADER¶: The flag indicates that the header should be copied to an internal buffer in case it’s needed after the send function returns. If this flag is specified, the header can be released immediately after the send function returns, even if the non-blocking send request is not completed.

enumerator UCP_AM_SEND_REPLY¶: Backward compatibility.

enum ucp_wakeup_event_types¶

UCP worker wakeup events mask.

The enumeration allows specifying which events are expected on wakeup. Empty events are possible for any type of event except for UCP_WAKEUP_TX and UCP_WAKEUP_RX.

Note

Send completions are reported by POLLIN-like events (see poll man page). Since outgoing operations can be initiated at any time, UCP does not generate POLLOUT-like events, although it must be noted that outgoing operations may be queued depending upon resource availability.

Values:

enumerator UCP_WAKEUP_RMA¶: Remote memory access send completion

enumerator UCP_WAKEUP_AMO¶: Atomic operation send completion

enumerator UCP_WAKEUP_TAG_SEND¶: Tag send completion

enumerator UCP_WAKEUP_TAG_RECV¶: Tag receive completion

enumerator UCP_WAKEUP_TX¶: This event type will generate an event on completion of any outgoing operation (complete or partial, according to the underlying protocol) for any type of transfer (send, atomic, or RMA).

enumerator UCP_WAKEUP_RX¶: This event type will generate an event on completion of any receive operation (complete or partial, according to the underlying protocol).

enumerator UCP_WAKEUP_EDGE¶: Use edge-triggered wakeup. The event file descriptor will be signaled only for new events, rather than existing ones.

Functions

ucs_status_t ucp_worker_create(ucp_context_h context, const ucp_worker_params_t *params, ucp_worker_h *worker_p)¶

Create a worker object.

This routine allocates and initializes a worker object. Each worker is associated with one and only one application context. In the same time, an application context can create multiple workers in order to enable concurrent access to communication resources. For example, application can allocate a dedicated worker for each application thread, where every worker can be progressed independently of others.

Parameters

context – [in] Handle to UCP application context.
params – [in] User defined ucp_worker_params_t configurations for the UCP worker.
worker_p – [out] A pointer to the worker object allocated by the UCP library

Returns

Error code as defined by ucs_status_t

Note

The worker object is allocated within context of the calling thread

void ucp_worker_destroy(ucp_worker_h worker)¶

Destroy a worker object.

This routine releases the resources associated with a UCP worker.

The destroy process releases and shuts down all resources associated with the worker.

Parameters: worker – [in] Worker object to destroy.

Warning

Once the UCP worker destroy the worker handle cannot be used with any UCP routine.

ucs_status_t ucp_worker_query(ucp_worker_h worker, ucp_worker_attr_t *attr)¶

Get attributes specific to a particular worker.

This routine fetches information about the worker.

Parameters

worker – [in] Worker object to query.
attr – [out] Filled with attributes of worker.

Returns

Error code as defined by ucs_status_t

void ucp_worker_print_info(ucp_worker_h worker, FILE *stream)¶

Print information about the worker.

This routine prints information about the protocols being used, thresholds, UCT transport methods, and other useful information associated with the worker.

Parameters

worker – [in] Worker object to print information for.
stream – [in] Output stream to print the information to.

void ucp_worker_release_address(ucp_worker_h worker, ucp_address_t *address)¶

Release an address of the worker object.

This routine release an address handle associated within the worker object.

Parameters

worker – [in] Worker object that is associated with the address object.
address – [in] Address to release; the address object has to be allocated using ucp_worker_query() routine.

Warning

Once the address released the address handle cannot be used with any UCP routine.

ucs_status_t ucp_worker_address_query(ucp_address_t *address, ucp_worker_address_attr_t *attr)¶

Get attributes of the particular worker address.

This routine fetches information about the worker address. The address can be either of local or remote worker.

Parameters

address – [in] Worker address to query.
attr – [out] Filled with attributes of the worker address.

Returns

Error code as defined by ucs_status_t.

unsigned ucp_worker_progress(ucp_worker_h worker)¶

Progress all communications on a specific worker.

This routine explicitly progresses all communication operations on a worker.

Parameters: worker – [in] Worker to progress.
Returns: Non-zero if any communication was progressed, zero otherwise.

Note

Typically, request wait and test routines call this routine to progress any outstanding operations.
Transport layers, implementing asynchronous progress using threads, require callbacks and other user code to be thread safe.
The state of communication can be advanced (progressed) by blocking routines. Nevertheless, the non-blocking routines can not be used for communication progress.

ssize_t ucp_stream_worker_poll(ucp_worker_h worker, ucp_stream_poll_ep_t *poll_eps, size_t max_eps, unsigned flags)¶

Poll for endpoints that are ready to consume streaming data.

This non-blocking routine returns endpoints on a worker which are ready to consume streaming data. The ready endpoints are placed in poll_eps array, and the function return value indicates how many are there.

Parameters

worker – [in] Worker to poll.
poll_eps – [out] Pointer to array of endpoints, should be allocated by user.
max_eps – [in] Maximum number of endpoints that should be filled in poll_eps.
flags – [in] Reserved for future use.

Returns

Negative value indicates an error according to ucs_status_t. On success, non-negative value (less or equal max_eps) indicates actual number of endpoints filled in poll_eps array.

ucs_status_t ucp_listener_create(ucp_worker_h worker, const ucp_listener_params_t *params, ucp_listener_h *listener_p)¶

Create a listener to accept connections on. Connection requests on the listener will arrive at a local address specified by the user.

This routine creates a new listener object that is bound to a specific local address. The listener will listen to incoming connection requests. After receiving a request from the remote peer, an endpoint to this peer will be created - either right away or by calling ucp_ep_create, as specified by the callback type in ucp_listener_params_t. The user’s callback will be invoked once the endpoint is created.

Parameters

worker – [in] Worker object to create the listener on.
params – [in] User defined ucp_listener_params_t configurations for the ucp_listener_h.
listener_p – [out] A handle to the created listener, can be released by calling ucp_listener_destroy

Returns

Error code as defined by ucs_status_t

Note

ucp_listener_params_t::conn_handler or ucp_listener_params_t::accept_handler must be provided to be able to handle incoming connections.

void ucp_listener_destroy(ucp_listener_h listener)¶

Stop accepting connections on a local address of the worker object.

This routine unbinds the worker from the given handle and stops listening for incoming connection requests on it.

Parameters: listener – [in] A handle to the listener to stop listening on.

ucs_status_t ucp_listener_query(ucp_listener_h listener, ucp_listener_attr_t *attr)¶

Get attributes specific to a particular listener.

This routine fetches information about the listener.

Parameters

listener – [in] listener object to query.
attr – [out] Filled with attributes of the listener.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_conn_request_query(ucp_conn_request_h conn_request, ucp_conn_request_attr_t *attr)¶

Get attributes specific to a particular connection request received on the server side.

This routine fetches information about the connection request.

Parameters

conn_request – [in] connection request object to query.
attr – [out] Filled with attributes of the connection request.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_listener_reject(ucp_listener_h listener, ucp_conn_request_h conn_request)¶

Reject an incoming connection request.

Reject the incoming connection request and release associated resources. If the remote initiator endpoint has set an ucp_ep_params_t::err_handler, it will be invoked with status UCS_ERR_REJECTED.

Parameters

listener – [in] Handle to the listener on which the connection request was received.
conn_request – [in] Handle to the connection request to reject.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_worker_set_am_recv_handler(ucp_worker_h worker, const ucp_am_handler_param_t *param)¶

Add user defined callback for Active Message.

This routine installs a user defined callback to handle incoming Active Messages with a specific id. This callback is called whenever an Active Message that was sent from the remote peer by ucp_am_send_nbx is received on this worker.

Parameters

worker – [in] UCP worker on which to set the Active Message handler.
param – [in] Active Message handler parameters, as defined by ucp_am_handler_param_t.

Returns

error code if the worker does not support Active Messages or requested callback flags.

Warning

Handlers set by this function are not compatible with ucp_am_send_nb routine.

ucs_status_t ucp_worker_fence(ucp_worker_h worker)¶

Assures ordering between non-blocking operations.

This routine ensures ordering of non-blocking communication operations on the UCP worker. Communication operations issued on a particular endpoint created on the worker prior to this call are guaranteed to be completed before any communication operations issued on the same endpoint after this call.

Parameters: worker – [in] UCP worker.
Returns: Error code as defined by ucs_status_t

Note

The primary difference between ucp_worker_fence() and the ucp_worker_flush_nb() is the fact the fence routine does not guarantee completion of the operations on the call return but only ensures the order between communication operations. The flush operation on return guarantees that all operations are completed and corresponding memory regions were updated.

ucs_status_ptr_t ucp_worker_flush_nbx(ucp_worker_h worker, const ucp_request_param_t *param)¶

Flush outstanding AMO and RMA operations on the worker.

This routine flushes all outstanding AMO and RMA communications on the worker. All the AMO and RMA operations issued on the worker prior to this call are completed both at the origin and at the target when this call returns.

Parameters

worker – [in] UCP worker.
param – [in] Operation parameters, see ucp_request_param_t

Returns

NULL - The flush operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The flush operation failed.

Returns

otherwise - Flush operation was scheduled and can be completed in any point in time. The request handle is returned to the application in order to track progress.

Note

For description of the differences between flush and fence operations please see ucp_worker_fence()

ucs_status_t ucp_worker_flush(ucp_worker_h worker)¶

Flush outstanding AMO and RMA operations on the worker.

Deprecated:: Replaced by ucp_worker_flush_nb. The following example implements the same functionality using ucp_worker_flush_nb :

ucs_status_t worker_flush(ucp_worker_h worker)
{
    void *request = ucp_worker_flush_nb(worker);
    if (request == NULL) {
        return UCS_OK;
    } else if (UCS_PTR_IS_ERR(request)) {
        return UCS_PTR_STATUS(request);
    } else {
        ucs_status_t status;
        do {
            ucp_worker_progress(worker);
            status = ucp_request_check_status(request);
        } while (status == UCS_INPROGRESS);
        ucp_request_release(request);
        return status;
    }
}

This routine flushes all outstanding AMO and RMA communications on the worker. All the AMO and RMA operations issued on the worker prior to this call are completed both at the origin and at the target when this call returns.

Parameters: worker – [in] UCP worker.
Returns: Error code as defined by ucs_status_t

Note

For description of the differences between flush and fence operations please see ucp_worker_fence()

ucs_status_t ucp_worker_get_address(ucp_worker_h worker, ucp_address_t **address_p, size_t *address_length_p)¶

Get the address of the worker object.

Deprecated:: Use ucp_worker_query with the flag UCP_WORKER_ATTR_FIELD_ADDRESS in order to obtain the worker address.

This routine returns the address of the worker object. This address can be passed to remote instances of the UCP library in order to connect to this worker. The memory for the address handle is allocated by this function, and must be released by using ucp_worker_release_address() routine.

Parameters

worker – [in] Worker object whose address to return.
address_p – [out] A pointer to the worker address.
address_length_p – [out] The size in bytes of the address.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_worker_set_am_handler(ucp_worker_h worker, uint16_t id, ucp_am_callback_t cb, void *arg, uint32_t flags)¶

Add user defined callback for Active Message.

Deprecated:: Use ucp_worker_set_am_recv_handler instead.

This routine installs a user defined callback to handle incoming Active Messages with a specific id. This callback is called whenever an Active Message that was sent from the remote peer by ucp_am_send_nb is received on this worker.

Parameters

worker – [in] UCP worker on which to set the Active Message handler.
id – [in] Active Message id.
cb – [in] Active Message callback. NULL to clear.
arg – [in] Active Message argument, which will be passed in to every invocation of the callback as the arg argument.
flags – [in] Dictates how an Active Message is handled on the remote endpoint. Currently only UCP_AM_FLAG_WHOLE_MSG is supported, which indicates the callback will not be invoked until all data has arrived.

Returns

error code if the worker does not support Active Messages or requested callback flags.

ucs_status_ptr_t ucp_worker_flush_nb(ucp_worker_h worker, unsigned flags, ucp_send_callback_t cb)¶

Flush outstanding AMO and RMA operations on the worker.

Deprecated:: Use ucp_worker_flush_nbx instead.

This routine flushes all outstanding AMO and RMA communications on the worker. All the AMO and RMA operations issued on the worker prior to this call are completed both at the origin and at the target when this call returns.

Parameters

worker – [in] UCP worker.
flags – [in] Flags for flush operation. Reserved for future use.
cb – [in] Callback which will be called when the flush operation completes.

Returns

NULL - The flush operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The flush operation failed.

Returns

otherwise - Flush operation was scheduled and can be completed in any point in time. The request handle is returned to the application in order to track progress. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

For description of the differences between flush and fence operations please see ucp_worker_fence()

struct ucp_worker_attr¶

#include <ucp.h>

UCP worker attributes.

The structure defines the attributes which characterize the particular worker.

struct ucp_worker_params¶

#include <ucp.h>

Tuning parameters for the UCP worker.

The structure defines the parameters that are used for the UCP worker tuning during the UCP worker creation.

struct ucp_worker_address_attr¶

#include <ucp.h>

UCP worker address attributes.

The structure defines the attributes of the particular worker address.

struct ucp_listener_attr¶

#include <ucp.h>

UCP listener attributes.

The structure defines the attributes which characterize the particular listener.

struct ucp_conn_request_attr¶

#include <ucp.h>

UCP listener’s connection request attributes.

The structure defines the attributes that characterize the particular connection request received on the server side.

struct ucp_listener_params¶

#include <ucp.h>

Parameters for a UCP listener object.

This structure defines parameters for ucp_listener_create, which is used to listen for incoming client/server connections.

struct ucp_am_handler_param¶: #include <ucp.h>
Active Message handler parameters passed to ucp_worker_set_am_recv_handler routine.

struct ucp_am_recv_param¶: #include <ucp.h>
Operation parameters provided in ucp_am_recv_callback_t callback.

struct ucp_listener_accept_handler¶

#include <ucp_compat.h>

Deprecated:: Replaced by ucp_listener_conn_handler_t.

struct ucp_listener_conn_handler¶

#include <ucp_def.h>

UCP callback to handle the connection request in a client-server connection establishment flow.

This structure is used for handling an incoming connection request on the listener. Setting this type of handler allows creating an endpoint on any other worker and not limited to the worker on which the listener was created.

Note

Other than communication progress routines, it is allowed to call all other communication routines from the callback in the struct.
The callback is thread safe with respect to the worker it is invoked on.
It is the user’s responsibility to avoid potential dead lock accessing different worker.

Endpoint routines ¶

group UCP_ENDPOINT

UCP Endpoint routines

Typedefs

typedef enum ucp_ep_perf_param_field ucp_ep_perf_param_field_t¶

UCP performance fields and flags.

The enumeration allows specifying which fields in ucp_ep_evaluate_perf_param_t are present and operation flags are used. It is used to enable backward compatibility support.

typedef enum ucp_ep_perf_attr_field ucp_ep_perf_attr_field_t¶

UCP performance fields and flags.

The enumeration allows specifying which fields in ucp_ep_evaluate_perf_attr_t are present and operation flags are used. It is used to enable backward compatibility support.

typedef struct ucp_stream_poll_ep ucp_stream_poll_ep_t¶

Output parameter of ucp_stream_worker_poll function.

The structure defines the endpoint and its user data.

typedef struct ucp_ep_attr ucp_ep_attr_t¶

UCP endpoint attributes.

The structure defines the attributes that characterize the particular endpoint.

typedef struct ucp_ep *ucp_ep_h¶

UCP Endpoint.

The endpoint handle is an opaque object that is used to address a remote worker. It typically provides a description of source, destination, or both. All UCP communication routines address a destination with the endpoint handle. The endpoint handle is associated with only one UCP context. UCP provides the endpoint create routine to create the endpoint handle and the destroy routine to destroy the endpoint handle.

typedef struct ucp_conn_request *ucp_conn_request_h¶

UCP connection request.

A server-side handle to incoming connection request. Can be used to create an endpoint which connects back to the client.

typedef ucs_status_t (*ucp_am_callback_t)(void *arg, void *data, size_t length, ucp_ep_h reply_ep, unsigned flags)¶

Callback to process incoming Active Message.

When the callback is called, flags indicates how data should be handled.

Parameters

arg – [in] User-defined argument.
data – [in] Points to the received data. This data may persist after the callback returns and needs to be freed with ucp_am_data_release.
length – [in] Length of data.
reply_ep – [in] If the Active Message is sent with the UCP_AM_SEND_FLAG_REPLY flag, the sending ep will be passed in. If not, NULL will be passed.
flags – [in] If this flag is set to UCP_CB_PARAM_FLAG_DATA, the callback can return UCS_INPROGRESS and data will persist after the callback returns.

Returns

UCS_OK data will not persist after the callback returns.

Returns

UCS_INPROGRESS Can only be returned if flags is set to UCP_CB_PARAM_FLAG_DATA. If UCP_INPROGRESS is returned, data will persist after the callback has returned. To free the memory, a pointer to the data must be passed into ucp_am_data_release.

Note

This callback should be set and released by ucp_worker_set_am_handler function.

typedef ucs_status_t (*ucp_am_recv_callback_t)(void *arg, const void *header, size_t header_length, void *data, size_t length, const ucp_am_recv_param_t *param)¶

Callback to process incoming Active Message sent by ucp_am_send_nbx routine.

The callback is always called from the progress context, therefore calling ucp_worker_progress() is not allowed. It is recommended to define callbacks with relatively short execution time to avoid blocking of communication progress.

Parameters

arg – [in] User-defined argument.
header – [in] User defined active message header. If header_length is 0, this value is undefined and must not be accessed.
header_length – [in] Active message header length in bytes.
data – [in] Points to the received data if UCP_AM_RECV_ATTR_FLAG_RNDV flag is not set in ucp_am_recv_param_t::recv_attr. Otherwise it points to the internal UCP descriptor which can further be used for initiating data receive by using ucp_am_recv_data_nbx routine.
length – [in] Length of data. If UCP_AM_RECV_ATTR_FLAG_RNDV flag is set in ucp_am_recv_param_t::recv_attr, it indicates the required receive buffer size for initiating rendezvous protocol.
param – [in] Data receive parameters.

Returns

UCS_OK data will not persist after the callback returns. If UCP_AM_RECV_ATTR_FLAG_RNDV flag is set in param->recv_attr and ucp_am_recv_data_nbx was not called for this data, the data descriptor will be dropped and the corresponding ucp_am_send_nbx call will complete with UCS_OK status.

Returns

UCS_INPROGRESS Can only be returned if param->recv_attr flags contains UCP_AM_RECV_ATTR_FLAG_DATA or UCP_AM_RECV_ATTR_FLAG_RNDV. The data will persist after the callback has returned. To free the memory, a pointer to the data must be passed into ucp_am_data_release or data receive is initiated by ucp_am_recv_data_nbx.

Returns

otherwise Can only be returned if param->recv_attr contains UCP_AM_RECV_ATTR_FLAG_RNDV. In this case data descriptor data will be dropped and the corresponding ucp_am_send_nbx call on the sender side will complete with the status returned from the callback.

Note

This callback should be set and released by ucp_worker_set_am_recv_handler function.

typedef struct ucp_ep_params ucp_ep_params_t¶

Tuning parameters for the UCP endpoint.

The structure defines the parameters that are used for the UCP endpoint tuning during the UCP ep creation.

Enums

enum ucp_ep_params_field¶

UCP endpoint parameters field mask.

The enumeration allows specifying which fields in ucp_ep_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_EP_PARAM_FIELD_REMOTE_ADDRESS¶: Address of remote peer

enumerator UCP_EP_PARAM_FIELD_ERR_HANDLING_MODE¶: Error handling mode. ucp_err_handling_mode_t

enumerator UCP_EP_PARAM_FIELD_ERR_HANDLER¶: Handler to process transport level errors

enumerator UCP_EP_PARAM_FIELD_USER_DATA¶: User data pointer

enumerator UCP_EP_PARAM_FIELD_SOCK_ADDR¶: Socket address field

enumerator UCP_EP_PARAM_FIELD_FLAGS¶: Endpoint flags Connection request field

enumerator UCP_EP_PARAM_FIELD_CONN_REQUEST¶

enumerator UCP_EP_PARAM_FIELD_NAME¶: Endpoint name

enumerator UCP_EP_PARAM_FIELD_LOCAL_SOCK_ADDR¶: Local socket Address

enum ucp_ep_params_flags_field¶

UCP endpoint parameters flags.

The enumeration list describes the endpoint’s parameters flags supported by ucp_ep_create() function.

Values:

enumerator UCP_EP_PARAMS_FLAGS_CLIENT_SERVER¶: Using a client-server connection establishment mechanism. ucs_sock_addr_t sockaddr field must be provided and contain the address of the remote peer

enumerator UCP_EP_PARAMS_FLAGS_NO_LOOPBACK¶: Avoid connecting the endpoint to itself when connecting the endpoint to the same worker it was created on. Affects protocols which send to a particular remote endpoint, for example stream

enumerator UCP_EP_PARAMS_FLAGS_SEND_CLIENT_ID¶: Send client id when connecting to remote socket address as part of the connection request payload. On the remote side value can be obtained from ucp_conn_request_h using ucp_conn_request_query

enum ucp_ep_close_flags_t¶

Close UCP endpoint modes.

The enumeration is used to specify the behavior of ucp_ep_close_nbx.

Values:

enumerator UCP_EP_CLOSE_FLAG_FORCE¶: ucp_ep_close_nbx releases the endpoint without any confirmation from the peer. All outstanding requests will be completed with UCS_ERR_CANCELED error.

Note

This mode may cause transport level errors on remote side, so it requires set UCP_ERR_HANDLING_MODE_PEER for all endpoints created on both (local and remote) sides to avoid undefined behavior. If this flag is not set then ucp_ep_close_nbx schedules flushes on all outstanding operations.

Note

this flag is incompatible with UCP_OP_ATTR_FLAG_NO_IMM_CMPL, since it forces immediate completion.

enum ucp_ep_perf_param_field¶

UCP performance fields and flags.

The enumeration allows specifying which fields in ucp_ep_evaluate_perf_param_t are present and operation flags are used. It is used to enable backward compatibility support.

Values:

enumerator UCP_EP_PERF_PARAM_FIELD_MESSAGE_SIZE¶: Enables ucp_ep_evaluate_perf_param_t::message_size

enum ucp_ep_perf_attr_field¶

UCP performance fields and flags.

The enumeration allows specifying which fields in ucp_ep_evaluate_perf_attr_t are present and operation flags are used. It is used to enable backward compatibility support.

Values:

enumerator UCP_EP_PERF_ATTR_FIELD_ESTIMATED_TIME¶: Enables ucp_ep_evaluate_perf_attr_t::estimated_time

enum ucp_cb_param_flags¶

Descriptor flags for Active Message callback.

In a callback, if flags is set to UCP_CB_PARAM_FLAG_DATA in a callback then data was allocated, so if UCS_INPROGRESS is returned from the callback, the data parameter will persist and the user has to call ucp_am_data_release when data is no longer needed.

Values:

enumerator UCP_CB_PARAM_FLAG_DATA¶

enum ucp_ep_attr_field¶

UCP endpoint attributes field mask.

The enumeration allows specifying which fields in ucp_ep_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_EP_ATTR_FIELD_NAME¶: UCP endpoint name

enumerator UCP_EP_ATTR_FIELD_LOCAL_SOCKADDR¶: Sockaddr used by the endpoint

enumerator UCP_EP_ATTR_FIELD_REMOTE_SOCKADDR¶: Sockaddr the endpoint is connected to

enumerator UCP_EP_ATTR_FIELD_TRANSPORTS¶: Transport and device used by endpoint

enum ucp_ep_close_mode¶

Close UCP endpoint modes.

Deprecated:: Use ucp_ep_close_nbx and ucp_ep_close_flags_t instead.

The enumeration is used to specify the behavior of ucp_ep_close_nb.

Values:

enumerator UCP_EP_CLOSE_MODE_FORCE¶: ucp_ep_close_nb releases the endpoint without any confirmation from the peer. All outstanding requests will be completed with UCS_ERR_CANCELED error.

Note

This mode may cause transport level errors on remote side, so it requires set UCP_ERR_HANDLING_MODE_PEER for all endpoints created on both (local and remote) sides to avoid undefined behavior.

enumerator UCP_EP_CLOSE_MODE_FLUSH¶: ucp_ep_close_nb schedules flushes on all outstanding operations.

enum ucp_err_handling_mode_t¶

Error handling mode for the UCP endpoint.

Specifies error handling mode for the UCP endpoint.

Values:

enumerator UCP_ERR_HANDLING_MODE_NONE¶: No guarantees about error reporting, imposes minimal overhead from a performance perspective.

Note

In this mode, any error reporting will not generate calls to ucp_ep_params_t::err_handler.

enumerator UCP_ERR_HANDLING_MODE_PEER¶: Guarantees that send requests are always completed (successfully or error) even in case of remote failure, disables protocols and APIs which may cause a hang or undefined behavior in case of peer failure, may affect performance and memory footprint

Functions

ucs_status_t ucp_ep_create(ucp_worker_h worker, const ucp_ep_params_t *params, ucp_ep_h *ep_p)¶

Create and connect an endpoint.

This routine creates and connects an endpoint on a local worker for a destination address that identifies the remote worker. This function is non-blocking, and communications may begin immediately after it returns. If the connection process is not completed, communications may be delayed. The created endpoint is associated with one and only one worker.

Parameters

worker – [in] Handle to the worker; the endpoint is associated with the worker.
params – [in] User defined ucp_ep_params_t configurations for the UCP endpoint.
ep_p – [out] A handle to the created endpoint.

Returns

Error code as defined by ucs_status_t

Note

One of the following fields has to be specified:

Note

By default, ucp_ep_create() will connect an endpoint to itself if the endpoint is destined to the same worker on which it was created, i.e. params.address belongs to worker. This behavior can be changed by passing the UCP_EP_PARAMS_FLAGS_NO_LOOPBACK flag in params.flags. In that case, the endpoint will be connected to the next endpoint created in the same way on the same worker.

ucs_status_ptr_t ucp_ep_close_nbx(ucp_ep_h ep, const ucp_request_param_t *param)¶

Non-blocking endpoint closure.

Parameters

ep – [in] Handle to the endpoint to close.
param – [in] Operation parameters, see ucp_request_param_t. This operation supports specific flags, which can be passed in param by ucp_request_param_t::flags. The exact set of flags is defined by ucp_ep_close_flags_t.

Returns

NULL - The endpoint is closed successfully.

Returns

UCS_PTR_IS_ERR(_ptr) - The closure failed and an error code indicates the transport level status. However, resources are released and the endpoint can no longer be used.

Returns

otherwise - The closure process is started, and can be completed at any point in time. A request handle is returned to the application in order to track progress of the endpoint closure.

void ucp_ep_print_info(ucp_ep_h ep, FILE *stream)¶

Print endpoint information.

This routine prints information about the endpoint transport methods, their thresholds, and other useful information associated with the endpoint.

Parameters

ep – [in] Endpoint object whose configuration to print.
stream – [in] Output stream to print the information to.

ucs_status_ptr_t ucp_ep_flush_nbx(ucp_ep_h ep, const ucp_request_param_t *param)¶

Non-blocking flush of outstanding AMO and RMA operations on the endpoint.

This routine flushes all outstanding AMO and RMA communications on the endpoint. All the AMO and RMA operations issued on the ep prior to this call are completed both at the origin and at the target endpoint when this call returns.

The following example demonstrates how blocking flush can be implemented using non-blocking flush:

ucs_status_t blocking_ep_flush(ucp_ep_h ep, ucp_worker_h worker)
{
    ucp_request_param_t param;
    void *request;

    param.op_attr_mask = 0;
    request            = ucp_ep_flush_nbx(ep, &param);
    if (request == NULL) {
        return UCS_OK;
    } else if (UCS_PTR_IS_ERR(request)) {
        return UCS_PTR_STATUS(request);
    } else {
        ucs_status_t status;
        do {
            ucp_worker_progress(worker);
            status = ucp_request_check_status(request);
        } while (status == UCS_INPROGRESS);
        ucp_request_free(request);
        return status;
    }
}

Parameters

ep – [in] UCP endpoint.
param – [in] Operation parameters, see ucp_request_param_t.

Returns

NULL - The flush operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The flush operation failed.

Returns

otherwise - Flush operation was scheduled and can be completed in any point in time. The request handle is returned to the application in order to track progress.

ucs_status_t ucp_ep_evaluate_perf(ucp_ep_h ep, const ucp_ep_evaluate_perf_param_t *param, ucp_ep_evaluate_perf_attr_t *attr)¶

Estimate performance characteristics of a specific endpoint.

This routine fetches information about the endpoint.

Parameters

ep – [in] Endpoint to query.
param – [in] Filled by the user with request params.
attr – [out] Filled with performance estimation of the given operation on the endpoint.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_ep_query(ucp_ep_h ep, ucp_ep_attr_t *attr)¶

Get attributes of a given endpoint.

This routine fetches information about the endpoint.

Parameters

ep – [in] Endpoint object to query.
attr – [out] Filled with attributes of the endpoint.

Returns

Error code as defined by ucs_status_t

void ucp_request_release(void *request)¶

Deprecated:: Replaced by ucp_request_free.

void ucp_ep_destroy(ucp_ep_h ep)¶

Deprecated:: Replaced by ucp_ep_close_nb.

ucs_status_ptr_t ucp_disconnect_nb(ucp_ep_h ep)¶

Deprecated:: Replaced by ucp_ep_close_nb.

ucs_status_t ucp_request_test(void *request, ucp_tag_recv_info_t *info)¶

Deprecated:: Replaced by ucp_tag_recv_request_test and ucp_request_check_status depends on use case.

Note

Please use ucp_request_check_status for cases that only need to check the completion status of an outstanding request. ucp_request_check_status can be used for any type of request. ucp_tag_recv_request_test should only be used for requests returned by ucp_tag_recv_nb (or request allocated by user for ucp_tag_recv_nbr) for which additional information (returned via the info pointer) is needed.

ucs_status_t ucp_ep_flush(ucp_ep_h ep)¶

Deprecated:: Replaced by ucp_ep_flush_nb.

ucs_status_ptr_t ucp_ep_modify_nb(ucp_ep_h ep, const ucp_ep_params_t *params)¶

Modify endpoint parameters.

Deprecated:: Use ucp_listener_conn_handler_t instead of ucp_listener_accept_handler_t, if you have other use case please submit an issue on https://github.com/openucx/ucx or report to ucx-group@elist.ornl.gov

This routine modifies endpoint created by ucp_ep_create or ucp_listener_accept_callback_t. For example, this API can be used to setup custom parameters like ucp_ep_params_t::user_data or ucp_ep_params_t::err_handler to endpoint created by ucp_listener_accept_callback_t.

Parameters

ep – [in] A handle to the endpoint.
params – [in] User defined ucp_ep_params_t configurations for the UCP endpoint.

Returns

NULL - The endpoint is modified successfully.

Returns

UCS_PTR_IS_ERR(_ptr) - The reconfiguration failed and an error code indicates the status. However, the endpoint is not modified and can be used further.

Returns

otherwise - The reconfiguration process is started, and can be completed at any point in time. A request handle is returned to the application in order to track progress of the endpoint modification. The application is responsible for releasing the handle using the ucp_request_free routine.

Note

See the documentation of ucp_ep_params_t for details, only some of the parameters can be modified.

ucs_status_ptr_t ucp_ep_close_nb(ucp_ep_h ep, unsigned mode)¶

Non-blocking endpoint closure.

Deprecated:: Use ucp_ep_close_nbx instead.

This routine releases the endpoint. The endpoint closure process depends on the selected mode.

Parameters

ep – [in] Handle to the endpoint to close.
mode – [in] One from ucp_ep_close_mode value.

Returns

UCS_OK - The endpoint is closed successfully.

Returns

UCS_PTR_IS_ERR(_ptr) - The closure failed and an error code indicates the transport level status. However, resources are released and the endpoint can no longer be used.

Returns

otherwise - The closure process is started, and can be completed at any point in time. A request handle is returned to the application in order to track progress of the endpoint closure. The application is responsible for releasing the handle using the ucp_request_free routine.

Note

ucp_ep_close_nb replaces deprecated ucp_disconnect_nb and ucp_ep_destroy

ucs_status_ptr_t ucp_ep_flush_nb(ucp_ep_h ep, unsigned flags, ucp_send_callback_t cb)¶

Non-blocking flush of outstanding AMO and RMA operations on the endpoint.

Deprecated:: Use ucp_ep_flush_nbx instead.

This routine flushes all outstanding AMO and RMA communications on the endpoint. All the AMO and RMA operations issued on the ep prior to this call are completed both at the origin and at the target endpoint when this call returns.

Parameters

ep – [in] UCP endpoint.
flags – [in] Flags for flush operation. Reserved for future use.
cb – [in] Callback which will be called when the flush operation completes.

Returns

NULL - The flush operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The flush operation failed.

Returns

otherwise - Flush operation was scheduled and can be completed in any point in time. The request handle is returned to the application in order to track progress. The application is responsible for releasing the handle using ucp_request_free() routine.

struct ucp_ep_evaluate_perf_param_t¶

#include <ucp.h>

UCP endpoint performance evaluation request attributes.

The structure defines the attributes which characterize the request for performance estimation of a particular endpoint.

struct ucp_ep_evaluate_perf_attr_t¶

#include <ucp.h>

UCP endpoint performance evaluation result attributes.

The structure defines the attributes which characterize the result of performance estimation of a particular endpoint.

struct ucp_stream_poll_ep¶

#include <ucp.h>

Output parameter of ucp_stream_worker_poll function.

The structure defines the endpoint and its user data.

struct ucp_ep_attr¶

#include <ucp.h>

UCP endpoint attributes.

The structure defines the attributes that characterize the particular endpoint.

struct ucp_ep_params¶

#include <ucp_def.h>

Tuning parameters for the UCP endpoint.

The structure defines the parameters that are used for the UCP endpoint tuning during the UCP ep creation.

struct ucp_transport_entry_t¶

#include <ucp_def.h>

The ucp_transports_t and ucp_transport_entry_t structures are used when ucp_ep_query is called to return an array of transport name and device name pairs that are used by an active endpoint.

The ucp_transport_t structure specifies the characteristics of the ucp_transport_entry_t array.

The caller is responsible for the allocation and de-allocation of the ucp_transport_entry_t array.

Example: Implementation of a function to query the set of transport and device name pairs used by the specified endpoint.

int query_transports(ucp_ep_h ep)
{
  ucs_status_t status;
  ucp_transport_entry_t *transport_entries;
  ucp_ep_attr_t ep_attrs;

  ep_attrs.field_mask = UCP_EP_ATTR_FIELD_TRANSPORTS;
  ep_attrs.transports.entries = (ucp_transport_entry_t *)
           malloc(10 * sizeof(ucp_transport_entry_t));
  ep_attrs.transports.num_entries = 10;
  ep_attrs.transports.entry_size = sizeof(ucp_transport_entry_t);
  status = ucp_ep_query(ep, &ep_attrs);
  if (status == UCS_OK) {
      // ep_attrs.transports.num_entries = number of returned entries 
      // ... process transport info ... 
  }
}

A transport name and device name pair used by this endpoint. The caller is responsible for the allocation and deallocation of an array of these structures large enough to contain the desired number of transport and device name pairs.

Any new fields must be added to the end of this structure.

struct ucp_transports_t¶

#include <ucp_def.h>

Structure containing an array of transport layers and device names used by an endpoint.

The caller is responsible for allocation and deallocation of this structure.

Data type routines ¶

group UCP_DATATYPE

UCP Data type routines

Defines

ucp_dt_make_contig(_elem_size)¶

Generate an identifier for contiguous data type.

This macro creates an identifier for contiguous datatype that is defined by the size of the basic element.

Parameters

_elem_size – [in] Size of the basic element of the type.

Returns

Data-type identifier.

Note

In case of partial receive, the buffer will be filled with integral count of elements.

ucp_dt_make_iov()¶

Generate an identifier for Scatter-gather IOV data type.

This macro creates an identifier for datatype of scatter-gather list with multiple pointers

Returns: Data-type identifier.

Note

In the event of partial receive, ucp_dt_iov_t::buffer can be filled with any number of bytes according to its ucp_dt_iov_t::length.

Typedefs

typedef struct ucp_dt_iov ucp_dt_iov_t¶

Structure for scatter-gather I/O.

This structure is used to specify a list of buffers which can be used within a single data transfer function call. This list should remain valid until the data transfer request is completed.

Note

If length is zero, the memory pointed to by buffer will not be accessed. Otherwise, buffer must point to valid memory.

typedef struct ucp_generic_dt_ops ucp_generic_dt_ops_t¶

UCP generic data type descriptor.

This structure provides a generic datatype descriptor that is used for definition of application defined datatypes.

Typically, the descriptor is used for an integration with datatype engines implemented within MPI and SHMEM implementations.

Note

In case of partial receive, any amount of received data is acceptable which matches buffer size.

typedef struct ucp_datatype_attr ucp_datatype_attr_t¶

UCP datatype attributes.

This structure provides attributes of a UCP datatype.

Enums

enum ucp_dt_type¶

UCP data type classification.

The enumeration list describes the datatypes supported by UCP.

Values:

enumerator UCP_DATATYPE_CONTIG¶: Contiguous datatype

enumerator UCP_DATATYPE_STRIDED¶: Strided datatype

enumerator UCP_DATATYPE_IOV¶: Scatter-gather list with multiple pointers

enumerator UCP_DATATYPE_GENERIC¶: Generic datatype with user-defined pack/unpack routines

enumerator UCP_DATATYPE_SHIFT¶: Number of bits defining the datatype classification

enumerator UCP_DATATYPE_CLASS_MASK¶: Data-type class mask

enum ucp_datatype_attr_field¶

UCP datatype attributes field mask.

The enumeration allows specifying which fields in ucp_datatype_attr_t are present and which datatype attributes are queried.

Values:

enumerator UCP_DATATYPE_ATTR_FIELD_PACKED_SIZE¶: ucp_datatype_attr_t::packed_size field is queried.

enumerator UCP_DATATYPE_ATTR_FIELD_BUFFER¶: ucp_datatype_attr_t::buffer field is set.

enumerator UCP_DATATYPE_ATTR_FIELD_COUNT¶: ucp_datatype_attr_t::count field is set.

Functions

ucs_status_t ucp_dt_create_generic(const ucp_generic_dt_ops_t *ops, void *context, ucp_datatype_t *datatype_p)¶

Create a generic datatype.

This routine create a generic datatype object. The generic datatype is described by the ops object which provides a table of routines defining the operations for generic datatype manipulation. Typically, generic datatypes are used for integration with datatype engines provided with MPI implementations (MPICH, Open MPI, etc). The application is responsible for releasing the datatype_p object using ucp_dt_destroy() routine.

Parameters

ops – [in] Generic datatype function table as defined by ucp_generic_dt_ops_t .
context – [in] Application defined context passed to this routine. The context is passed as a parameter to the routines in the ops table.
datatype_p – [out] A pointer to datatype object.

Returns

Error code as defined by ucs_status_t

void ucp_dt_destroy(ucp_datatype_t datatype)¶

Destroy a datatype and release its resources.

This routine destroys the datatype object and releases any resources that are associated with the object. The datatype object must be allocated using ucp_dt_create_generic() routine.

Parameters: datatype – [in] Datatype object to destroy.

Warning

Once the datatype object is released an access to this object may cause an undefined failure.

ucs_status_t ucp_dt_query(ucp_datatype_t datatype, ucp_datatype_attr_t *attr)¶

Query attributes of a datatype.

This routine fetches information about the attributes of a datatype. When UCP_DATATYPE_ATTR_FIELD_PACKED_SIZE is set in field_mask of attr, the field packed_size is set to the packed size (bytes) of the datatype.

Parameters

datatype – [in] Datatype object to query.
attr – [inout] Filled with attributes of the datatype.

Returns

Error code as defined by ucs_status_t

Variables

void *(*start_pack)(void *context, const void *buffer, size_t count)¶

Start a packing request.

The pointer refers to application defined start-to-pack routine. It will be called from the ucp_tag_send_nb routine.

Parameters

context – [in] User-defined context.
buffer – [in] Buffer to pack.
count – [in] Number of elements to pack into the buffer.

Returns

A custom state that is passed to the following pack() routine.

void *(*start_unpack)(void *context, void *buffer, size_t count)¶

Start an unpacking request.

The pointer refers to application defined start-to-unpack routine. It will be called from the ucp_tag_recv_nb routine.

Parameters

context – [in] User-defined context.
buffer – [in] Buffer to unpack to.
count – [in] Number of elements to unpack in the buffer.

Returns

A custom state that is passed later to the following unpack() routine.

size_t (*packed_size)(void *state)¶

Get the total size of packed data.

The pointer refers to user defined routine that returns the size of data in a packed format.

Parameters: state – [in] State as returned by start_pack() routine.
Returns: The size of the data in a packed form.

size_t (*pack)(void *state, size_t offset, void *dest, size_t max_length)¶

Pack data.

The pointer refers to application defined pack routine.

Parameters

state – [in] State as returned by start_pack() routine.
offset – [in] Virtual offset in the output stream.
dest – [in] Destination buffer to pack the data.
max_length – [in] Maximum length to pack.

Returns

The size of the data that was written to the destination buffer. Must be less than or equal to max_length.

ucs_status_t (*unpack)(void *state, size_t offset, const void *src, size_t length)¶

Unpack data.

The pointer refers to application defined unpack routine.

Parameters

state – [in] State as returned by start_unpack() routine.
offset – [in] Virtual offset in the input stream.
src – [in] Source to unpack the data from.
length – [in] Length to unpack.

Returns

UCS_OK or an error if unpacking failed.

void (*finish)(void *state)¶

Finish packing/unpacking.

The pointer refers to application defined finish routine.

Parameters: state – [in] State as returned by start_pack() and start_unpack() routines.

struct ucp_dt_iov¶

#include <ucp.h>

Structure for scatter-gather I/O.

This structure is used to specify a list of buffers which can be used within a single data transfer function call. This list should remain valid until the data transfer request is completed.

Note

If length is zero, the memory pointed to by buffer will not be accessed. Otherwise, buffer must point to valid memory.

struct ucp_generic_dt_ops¶

#include <ucp.h>

UCP generic data type descriptor.

This structure provides a generic datatype descriptor that is used for definition of application defined datatypes.

Typically, the descriptor is used for an integration with datatype engines implemented within MPI and SHMEM implementations.

Note

In case of partial receive, any amount of received data is acceptable which matches buffer size.

struct ucp_datatype_attr¶

#include <ucp.h>

UCP datatype attributes.

This structure provides attributes of a UCP datatype.

Memory routines ¶

group UCP_MEM

UCP Memory routines

Typedefs

typedef struct ucp_rkey_compare_params ucp_rkey_compare_params_t¶

Tuning parameters for the comparison function ucp_rkey_compare.

The structure defines the parameters that can be used for UCP library remote keys comparison using ucp_rkey_compare routine.

typedef struct ucp_mem_map_params ucp_mem_map_params_t¶

Tuning parameters for the UCP memory mapping.

The structure defines the parameters that are used for the UCP memory mapping tuning during the ucp_mem_map routine.

typedef enum ucp_mem_advice ucp_mem_advice_t¶

list of UCP memory use advice.

The enumeration list describes memory advice supported by ucp_mem_advise() function.

typedef struct ucp_mem_advise_params ucp_mem_advise_params_t¶

Tuning parameters for the UCP memory advice.

This structure defines the parameters that are used for the UCP memory advice tuning during the ucp_mem_advise routine.

typedef struct ucp_memh_pack_params ucp_memh_pack_params_t¶

Memory handle pack parameters passed to ucp_memh_pack.

This structure defines the parameters that are used for packing the UCP memory handle during the ucp_memh_pack routine.

typedef struct ucp_memh_buffer_release_params ucp_memh_buffer_release_params_t¶

Memory handle release parameters passed to ucp_memh_buffer_release.

This structure defines the parameters that are used for releasing the UCP memory handle buffer during the ucp_memh_buffer_release routine.

typedef struct ucp_rkey *ucp_rkey_h¶

UCP Remote memory handle.

Remote memory handle is an opaque object representing remote memory access information. Typically, the handle includes a memory access key and other network hardware specific information, which are input to remote memory access operations, such as PUT, GET, and ATOMIC. The object is communicated to remote peers to enable an access to the memory region.

typedef struct ucp_mem *ucp_mem_h¶

UCP Memory handle.

Memory handle is an opaque object representing a memory region allocated through UCP library, which is optimized for remote memory access operations (zero-copy operations). The memory handle is a self-contained object, which includes the information required to access the memory region locally, while remote key is used to access it remotely. The memory could be registered to one or multiple network resources that are supported by UCP, such as InfiniBand, Gemini, and others.

typedef struct ucp_mem_attr ucp_mem_attr_t¶: Attributes of the UCP Memory handle, filled by ucp_mem_query function.

Enums

enum ucp_mem_map_params_field¶

UCP memory mapping parameters field mask.

The enumeration allows specifying which fields in ucp_mem_map_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_MEM_MAP_PARAM_FIELD_ADDRESS¶: Address of the memory that will be used in the ucp_mem_map routine.

enumerator UCP_MEM_MAP_PARAM_FIELD_LENGTH¶: The size of memory that will be allocated or registered in the ucp_mem_map routine.

enumerator UCP_MEM_MAP_PARAM_FIELD_FLAGS¶: Allocation flags.

enumerator UCP_MEM_MAP_PARAM_FIELD_PROT¶: Memory protection mode.

enumerator UCP_MEM_MAP_PARAM_FIELD_MEMORY_TYPE¶: Memory type.

enumerator UCP_MEM_MAP_PARAM_FIELD_EXPORTED_MEMH_BUFFER¶: Exported memory handle buffer.

enum ucp_mem_advise_params_field¶

UCP memory advice parameters field mask.

The enumeration allows specifying which fields in ucp_mem_advise_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_MEM_ADVISE_PARAM_FIELD_ADDRESS¶: Address of the memory

enumerator UCP_MEM_ADVISE_PARAM_FIELD_LENGTH¶: The size of memory

enumerator UCP_MEM_ADVISE_PARAM_FIELD_ADVICE¶: Advice on memory usage

enum [anonymous]¶

UCP memory mapping flags.

The enumeration list describes the memory mapping flags supported by ucp_mem_map() function.

Values:

enumerator UCP_MEM_MAP_NONBLOCK¶: Complete the mapping faster, possibly by not populating the pages in the mapping up-front, and mapping them later when they are accessed by communication routines.

enumerator UCP_MEM_MAP_ALLOCATE¶: Identify requirement for allocation, if passed address is not a null-pointer, then it will be used as a hint or direct address for allocation.

enumerator UCP_MEM_MAP_FIXED¶: Don’t interpret address as a hint: place the mapping at exactly that address. The address must be a multiple of the page size.

enumerator UCP_MEM_MAP_SYMMETRIC_RKEY¶: Register the memory region so its remote access key would likely be equal to remote access keys received from other peers, when compared with ucp_rkey_compare. This flag is a hint. When remote access keys received from different peers are compared equal, they can be used interchangeably, avoiding the need to keep all of them in memory.

enum [anonymous]¶

UCP memory mapping protection mode.

The enumeration list describes the memory mapping protections supported by the ucp_mem_map() function.

Values:

enumerator UCP_MEM_MAP_PROT_LOCAL_READ¶: Enable local read access.

enumerator UCP_MEM_MAP_PROT_LOCAL_WRITE¶: Enable local write access.

enumerator UCP_MEM_MAP_PROT_REMOTE_READ¶: Enable remote read access.

enumerator UCP_MEM_MAP_PROT_REMOTE_WRITE¶: Enable remote write access.

enum ucp_mem_advice¶

list of UCP memory use advice.

The enumeration list describes memory advice supported by ucp_mem_advise() function.

Values:

enumerator UCP_MADV_NORMAL¶: No special treatment

enumerator UCP_MADV_WILLNEED¶: can be used on the memory mapped with UCP_MEM_MAP_NONBLOCK to speed up memory mapping and to avoid page faults when the memory is accessed for the first time.

enum ucp_memh_pack_params_field¶

UCP memory handle packing parameters field mask.

The enumeration allows specifying which fields in ucp_memh_pack_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_MEMH_PACK_PARAM_FIELD_FLAGS¶: Memory handle packing field that will be used in the ucp_memh_pack routine.

enum ucp_memh_pack_flags¶

UCP memory handle flags.

The enumeration list describes the memory handle packing flags supported by ucp_memh_pack() function.

Values:

enumerator UCP_MEMH_PACK_FLAG_EXPORT¶: Pack a memory handle to be exported and used by peers for their local operations on a memory buffer allocated from same or another virtual memory space, but physically registered on the same network device. A peer should call ucp_mem_map with the flag UCP_MEM_MAP_PARAM_FIELD_EXPORTED_MEMH_BUFFER in order to import and use a memory handle buffer obtained from ucp_memh_pack.

enum ucp_mem_attr_field¶

UCP Memory handle attributes field mask.

The enumeration allows specifying which fields in ucp_mem_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_MEM_ATTR_FIELD_ADDRESS¶: Virtual address

enumerator UCP_MEM_ATTR_FIELD_LENGTH¶: The size of memory region

enumerator UCP_MEM_ATTR_FIELD_MEM_TYPE¶: Type of allocated or registered memory

Functions

ucs_status_t ucp_rkey_compare(ucp_worker_h worker, ucp_rkey_h rkey1, ucp_rkey_h rkey2, const ucp_rkey_compare_params_t *params, int *result)¶

Compare two remote keys.

This routine compares two remote keys. They must belong to the same worker.

It sets the result argument to < 0 if rkey1 is lower than rkey2, 0 if they are equal or > 0 if rkey1 is greater than rkey2. The result value can be used for sorting remote keys.

Parameters

worker – [in] Worker object both rkeys are referring to
rkey1 – [in] First rkey to compare
rkey2 – [in] Second rkey to compare
params – [in] Additional parameters to the comparison
result – [out] Result of the comparison

Returns

UCS_OK - result contains the comparison result

Returns

UCS_ERR_INVALID_PARAM - The routine arguments are invalid

Returns

Other - Error code as defined by ucs_status_t

ucs_status_t ucp_mem_map(ucp_context_h context, const ucp_mem_map_params_t *params, ucp_mem_h *memh_p)¶

Map or allocate memory for zero-copy operations.

This routine maps or/and allocates a user-specified memory segment with UCP application context and the network resources associated with it. If the application specifies NULL as an address for the memory segment, the routine allocates a mapped memory segment and returns its address in the address_p argument. The network stack associated with an application context can typically send and receive data from the mapped memory without CPU intervention; some devices and associated network stacks require the memory to be mapped to send and receive data. The memory handle includes all information required to access the memory locally using UCP routines, while remote registration handle provides an information that is necessary for remote memory access.

Memory mapping assumptions:

A given memory segment can be mapped by several different communication stacks, if these are compatible.
The memh_p handle returned may be used with any sub-region of the mapped memory.
If a large segment is registered, and then segmented for subsequent use by a user, then the user is responsible for segmentation and subsequent management.

parameter/flag	NONBLOCK	ALLOCATE	FIXED	address	result
value	0/1 - the value only affects the register/map phase	0	0	0	error if length > 0
		1	0	0	alloc+register
		0	1	0	error
		0	0	defined	register
		1	1	0	error
		1	0	defined	alloc+register,hint
		0	1	defined	error
		1	1	defined	alloc+register,fixed

Parameters

context – [in] Application context to map (register) and allocate the memory on.
params – [in] User defined ucp_mem_map_params_t configurations for the UCP memory handle.
memh_p – [out] UCP handle for the allocated segment.

Returns

Error code as defined by ucs_status_t

Note

Another well know terminology for the “map” operation that is typically used in the context of networking is memory “registration” or “pinning”. The UCP library registers the memory the available hardware so it can be assessed directly by the hardware.

Note

register means that the memory will be registered in corresponding transports for RMA/AMO operations. This case intends that the memory was allocated by user before.
alloc+register means that the memory will be allocated in the memory provided by the system and registered in corresponding transports for RMA/AMO operations.
alloc+register,hint means that the memory will be allocated with using ucp_mem_map_params::address as a hint and registered in corresponding transports for RMA/AMO operations.
alloc+register,fixed means that the memory will be allocated and registered in corresponding transports for RMA/AMO operations.
error is an erroneous combination of the parameters.

ucs_status_t ucp_mem_unmap(ucp_context_h context, ucp_mem_h memh)¶

Unmap memory segment.

This routine unmaps a user specified memory segment, that was previously mapped using the ucp_mem_map() routine. The unmap routine will also release the resources associated with the memory handle. When the function returns, the ucp_mem_h and associated remote key will be invalid and cannot be used with any UCP routine.

Error cases:

Once memory is unmapped a network access to the region may cause a failure.

Parameters

context – [in] Application context which was used to allocate/map the memory.
memh – [in] Handle to memory region.

Returns

Error code as defined by ucs_status_t

Note

Another well know terminology for the “unmap” operation that is typically used in the context of networking is memory “de-registration”. The UCP library de-registers the memory the available hardware so it can be returned back to the operation system.

ucs_status_t ucp_mem_query(const ucp_mem_h memh, ucp_mem_attr_t *attr)¶

query mapped memory segment

This routine returns address and length of memory segment mapped with ucp_mem_map() routine.

Parameters

memh – [in] Handle to memory region.
attr – [out] Filled with attributes of the UCP memory handle.

Returns

Error code as defined by ucs_status_t

void ucp_mem_print_info(const char *mem_spec, ucp_context_h context, FILE *stream)¶

Print memory mapping information.

This routine maps memory and prints information about the created memory handle: including the mapped memory length, the allocation method, and other useful information associated with the memory handle.

Parameters

mem_spec – [in] Size and optional type of the memory to map. The format of the string is: “<size>[,<type>]”. For example:
- ”32768” : allocate 32 kilobytes of host memory.
- ”1m,cuda” : allocate 1 megabyte of cuda memory.
context – [in] The context on which the memory is mapped.
stream – [in] Output stream on which to print the information.

ucs_status_t ucp_mem_advise(ucp_context_h context, ucp_mem_h memh, ucp_mem_advise_params_t *params)¶

give advice about the use of memory

This routine advises the UCP about how to handle memory range beginning at address and size of length bytes. This call does not influence the semantics of the application, but may influence its performance. The UCP may ignore the advice.

Parameters

context – [in] Application context which was used to allocate/map the memory.
memh – [in] Handle to memory region.
params – [in] Memory base address and length. The advice field is used to pass memory use advice as defined in the ucp_mem_advice list The memory range must belong to the memh

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_memh_pack(ucp_mem_h memh, const ucp_memh_pack_params_t *params, void **buffer_p, size_t *buffer_size_p)¶

Pack a memory handle to a buffer specified by the user.

This routine allocates a memory buffer and packs a memory handle into the buffer. A packed memory key is an opaque object that provides the information that is necessary for a peer. This routine packs the memory handle in a portable format such that the object can be unpacked on any platform supported by the UCP library, e.g. if the memory handle was packed as a remote memory key (RKEY), it should be unpacked by ucp_ep_rkey_unpack(). In order to release the memory buffer allocated by this routine, the application is responsible for calling the ucp_memh_buffer_release() routine.

Parameters

memh – [in] Handle to memory region.
params – [in] Memory handle packing parameters, as defined by ucp_memh_pack_params_t.
buffer_p – [out] Memory buffer allocated by the library. The buffer contains the packed memory handle.
buffer_size_p – [out] Size (in bytes) of the buffer which contains packed memory handle.

Returns

Error code as defined by ucs_status_t

Note

RKEYs for InfiniBand and Cray Aries networks typically includes InfiniBand and Aries key.
In order to enable remote direct memory access to the memory associated with the memory handle the application is responsible for sharing the RKEY with the peers that will initiate the access.

void ucp_memh_buffer_release(void *buffer, const ucp_memh_buffer_release_params_t *params)¶

Release packed memory handle buffer.

This routine releases the buffer that was allocated using ucp_memh_pack().

Parameters

buffer – [in] Buffer to release.
params – [in] Memory handle buffer release parameters, as defined by ucp_memh_buffer_release_params_t.

Warning

Once memory is released, an access to the memory may cause undefined behavior.
If the input memory address was not allocated using ucp_memh_pack() routine, the behavior of this routine is undefined.

ucs_status_t ucp_ep_rkey_unpack(ucp_ep_h ep, const void *rkey_buffer, ucp_rkey_h *rkey_p)¶

Create remote access key from packed buffer.

This routine unpacks the remote key (RKEY) object into the local memory such that it can be accessed and used by UCP routines. The RKEY object has to be packed using the ucp_rkey_pack() routine. Application code should not make any changes to the content of the RKEY buffer.

Parameters

ep – [in] Endpoint to access using the remote key.
rkey_buffer – [in] Packed rkey.
rkey_p – [out] Remote key handle.

Returns

Error code as defined by ucs_status_t

Note

The application is responsible for releasing the RKEY object when it is no longer needed, by calling the ucp_rkey_destroy() routine.

Note

The remote key object can be used for communications only on the endpoint on which it was unpacked.

ucs_status_t ucp_rkey_ptr(ucp_rkey_h rkey, uint64_t raddr, void **addr_p)¶

Get a local pointer to remote memory.

This routine returns a local pointer to the remote memory described by the rkey.

Parameters

rkey – [in] A remote key handle.
raddr – [in] A remote memory address within the memory area described by the rkey.
addr_p – [out] A pointer that can be used for direct access to the remote memory.

Returns

Error code as defined by ucs_status_t if the remote memory cannot be accessed directly or the remote memory address is not valid.

Note

This routine can return a valid pointer only for the endpoints that are reachable via shared memory.

void ucp_rkey_destroy(ucp_rkey_h rkey)¶

Destroy the remote key.

This routine destroys the RKEY object and the memory that was allocated using the ucp_ep_rkey_unpack() routine. This routine also releases any resources that are associated with the RKEY object.

Parameters: rkey – [in] Remote key to destroy.

Warning

Once the RKEY object is released an access to the memory will cause an undefined failure.
If the RKEY object was not created using ucp_ep_rkey_unpack() routine the behavior of this routine is undefined.
The RKEY object must be destroyed after all outstanding operations which are using it are flushed, and before the endpoint on which it was unpacked is destroyed.

ucs_status_t ucp_rkey_pack(ucp_context_h context, ucp_mem_h memh, void **rkey_buffer_p, size_t *size_p)¶

Pack memory region remote access key.

Deprecated:: Replaced by ucp_memh_pack().

This routine allocates a memory buffer and packs a remote access key (RKEY) object into it. RKEY is an opaque object that provides the information that is necessary for remote memory access. This routine packs the RKEY object in a portable format such that the object can be unpacked on any platform supported by the UCP library. In order to release the memory buffer allocated by this routine, the application is responsible for calling the ucp_rkey_buffer_release() routine.

Parameters

context – [in] Application context which was used to allocate/map the memory.
memh – [in] Handle to the memory region.
rkey_buffer_p – [out] Memory buffer allocated by the library. The buffer contains the packed RKEY.
size_p – [out] Size (in bytes) of the packed RKEY.

Returns

Error code as defined by ucs_status_t

Note

RKEYs for InfiniBand and Cray Aries networks typically include the InfiniBand and Aries key.
In order to enable remote direct memory access to the memory associated with the memory handle, the application is responsible for sharing the RKEY with the peers that will initiate the access.

void ucp_rkey_buffer_release(void *rkey_buffer)¶

Release packed remote key buffer.

Deprecated:: Replaced by ucp_memh_buffer_release().

This routine releases the buffer that was allocated using ucp_rkey_pack().

Parameters: rkey_buffer – [in] Buffer to release.

Warning

Once memory is released, an access to the memory may cause undefined behavior.
If the input memory address was not allocated using ucp_rkey_pack() routine, the behavior of this routine is undefined.

struct ucp_rkey_compare_params¶

#include <ucp.h>

Tuning parameters for the comparison function ucp_rkey_compare.

The structure defines the parameters that can be used for UCP library remote keys comparison using ucp_rkey_compare routine.

struct ucp_mem_map_params¶

#include <ucp.h>

Tuning parameters for the UCP memory mapping.

The structure defines the parameters that are used for the UCP memory mapping tuning during the ucp_mem_map routine.

struct ucp_mem_advise_params¶

#include <ucp.h>

Tuning parameters for the UCP memory advice.

This structure defines the parameters that are used for the UCP memory advice tuning during the ucp_mem_advise routine.

struct ucp_memh_pack_params¶

#include <ucp.h>

Memory handle pack parameters passed to ucp_memh_pack.

This structure defines the parameters that are used for packing the UCP memory handle during the ucp_memh_pack routine.

struct ucp_memh_buffer_release_params¶

#include <ucp.h>

Memory handle release parameters passed to ucp_memh_buffer_release.

This structure defines the parameters that are used for releasing the UCP memory handle buffer during the ucp_memh_buffer_release routine.

struct ucp_mem_attr¶: #include <ucp_def.h>
Attributes of the UCP Memory handle, filled by ucp_mem_query function.

Configuration routines ¶

group UCP_CONFIG

This section describes routines for configuration of the UCP network layer

Typedefs

typedef struct ucp_params ucp_params_t¶

Tuning parameters for UCP library.

The structure defines the parameters that are used for UCP library tuning during UCP library initialization.

Note

UCP library implementation uses the features parameter to optimize the library functionality that minimize memory footprint. For example, if the application does not require send/receive semantics UCP library may avoid allocation of expensive resources associated with send/receive queues.

typedef struct ucp_config ucp_config_t¶

UCP configuration descriptor.

This descriptor defines the configuration for UCP application context. The configuration is loaded from the run-time environment (using configuration files of environment variables) using ucp_config_read routine and can be printed using ucp_config_print routine. In addition, application is responsible to release the descriptor using ucp_config_release routine.

Functions

ucs_status_t ucp_config_read(const char *env_prefix, const char *filename, ucp_config_t **config_p)¶

Read UCP configuration descriptor.

The routine fetches the information about UCP library configuration from the run-time environment. Then, the fetched descriptor is used for UCP library initialization. The Application can print out the descriptor using print routine. In addition the application is responsible for releasing the descriptor back to the UCP library.

Parameters

env_prefix – [in] If non-NULL, the routine searches for the environment variables that start with <env_prefix> prefix. Otherwise, the routine searches for the environment variables that start with UCX_ prefix.
filename – [in] If non-NULL, read configuration from the file defined by filename. If the file does not exist, it will be ignored and no error reported to the application.
config_p – [out] Pointer to configuration descriptor as defined by ucp_config_t.

Returns

Error code as defined by ucs_status_t

void ucp_config_release(ucp_config_t *config)¶

Release configuration descriptor.

The routine releases the configuration descriptor that was allocated through ucp_config_read() routine.

Parameters: config – [out] Configuration descriptor as defined by ucp_config_t.

ucs_status_t ucp_config_modify(ucp_config_t *config, const char *name, const char *value)¶

Modify context configuration.

The routine changes one configuration setting stored in configuration descriptor.

Parameters

config – [in] Configuration to modify.
name – [in] Configuration variable name.
value – [in] Value to set.

Returns

Error code.

void ucp_config_print(const ucp_config_t *config, FILE *stream, const char *title, ucs_config_print_flags_t print_flags)¶

Print configuration information.

The routine prints the configuration information that is stored in configuration descriptor.

Parameters

config – [in] Configuration descriptor to print.
stream – [in] Output stream to print the configuration to.
title – [in] Configuration title to print.
print_flags – [in] Flags that control various printing options.

struct ucp_params¶

#include <ucp.h>

Tuning parameters for UCP library.

The structure defines the parameters that are used for UCP library tuning during UCP library initialization.

Note

UCP library implementation uses the features parameter to optimize the library functionality that minimize memory footprint. For example, if the application does not require send/receive semantics UCP library may avoid allocation of expensive resources associated with send/receive queues.

Communication routines ¶

group UCP_COMM

UCP Communication routines

Typedefs

typedef uint64_t ucp_tag_t¶

UCP Tag Identifier.

UCP tag identifier is a 64bit object used for message identification. UCP tag send and receive operations use the object for an implementation tag matching semantics (derivative of MPI tag matching semantics).

typedef struct ucp_recv_desc *ucp_tag_message_h¶

UCP Message descriptor.

UCP Message descriptor is an opaque handle for a message returned by ucp_tag_probe_nb. This handle can be passed to ucp_tag_msg_recv_nb in order to receive the message data to a specific buffer.

typedef uint64_t ucp_datatype_t¶

UCP Datatype Identifier.

UCP datatype identifier is a 64bit object used for datatype identification. Predefined UCP identifiers are defined by ucp_dt_type.

typedef void (*ucp_send_callback_t)(void *request, ucs_status_t status)¶

Completion callback for non-blocking sends.

This callback routine is invoked whenever the send operation is completed. It is important to note that the call-back is only invoked in a case when the operation cannot be completed in place.

Parameters

request – [in] The completed send request.
status – [in] Completion status. If the send operation was completed successfully UCS_OK is returned. If send operation was canceled UCS_ERR_CANCELED is returned. Otherwise, an error status is returned.

typedef void (*ucp_send_nbx_callback_t)(void *request, ucs_status_t status, void *user_data)¶

Completion callback for non-blocking sends.

This callback routine is invoked whenever the ucp_tag_send_nbx, ucp_am_send_nbx, ucp_stream_send_nbx, ucp_put_nbx, ucp_get_nbx, ucp_atomic_op_nbx or any other “send operation” is completed.

Parameters

request – [in] The completed send request.
status – [in] Completion status. If the send operation was completed successfully UCS_OK is returned. If send operation was canceled UCS_ERR_CANCELED is returned. Otherwise, an error status is returned.
user_data – [in] User data passed to “user_data” value, see ucp_request_param_t

typedef void (*ucp_err_handler_cb_t)(void *arg, ucp_ep_h ep, ucs_status_t status)¶

Callback to process peer failure.

This callback routine is invoked when transport level error detected.

Parameters

arg – [in] User argument to be passed to the callback.
ep – [in] Endpoint to handle transport level error. Upon return from the callback, this ep is no longer usable and all subsequent operations on this ep will fail with the error code passed in status.
status – [in] error status.

typedef struct ucp_err_handler ucp_err_handler_t¶

UCP endpoint error handling context.

This structure should be initialized in ucp_ep_params_t to handle peer failure

typedef void (*ucp_stream_recv_callback_t)(void *request, ucs_status_t status, size_t length)¶

Completion callback for non-blocking stream oriented receives.

This callback routine is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Parameters

request – [in] The completed receive request.
status – [in] Completion status. If the send operation was completed successfully UCS_OK is returned. Otherwise, an error status is returned.
length – [in] The size of the received data in bytes, always boundary of base datatype size. The value is valid only if the status is UCS_OK.

typedef void (*ucp_stream_recv_nbx_callback_t)(void *request, ucs_status_t status, size_t length, void *user_data)¶

Completion callback for non-blocking stream receives ucp_stream_recv_nbx call.

This callback routine is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Parameters

request – [in] The completed receive request.
status – [in] Completion status. If the send operation was completed successfully UCS_OK is returned. Otherwise, an error status is returned.
length – [in] The size of the received data in bytes, always on the boundary of base datatype size. The value is valid only if the status is UCS_OK.
user_data – [in] User data passed to “user_data” value, see ucp_request_param_t.

typedef void (*ucp_tag_recv_callback_t)(void *request, ucs_status_t status, ucp_tag_recv_info_t *info)¶

Completion callback for non-blocking tag receives.

This callback routine is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Parameters

request – [in] The completed receive request.
status – [in] Completion status. If the send operation was completed successfully UCS_OK is returned. If send operation was canceled UCS_ERR_CANCELED is returned. If the data can not fit into the receive buffer the UCS_ERR_MESSAGE_TRUNCATED error code is returned. Otherwise, an error status is returned.
info – [in] Completion information The info descriptor is Valid only if the status is UCS_OK.

typedef void (*ucp_tag_recv_nbx_callback_t)(void *request, ucs_status_t status, const ucp_tag_recv_info_t *tag_info, void *user_data)¶

Completion callback for non-blocking tag receives ucp_tag_recv_nbx call.

This callback routine is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Parameters

request – [in] The completed receive request.
status – [in] Completion status. If the receive operation was completed successfully UCS_OK is returned. If send operation was canceled, UCS_ERR_CANCELED is returned. If the data can not fit into the receive buffer the UCS_ERR_MESSAGE_TRUNCATED error code is returned. Otherwise, an error status is returned.
tag_info – [in] Completion information The info descriptor is Valid only if the status is UCS_OK.
user_data – [in] User data passed to “user_data” value, see ucp_request_param_t

typedef void (*ucp_am_recv_data_nbx_callback_t)(void *request, ucs_status_t status, size_t length, void *user_data)¶

Completion callback for non-blocking Active Message receives.

This callback routine is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Parameters

request – [in] The completed receive request.
status – [in] Completion status. If the receive operation was completed successfully UCS_OK is returned. Otherwise, an error status is returned.
length – [in] The size of the received data in bytes, always boundary of base datatype size. The value is valid only if the status is UCS_OK.
user_data – [in] User data passed to “user_data” value, see ucp_request_param_t

Enums

enum ucp_atomic_op_t¶

Atomic operation requested for ucp_atomic_op_nbx.

This enumeration defines which atomic memory operation should be performed by the ucp_atomic_op_nbx routine.

Values:

enumerator UCP_ATOMIC_OP_ADD¶: Atomic add

enumerator UCP_ATOMIC_OP_SWAP¶: Atomic swap

enumerator UCP_ATOMIC_OP_CSWAP¶: Atomic conditional swap

enumerator UCP_ATOMIC_OP_AND¶: Atomic and

enumerator UCP_ATOMIC_OP_OR¶: Atomic or

enumerator UCP_ATOMIC_OP_XOR¶: Atomic xor

enumerator UCP_ATOMIC_OP_LAST¶

enum ucp_stream_recv_flags_t¶

Flags to define behavior of ucp_stream_recv_nb function.

This enumeration defines behavior of ucp_stream_recv_nb function.

Values:

enumerator UCP_STREAM_RECV_FLAG_WAITALL¶: This flag requests that the operation will not be completed until all requested data is received and placed in the user buffer.

enum ucp_op_attr_t¶

UCP operation fields and flags.

The enumeration allows specifying which fields in ucp_request_param_t are present and operation flags are used. It is used to enable backward compatibility support.

Values:

enumerator UCP_OP_ATTR_FIELD_REQUEST¶: request field

enumerator UCP_OP_ATTR_FIELD_CALLBACK¶: cb field

enumerator UCP_OP_ATTR_FIELD_USER_DATA¶: user_data field

enumerator UCP_OP_ATTR_FIELD_DATATYPE¶: datatype field

enumerator UCP_OP_ATTR_FIELD_FLAGS¶: operation-specific flags

enumerator UCP_OP_ATTR_FIELD_REPLY_BUFFER¶: reply_buffer field

enumerator UCP_OP_ATTR_FIELD_MEMORY_TYPE¶: memory type field

enumerator UCP_OP_ATTR_FIELD_RECV_INFO¶: recv_info field

enumerator UCP_OP_ATTR_FIELD_MEMH¶: memory handle field

enumerator UCP_OP_ATTR_FLAG_NO_IMM_CMPL¶: Deny immediate completion, i.e NULL cannot be returned. If a completion callback is provided, it can be called before the function returns.

enumerator UCP_OP_ATTR_FLAG_FAST_CMPL¶: expedite local completion, even if it delays remote data delivery. Note for implementer: this option can disable zero copy and/or rendezvous protocols which require synchronization with the remote peer before releasing the local send buffer

enumerator UCP_OP_ATTR_FLAG_FORCE_IMM_CMPL¶: force immediate complete operation, fail if the operation cannot be completed immediately

enumerator UCP_OP_ATTR_FLAG_MULTI_SEND¶: optimize for bandwidth of multiple in-flight operations, rather than for the latency of a single operation. This flag and UCP_OP_ATTR_FLAG_FAST_CMPL are mutually exclusive.

enum ucp_req_attr_field¶

UCP request query attributes.

The enumeration allows specifying which fields in ucp_request_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_REQUEST_ATTR_FIELD_INFO_STRING¶

enumerator UCP_REQUEST_ATTR_FIELD_INFO_STRING_SIZE¶

enumerator UCP_REQUEST_ATTR_FIELD_STATUS¶

enumerator UCP_REQUEST_ATTR_FIELD_MEM_TYPE¶

enum ucp_am_recv_attr_t¶

UCP AM receive data parameter fields and flags.

The enumeration allows specifying which fields in ucp_am_recv_param_t are present and receive operation flags are used. It is used to enable backward compatibility support.

Values:

enumerator UCP_AM_RECV_ATTR_FIELD_REPLY_EP¶: reply_ep field

enumerator UCP_AM_RECV_ATTR_FLAG_DATA¶: Indicates that the data provided in ucp_am_recv_callback_t callback can be held by the user. If UCS_INPROGRESS is returned from the callback, the data parameter will persist and the user has to call ucp_am_data_release when data is no longer needed. This flag is mutually exclusive with UCP_AM_RECV_ATTR_FLAG_RNDV.

enumerator UCP_AM_RECV_ATTR_FLAG_RNDV¶: Indicates that the arriving data was sent using rendezvous protocol. In this case data parameter of the ucp_am_recv_callback_t points to the internal UCP descriptor, which can be used for obtaining the actual data by calling ucp_am_recv_data_nbx routine. This flag is mutually exclusive with UCP_AM_RECV_ATTR_FLAG_DATA.

enum ucp_am_handler_param_field¶

UCP AM receive data parameters fields and flags.

The enumeration allows specifying which fields in ucp_am_handler_param_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_AM_HANDLER_PARAM_FIELD_ID¶: Indicates that ucp_am_handler_param_t::id field is valid.

enumerator UCP_AM_HANDLER_PARAM_FIELD_FLAGS¶: Indicates that ucp_am_handler_param_t::flags field is valid.

enumerator UCP_AM_HANDLER_PARAM_FIELD_CB¶: Indicates that ucp_am_handler_param_t::cb field is valid.

enumerator UCP_AM_HANDLER_PARAM_FIELD_ARG¶: Indicates that ucp_am_handler_param_t::arg field is valid.

enum ucp_atomic_post_op_t¶

Atomic operation requested for ucp_atomic_post.

Deprecated:: Use ucp_atomic_op_nbx and ucp_atomic_op_t instead.

This enumeration defines which atomic memory operation should be performed by the ucp_atomic_post family of functions. All of these are non-fetching atomics and will not result in a request handle.

Values:

enumerator UCP_ATOMIC_POST_OP_ADD¶: Atomic add

enumerator UCP_ATOMIC_POST_OP_AND¶: Atomic and

enumerator UCP_ATOMIC_POST_OP_OR¶: Atomic or

enumerator UCP_ATOMIC_POST_OP_XOR¶: Atomic xor

enumerator UCP_ATOMIC_POST_OP_LAST¶

enum ucp_atomic_fetch_op_t¶

Atomic operation requested for ucp_atomic_fetch.

Deprecated:: Use ucp_atomic_op_nbx and ucp_atomic_op_t instead.

This enumeration defines which atomic memory operation should be performed by the ucp_atomic_fetch family of functions. All of these functions will fetch data from the remote node.

Values:

enumerator UCP_ATOMIC_FETCH_OP_FADD¶: Atomic Fetch and add

enumerator UCP_ATOMIC_FETCH_OP_SWAP¶: Atomic swap

enumerator UCP_ATOMIC_FETCH_OP_CSWAP¶: Atomic conditional swap

enumerator UCP_ATOMIC_FETCH_OP_FAND¶: Atomic Fetch and and

enumerator UCP_ATOMIC_FETCH_OP_FOR¶: Atomic Fetch and or

enumerator UCP_ATOMIC_FETCH_OP_FXOR¶: Atomic Fetch and xor

enumerator UCP_ATOMIC_FETCH_OP_LAST¶

Functions

ucs_status_t ucp_request_query(void *request, ucp_request_attr_t *attr)¶

Get information about ucp_request.

Parameters

request – [in] Non-blocking request to query.
attr – [out] Filled with attributes of the request.

Returns

Error code as defined by ucs_status_t

ucs_status_ptr_t ucp_am_send_nbx(ucp_ep_h ep, unsigned id, const void *header, size_t header_length, const void *buffer, size_t count, const ucp_request_param_t *param)¶

Send Active Message.

This routine sends an Active Message to an ep. If the operation completes immediately, then the routine returns NULL and the callback function is ignored, even if specified. Otherwise, if no error is reported and a callback is requested (i.e. the UCP_OP_ATTR_FIELD_CALLBACK flag is set in the op_attr_mask field of param), then the UCP library will schedule invocation of the callback routine param->cb.send upon completion of the operation.

Parameters

ep – [in] UCP endpoint where the Active Message will be run.
id – [in] Active Message id. Specifies which registered callback to run.
header – [in] User defined Active Message header. NULL value is allowed if no header needed. In this case header_length must be set to 0. By default the header must be valid until the active message send operation completes. If the flag UCP_AM_SEND_FLAG_COPY_HEADER is specified, the header is only required to be valid until this function call returns.
header_length – [in] Active message header length in bytes.
buffer – [in] Pointer to the data to be sent to the target node of the Active Message.
count – [in] Number of elements to send.
param – [in] Operation parameters, see ucp_request_param_t.

Returns

NULL - Active Message was sent immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - Error sending Active Message.

Returns

otherwise - Operation was scheduled for send and can be completed at any point in time. The request handle is returned to the application in order to track progress of the message. If user request was not provided in param->request, the application is responsible for releasing the handle using ucp_request_free routine.

Note

If UCP_OP_ATTR_FLAG_NO_IMM_CMPL flag is set in the op_attr_mask field of param, then the operation will return a request handle, even if it completes immediately.

Note

This operation supports specific flags, which can be passed in param by ucp_request_param_t::flags. The exact set of flags is defined by ucp_send_am_flags.

Note

Sending only header without actual data is allowed and is recommended for transferring a latency-critical amount of data.

Note

The maximum allowed header size can be obtained by querying worker attributes by the ucp_worker_query routine.

ucs_status_ptr_t ucp_am_recv_data_nbx(ucp_worker_h worker, void *data_desc, void *buffer, size_t count, const ucp_request_param_t *param)¶

Receive Active Message as defined by provided data descriptor.

This routine receives a message that is described by the data descriptor data_desc, local address buffer, size count and param parameters on the worker. The routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. If the receive operation cannot be started the routine returns an error.

Parameters

worker – [in] Worker that is used for the receive operation.
data_desc – [in] Data descriptor, provided in ucp_am_recv_callback_t routine.
buffer – [in] Pointer to the buffer to receive the data.
count – [in] Number of elements to receive into buffer.
param – [in] Operation parameters, see ucp_request_param_t.

Returns

NULL - The receive operation was completed immediately. In this case, if param->recv_info.length is specified in the param, the value to which it points is updated with the size of the received message.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Receive operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track operation progress. If user request was not provided in param->request, the application is responsible for releasing the handle using ucp_request_free routine.

Note

This routine can be performed on any valid data descriptor delivered in ucp_am_recv_callback_t. Data descriptor is considered to be valid if:

It is a rendezvous request (UCP_AM_RECV_ATTR_FLAG_RNDV is set in ucp_am_recv_param_t::recv_attr) or
It is a persistent data pointer (UCP_AM_RECV_ATTR_FLAG_DATA is set in ucp_am_recv_param_t::recv_attr). In this case receive operation may be needed to unpack data to device memory (for example GPU device) or some specific datatype.

Note

After this call UCP takes ownership of data_desc descriptor, so there is no need to release it even if the operation fails. The routine returns a request handle instead, which can be used for tracking operation progress.

void ucp_am_data_release(ucp_worker_h worker, void *data)¶

Releases Active Message data.

This routine releases data that persisted through an Active Message callback because that callback returned UCS_INPROGRESS.

Parameters

worker – [in] Worker which received the Active Message.
data – [in] Pointer to data that was passed into the Active Message callback as the data parameter.

ucs_status_ptr_t ucp_stream_send_nbx(ucp_ep_h ep, const void *buffer, size_t count, const ucp_request_param_t *param)¶

Non-blocking stream send operation.

This routine sends data that is described by the local address buffer, size count object to the destination endpoint ep. The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine returns UCS_OK.

Parameters

ep – [in] Destination endpoint handle.
buffer – [in] Pointer to the message buffer (payload).
count – [in] Number of elements to send.
param – [in] Operation parameters, see ucp_request_param_t.

Returns

NULL - The send operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed at any point in time. The request handle is returned to the application in order to track progress of the message.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_ptr_t ucp_tag_send_nbx(ucp_ep_h ep, const void *buffer, size_t count, ucp_tag_t tag, const ucp_request_param_t *param)¶

Non-blocking tagged-send operation.

This routine sends a messages that is described by the local address buffer, size count object to the destination endpoint ep. Each message is associated with a tag value that is used for message matching on the ucp_tag_recv_nb or receiver. The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine returns UCS_OK and the call-back function is not invoked. If the operation is not completed immediately and no error reported then the UCP library will schedule to invoke the call-back whenever the send operation is completed. In other words, the completion of a message can be signaled by the return code or the call-back. Immediate completion signals can be fine-tuned via the ucp_request_param_t::op_attr_mask field in the ucp_request_param_t structure. The values of this field are a bit-wise OR of the ucp_op_attr_t enumeration.

Parameters

ep – [in] Destination endpoint handle.
buffer – [in] Pointer to the message buffer (payload).
count – [in] Number of elements to send
tag – [in] Message tag.
param – [in] Operation parameters, see ucp_request_param_t

Returns

UCS_OK - The send operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_ptr_t ucp_tag_send_sync_nbx(ucp_ep_h ep, const void *buffer, size_t count, ucp_tag_t tag, const ucp_request_param_t *param)¶

Non-blocking synchronous tagged-send operation.

Same as ucp_tag_send_nbx, except the request completes only after there is a remote tag match on the message (which does not always mean the remote receive has been completed). This function never completes “in-place”, and always returns a request handle.

Parameters

ep – [in] Destination endpoint handle.
buffer – [in] Pointer to the message buffer (payload).
count – [in] Number of elements to send
tag – [in] Message tag.
param – [in] Operation parameters, see ucp_request_param_t

Returns

UCS_OK - The send operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

Note

Returns UCS_ERR_UNSUPPORTED if UCP_ERR_HANDLING_MODE_PEER is enabled. This is a temporary implementation-related constraint that will be addressed in future releases.

ucs_status_ptr_t ucp_stream_recv_nbx(ucp_ep_h ep, void *buffer, size_t count, size_t *length, const ucp_request_param_t *param)¶

Non-blocking stream receive operation of structured data into a user-supplied buffer.

This routine receives data that is described by the local address buffer, size count object on the endpoint ep. The routine is non-blocking and therefore returns immediately. The receive operation is considered complete when the message is delivered to the buffer. If the receive operation cannot be started, then the routine returns an error.

Parameters

ep – [in] UCP endpoint that is used for the receive operation.
buffer – [in] Pointer to the buffer that will receive the data.
count – [in] Number of elements to receive into buffer.
length – [out] Size of the received data in bytes. The value is valid only if return code is NULL.
param – [in] Operation parameters, see ucp_request_param_t. This operation supports specific flags, which can be passed in param by ucp_request_param_t::flags. The exact set of flags is defined by ucp_stream_recv_flags_t.

Returns

NULL - The receive operation was completed immediately. In this case the value pointed by length is updated by the size of received data. Note param->recv_info is not relevant for this function.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. A request handle is returned to the application in order to track progress of the operation.

Note

The amount of data received, in bytes, is always an integral multiple of the datatype size.

ucs_status_ptr_t ucp_stream_recv_data_nb(ucp_ep_h ep, size_t *length)¶

Non-blocking stream receive operation of unstructured data into a UCP-supplied buffer.

This routine receives any available data from endpoint ep. Unlike ucp_stream_recv_nb, the returned data is unstructured and is treated as an array of bytes. If data is immediately available, UCS_STATUS_PTR(_ptr) is returned as a pointer to the data, and length is set to the size of the returned data buffer. The routine is non-blocking and therefore returns immediately.

Parameters

ep – [in] UCP endpoint that is used for the receive operation.
length – [out] Length of received data.

Returns

NULL - No received data available on the ep.

Returns

UCS_PTR_IS_ERR(_ptr) - the receive operation failed and UCS_PTR_STATUS(_ptr) indicates an error.

Returns

otherwise - The pointer to the data UCS_STATUS_PTR(_ptr) is returned to the application. After the data is processed, the application is responsible for releasing the data buffer by calling the ucp_stream_data_release routine.

Note

This function returns packed data (equivalent to ucp_dt_make_contig(1)).

Note

This function returns a pointer to a UCP-supplied buffer, whereas ucp_stream_recv_nb places the data into a user-provided buffer. In some cases, receiving data directly into a UCP-supplied buffer can be more optimal, for example by processing the incoming data in-place and thus avoiding extra memory copy operations.

ucs_status_ptr_t ucp_tag_recv_nbx(ucp_worker_h worker, void *buffer, size_t count, ucp_tag_t tag, ucp_tag_t tag_mask, const ucp_request_param_t *param)¶

Non-blocking tagged-receive operation.

This routine receives a message that is described by the local address buffer, size count, and info object on the worker. The tag value of the receive message has to match the tag and tag_mask values, where the tag_mask indicates what bits of the tag have to be matched. The routine is a non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to notify the application about completion of the receive operation the UCP library will invoke the call-back cb when the received message is in the receive buffer and ready for application access. If the receive operation cannot be started, then the routine returns an error.

Parameters

worker – [in] UCP worker that is used for the receive operation.
buffer – [in] Pointer to the buffer to receive the data.
count – [in] Number of elements to receive
tag – [in] Message tag to expect.
tag_mask – [in] Bit mask that indicates the bits that are used for the matching of the incoming tag against the expected tag.
param – [in] Operation parameters, see ucp_request_param_t

Returns

NULL - The receive operation was completed immediately. In this case, if param->recv_info.tag_info is specified in the param, the value to which it points is updated with the information about the received message.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

ucp_tag_message_h ucp_tag_probe_nb(ucp_worker_h worker, ucp_tag_t tag, ucp_tag_t tag_mask, int remove, ucp_tag_recv_info_t *info)¶

Non-blocking probe and return a message.

This routine probes (checks) if a messages described by the tag and tag_mask was received (fully or partially) on the worker. The tag value of the received message has to match the tag and tag_mask values, where the tag_mask indicates what bits of the tag have to be matched. The function returns immediately and if the message is matched it returns a handle for the message.

Parameters

worker – [in] UCP worker that is used for the probe operation.
tag – [in] Message tag to probe for.
tag_mask – [in] Bit mask that indicates the bits that are used for the matching of the incoming tag against the expected tag.
remove – [in] The flag indicates if the matched message has to be removed from UCP library. If true (1), the message handle is removed from the UCP library and the application is responsible to call ucp_tag_msg_recv_nb() in order to receive the data and release the resources associated with the message handle. If false (0), the return value is merely an indication to whether a matching message is present, and it cannot be used in any other way, and in particular it cannot be passed to ucp_tag_msg_recv_nb().
info – [out] If the matching message is found the descriptor is filled with the details about the message.

Returns

NULL - No match found.

Returns

Message handle (not NULL) - If message is matched the message handle is returned.

Note

This function does not advance the communication state of the network. If this routine is used in busy-poll mode, need to make sure ucp_worker_progress() is called periodically to extract messages from the transport.

ucs_status_ptr_t ucp_tag_msg_recv_nbx(ucp_worker_h worker, void *buffer, size_t count, ucp_tag_message_h message, const ucp_request_param_t *param)¶

Non-blocking receive operation for a probed message.

This routine receives a message that is described by the local address buffer, size count, and message handle on the worker. The message handle can be obtained by calling the ucp_tag_probe_nb() routine. The ucp_tag_msg_recv_nbx() routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to notify the application about completion of the receive operation the UCP library will invoke the call-back cb when the received message is in the receive buffer and ready for application access. If the receive operation cannot be started, then the routine returns an error.

Parameters

worker – [in] UCP worker that is used for the receive operation.
buffer – [in] Pointer to the buffer that will receive the data.
count – [in] Number of elements to receive
message – [in] Message handle.
param – [in] Operation parameters, see ucp_request_param_t

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

ucs_status_ptr_t ucp_put_nbx(ucp_ep_h ep, const void *buffer, size_t count, uint64_t remote_addr, ucp_rkey_h rkey, const ucp_request_param_t *param)¶

Non-blocking remote memory put operation.

This routine initiates a storage of contiguous block of data that is described by the local address buffer in the remote contiguous memory region described by remote_addr address and the memory handle rkey. The routine returns immediately and does not guarantee re-usability of the source address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user. If the put operation completes immediately, the routine returns UCS_OK and the call-back routine param.cb.send is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine param.cb.send upon completion of the put operation. In other words, the completion of a put operation can be signaled by the return code or execution of the call-back. Immediate completion signals can be fine-tuned via the ucp_request_param_t::op_attr_mask field in the ucp_request_param_t structure. The values of this field are a bit-wise OR of the ucp_op_attr_t enumeration.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local source address.
count – [in] Number of elements of type ucp_request_param_t::datatype to put. If ucp_request_param_t::datatype is not specified, the type defaults to ucp_dt_make_contig(1), which corresponds to byte elements.
remote_addr – [in] Pointer to the destination remote memory address to write to.
rkey – [in] Remote memory key associated with the remote memory address.
param – [in] Operation parameters, see ucp_request_param_t

Returns

UCS_OK - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

The completion of a put operation signals the local buffer can be reused. The completion of the operation on the remote address requires use of ucp_worker_flush_nbx() or ucp_ep_flush_nbx(), after completion of which the data in remote_addr is guaranteed to be available.

Note

Only the datatype ucp_dt_make_contig(1) is supported for param->datatype, see ucp_dt_make_contig.

ucs_status_ptr_t ucp_get_nbx(ucp_ep_h ep, void *buffer, size_t count, uint64_t remote_addr, ucp_rkey_h rkey, const ucp_request_param_t *param)¶

Non-blocking remote memory get operation.

This routine initiates a load of a contiguous block of data that is described by the remote memory address remote_addr and the memory handle rkey in the local contiguous memory region described by buffer address. The routine returns immediately and does not guarantee that remote data is loaded and stored under the local address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user. If the get operation completes immediately, the routine returns UCS_OK and the call-back routine param.cb.send is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine param.cb.send upon completion of the get operation. In other words, the completion of a get operation can be signaled by the return code or execution of the call-back.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local destination address.
count – [in] Number of elements of type ucp_request_param_t::datatype to put. If ucp_request_param_t::datatype is not specified, the type defaults to ucp_dt_make_contig(1), which corresponds to byte elements.
remote_addr – [in] Pointer to the source remote memory address to read from.
rkey – [in] Remote memory key associated with the remote memory address.
param – [in] Operation parameters, see ucp_request_param_t.

Returns

UCS_OK - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

A user can use ucp_worker_flush_nb() in order to guarantee re-usability of the source address buffer.

Note

The completion of a get operation signals the local buffer holds the the expected data and that both local buffer and remote remote_addr are safe to be reused, unlike with ucp_put_nbx where the use of ucp_worker_flush_nbx() or ucp_ep_flush_nbx() is required before the remote data is available.

Note

Only the datatype ucp_dt_make_contig(1) is supported for param->datatype, see ucp_dt_make_contig.

ucs_status_ptr_t ucp_atomic_op_nbx(ucp_ep_h ep, ucp_atomic_op_t opcode, const void *buffer, size_t count, uint64_t remote_addr, ucp_rkey_h rkey, const ucp_request_param_t *param)¶

Post an atomic memory operation.

This routine will post an atomic operation to remote memory. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The routine is non-blocking and therefore returns immediately. However, the actual atomic operation may be delayed. In order to enable fetching semantics for atomic operations user has to specify param.reply_buffer. Please see table below for more details.

Atomic Operation	Pseudo code	X	Y	Z	Result
UCP_ATOMIC_OP_ADD	Result=Y; Y+=X	buffer	remote_addr	-	param.reply_buffer(optional)
UCP_ATOMIC_OP_SWAP	Result=Y; Y=X	buffer	remote_addr	-	param.reply_buffer
UCP_ATOMIC_OP_CSWAP	Result=Y; if (X==Y) then Y=Z	buffer	remote_addr	param.reply_buffer	param.reply_buffer
UCP_ATOMIC_OP_AND	Result=Y; Y&=X	buffer	remote_addr	-	param.reply_buffer(optional)
UCP_ATOMIC_OP_OR	Result=Y; Y\|=X	buffer	remote_addr	-	param.reply_buffer(optional)
UCP_ATOMIC_OP_XOR	Result=Y; Y^=X	buffer	remote_addr	-	param.reply_buffer(optional)

Parameters

ep – [in] UCP endpoint.
opcode – [in] One of ucp_atomic_op_t.
buffer – [in] Address of operand for the atomic operation. See Atomic Operations Semantic table for exact usage by different atomic operations.
count – [in] Number of elements in buffer and result. The size of each element is specified by ucp_request_param_t::datatype
remote_addr – [in] Remote address to operate on.
rkey – [in] Remote key handle for the remote memory address.
param – [in] Operation parameters, see ucp_request_param_t.

Returns

NULL - The operation completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at some time in the future. The request handle is returned to the application in order to track progress of the operation.

Note

The user should not modify any part of the buffer (or also param->reply_buffer for fetch operations), until the operation completes.

Note

Only ucp_dt_make_config(4) and ucp_dt_make_contig(8) are supported in param->datatype, see ucp_dt_make_contig. Also, currently atomic operations can handle one element only. Thus, count argument must be set to 1.

ucs_status_t ucp_request_check_status(void *request)¶

Check the status of non-blocking request.

This routine checks the state of the request and returns its current status. Any value different from UCS_INPROGRESS means that request is in a completed state.

Parameters: request – [in] Non-blocking request to check.
Returns: Error code as defined by ucs_status_t

ucs_status_t ucp_tag_recv_request_test(void *request, ucp_tag_recv_info_t *info)¶

Check the status and currently available state of non-blocking request returned from ucp_tag_recv_nb routine.

This routine checks the state and returns current status of the request returned from ucp_tag_recv_nb routine or the user allocated request for ucp_tag_recv_nbr. Any value different from UCS_INPROGRESS means that the request is in a completed state.

Parameters

request – [in] Non-blocking request to check.
info – [out] It is filled with the details about the message available at the moment of calling.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_stream_recv_request_test(void *request, size_t *length_p)¶

Check the status and currently available state of non-blocking request returned from ucp_stream_recv_nb routine.

This routine checks the state and returns current status of the request returned from ucp_stream_recv_nb routine. Any value different from UCS_INPROGRESS means that the request is in a completed state.

Parameters

request – [in] Non-blocking request to check.
length_p – [out] The size of the received data in bytes. This value is only valid if the status is UCS_OK. If valid, it is always an integral multiple of the datatype size associated with the request.

Returns

Error code as defined by ucs_status_t

void ucp_request_cancel(ucp_worker_h worker, void *request)¶

Cancel an outstanding communications request.

This routine tries to cancels an outstanding communication request. After calling this routine, the request will be in completed or canceled (but not both) state regardless of the status of the target endpoint associated with the communication request. If the request is completed successfully, the send or receive completion callbacks (based on the type of the request) will be called with the status argument of the callback set to UCS_OK, and in a case it is canceled the status argument is set to UCS_ERR_CANCELED. It is important to note that in order to release the request back to the library the application is responsible for calling ucp_request_free().

Parameters

worker – [in] UCP worker.
request – [in] Non-blocking request to cancel.

void ucp_stream_data_release(ucp_ep_h ep, void *data)¶

Release UCP data buffer returned by ucp_stream_recv_data_nb.

This routine releases internal UCP data buffer returned by ucp_stream_recv_data_nb when data is processed, the application can’t use this buffer after calling this function.

Parameters

ep – [in] Endpoint data received from.
data – [in] Data pointer to release, which was returned from ucp_stream_recv_data_nb.

void ucp_request_free(void *request)¶

Release a communications request.

This routine releases the non-blocking request back to the library, regardless of its current state. Communications operations associated with this request will make progress internally, however no further notifications or callbacks will be invoked for this request.

Parameters: request – [in] Non-blocking request to release.

void *ucp_request_alloc(ucp_worker_h worker)¶

Create an empty communications request.

This routine creates request which may be used in functions ucp_tag_send_nbx, ucp_tag_recv_nbx, etc. The application is responsible for releasing the handle using the ucp_request_free routine

Parameters: worker – [in] UCP worker.
Returns: Error code as defined by ucs_status_t

int ucp_request_is_completed(void *request)¶

Deprecated:: Replaced by ucp_request_test.

ucs_status_t ucp_put(ucp_ep_h ep, const void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey)¶

Blocking remote memory put operation.

Deprecated:: Replaced by ucp_put_nb. The following example implements the same functionality using ucp_put_nb :

void empty_callback(void *request, ucs_status_t status)
{
}

ucs_status_t put(ucp_ep_h ep, const void *buffer, size_t length,
                  uint64_t remote_addr, ucp_rkey_h rkey)
{
    void *request = ucp_put_nb(ep, buffer, length, remote_addr, rkey,
                               empty_callback),
    if (request == NULL) {
        return UCS_OK;
    } else if (UCS_PTR_IS_ERR(request)) {
        return UCS_PTR_STATUS(request);
    } else {
        ucs_status_t status;
        do {
            ucp_worker_progress(worker);
            status = ucp_request_check_status(request);
        } while (status == UCS_INPROGRESS);
        ucp_request_release(request);
        return status;
    }
}

This routine stores contiguous block of data that is described by the local address buffer in the remote contiguous memory region described by remote_addr address and the memory handle rkey. The routine returns when it is safe to reuse the source address buffer.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local source address.
length – [in] Length of the data (in bytes) stored under the source address.
remote_addr – [in] Pointer to the destination remote address to write to.
rkey – [in] Remote memory key associated with the remote address.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_get(ucp_ep_h ep, void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey)¶

Blocking remote memory get operation.

Deprecated:: Replaced by ucp_get_nb.

This routine loads contiguous block of data that is described by the remote address remote_addr and the memory handle rkey in the local contiguous memory region described by buffer address. The routine returns when remote data is loaded and stored under the local address buffer.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local source address.
length – [in] Length of the data (in bytes) stored under the source address.
remote_addr – [in] Pointer to the destination remote address to write to.
rkey – [in] Remote memory key associated with the remote address.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_atomic_add32(ucp_ep_h ep, uint32_t add, uint64_t remote_addr, ucp_rkey_h rkey)¶

Blocking atomic add operation for 32 bit integers.

Deprecated:: Replaced by ucp_atomic_post with opcode UCP_ATOMIC_POST_OP_ADD.

This routine performs an add operation on a 32 bit integer value atomically. The remote integer value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The add value is the value that is used for the add operation. When the operation completes the sum of the original remote value and the operand value (add) is stored in remote memory. The call to the routine returns immediately, independent of operation completion.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
add – [in] Value to add.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 32 bit.

ucs_status_t ucp_atomic_add64(ucp_ep_h ep, uint64_t add, uint64_t remote_addr, ucp_rkey_h rkey)¶

Blocking atomic add operation for 64 bit integers.

Deprecated:: Replaced by ucp_atomic_post with opcode UCP_ATOMIC_POST_OP_ADD.

This routine performs an add operation on a 64 bit integer value atomically. The remote integer value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The add value is the value that is used for the add operation. When the operation completes the sum of the original remote value and the operand value (add) is stored in remote memory. The call to the routine returns immediately, independent of operation completion.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
add – [in] Value to add.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 64 bit.

ucs_status_t ucp_atomic_fadd32(ucp_ep_h ep, uint32_t add, uint64_t remote_addr, ucp_rkey_h rkey, uint32_t *result)¶

Blocking atomic fetch and add operation for 32 bit integers.

Deprecated:: Replaced by ucp_atomic_fetch_nb with opcode UCP_ATOMIC_FETCH_OP_FADD.

This routine performs an add operation on a 32 bit integer value atomically. The remote integer value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The add value is the value that is used for the add operation. When the operation completes, the original remote value is stored in the local memory result, and the sum of the original remote value and the operand value is stored in remote memory. The call to the routine returns when the operation is completed and the result value is updated.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
add – [in] Value to add.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.
result – [out] Pointer to the address that is used to store the previous value of the atomic variable described by the remote_addr

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 32 bit.

ucs_status_t ucp_atomic_fadd64(ucp_ep_h ep, uint64_t add, uint64_t remote_addr, ucp_rkey_h rkey, uint64_t *result)¶

Blocking atomic fetch and add operation for 64 bit integers.

Deprecated:: Replaced by ucp_atomic_fetch_nb with opcode UCP_ATOMIC_FETCH_OP_FADD.

This routine performs an add operation on a 64 bit integer value atomically. The remote integer value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The add value is the value that is used for the add operation. When the operation completes, the original remote value is stored in the local memory result, and the sum of the original remote value and the operand value is stored in remote memory. The call to the routine returns when the operation is completed and the result value is updated.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
add – [in] Value to add.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.
result – [out] Pointer to the address that is used to store the previous value of the atomic variable described by the remote_addr

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 64 bit.

ucs_status_t ucp_atomic_swap32(ucp_ep_h ep, uint32_t swap, uint64_t remote_addr, ucp_rkey_h rkey, uint32_t *result)¶

Blocking atomic swap operation for 32 bit values.

Deprecated:: Replaced by ucp_atomic_fetch_nb with opcode UCP_ATOMIC_FETCH_OP_SWAP.

This routine swaps a 32 bit value between local and remote memory. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The swap value is the value that is used for the swap operation. When the operation completes, the remote value is stored in the local memory result, and the operand value (swap) is stored in remote memory. The call to the routine returns when the operation is completed and the result value is updated.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
swap – [in] Value to swap.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.
result – [out] Pointer to the address that is used to store the previous value of the atomic variable described by the remote_addr

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 32 bit.

ucs_status_t ucp_atomic_swap64(ucp_ep_h ep, uint64_t swap, uint64_t remote_addr, ucp_rkey_h rkey, uint64_t *result)¶

Blocking atomic swap operation for 64 bit values.

Deprecated:: Replaced by ucp_atomic_fetch_nb with opcode UCP_ATOMIC_FETCH_OP_SWAP.

This routine swaps a 64 bit value between local and remote memory. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The swap value is the value that is used for the swap operation. When the operation completes, the remote value is stored in the local memory result, and the operand value (swap) is stored in remote memory. The call to the routine returns when the operation is completed and the result value is updated.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
swap – [in] Value to swap.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.
result – [out] Pointer to the address that is used to store the previous value of the atomic variable described by the remote_addr

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 64 bit.

ucs_status_t ucp_atomic_cswap32(ucp_ep_h ep, uint32_t compare, uint32_t swap, uint64_t remote_addr, ucp_rkey_h rkey, uint32_t *result)¶

Blocking atomic conditional swap (cswap) operation for 32 bit values.

Deprecated:: Replaced by ucp_atomic_fetch_nb with opcode UCP_ATOMIC_FETCH_OP_CSWAP.

This routine conditionally swaps a 32 bit value between local and remote memory. The swap occurs only if the condition value (continue) is equal to the remote value, otherwise the remote memory is not modified. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The swap value is the value that is used to update the remote memory if the condition is true. The call to the routine returns when the operation is completed and the result value is updated.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
compare – [in] Value to compare to.
swap – [in] Value to swap.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.
result – [out] Pointer to the address that is used to store the previous value of the atomic variable described by the remote_addr

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 32 bit.

ucs_status_t ucp_atomic_cswap64(ucp_ep_h ep, uint64_t compare, uint64_t swap, uint64_t remote_addr, ucp_rkey_h rkey, uint64_t *result)¶

Blocking atomic conditional swap (cswap) operation for 64 bit values.

Deprecated:: Replaced by ucp_atomic_fetch_nb with opcode UCP_ATOMIC_FETCH_OP_CSWAP.

This routine conditionally swaps a 64 bit value between local and remote memory. The swap occurs only if the condition value (continue) is equal to the remote value, otherwise the remote memory is not modified. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The swap value is the value that is used to update the remote memory if the condition is true. The call to the routine returns when the operation is completed and the result value is updated.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
compare – [in] Value to compare to.
swap – [in] Value to swap.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.
result – [out] Pointer to the address that is used to store the previous value of the atomic variable described by the remote_addr

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 64 bit.

ucs_status_ptr_t ucp_am_send_nb(ucp_ep_h ep, uint16_t id, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_send_callback_t cb, unsigned flags)¶

Send Active Message.

Deprecated:: Use ucp_am_send_nbx instead.

This routine sends an Active Message to an ep. It does not support CUDA memory.

Parameters

ep – [in] UCP endpoint where the Active Message will be run.
id – [in] Active Message id. Specifies which registered callback to run.
buffer – [in] Pointer to the data to be sent to the target node of the Active Message.
count – [in] Number of elements to send.
datatype – [in] Datatype descriptor for the elements in the buffer.
cb – [in] Callback that is invoked upon completion of the data transfer if it is not completed immediately.
flags – [in] Operation flags as defined by ucp_send_am_flags.

Returns

NULL Active Message was sent immediately.

Returns

UCS_PTR_IS_ERR(_ptr) Error sending Active Message.

Returns

otherwise Pointer to request, and Active Message is known to be completed after cb is run.

ucs_status_ptr_t ucp_stream_send_nb(ucp_ep_h ep, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_send_callback_t cb, unsigned flags)¶

Non-blocking stream send operation.

Deprecated:: Use ucp_stream_send_nbx instead.

This routine sends data that is described by the local address buffer, size count, and datatype object to the destination endpoint ep. The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine returns UCS_OK and the callback function cb is not invoked. If the operation is not completed immediately and no error reported, then the UCP library will schedule invocation of the callback cb upon completion of the send operation. In other words, the completion of the operation will be signaled either by the return code or by the callback.

Parameters

ep – [in] Destination endpoint handle.
buffer – [in] Pointer to the message buffer (payload).
count – [in] Number of elements to send.
datatype – [in] Datatype descriptor for the elements in the buffer.
cb – [in] Callback function that is invoked whenever the send operation is completed. It is important to note that the callback is only invoked in the event that the operation cannot be completed in place.
flags – [in] Reserved for future use.

Returns

NULL - The send operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message. The application is responsible for releasing the handle using ucp_request_free routine.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_ptr_t ucp_stream_recv_nb(ucp_ep_h ep, void *buffer, size_t count, ucp_datatype_t datatype, ucp_stream_recv_callback_t cb, size_t *length, unsigned flags)¶

Non-blocking stream receive operation of structured data into a user-supplied buffer.

Deprecated:: Use ucp_stream_recv_nbx instead.

This routine receives data that is described by the local address buffer, size count, and datatype object on the endpoint ep. The routine is non-blocking and therefore returns immediately. The receive operation is considered complete when the message is delivered to the buffer. If data is not immediately available, the operation will be scheduled for receive and a request handle will be returned. In order to notify the application about completion of a scheduled receive operation, the UCP library will invoke the call-back cb when data is in the receive buffer and ready for application access. If the receive operation cannot be started, the routine returns an error.

Parameters

ep – [in] UCP endpoint that is used for the receive operation.
buffer – [in] Pointer to the buffer to receive the data.
count – [in] Number of elements to receive into buffer.
datatype – [in] Datatype descriptor for the elements in the buffer.
cb – [in] Callback function that is invoked whenever the receive operation is completed and the data is ready in the receive buffer. It is important to note that the call-back is only invoked in a case when the operation cannot be completed immediately.
length – [out] Size of the received data in bytes. The value is valid only if return code is UCS_OK.
flags – [in] Flags defined in ucp_stream_recv_flags_t.

Returns

NULL - The receive operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. A request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle by calling the ucp_request_free routine.

Note

The amount of data received, in bytes, is always an integral multiple of the datatype size.

ucs_status_ptr_t ucp_tag_send_nb(ucp_ep_h ep, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, ucp_send_callback_t cb)¶

Non-blocking tagged-send operations.

Deprecated:: Use ucp_tag_send_nbx instead.

This routine sends a messages that is described by the local address buffer, size count, and datatype object to the destination endpoint ep. Each message is associated with a tag value that is used for message matching on the receiver. The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine return UCS_OK and the call-back function cb is not invoked. If the operation is not completed immediately and no error reported then the UCP library will schedule to invoke the call-back cb whenever the send operation will be completed. In other words, the completion of a message can be signaled by the return code or the call-back.

Parameters

ep – [in] Destination endpoint handle.
buffer – [in] Pointer to the message buffer (payload).
count – [in] Number of elements to send
datatype – [in] Datatype descriptor for the elements in the buffer.
tag – [in] Message tag.
cb – [in] Callback function that is invoked whenever the send operation is completed. It is important to note that the call-back is only invoked in a case when the operation cannot be completed in place.

Returns

NULL - The send operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_t ucp_tag_send_nbr(ucp_ep_h ep, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, void *req)¶

Non-blocking tagged-send operations with user provided request.

Deprecated:: Use ucp_tag_send_nbx with the flag UCP_OP_ATTR_FIELD_REQUEST instead.

This routine provides a convenient and efficient way to implement a blocking send pattern. It also completes requests faster than ucp_tag_send_nb() because:

it always uses eager protocol to send data up to the rendezvous threshold.
its rendezvous threshold is higher than the one used by the ucp_tag_send_nb(). The threshold is controlled by the UCX_SEND_NBR_RNDV_THRESH environment variable.
its request handling is simpler. There is no callback and no need to allocate and free requests. In fact request can be allocated by caller on the stack.

This routine sends a messages that is described by the local address buffer, size count, and datatype object to the destination endpoint ep. Each message is associated with a tag value that is used for message matching on the receiver.

The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine returns UCS_OK.

If the operation is not completed immediately and no error reported then the UCP library will fill a user provided req and return UCS_INPROGRESS status. In order to monitor completion of the operation ucp_request_check_status() should be used.

Following pseudo code implements a blocking send function:

MPI_send(...)
{
    char *request;
    ucs_status_t status;

    // allocate request on the stack
    // ucp_context_query() was used to get ucp_request_size
    request = alloca(ucp_request_size);

    // note: make sure that there is enough memory before the
    // request handle
    status = ucp_tag_send_nbr(ep, ..., request + ucp_request_size);
    if (status != UCS_INPROGRESS) {
        return status;
    }

    do {
        ucp_worker_progress(worker);
        status = ucp_request_check_status(request + ucp_request_size);
    } while (status == UCS_INPROGRESS);

    return status;
}

Parameters

ep – [in] Destination endpoint handle.
buffer – [in] Pointer to the message buffer (payload).
count – [in] Number of elements to send
datatype – [in] Datatype descriptor for the elements in the buffer.
tag – [in] Message tag.
req – [in] Request handle allocated by the user. There should be at least UCP request size bytes of available space before the req. The size of UCP request can be obtained by ucp_context_query function.

Returns

UCS_OK - The send operation was completed immediately.

Returns

UCS_INPROGRESS - The send was not completed and is in progress. ucp_request_check_status() should be used to monitor req status.

Returns

Error code as defined by ucs_status_t

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_ptr_t ucp_tag_send_sync_nb(ucp_ep_h ep, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, ucp_send_callback_t cb)¶

Non-blocking synchronous tagged-send operation.

Deprecated:: Use ucp_tag_send_sync_nbx instead.

Same as ucp_tag_send_nb, except the request completes only after there is a remote tag match on the message (which does not always mean the remote receive has been completed). This function never completes “in-place”, and always returns a request handle.

Parameters

ep – [in] Destination endpoint handle.
buffer – [in] Pointer to the message buffer (payload).
count – [in] Number of elements to send
datatype – [in] Datatype descriptor for the elements in the buffer.
tag – [in] Message tag.
cb – [in] Callback function that is invoked whenever the send operation is completed.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

Note

Returns UCS_ERR_UNSUPPORTED if UCP_ERR_HANDLING_MODE_PEER is enabled. This is a temporary implementation-related constraint that will be addressed in future releases.

ucs_status_ptr_t ucp_tag_recv_nb(ucp_worker_h worker, void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, ucp_tag_t tag_mask, ucp_tag_recv_callback_t cb)¶

Non-blocking tagged-receive operation.

Deprecated:: Use ucp_tag_recv_nbx instead.

This routine receives a message that is described by the local address buffer, size count, and datatype object on the worker. The tag value of the receive message has to match the tag and tag_mask values, where the tag_mask indicates which bits of the tag have to be matched. The routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to notify the application about completion of the receive operation the UCP library will invoke the call-back cb when the received message is in the receive buffer and ready for application access. If the receive operation cannot be stated the routine returns an error.

Parameters

worker – [in] UCP worker that is used for the receive operation.
buffer – [in] Pointer to the buffer to receive the data.
count – [in] Number of elements to receive
datatype – [in] Datatype descriptor for the elements in the buffer.
tag – [in] Message tag to expect.
tag_mask – [in] Bit mask that indicates the bits that are used for the matching of the incoming tag against the expected tag.
cb – [in] Callback function that is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

This routine cannot return UCS_OK. It always returns a request handle or an error.

ucs_status_t ucp_tag_recv_nbr(ucp_worker_h worker, void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, ucp_tag_t tag_mask, void *req)¶

Non-blocking tagged-receive operation.

Deprecated:: Use ucp_tag_recv_nbx with the flag UCP_OP_ATTR_FIELD_REQUEST instead.

This routine receives a message that is described by the local address buffer, size count, and datatype object on the worker. The tag value of the receive message has to match the tag and tag_mask values, where the tag_mask indicates which bits of the tag have to be matched. The routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to monitor completion of the operation ucp_request_check_status or ucp_tag_recv_request_test should be used.

Parameters

worker – [in] UCP worker that is used for the receive operation.
buffer – [in] Pointer to the buffer to receive the data.
count – [in] Number of elements to receive
datatype – [in] Datatype descriptor for the elements in the buffer.
tag – [in] Message tag to expect.
tag_mask – [in] Bit mask that indicates the bits that are used for the matching of the incoming tag against the expected tag.
req – [in] Request handle allocated by the user. There should be at least UCP request size bytes of available space before the req. The size of UCP request can be obtained by ucp_context_query function.

Returns

Error code as defined by ucs_status_t

ucs_status_ptr_t ucp_tag_msg_recv_nb(ucp_worker_h worker, void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_message_h message, ucp_tag_recv_callback_t cb)¶

Non-blocking receive operation for a probed message.

Deprecated:: Use ucp_tag_recv_nbx instead.

This routine receives a message that is described by the local address buffer, size count, message handle, and datatype object on the worker. The message handle can be obtained by calling the ucp_tag_probe_nb() routine. The ucp_tag_msg_recv_nb() routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to notify the application about completion of the receive operation the UCP library will invoke the call-back cb when the received message is in the receive buffer and ready for application access. If the receive operation cannot be started the routine returns an error.

Parameters

worker – [in] UCP worker that is used for the receive operation.
buffer – [in] Pointer to the buffer that will receive the data.
count – [in] Number of elements to receive
datatype – [in] Datatype descriptor for the elements in the buffer.
message – [in] Message handle.
cb – [in] Callback function that is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

ucs_status_t ucp_put_nbi(ucp_ep_h ep, const void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey)¶

Non-blocking implicit remote memory put operation.

Deprecated:: Use ucp_put_nbx without passing the flag UCP_OP_ATTR_FIELD_CALLBACK instead. If a request pointer is returned, release it immediately by ucp_request_free.

This routine initiates a storage of contiguous block of data that is described by the local address buffer in the remote contiguous memory region described by remote_addr address and the memoryhandle” rkey. The routine returns immediately and does not guarantee re-usability of the source address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local source address.
length – [in] Length of the data (in bytes) stored under the source address.
remote_addr – [in] Pointer to the destination remote memory address to write to.
rkey – [in] Remote memory key associated with the remote memory address.

Returns

Error code as defined by ucs_status_t

Note

A user can use ucp_worker_flush_nb() in order to guarantee re-usability of the source address buffer.

ucs_status_ptr_t ucp_put_nb(ucp_ep_h ep, const void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey, ucp_send_callback_t cb)¶

Non-blocking remote memory put operation.

Deprecated:: Use ucp_put_nbx instead.

This routine initiates a storage of contiguous block of data that is described by the local address buffer in the remote contiguous memory region described by remote_addr address and the memoryhandle” rkey. The routine returns immediately and does not guarantee re-usability of the source address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user. If the put operation completes immediately, the routine returns UCS_OK and the call-back routine cb is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine cb upon completion of the put operation. In other words, the completion of a put operation can be signaled by the return code or execution of the call-back.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local source address.
length – [in] Length of the data (in bytes) stored under the source address.
remote_addr – [in] Pointer to the destination remote memory address to write to.
rkey – [in] Remote memory key associated with the remote memory address.
cb – [in] Call-back function that is invoked whenever the put operation is completed and the local buffer can be modified. Does not guarantee remote completion.

Returns

NULL - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

A user can use ucp_worker_flush_nb() in order to guarantee re-usability of the source address buffer.

ucs_status_t ucp_get_nbi(ucp_ep_h ep, void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey)¶

Non-blocking implicit remote memory get operation.

Deprecated:: Use ucp_get_nbx without passing the flag UCP_OP_ATTR_FIELD_CALLBACK instead. If a request pointer is returned, release it immediately by ucp_request_free.

This routine initiate a load of contiguous block of data that is described by the remote memory address remote_addr and the memory handle rkey in the local contiguous memory region described by buffer address. The routine returns immediately and does not guarantee that remote data is loaded and stored under the local address buffer.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local destination address.
length – [in] Length of the data (in bytes) stored under the destination address.
remote_addr – [in] Pointer to the source remote memory address to read from.
rkey – [in] Remote memory key associated with the remote memory address.

Returns

Error code as defined by ucs_status_t

Note

A user can use ucp_worker_flush_nb() in order guarantee that remote data is loaded and stored under the local address buffer.

ucs_status_ptr_t ucp_get_nb(ucp_ep_h ep, void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey, ucp_send_callback_t cb)¶

Non-blocking remote memory get operation.

Deprecated:: Use ucp_get_nbx instead.

This routine initiates a load of a contiguous block of data that is described by the remote memory address remote_addr and the memory handle rkey in the local contiguous memory region described by buffer address. The routine returns immediately and does not guarantee that remote data is loaded and stored under the local address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user. If the get operation completes immediately, the routine returns UCS_OK and the call-back routine cb is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine cb upon completion of the get operation. In other words, the completion of a get operation can be signaled by the return code or execution of the call-back.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local destination address.
length – [in] Length of the data (in bytes) stored under the destination address.
remote_addr – [in] Pointer to the source remote memory address to read from.
rkey – [in] Remote memory key associated with the remote memory address.
cb – [in] Call-back function that is invoked whenever the get operation is completed and the data is visible to the local process.

Returns

NULL - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

A user can use ucp_worker_flush_nb() in order to guarantee re-usability of the source address buffer.

ucs_status_t ucp_atomic_post(ucp_ep_h ep, ucp_atomic_post_op_t opcode, uint64_t value, size_t op_size, uint64_t remote_addr, ucp_rkey_h rkey)¶

Post an atomic memory operation.

Deprecated:: Use ucp_atomic_op_nbx without the flag UCP_OP_ATTR_FIELD_REPLY_BUFFER instead.

This routine posts an atomic memory operation to a remote value. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. Return from the function does not guarantee completion. A user must call ucp_ep_flush_nb or ucp_worker_flush_nb to guarantee that the remote value has been updated.

Parameters

ep – [in] UCP endpoint.
opcode – [in] One of ucp_atomic_post_op_t.
value – [in] Source operand for the atomic operation.
op_size – [in] Size of value in bytes
remote_addr – [in] Remote address to operate on.
rkey – [in] Remote key handle for the remote memory address.

Returns

Error code as defined by ucs_status_t

ucs_status_ptr_t ucp_atomic_fetch_nb(ucp_ep_h ep, ucp_atomic_fetch_op_t opcode, uint64_t value, void *result, size_t op_size, uint64_t remote_addr, ucp_rkey_h rkey, ucp_send_callback_t cb)¶

Post an atomic fetch operation.

Deprecated:: Use ucp_atomic_op_nbx with the flag UCP_OP_ATTR_FIELD_REPLY_BUFFER instead.

This routine will post an atomic fetch operation to remote memory. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The routine is non-blocking and therefore returns immediately. However the actual atomic operation may be delayed. The atomic operation is not considered complete until the values in remote and local memory are completed. If the atomic operation completes immediately, the routine returns UCS_OK and the call-back routine cb is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine cb upon completion of the atomic operation. In other words, the completion of an atomic operation can be signaled by the return code or execution of the call-back.

Parameters

ep – [in] UCP endpoint.
opcode – [in] One of ucp_atomic_fetch_op_t.
value – [in] Source operand for atomic operation. In the case of CSWAP this is the conditional for the swap. For SWAP this is the value to be placed in remote memory.
result – [inout] Local memory address to store resulting fetch to. In the case of CSWAP the value in result will be swapped into the remote_addr if the condition is true.
op_size – [in] Size of value in bytes and pointer type for result
remote_addr – [in] Remote address to operate on.
rkey – [in] Remote key handle for the remote memory address.
cb – [in] Call-back function that is invoked whenever the send operation is completed. It is important to note that the call-back function is only invoked in a case when the operation cannot be completed in place.

Returns

NULL - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

The user should not modify any part of the result after this operation is called, until the operation completes.

struct ucp_request_attr_t¶: #include <ucp.h>
Attributes of a particular request.

struct ucp_err_handler¶

#include <ucp_def.h>

UCP endpoint error handling context.

This structure should be initialized in ucp_ep_params_t to handle peer failure

Wake-up routines ¶

group UCP_WAKEUP

UCP Wake-up routines

Functions

ucs_status_t ucp_worker_get_efd(ucp_worker_h worker, int *fd)¶

Obtain an event file descriptor for event notification.

This routine returns a valid file descriptor for polling functions. The file descriptor will get signaled when an event occurs, as part of the wake-up mechanism. Signaling means a call to poll() or select() with this file descriptor will return at this point, with this descriptor marked as the reason (or one of the reasons) the function has returned. The user does not need to release the obtained file descriptor.

The wake-up mechanism exists to allow for the user process to register for notifications on events of the underlying interfaces, and wait until such occur. This is an alternative to repeated polling for request completion. The goal is to allow for waiting while consuming minimal resources from the system. This is recommended for cases where traffic is infrequent, and latency can be traded for lower resource consumption while waiting for it.

There are two alternative ways to use the wakeup mechanism: the first is the file descriptor obtained per worker (this function) and the second is the ucp_worker_wait function for waiting on the next event internally.

Parameters

worker – [in] Worker of notified events.
fd – [out] File descriptor.

Returns

Error code as defined by ucs_status_t

Note

UCP features have to be triggered with UCP_FEATURE_WAKEUP to select proper transport

ucs_status_t ucp_worker_wait(ucp_worker_h worker)¶

Wait for an event of the worker.

This routine waits (blocking) until an event has happened, as part of the wake-up mechanism.

This function is guaranteed to return only if new communication events occur on the worker. Therefore one must drain all existing events before waiting on the file descriptor. This can be achieved by calling ucp_worker_progress repeatedly until it returns 0.

There are two alternative ways to use the wakeup mechanism. The first is by polling on a per-worker file descriptor obtained from ucp_worker_get_efd. The second is by using this function to perform an internal wait for the next event associated with the specified worker.

Parameters: worker – [in] Worker to wait for events on.
Returns: Error code as defined by ucs_status_t

Note

During the blocking call the wake-up mechanism relies on other means of notification and may not progress some of the requests as it would when calling ucp_worker_progress (which is not invoked in that duration).

Note

UCP features have to be triggered with UCP_FEATURE_WAKEUP to select proper transport

void ucp_worker_wait_mem(ucp_worker_h worker, void *address)¶

Wait for memory update on the address.

This routine waits for a memory update at the local memory address. This is a blocking routine. The routine returns when the memory address is updated (“write”) or an event occurs in the system.

This function is guaranteed to return only if new communication events occur on the worker or address is modified. Therefore one must drain all existing events before waiting on the file descriptor. This can be achieved by calling ucp_worker_progress repeatedly until it returns 0.

Parameters

worker – [in] Worker to wait for updates on.
address – [in] Local memory address

Note

This routine can be used by an application that executes busy-waiting loop checking for a memory update. Instead of continuous busy-waiting on an address the application can use ucp_worker_wait_mem, which may suspend execution until the memory is updated. The goal of the routine is to provide an opportunity for energy savings for architectures that support this functionality.

ucs_status_t ucp_worker_arm(ucp_worker_h worker)¶

Turn on event notification for the next event.

This routine needs to be called before waiting on each notification on this worker, so will typically be called once the processing of the previous event is over, as part of the wake-up mechanism.

The worker must be armed before waiting on an event (must be re-armed after it has been signaled for re-use) with ucp_worker_arm. The events triggering a signal of the file descriptor from ucp_worker_get_efd depend on the interfaces used by the worker and defined in the transport layer, and typically represent a request completion or newly available resources. It can also be triggered by calling ucp_worker_signal .

The file descriptor is guaranteed to become signaled only if new communication events occur on the worker. Therefore one must drain all existing events before waiting on the file descriptor. This can be achieved by calling ucp_worker_progress repeatedly until it returns 0.

void application_initialization() {
// should be called once in application init flow and before
// process_communication() is used
    ...
    status = ucp_worker_get_efd(worker, &fd);
    ...
}

void process_communication() {
// should be called every time need to wait for some condition such as
// ucp request completion in sleep mode.

    for (;;) {
        // check for stop condition as long as progress is made
        if (check_for_events()) {
             break;
        } else if (ucp_worker_progress(worker)) {
             continue;                 // some progress happened but condition not met
        }

        // arm the worker and clean-up fd
        status = ucp_worker_arm(worker);
        if (UCS_OK == status) {
            poll(&fds, nfds, timeout);  // wait for events (sleep mode)
        } else if (UCS_ERR_BUSY == status) {
            continue;                   // could not arm, need to progress more
        } else {
            abort();
        }
    }
}

Parameters: worker – [in] Worker of notified events.
Returns: UCS_OK The operation completed successfully. File descriptor will be signaled by new events.
Returns: UCS_ERR_BUSY There are unprocessed events which prevent the file descriptor from being armed. These events should be removed by calling ucp_worker_progress(). The operation is not completed. File descriptor will not be signaled by new events.
Returns: Other different error codes in case of issues.

Note

UCP features have to be triggered with UCP_FEATURE_WAKEUP to select proper transport

ucs_status_t ucp_worker_signal(ucp_worker_h worker)¶

Cause an event of the worker.

This routine signals that the event has happened, as part of the wake-up mechanism. This function causes a blocking call to ucp_worker_wait or waiting on a file descriptor from ucp_worker_get_efd to return, even if no event from the underlying interfaces has taken place.

Parameters: worker – [in] Worker to wait for events on.
Returns: Error code as defined by ucs_status_t

Note

It’s safe to use this routine from any thread, even if UCX is compiled without multi-threading support and/or initialized with any value of ucp_params_t::mt_workers_shared and ucp_worker_params_t::thread_mode parameters

Resources ¶

group UCS_RESOURCE

This section describes a concept of the Communication Resource and routines associated with the concept.

Typedefs

typedef void (*ucs_async_event_cb_t)(int id, ucs_event_set_types_t events, void *arg)¶

Async event callback.

Parameters

id – Event id (timer or file descriptor).
events – The events that triggered the callback.
arg – User-defined argument.

typedef struct ucs_sock_addr ucs_sock_addr_t¶: BSD socket address specification.

typedef enum ucs_memory_type ucs_memory_type_t¶

Memory types.

List of supported memory types.

typedef unsigned long ucs_time_t¶: UCS time units. These are not necessarily aligned with metric time units. MUST compare short time values with UCS_SHORT_TIME_CMP to handle wrap-around.

typedef void *ucs_status_ptr_t¶

Status pointer.

A pointer can represent one of these values:

NULL / UCS_OK
Error code pointer (UCS_ERR_xx)
Valid pointer

Enums

enum ucs_callbackq_flags¶

See file LICENSE for terms.

Callback backward compatibility flags

Values:

enumerator UCS_CALLBACKQ_FLAG_FAST¶: Fast-path (best effort)

enumerator UCS_CALLBACKQ_FLAG_ONESHOT¶: Call the callback only once (cannot be used with FAST)

enum ucs_memory_type¶

Memory types.

List of supported memory types.

Values:

enumerator UCS_MEMORY_TYPE_HOST¶: Default system memory

enumerator UCS_MEMORY_TYPE_CUDA¶: NVIDIA CUDA memory

enumerator UCS_MEMORY_TYPE_CUDA_MANAGED¶: NVIDIA CUDA managed (or unified) memory

enumerator UCS_MEMORY_TYPE_ROCM¶: AMD ROCM memory

enumerator UCS_MEMORY_TYPE_ROCM_MANAGED¶: AMD ROCM managed system memory

enumerator UCS_MEMORY_TYPE_RDMA¶: RDMA device memory

enumerator UCS_MEMORY_TYPE_ZE_HOST¶: Intel ZE memory (USM host)

enumerator UCS_MEMORY_TYPE_ZE_DEVICE¶: Intel ZE memory (USM device)

enumerator UCS_MEMORY_TYPE_ZE_MANAGED¶: Intel ZE managed memory (USM shared)

enumerator UCS_MEMORY_TYPE_LAST¶

enumerator UCS_MEMORY_TYPE_UNKNOWN¶

enum ucs_status_t¶

Status codes.

Note

In order to evaluate the necessary steps to recover from a certain error, all error codes which can be returned by the external API are grouped by the largest entity permanently effected by the error. Each group ranges between its UCS_ERR_FIRST_<name> and UCS_ERR_LAST_<name> enum values. For example, if a link fails it may be sufficient to destroy (and possibly replace) it, in contrast to an endpoint-level error.

Values:

enumerator UCS_OK¶

enumerator UCS_INPROGRESS¶

enumerator UCS_ERR_NO_MESSAGE¶

enumerator UCS_ERR_NO_RESOURCE¶

enumerator UCS_ERR_IO_ERROR¶

enumerator UCS_ERR_NO_MEMORY¶

enumerator UCS_ERR_INVALID_PARAM¶

enumerator UCS_ERR_UNREACHABLE¶

enumerator UCS_ERR_INVALID_ADDR¶

enumerator UCS_ERR_NOT_IMPLEMENTED¶

enumerator UCS_ERR_MESSAGE_TRUNCATED¶

enumerator UCS_ERR_NO_PROGRESS¶

enumerator UCS_ERR_BUFFER_TOO_SMALL¶

enumerator UCS_ERR_NO_ELEM¶

enumerator UCS_ERR_SOME_CONNECTS_FAILED¶

enumerator UCS_ERR_NO_DEVICE¶

enumerator UCS_ERR_BUSY¶

enumerator UCS_ERR_CANCELED¶

enumerator UCS_ERR_SHMEM_SEGMENT¶

enumerator UCS_ERR_ALREADY_EXISTS¶

enumerator UCS_ERR_OUT_OF_RANGE¶

enumerator UCS_ERR_TIMED_OUT¶

enumerator UCS_ERR_EXCEEDS_LIMIT¶

enumerator UCS_ERR_UNSUPPORTED¶

enumerator UCS_ERR_REJECTED¶

enumerator UCS_ERR_NOT_CONNECTED¶

enumerator UCS_ERR_CONNECTION_RESET¶

enumerator UCS_ERR_FIRST_LINK_FAILURE¶

enumerator UCS_ERR_LAST_LINK_FAILURE¶

enumerator UCS_ERR_FIRST_ENDPOINT_FAILURE¶

enumerator UCS_ERR_ENDPOINT_TIMEOUT¶

enumerator UCS_ERR_LAST_ENDPOINT_FAILURE¶

enumerator UCS_ERR_LAST¶

enum ucs_thread_mode_t¶

Thread sharing mode.

Specifies thread sharing mode of an object.

Values:

enumerator UCS_THREAD_MODE_SINGLE¶: Only the master thread can access (i.e. the thread that initialized the context; multiple threads may exist and never access)

enumerator UCS_THREAD_MODE_SERIALIZED¶: Multiple threads can access, but only one at a time

enumerator UCS_THREAD_MODE_MULTI¶: Multiple threads can access concurrently

enumerator UCS_THREAD_MODE_LAST¶

Functions

ucs_status_t ucs_async_set_event_handler(ucs_async_mode_t mode, int event_fd, ucs_event_set_types_t events, ucs_async_event_cb_t cb, void *arg, ucs_async_context_t *async)¶

Register a file descriptor for monitoring (call handler upon events). Every fd can have only one handler.

Parameters

mode – Thread or signal.
event_fd – File descriptor to set handler for.
events – Events to wait on (UCS_EVENT_SET_EVxxx bits).
cb – Callback function to execute.
arg – Argument to callback.
async – Async context to which events are delivered. If NULL, safety is up to the user.

Returns

Error code as defined by ucs_status_t.

ucs_status_t ucs_async_add_timer(ucs_async_mode_t mode, ucs_time_t interval, ucs_async_event_cb_t cb, void *arg, ucs_async_context_t *async, int *timer_id_p)¶

Add timer handler.

Parameters

mode – Thread or signal.
interval – Timer interval.
cb – Callback function to execute.
arg – Argument to callback.
async – Async context to which events are delivered. If NULL, safety is up to the user.
timer_id_p – Filled with timer id.

Returns

Error code as defined by ucs_status_t.

ucs_status_t ucs_async_remove_handler(int id, int sync)¶

Remove an event handler (Timer or event file).

Parameters

id – Timer/FD to remove.
sync – If nonzero, wait until the handler for this event is not running anymore. If called from the context of the callback, the handler will be removed immediately after the current callback returns.

Returns

Error code as defined by ucs_status_t.

ucs_status_t ucs_async_modify_handler(int fd, ucs_event_set_types_t events)¶

Modify events mask for an existing event handler (event file).

Parameters

fd – File descriptor modify events for.
events – New set of events to wait on (UCS_EVENT_SET_EVxxx bits).

Returns

Error code as defined by ucs_status_t.

ucs_status_t ucs_async_context_create(ucs_async_mode_t mode, ucs_async_context_t **async_p)¶

Create an asynchronous execution context.

Allocate and initialize an asynchronous execution context. This can be used to ensure safe event delivery.

Parameters

mode – Indicates whether to use signals or polling threads for waiting.
async_p – Event context pointer to initialize.

Returns

Error code as defined by ucs_status_t.

void ucs_async_context_destroy(ucs_async_context_t *async)¶

Destroy the asynchronous execution context.

Clean up the async context, and release system resources if possible. The context memory released.

Parameters: async – Asynchronous context to clean up.

void ucs_async_poll(ucs_async_context_t *async)¶

Poll on async context.

Parameters: async – Async context to poll on. NULL polls on all.

struct ucs_sock_addr¶: #include <types.h>
BSD socket address specification.