File ucp.h

Defines

ucp_dt_make_contig(_elem_size)

Generate an identifier for contiguous data type.

This macro creates an identifier for contiguous datatype that is defined by the size of the basic element.

Parameters
  • _elem_size[in] Size of the basic element of the type.

Returns

Data-type identifier.

Note

In case of partial receive, the buffer will be filled with integral count of elements.

ucp_dt_make_iov()

Generate an identifier for Scatter-gather IOV data type.

This macro creates an identifier for datatype of scatter-gather list with multiple pointers

Returns

Data-type identifier.

Note

In the event of partial receive, ucp_dt_iov_t::buffer can be filled with any number of bytes according to its ucp_dt_iov_t::length.

Typedefs

typedef enum ucp_ep_perf_param_field ucp_ep_perf_param_field_t

UCP performance fields and flags.

The enumeration allows specifying which fields in ucp_ep_evaluate_perf_param_t are present and operation flags are used. It is used to enable backward compatibility support.

typedef enum ucp_ep_perf_attr_field ucp_ep_perf_attr_field_t

UCP performance fields and flags.

The enumeration allows specifying which fields in ucp_ep_evaluate_perf_attr_t are present and operation flags are used. It is used to enable backward compatibility support.

typedef struct ucp_dt_iov ucp_dt_iov_t

Structure for scatter-gather I/O.

This structure is used to specify a list of buffers which can be used within a single data transfer function call. This list should remain valid until the data transfer request is completed.

Note

If length is zero, the memory pointed to by buffer will not be accessed. Otherwise, buffer must point to valid memory.

typedef struct ucp_generic_dt_ops ucp_generic_dt_ops_t

UCP generic data type descriptor.

This structure provides a generic datatype descriptor that is used for definition of application defined datatypes.

Typically, the descriptor is used for an integration with datatype engines implemented within MPI and SHMEM implementations.

Note

In case of partial receive, any amount of received data is acceptable which matches buffer size.

typedef struct ucp_params ucp_params_t

Tuning parameters for UCP library.

The structure defines the parameters that are used for UCP library tuning during UCP library initialization.

Note

UCP library implementation uses the features parameter to optimize the library functionality that minimize memory footprint. For example, if the application does not require send/receive semantics UCP library may avoid allocation of expensive resources associated with send/receive queues.

typedef struct ucp_lib_attr ucp_lib_attr_t

Lib attributes.

The structure defines the attributes that characterize the Library.

typedef struct ucp_context_attr ucp_context_attr_t

Context attributes.

The structure defines the attributes that characterize the particular context.

typedef struct ucp_worker_attr ucp_worker_attr_t

UCP worker attributes.

The structure defines the attributes which characterize the particular worker.

typedef struct ucp_worker_params ucp_worker_params_t

Tuning parameters for the UCP worker.

The structure defines the parameters that are used for the UCP worker tuning during the UCP worker creation.

typedef struct ucp_worker_address_attr ucp_worker_address_attr_t

UCP worker address attributes.

The structure defines the attributes of the particular worker address.

typedef struct ucp_listener_attr ucp_listener_attr_t

UCP listener attributes.

The structure defines the attributes which characterize the particular listener.

typedef struct ucp_conn_request_attr ucp_conn_request_attr_t

UCP listener’s connection request attributes.

The structure defines the attributes that characterize the particular connection request received on the server side.

typedef struct ucp_listener_params ucp_listener_params_t

Parameters for a UCP listener object.

This structure defines parameters for ucp_listener_create, which is used to listen for incoming client/server connections.

typedef struct ucp_stream_poll_ep ucp_stream_poll_ep_t

Output parameter of ucp_stream_worker_poll function.

The structure defines the endpoint and its user data.

typedef struct ucp_mem_map_params ucp_mem_map_params_t

Tuning parameters for the UCP memory mapping.

The structure defines the parameters that are used for the UCP memory mapping tuning during the ucp_mem_map routine.

typedef struct ucp_am_handler_param ucp_am_handler_param_t

Active Message handler parameters passed to ucp_worker_set_am_recv_handler routine.

typedef enum ucp_mem_advice ucp_mem_advice_t

list of UCP memory use advice.

The enumeration list describes memory advice supported by ucp_mem_advise() function.

typedef struct ucp_mem_advise_params ucp_mem_advise_params_t

Tuning parameters for the UCP memory advice.

This structure defines the parameters that are used for the UCP memory advice tuning during the ucp_mem_advise routine.

typedef struct ucp_ep_attr ucp_ep_attr_t

UCP endpoint attributes.

The structure defines the attributes that characterize the particular endpoint.

Enums

enum ucp_params_field

UCP context parameters field mask.

The enumeration allows specifying which fields in ucp_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_PARAM_FIELD_FEATURES

features

enumerator UCP_PARAM_FIELD_REQUEST_SIZE

request_size

enumerator UCP_PARAM_FIELD_REQUEST_INIT

request_init

enumerator UCP_PARAM_FIELD_REQUEST_CLEANUP

request_cleanup

enumerator UCP_PARAM_FIELD_TAG_SENDER_MASK

tag_sender_mask

enumerator UCP_PARAM_FIELD_MT_WORKERS_SHARED

mt_workers_shared

enumerator UCP_PARAM_FIELD_ESTIMATED_NUM_EPS

estimated_num_eps

enumerator UCP_PARAM_FIELD_ESTIMATED_NUM_PPN

estimated_num_ppn

enumerator UCP_PARAM_FIELD_NAME

name

enum ucp_feature

UCP configuration features.

The enumeration list describes the features supported by UCP. An application can request the features using UCP parameters during UCP initialization process.

Values:

enumerator UCP_FEATURE_TAG

Request tag matching support

enumerator UCP_FEATURE_RMA

Request remote memory access support

enumerator UCP_FEATURE_AMO32

Request 32-bit atomic operations support

enumerator UCP_FEATURE_AMO64

Request 64-bit atomic operations support

enumerator UCP_FEATURE_WAKEUP

Request interrupt notification support

enumerator UCP_FEATURE_STREAM

Request stream support

enumerator UCP_FEATURE_AM

Request Active Message support

enum ucp_worker_params_field

UCP worker parameters field mask.

The enumeration allows specifying which fields in ucp_worker_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_WORKER_PARAM_FIELD_THREAD_MODE

UCP thread mode

enumerator UCP_WORKER_PARAM_FIELD_CPU_MASK

Worker’s CPU bitmap

enumerator UCP_WORKER_PARAM_FIELD_EVENTS

Worker’s events bitmap

enumerator UCP_WORKER_PARAM_FIELD_USER_DATA

User data

enumerator UCP_WORKER_PARAM_FIELD_EVENT_FD

External event file descriptor

enumerator UCP_WORKER_PARAM_FIELD_FLAGS

Worker flags

enumerator UCP_WORKER_PARAM_FIELD_NAME

Worker name

enumerator UCP_WORKER_PARAM_FIELD_AM_ALIGNMENT

Alignment of active messages on the receiver

enumerator UCP_WORKER_PARAM_FIELD_CLIENT_ID

Client id

enum ucp_worker_flags_t

UCP worker flags.

This enumeration allows specifying flags for ucp_worker_params_t::flags, which is used as parameter for ucp_worker_create.

Values:

enumerator UCP_WORKER_FLAG_IGNORE_REQUEST_LEAK

Do not print warnings about request leaks

enum ucp_listener_params_field

UCP listener parameters field mask.

The enumeration allows specifying which fields in ucp_listener_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_LISTENER_PARAM_FIELD_SOCK_ADDR

Sock address and length.

enumerator UCP_LISTENER_PARAM_FIELD_ACCEPT_HANDLER

User’s callback and argument for handling the creation of an endpoint.User’s callback and argument for handling the incoming connection request.

enumerator UCP_LISTENER_PARAM_FIELD_CONN_HANDLER
enum ucp_worker_address_flags_t

UCP worker address flags.

The enumeration list describes possible UCP worker address flags, indicating what needs to be included to the worker address returned by ucp_worker_query() routine.

Values:

enumerator UCP_WORKER_ADDRESS_FLAG_NET_ONLY

Pack addresses of network devices only. Using such shortened addresses for the remote node peers will reduce the amount of wireup data being exchanged during connection establishment phase.

enum ucp_ep_params_field

UCP endpoint parameters field mask.

The enumeration allows specifying which fields in ucp_ep_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_EP_PARAM_FIELD_REMOTE_ADDRESS

Address of remote peer

enumerator UCP_EP_PARAM_FIELD_ERR_HANDLING_MODE

Error handling mode. ucp_err_handling_mode_t

enumerator UCP_EP_PARAM_FIELD_ERR_HANDLER

Handler to process transport level errors

enumerator UCP_EP_PARAM_FIELD_USER_DATA

User data pointer

enumerator UCP_EP_PARAM_FIELD_SOCK_ADDR

Socket address field

enumerator UCP_EP_PARAM_FIELD_FLAGS

Endpoint flags Connection request field

enumerator UCP_EP_PARAM_FIELD_CONN_REQUEST
enumerator UCP_EP_PARAM_FIELD_NAME

Endpoint name

enum ucp_ep_params_flags_field

UCP endpoint parameters flags.

The enumeration list describes the endpoint’s parameters flags supported by ucp_ep_create() function.

Values:

enumerator UCP_EP_PARAMS_FLAGS_CLIENT_SERVER

Using a client-server connection establishment mechanism. ucs_sock_addr_t sockaddr field must be provided and contain the address of the remote peer

enumerator UCP_EP_PARAMS_FLAGS_NO_LOOPBACK

Avoid connecting the endpoint to itself when connecting the endpoint to the same worker it was created on. Affects protocols which send to a particular remote endpoint, for example stream

enumerator UCP_EP_PARAMS_FLAGS_SEND_CLIENT_ID

Send client id when connecting to remote socket address as part of the connection request payload. On the remote side value can be obtained from ucp_conn_request_h using ucp_conn_request_query

enum ucp_ep_close_flags_t

Close UCP endpoint modes.

The enumeration is used to specify the behavior of ucp_ep_close_nbx.

Values:

enumerator UCP_EP_CLOSE_FLAG_FORCE

ucp_ep_close_nbx releases the endpoint without any confirmation from the peer. All outstanding requests will be completed with UCS_ERR_CANCELED error.

Note

This mode may cause transport level errors on remote side, so it requires set UCP_ERR_HANDLING_MODE_PEER for all endpoints created on both (local and remote) sides to avoid undefined behavior. If this flag is not set then ucp_ep_close_nbx schedules flushes on all outstanding operations.

Note

this flag is incompatible with UCP_OP_ATTR_FLAG_NO_IMM_CMPL, since it forces immediate completion.

enum ucp_ep_close_mode

Close UCP endpoint modes.

The enumeration is used to specify the behavior of ucp_ep_close_nb.

Values:

enumerator UCP_EP_CLOSE_MODE_FORCE

ucp_ep_close_nb releases the endpoint without any confirmation from the peer. All outstanding requests will be completed with UCS_ERR_CANCELED error.

Note

This mode may cause transport level errors on remote side, so it requires set UCP_ERR_HANDLING_MODE_PEER for all endpoints created on both (local and remote) sides to avoid undefined behavior.

enumerator UCP_EP_CLOSE_MODE_FLUSH

ucp_ep_close_nb schedules flushes on all outstanding operations.

enum ucp_ep_perf_param_field

UCP performance fields and flags.

The enumeration allows specifying which fields in ucp_ep_evaluate_perf_param_t are present and operation flags are used. It is used to enable backward compatibility support.

Values:

enumerator UCP_EP_PERF_PARAM_FIELD_MESSAGE_SIZE

Enables ucp_ep_evaluate_perf_param_t::message_size

enum ucp_ep_perf_attr_field

UCP performance fields and flags.

The enumeration allows specifying which fields in ucp_ep_evaluate_perf_attr_t are present and operation flags are used. It is used to enable backward compatibility support.

Values:

enumerator UCP_EP_PERF_ATTR_FIELD_ESTIMATED_TIME

Enables ucp_ep_evaluate_perf_attr_t::estimated_time

enum ucp_mem_map_params_field

UCP memory mapping parameters field mask.

The enumeration allows specifying which fields in ucp_mem_map_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_MEM_MAP_PARAM_FIELD_ADDRESS

Address of the memory that will be used in the ucp_mem_map routine.

enumerator UCP_MEM_MAP_PARAM_FIELD_LENGTH

The size of memory that will be allocated or registered in the ucp_mem_map routine.

enumerator UCP_MEM_MAP_PARAM_FIELD_FLAGS

Allocation flags.

enumerator UCP_MEM_MAP_PARAM_FIELD_PROT

Memory protection mode.

enumerator UCP_MEM_MAP_PARAM_FIELD_MEMORY_TYPE

Memory type.

enum ucp_mem_advise_params_field

UCP memory advice parameters field mask.

The enumeration allows specifying which fields in ucp_mem_advise_params_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_MEM_ADVISE_PARAM_FIELD_ADDRESS

Address of the memory

enumerator UCP_MEM_ADVISE_PARAM_FIELD_LENGTH

The size of memory

enumerator UCP_MEM_ADVISE_PARAM_FIELD_ADVICE

Advice on memory usage

enum ucp_lib_attr_field

UCP library attributes field mask.

The enumeration allows specifying which fields in ucp_lib_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_LIB_ATTR_FIELD_MAX_THREAD_LEVEL

UCP library maximum supported thread level flag

enum ucp_context_attr_field

UCP context attributes field mask.

The enumeration allows specifying which fields in ucp_context_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_ATTR_FIELD_REQUEST_SIZE

UCP request size

enumerator UCP_ATTR_FIELD_THREAD_MODE

UCP context thread flag

enumerator UCP_ATTR_FIELD_MEMORY_TYPES

UCP supported memory types

enumerator UCP_ATTR_FIELD_NAME

UCP context name

enum ucp_worker_attr_field

UCP worker attributes field mask.

The enumeration allows specifying which fields in ucp_worker_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_WORKER_ATTR_FIELD_THREAD_MODE

UCP thread mode

enumerator UCP_WORKER_ATTR_FIELD_ADDRESS

UCP address

enumerator UCP_WORKER_ATTR_FIELD_ADDRESS_FLAGS

UCP address flags

enumerator UCP_WORKER_ATTR_FIELD_MAX_AM_HEADER

Maximum header size used by UCP AM API

enumerator UCP_WORKER_ATTR_FIELD_NAME

UCP worker name

enumerator UCP_WORKER_ATTR_FIELD_MAX_INFO_STRING

Maximum size of info string

enum ucp_worker_address_attr_field

UCP worker address attributes field mask.

The enumeration allows specifying which fields in ucp_worker_address_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_WORKER_ADDRESS_ATTR_FIELD_UID

Unique id of the worker

enum ucp_listener_attr_field

UCP listener attributes field mask.

The enumeration allows specifying which fields in ucp_listener_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_LISTENER_ATTR_FIELD_SOCKADDR

Sockaddr used for listening

enum ucp_conn_request_attr_field

UCP listener’s connection request attributes field mask.

The enumeration allows specifying which fields in ucp_conn_request_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_CONN_REQUEST_ATTR_FIELD_CLIENT_ADDR

Client’s address

enumerator UCP_CONN_REQUEST_ATTR_FIELD_CLIENT_ID

Remote client id

enum ucp_dt_type

UCP data type classification.

The enumeration list describes the datatypes supported by UCP.

Values:

enumerator UCP_DATATYPE_CONTIG

Contiguous datatype

enumerator UCP_DATATYPE_STRIDED

Strided datatype

enumerator UCP_DATATYPE_IOV

Scatter-gather list with multiple pointers

enumerator UCP_DATATYPE_GENERIC

Generic datatype with user-defined pack/unpack routines

enumerator UCP_DATATYPE_SHIFT

Number of bits defining the datatype classification

enumerator UCP_DATATYPE_CLASS_MASK

Data-type class mask

enum [anonymous]

UCP memory mapping flags.

The enumeration list describes the memory mapping flags supported by ucp_mem_map() function.

Values:

enumerator UCP_MEM_MAP_NONBLOCK

Complete the mapping faster, possibly by not populating the pages in the mapping up-front, and mapping them later when they are accessed by communication routines.

enumerator UCP_MEM_MAP_ALLOCATE

Identify requirement for allocation, if passed address is not a null-pointer then it will be used as a hint or direct address for allocation.

enumerator UCP_MEM_MAP_FIXED

Don’t interpret address as a hint: place the mapping at exactly that address. The address must be a multiple of the page size.

enum [anonymous]

UCP memory mapping protection mode.

The enumeration list describes the memory mapping protections supported by the ucp_mem_map() function.

Values:

enumerator UCP_MEM_MAP_PROT_LOCAL_READ

Enable local read access.

enumerator UCP_MEM_MAP_PROT_LOCAL_WRITE

Enable local write access.

enumerator UCP_MEM_MAP_PROT_REMOTE_READ

Enable remote read access.

enumerator UCP_MEM_MAP_PROT_REMOTE_WRITE

Enable remote write access.

enum ucp_am_cb_flags

Flags for a UCP Active Message callback.

Flags that indicate how to handle UCP Active Messages.

Values:

enumerator UCP_AM_FLAG_WHOLE_MSG

Indicates that the entire message will be handled in one callback.

enumerator UCP_AM_FLAG_PERSISTENT_DATA

Guarantees that the specified ucp_am_recv_callback_t callback, will always be called with UCP_AM_RECV_ATTR_FLAG_DATA flag set, so the data will be accessible outside the callback, until ucp_am_data_release is called.

enum ucp_send_am_flags

Flags for sending a UCP Active Message.

Flags dictate the behavior of ucp_am_send_nb and ucp_am_send_nbx routines.

Values:

enumerator UCP_AM_SEND_FLAG_REPLY

Force relevant reply endpoint to be passed to the data callback on the receiver.

enumerator UCP_AM_SEND_FLAG_EAGER

Force UCP to use only eager protocol for AM sends.

enumerator UCP_AM_SEND_FLAG_RNDV

Force UCP to use only rendezvous protocol for AM sends.

enumerator UCP_AM_SEND_REPLY

Backward compatibility.

enum ucp_cb_param_flags

Descriptor flags for Active Message callback.

In a callback, if flags is set to UCP_CB_PARAM_FLAG_DATA in a callback then data was allocated, so if UCS_INPROGRESS is returned from the callback, the data parameter will persist and the user has to call ucp_am_data_release when data is no longer needed.

Values:

enumerator UCP_CB_PARAM_FLAG_DATA
enum ucp_atomic_post_op_t

Atomic operation requested for ucp_atomic_post.

This enumeration defines which atomic memory operation should be performed by the ucp_atomic_post family of functions. All of these are non-fetching atomics and will not result in a request handle.

Values:

enumerator UCP_ATOMIC_POST_OP_ADD

Atomic add

enumerator UCP_ATOMIC_POST_OP_AND

Atomic and

enumerator UCP_ATOMIC_POST_OP_OR

Atomic or

enumerator UCP_ATOMIC_POST_OP_XOR

Atomic xor

enumerator UCP_ATOMIC_POST_OP_LAST
enum ucp_atomic_fetch_op_t

Atomic operation requested for ucp_atomic_fetch.

This enumeration defines which atomic memory operation should be performed by the ucp_atomic_fetch family of functions. All of these functions will fetch data from the remote node.

Values:

enumerator UCP_ATOMIC_FETCH_OP_FADD

Atomic Fetch and add

enumerator UCP_ATOMIC_FETCH_OP_SWAP

Atomic swap

enumerator UCP_ATOMIC_FETCH_OP_CSWAP

Atomic conditional swap

enumerator UCP_ATOMIC_FETCH_OP_FAND

Atomic Fetch and and

enumerator UCP_ATOMIC_FETCH_OP_FOR

Atomic Fetch and or

enumerator UCP_ATOMIC_FETCH_OP_FXOR

Atomic Fetch and xor

enumerator UCP_ATOMIC_FETCH_OP_LAST
enum ucp_atomic_op_t

Atomic operation requested for ucp_atomic_op_nbx.

This enumeration defines which atomic memory operation should be performed by the ucp_atomic_op_nbx routine.

Values:

enumerator UCP_ATOMIC_OP_ADD

Atomic add

enumerator UCP_ATOMIC_OP_SWAP

Atomic swap

enumerator UCP_ATOMIC_OP_CSWAP

Atomic conditional swap

enumerator UCP_ATOMIC_OP_AND

Atomic and

enumerator UCP_ATOMIC_OP_OR

Atomic or

enumerator UCP_ATOMIC_OP_XOR

Atomic xor

enumerator UCP_ATOMIC_OP_LAST
enum ucp_stream_recv_flags_t

Flags to define behavior of ucp_stream_recv_nb function.

This enumeration defines behavior of ucp_stream_recv_nb function.

Values:

enumerator UCP_STREAM_RECV_FLAG_WAITALL

This flag requests that the operation will not be completed until all requested data is received and placed in the user buffer.

enum ucp_op_attr_t

UCP operation fields and flags.

The enumeration allows specifying which fields in ucp_request_param_t are present and operation flags are used. It is used to enable backward compatibility support.

Values:

enumerator UCP_OP_ATTR_FIELD_REQUEST

request field

enumerator UCP_OP_ATTR_FIELD_CALLBACK

cb field

enumerator UCP_OP_ATTR_FIELD_USER_DATA

user_data field

enumerator UCP_OP_ATTR_FIELD_DATATYPE

datatype field

enumerator UCP_OP_ATTR_FIELD_FLAGS

operation-specific flags

enumerator UCP_OP_ATTR_FIELD_REPLY_BUFFER

reply_buffer field

enumerator UCP_OP_ATTR_FIELD_MEMORY_TYPE

memory type field

enumerator UCP_OP_ATTR_FIELD_RECV_INFO

recv_info field

enumerator UCP_OP_ATTR_FLAG_NO_IMM_CMPL

deny immediate completion

enumerator UCP_OP_ATTR_FLAG_FAST_CMPL

expedite local completion, even if it delays remote data delivery. Note for implementer: this option can disable zero copy and/or rendezvous protocols which require synchronization with the remote peer before releasing the local send buffer

enumerator UCP_OP_ATTR_FLAG_FORCE_IMM_CMPL

force immediate complete operation, fail if the operation cannot be completed immediately

enumerator UCP_OP_ATTR_FLAG_MULTI_SEND

optimize for bandwidth of multiple in-flight operations, rather than for the latency of a single operation

enum ucp_req_attr_field

UCP request query attributes.

The enumeration allows specifying which fields in ucp_request_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_REQUEST_ATTR_FIELD_INFO_STRING
enumerator UCP_REQUEST_ATTR_FIELD_INFO_STRING_SIZE
enumerator UCP_REQUEST_ATTR_FIELD_STATUS
enumerator UCP_REQUEST_ATTR_FIELD_MEM_TYPE
enum ucp_am_recv_attr_t

UCP AM receive data parameter fields and flags.

The enumeration allows specifying which fields in ucp_am_recv_param_t are present and receive operation flags are used. It is used to enable backward compatibility support.

Values:

enumerator UCP_AM_RECV_ATTR_FIELD_REPLY_EP

reply_ep field

enumerator UCP_AM_RECV_ATTR_FLAG_DATA

Indicates that the data provided in ucp_am_recv_callback_t callback can be held by the user. If UCS_INPROGRESS is returned from the callback, the data parameter will persist and the user has to call ucp_am_data_release when data is no longer needed. This flag is mutually exclusive with UCP_AM_RECV_ATTR_FLAG_RNDV.

enumerator UCP_AM_RECV_ATTR_FLAG_RNDV

Indicates that the arriving data was sent using rendezvous protocol. In this case data parameter of the ucp_am_recv_callback_t points to the internal UCP descriptor, which can be used for obtaining the actual data by calling ucp_am_recv_data_nbx routine. This flag is mutually exclusive with UCP_AM_RECV_ATTR_FLAG_DATA.

enum ucp_am_handler_param_field

UCP AM receive data parameters fields and flags.

The enumeration allows specifying which fields in ucp_am_handler_param_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_AM_HANDLER_PARAM_FIELD_ID

Indicates that ucp_am_handler_param_t::id field is valid.

enumerator UCP_AM_HANDLER_PARAM_FIELD_FLAGS

Indicates that ucp_am_handler_param_t::flags field is valid.

enumerator UCP_AM_HANDLER_PARAM_FIELD_CB

Indicates that ucp_am_handler_param_t::cb field is valid.

enumerator UCP_AM_HANDLER_PARAM_FIELD_ARG

Indicates that ucp_am_handler_param_t::arg field is valid.

enum ucp_mem_advice

list of UCP memory use advice.

The enumeration list describes memory advice supported by ucp_mem_advise() function.

Values:

enumerator UCP_MADV_NORMAL

No special treatment

enumerator UCP_MADV_WILLNEED

can be used on the memory mapped with UCP_MEM_MAP_NONBLOCK to speed up memory mapping and to avoid page faults when the memory is accessed for the first time.

enum ucp_ep_attr_field

UCP endpoint attributes field mask.

The enumeration allows specifying which fields in ucp_ep_attr_t are present. It is used to enable backward compatibility support.

Values:

enumerator UCP_EP_ATTR_FIELD_NAME

UCP endpoint name

enumerator UCP_EP_ATTR_FIELD_LOCAL_SOCKADDR

Sockaddr used by the endpoint

enumerator UCP_EP_ATTR_FIELD_REMOTE_SOCKADDR

Sockaddr the endpoint is connected to

Functions

ucs_status_t ucp_lib_query(ucp_lib_attr_t *attr)

Get attributes of the UCP library.

This routine fetches information about the UCP library attributes.

Parameters

attr[out] Filled with attributes of the UCP library.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_config_read(const char *env_prefix, const char *filename, ucp_config_t **config_p)

Read UCP configuration descriptor.

The routine fetches the information about UCP library configuration from the run-time environment. Then, the fetched descriptor is used for UCP library initialization. The Application can print out the descriptor using print routine. In addition the application is responsible for releasing the descriptor back to the UCP library.

Parameters
  • env_prefix[in] If non-NULL, the routine searches for the environment variables that start with <env_prefix>_UCX_ prefix. Otherwise, the routine searches for the environment variables that start with UCX_ prefix.

  • filename[in] If non-NULL, read configuration from the file defined by filename. If the file does not exist, it will be ignored and no error reported to the application.

  • config_p[out] Pointer to configuration descriptor as defined by ucp_config_t.

Returns

Error code as defined by ucs_status_t

void ucp_config_release(ucp_config_t *config)

Release configuration descriptor.

The routine releases the configuration descriptor that was allocated through ucp_config_read() routine.

Parameters

config[out] Configuration descriptor as defined by ucp_config_t.

ucs_status_t ucp_config_modify(ucp_config_t *config, const char *name, const char *value)

Modify context configuration.

The routine changes one configuration setting stored in configuration descriptor.

Parameters
  • config[in] Configuration to modify.

  • name[in] Configuration variable name.

  • value[in] Value to set.

Returns

Error code.

void ucp_config_print(const ucp_config_t *config, FILE *stream, const char *title, ucs_config_print_flags_t print_flags)

Print configuration information.

The routine prints the configuration information that is stored in configuration descriptor.

Parameters
  • config[in] Configuration descriptor to print.

  • stream[in] Output stream to print the configuration to.

  • title[in] Configuration title to print.

  • print_flags[in] Flags that control various printing options.

void ucp_get_version(unsigned *major_version, unsigned *minor_version, unsigned *release_number)

Get UCP library version.

This routine returns the UCP library version.

Parameters
  • major_version[out] Filled with library major version.

  • minor_version[out] Filled with library minor version.

  • release_number[out] Filled with library release number.

const char *ucp_get_version_string(void)

Get UCP library version as a string.

This routine returns the UCP library version as a string which consists of: “major.minor.release”.

static inline ucs_status_t ucp_init(const ucp_params_t *params, const ucp_config_t *config, ucp_context_h *context_p)

UCP context initialization.

This routine creates and initializes a UCP application context.

This routine checks API version compatibility, then discovers the available network interfaces, and initializes the network resources required for discovering of the network and memory related devices. This routine is responsible for initialization all information required for a particular application scope, for example, MPI application, OpenSHMEM application, etc.

Parameters
Returns

Error code as defined by ucs_status_t

Warning

This routine must be called before any other UCP function call in the application.

Note

  • Higher level protocols can add additional communication isolation, as MPI does with it’s communicator object. A single communication context may be used to support multiple MPI communicators.

  • The context can be used to isolate the communication that corresponds to different protocols. For example, if MPI and OpenSHMEM are using UCP to isolate the MPI communication from the OpenSHMEM communication, users should use different application context for each of the communication libraries.

void ucp_cleanup(ucp_context_h context_p)

Release UCP application context.

This routine finalizes and releases the resources associated with a UCP application context.

The cleanup process releases and shuts down all resources associated with the application context. After calling this routine, calling any UCP routine without calling UCP initialization routine is invalid.

Parameters

context_p[in] Handle to UCP application context.

Warning

An application cannot call any UCP routine once the UCP application context released.

ucs_status_t ucp_context_query(ucp_context_h context_p, ucp_context_attr_t *attr)

Get attributes specific to a particular context.

This routine fetches information about the context.

Parameters
  • context_p[in] Handle to UCP application context.

  • attr[out] Filled with attributes of context_p context.

Returns

Error code as defined by ucs_status_t

void ucp_context_print_info(const ucp_context_h context, FILE *stream)

Print context information.

This routine prints information about the context configuration: including memory domains, transport resources, and other useful information associated with the context.

Parameters
  • context[in] Print this context object’s configuration.

  • stream[in] Output stream on which to print the information.

ucs_status_t ucp_worker_create(ucp_context_h context, const ucp_worker_params_t *params, ucp_worker_h *worker_p)

Create a worker object.

This routine allocates and initializes a worker object. Each worker is associated with one and only one application context. In the same time, an application context can create multiple workers in order to enable concurrent access to communication resources. For example, application can allocate a dedicated worker for each application thread, where every worker can be progressed independently of others.

Parameters
Returns

Error code as defined by ucs_status_t

Note

The worker object is allocated within context of the calling thread

void ucp_worker_destroy(ucp_worker_h worker)

Destroy a worker object.

This routine releases the resources associated with a UCP worker.

The destroy process releases and shuts down all resources associated with the worker.

Parameters

worker[in] Worker object to destroy.

Warning

Once the UCP worker destroy the worker handle cannot be used with any UCP routine.

ucs_status_t ucp_worker_query(ucp_worker_h worker, ucp_worker_attr_t *attr)

Get attributes specific to a particular worker.

This routine fetches information about the worker.

Parameters
  • worker[in] Worker object to query.

  • attr[out] Filled with attributes of worker.

Returns

Error code as defined by ucs_status_t

void ucp_worker_print_info(ucp_worker_h worker, FILE *stream)

Print information about the worker.

This routine prints information about the protocols being used, thresholds, UCT transport methods, and other useful information associated with the worker.

Parameters
  • worker[in] Worker object to print information for.

  • stream[in] Output stream to print the information to.

ucs_status_t ucp_worker_get_address(ucp_worker_h worker, ucp_address_t **address_p, size_t *address_length_p)

Get the address of the worker object.

This routine returns the address of the worker object. This address can be passed to remote instances of the UCP library in order to connect to this worker. The memory for the address handle is allocated by this function, and must be released by using ucp_worker_release_address() routine.

Parameters
  • worker[in] Worker object whose address to return.

  • address_p[out] A pointer to the worker address.

  • address_length_p[out] The size in bytes of the address.

Returns

Error code as defined by ucs_status_t

void ucp_worker_release_address(ucp_worker_h worker, ucp_address_t *address)

Release an address of the worker object.

This routine release an address handle associated within the worker object.

Parameters
  • worker[in] Worker object that is associated with the address object.

  • address[in] Address to release; the address object has to be allocated using ucp_worker_get_address() routine.

Warning

Once the address released the address handle cannot be used with any UCP routine.

ucs_status_t ucp_worker_address_query(ucp_address_t *address, ucp_worker_address_attr_t *attr)

Get attributes of the particular worker address.

This routine fetches information about the worker address. The address can be either of local or remote worker.

Parameters
  • address[in] Worker address to query.

  • attr[out] Filled with attributes of the worker address.

Returns

Error code as defined by ucs_status_t.

unsigned ucp_worker_progress(ucp_worker_h worker)

Progress all communications on a specific worker.

This routine explicitly progresses all communication operations on a worker.

Parameters

worker[in] Worker to progress.

Returns

Non-zero if any communication was progressed, zero otherwise.

Note

  • Typically, request wait and test routines call this routine to progress any outstanding operations.

  • Transport layers, implementing asynchronous progress using threads, require callbacks and other user code to be thread safe.

  • The state of communication can be advanced (progressed) by blocking routines. Nevertheless, the non-blocking routines can not be used for communication progress.

ssize_t ucp_stream_worker_poll(ucp_worker_h worker, ucp_stream_poll_ep_t *poll_eps, size_t max_eps, unsigned flags)

Poll for endpoints that are ready to consume streaming data.

This non-blocking routine returns endpoints on a worker which are ready to consume streaming data. The ready endpoints are placed in poll_eps array, and the function return value indicates how many are there.

Parameters
  • worker[in] Worker to poll.

  • poll_eps[out] Pointer to array of endpoints, should be allocated by user.

  • max_eps[in] Maximum number of endpoints that should be filled in poll_eps.

  • flags[in] Reserved for future use.

Returns

Negative value indicates an error according to ucs_status_t. On success, non-negative value (less or equal max_eps) indicates actual number of endpoints filled in poll_eps array.

ucs_status_t ucp_worker_get_efd(ucp_worker_h worker, int *fd)

Obtain an event file descriptor for event notification.

This routine returns a valid file descriptor for polling functions. The file descriptor will get signaled when an event occurs, as part of the wake-up mechanism. Signaling means a call to poll() or select() with this file descriptor will return at this point, with this descriptor marked as the reason (or one of the reasons) the function has returned. The user does not need to release the obtained file descriptor.

The wake-up mechanism exists to allow for the user process to register for notifications on events of the underlying interfaces, and wait until such occur. This is an alternative to repeated polling for request completion. The goal is to allow for waiting while consuming minimal resources from the system. This is recommended for cases where traffic is infrequent, and latency can be traded for lower resource consumption while waiting for it.

There are two alternative ways to use the wakeup mechanism: the first is the file descriptor obtained per worker (this function) and the second is the ucp_worker_wait function for waiting on the next event internally.

Parameters
  • worker[in] Worker of notified events.

  • fd[out] File descriptor.

Returns

Error code as defined by ucs_status_t

Note

UCP features have to be triggered with UCP_FEATURE_WAKEUP to select proper transport

ucs_status_t ucp_worker_wait(ucp_worker_h worker)

Wait for an event of the worker.

This routine waits (blocking) until an event has happened, as part of the wake-up mechanism.

This function is guaranteed to return only if new communication events occur on the worker. Therefore one must drain all existing events before waiting on the file descriptor. This can be achieved by calling ucp_worker_progress repeatedly until it returns 0.

There are two alternative ways to use the wakeup mechanism. The first is by polling on a per-worker file descriptor obtained from ucp_worker_get_efd. The second is by using this function to perform an internal wait for the next event associated with the specified worker.

Parameters

worker[in] Worker to wait for events on.

Returns

Error code as defined by ucs_status_t

Note

During the blocking call the wake-up mechanism relies on other means of notification and may not progress some of the requests as it would when calling ucp_worker_progress (which is not invoked in that duration).

Note

UCP features have to be triggered with UCP_FEATURE_WAKEUP to select proper transport

void ucp_worker_wait_mem(ucp_worker_h worker, void *address)

Wait for memory update on the address.

This routine waits for a memory update at the local memory address. This is a blocking routine. The routine returns when the memory address is updated (“write”) or an event occurs in the system.

This function is guaranteed to return only if new communication events occur on the worker or address is modified. Therefore one must drain all existing events before waiting on the file descriptor. This can be achieved by calling ucp_worker_progress repeatedly until it returns 0.

Parameters
  • worker[in] Worker to wait for updates on.

  • address[in] Local memory address

Note

This routine can be used by an application that executes busy-waiting loop checking for a memory update. Instead of continuous busy-waiting on an address the application can use ucp_worker_wait_mem, which may suspend execution until the memory is updated. The goal of the routine is to provide an opportunity for energy savings for architectures that support this functionality.

ucs_status_t ucp_worker_arm(ucp_worker_h worker)

Turn on event notification for the next event.

This routine needs to be called before waiting on each notification on this worker, so will typically be called once the processing of the previous event is over, as part of the wake-up mechanism.

The worker must be armed before waiting on an event (must be re-armed after it has been signaled for re-use) with ucp_worker_arm. The events triggering a signal of the file descriptor from ucp_worker_get_efd depend on the interfaces used by the worker and defined in the transport layer, and typically represent a request completion or newly available resources. It can also be triggered by calling ucp_worker_signal .

The file descriptor is guaranteed to become signaled only if new communication events occur on the worker. Therefore one must drain all existing events before waiting on the file descriptor. This can be achieved by calling ucp_worker_progress repeatedly until it returns 0.

void application_initialization() {
// should be called once in application init flow and before
// process_communication() is used
    ...
    status = ucp_worker_get_efd(worker, &fd);
    ...
}

void process_communication() {
// should be called every time need to wait for some condition such as
// ucp request completion in sleep mode.

    for (;;) {
        // check for stop condition as long as progress is made
        if (check_for_events()) {
             break;
        } else if (ucp_worker_progress(worker)) {
             continue;                 // some progress happened but condition not met
        }

        // arm the worker and clean-up fd
        status = ucp_worker_arm(worker);
        if (UCS_OK == status) {
            poll(&fds, nfds, timeout);  // wait for events (sleep mode)
        } else if (UCS_ERR_BUSY == status) {
            continue;                   // could not arm, need to progress more
        } else {
            abort();
        }
    }
}

Parameters

worker[in] Worker of notified events.

Returns

UCS_OK The operation completed successfully. File descriptor will be signaled by new events.

Returns

UCS_ERR_BUSY There are unprocessed events which prevent the file descriptor from being armed. These events should be removed by calling ucp_worker_progress(). The operation is not completed. File descriptor will not be signaled by new events.

Returns

Other different error codes in case of issues.

Note

UCP features have to be triggered with UCP_FEATURE_WAKEUP to select proper transport

ucs_status_t ucp_worker_signal(ucp_worker_h worker)

Cause an event of the worker.

This routine signals that the event has happened, as part of the wake-up mechanism. This function causes a blocking call to ucp_worker_wait or waiting on a file descriptor from ucp_worker_get_efd to return, even if no event from the underlying interfaces has taken place.

Parameters

worker[in] Worker to wait for events on.

Returns

Error code as defined by ucs_status_t

Note

It’s safe to use this routine from any thread, even if UCX is compiled without multi-threading support and/or initialized with any value of ucp_params_t::mt_workers_shared and ucp_worker_params_t::thread_mode parameters

ucs_status_t ucp_listener_create(ucp_worker_h worker, const ucp_listener_params_t *params, ucp_listener_h *listener_p)

Create a listener to accept connections on. Connection requests on the listener will arrive at a local address specified by the user.

This routine creates a new listener object that is bound to a specific local address. The listener will listen to incoming connection requests. After receiving a request from the remote peer, an endpoint to this peer will be created - either right away or by calling ucp_ep_create, as specified by the callback type in ucp_listener_params_t. The user’s callback will be invoked once the endpoint is created.

Parameters
Returns

Error code as defined by ucs_status_t

Note

ucp_listener_params_t::conn_handler or ucp_listener_params_t::accept_handler must be provided to be able to handle incoming connections.

void ucp_listener_destroy(ucp_listener_h listener)

Stop accepting connections on a local address of the worker object.

This routine unbinds the worker from the given handle and stops listening for incoming connection requests on it.

Parameters

listener[in] A handle to the listener to stop listening on.

ucs_status_t ucp_listener_query(ucp_listener_h listener, ucp_listener_attr_t *attr)

Get attributes specific to a particular listener.

This routine fetches information about the listener.

Parameters
  • listener[in] listener object to query.

  • attr[out] Filled with attributes of the listener.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_conn_request_query(ucp_conn_request_h conn_request, ucp_conn_request_attr_t *attr)

Get attributes specific to a particular connection request received on the server side.

This routine fetches information about the connection request.

Parameters
  • conn_request[in] connection request object to query.

  • attr[out] Filled with attributes of the connection request.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_request_query(void *request, ucp_request_attr_t *attr)

Get information about ucp_request.

Parameters
  • request[in] Non-blocking request to query.

  • attr[out] Filled with attributes of the request.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_ep_create(ucp_worker_h worker, const ucp_ep_params_t *params, ucp_ep_h *ep_p)

Create and connect an endpoint.

This routine creates and connects an endpoint on a local worker for a destination address that identifies the remote worker. This function is non-blocking, and communications may begin immediately after it returns. If the connection process is not completed, communications may be delayed. The created endpoint is associated with one and only one worker.

Parameters
  • worker[in] Handle to the worker; the endpoint is associated with the worker.

  • params[in] User defined ucp_ep_params_t configurations for the UCP endpoint.

  • ep_p[out] A handle to the created endpoint.

Returns

Error code as defined by ucs_status_t

Note

One of the following fields has to be specified:

Note

By default, ucp_ep_create() will connect an endpoint to itself if the endpoint is destined to the same worker on which it was created, i.e. params.address belongs to worker. This behavior can be changed by passing the UCP_EP_PARAMS_FLAGS_NO_LOOPBACK flag in params.flags. In that case, the endpoint will be connected to the next endpoint created in the same way on the same worker.

ucs_status_ptr_t ucp_ep_close_nb(ucp_ep_h ep, unsigned mode)

Non-blocking endpoint closure.

This routine releases the endpoint. The endpoint closure process depends on the selected mode.

Parameters
  • ep[in] Handle to the endpoint to close.

  • mode[in] One from ucp_ep_close_mode value.

Returns

UCS_OK - The endpoint is closed successfully.

Returns

UCS_PTR_IS_ERR(_ptr) - The closure failed and an error code indicates the transport level status. However, resources are released and the endpoint can no longer be used.

Returns

otherwise - The closure process is started, and can be completed at any point in time. A request handle is returned to the application in order to track progress of the endpoint closure. The application is responsible for releasing the handle using the ucp_request_free routine.

Note

ucp_ep_close_nb replaces deprecated ucp_disconnect_nb and ucp_ep_destroy

ucs_status_ptr_t ucp_ep_close_nbx(ucp_ep_h ep, const ucp_request_param_t *param)

Non-blocking endpoint closure.

Parameters
Returns

NULL - The endpoint is closed successfully.

Returns

UCS_PTR_IS_ERR(_ptr) - The closure failed and an error code indicates the transport level status. However, resources are released and the endpoint can no longer be used.

Returns

otherwise - The closure process is started, and can be completed at any point in time. A request handle is returned to the application in order to track progress of the endpoint closure.

ucs_status_t ucp_listener_reject(ucp_listener_h listener, ucp_conn_request_h conn_request)

Reject an incoming connection request.

Reject the incoming connection request and release associated resources. If the remote initiator endpoint has set an ucp_ep_params_t::err_handler, it will be invoked with status UCS_ERR_REJECTED.

Parameters
  • listener[in] Handle to the listener on which the connection request was received.

  • conn_request[in] Handle to the connection request to reject.

Returns

Error code as defined by ucs_status_t

void ucp_ep_print_info(ucp_ep_h ep, FILE *stream)

Print endpoint information.

This routine prints information about the endpoint transport methods, their thresholds, and other useful information associated with the endpoint.

Parameters
  • ep[in] Endpoint object whose configuration to print.

  • stream[in] Output stream to print the information to.

ucs_status_ptr_t ucp_ep_flush_nb(ucp_ep_h ep, unsigned flags, ucp_send_callback_t cb)

Non-blocking flush of outstanding AMO and RMA operations on the endpoint.

This routine flushes all outstanding AMO and RMA communications on the endpoint. All the AMO and RMA operations issued on the ep prior to this call are completed both at the origin and at the target endpoint when this call returns.

The following example demonstrates how blocking flush can be implemented using non-blocking flush:

void empty_function(void *request, ucs_status_t status)
{
}

ucs_status_t blocking_ep_flush(ucp_ep_h ep, ucp_worker_h worker)
{
    void *request;

    request = ucp_ep_flush_nb(ep, 0, empty_function);
    if (request == NULL) {
        return UCS_OK;
    } else if (UCS_PTR_IS_ERR(request)) {
        return UCS_PTR_STATUS(request);
    } else {
        ucs_status_t status;
        do {
            ucp_worker_progress(worker);
            status = ucp_request_check_status(request);
        } while (status == UCS_INPROGRESS);
        ucp_request_free(request);
        return status;
    }
}

Parameters
  • ep[in] UCP endpoint.

  • flags[in] Flags for flush operation. Reserved for future use.

  • cb[in] Callback which will be called when the flush operation completes.

Returns

NULL - The flush operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The flush operation failed.

Returns

otherwise - Flush operation was scheduled and can be completed in any point in time. The request handle is returned to the application in order to track progress. The application is responsible for releasing the handle using ucp_request_free() routine.

ucs_status_ptr_t ucp_ep_flush_nbx(ucp_ep_h ep, const ucp_request_param_t *param)

Non-blocking flush of outstanding AMO and RMA operations on the endpoint.

This routine flushes all outstanding AMO and RMA communications on the endpoint. All the AMO and RMA operations issued on the ep prior to this call are completed both at the origin and at the target endpoint when this call returns.

Parameters
Returns

NULL - The flush operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The flush operation failed.

Returns

otherwise - Flush operation was scheduled and can be completed in any point in time. The request handle is returned to the application in order to track progress.

ucs_status_t ucp_ep_evaluate_perf(ucp_ep_h ep, const ucp_ep_evaluate_perf_param_t *param, ucp_ep_evaluate_perf_attr_t *attr)

Estimate performance characteristics of a specific endpoint.

This routine fetches information about the endpoint.

Parameters
  • ep[in] Endpoint to query.

  • param[in] Filled by the user with request params.

  • attr[out] Filled with performance estimation of the given operation on the endpoint.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_mem_map(ucp_context_h context, const ucp_mem_map_params_t *params, ucp_mem_h *memh_p)

Map or allocate memory for zero-copy operations.

This routine maps or/and allocates a user-specified memory segment with UCP application context and the network resources associated with it. If the application specifies NULL as an address for the memory segment, the routine allocates a mapped memory segment and returns its address in the address_p argument. The network stack associated with an application context can typically send and receive data from the mapped memory without CPU intervention; some devices and associated network stacks require the memory to be mapped to send and receive data. The memory handle includes all information required to access the memory locally using UCP routines, while remote registration handle provides an information that is necessary for remote memory access.

Memory mapping assumptions:

  • A given memory segment can be mapped by several different communication stacks, if these are compatible.

  • The memh_p handle returned may be used with any sub-region of the mapped memory.

  • If a large segment is registered, and then segmented for subsequent use by a user, then the user is responsible for segmentation and subsequent management.

parameter/flag

NONBLOCK

ALLOCATE

FIXED

address

result

value

0/1 - the value

only affects the

register/map

phase

0

0

0

error if length > 0

1

0

0

alloc+register

0

1

0

error

0

0

defined

register

1

1

0

error

1

0

defined

alloc+register,hint

0

1

defined

error

1

1

defined

alloc+register,fixed

Parameters
Returns

Error code as defined by ucs_status_t

Note

Another well know terminology for the “map” operation that is typically used in the context of networking is memory “registration” or “pinning”. The UCP library registers the memory the available hardware so it can be assessed directly by the hardware.

Note

  • register means that the memory will be registered in corresponding transports for RMA/AMO operations. This case intends that the memory was allocated by user before.

  • alloc+register means that the memory will be allocated in the memory provided by the system and registered in corresponding transports for RMA/AMO operations.

  • alloc+register,hint means that the memory will be allocated with using ucp_mem_map_params::address as a hint and registered in corresponding transports for RMA/AMO operations.

  • alloc+register,fixed means that the memory will be allocated and registered in corresponding transports for RMA/AMO operations.

  • error is an erroneous combination of the parameters.

ucs_status_t ucp_mem_unmap(ucp_context_h context, ucp_mem_h memh)

Unmap memory segment.

This routine unmaps a user specified memory segment, that was previously mapped using the ucp_mem_map() routine. The unmap routine will also release the resources associated with the memory handle. When the function returns, the ucp_mem_h and associated remote key will be invalid and cannot be used with any UCP routine.

Error cases:

  • Once memory is unmapped a network access to the region may cause a failure.

Parameters
  • context[in] Application context which was used to allocate/map the memory.

  • memh[in] Handle to memory region.

Returns

Error code as defined by ucs_status_t

Note

Another well know terminology for the “unmap” operation that is typically used in the context of networking is memory “de-registration”. The UCP library de-registers the memory the available hardware so it can be returned back to the operation system.

ucs_status_t ucp_mem_query(const ucp_mem_h memh, ucp_mem_attr_t *attr)

query mapped memory segment

This routine returns address and length of memory segment mapped with ucp_mem_map() routine.

Parameters
Returns

Error code as defined by ucs_status_t

void ucp_mem_print_info(const char *mem_size, ucp_context_h context, FILE *stream)

Print memory mapping information.

This routine maps memory and prints information about the created memory handle: including the mapped memory length, the allocation method, and other useful information associated with the memory handle.

Parameters
  • mem_size[in] Size of the memory to map.

  • context[in] The context on which the memory is mapped.

  • stream[in] Output stream on which to print the information.

ucs_status_t ucp_mem_advise(ucp_context_h context, ucp_mem_h memh, ucp_mem_advise_params_t *params)

give advice about the use of memory

This routine advises the UCP about how to handle memory range beginning at address and size of length bytes. This call does not influence the semantics of the application, but may influence its performance. The UCP may ignore the advice.

Parameters
  • context[in] Application context which was used to allocate/map the memory.

  • memh[in] Handle to memory region.

  • params[in] Memory base address and length. The advice field is used to pass memory use advice as defined in the ucp_mem_advice list The memory range must belong to the memh

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_rkey_pack(ucp_context_h context, ucp_mem_h memh, void **rkey_buffer_p, size_t *size_p)

Pack memory region remote access key.

This routine allocates memory buffer and packs into the buffer a remote access key (RKEY) object. RKEY is an opaque object that provides the information that is necessary for remote memory access. This routine packs the RKEY object in a portable format such that the object can be unpacked on any platform supported by the UCP library. In order to release the memory buffer allocated by this routine the application is responsible for calling the ucp_rkey_buffer_release() routine.

Parameters
  • context[in] Application context which was used to allocate/map the memory.

  • memh[in] Handle to memory region.

  • rkey_buffer_p[out] Memory buffer allocated by the library. The buffer contains packed RKEY.

  • size_p[out] Size (in bytes) of the packed RKEY.

Returns

Error code as defined by ucs_status_t

Note

  • RKEYs for InfiniBand and Cray Aries networks typically includes InfiniBand and Aries key.

  • In order to enable remote direct memory access to the memory associated with the memory handle the application is responsible for sharing the RKEY with the peers that will initiate the access.

void ucp_rkey_buffer_release(void *rkey_buffer)

Release packed remote key buffer.

This routine releases the buffer that was allocated using ucp_rkey_pack().

Parameters

rkey_buffer[in] Buffer to release.

Warning

  • Once memory is released an access to the memory may cause a failure.

  • If the input memory address was not allocated using ucp_rkey_pack() routine the behaviour of this routine is undefined.

ucs_status_t ucp_ep_rkey_unpack(ucp_ep_h ep, const void *rkey_buffer, ucp_rkey_h *rkey_p)

Create remote access key from packed buffer.

This routine unpacks the remote key (RKEY) object into the local memory such that it can be accessed and used by UCP routines. The RKEY object has to be packed using the ucp_rkey_pack() routine. Application code should not make any changes to the content of the RKEY buffer.

Parameters
  • ep[in] Endpoint to access using the remote key.

  • rkey_buffer[in] Packed rkey.

  • rkey_p[out] Remote key handle.

Returns

Error code as defined by ucs_status_t

Note

The application is responsible for releasing the RKEY object when it is no longer needed, by calling the ucp_rkey_destroy() routine.

Note

The remote key object can be used for communications only on the endpoint on which it was unpacked.

ucs_status_t ucp_rkey_ptr(ucp_rkey_h rkey, uint64_t raddr, void **addr_p)

Get a local pointer to remote memory.

This routine returns a local pointer to the remote memory described by the rkey.

Parameters
  • rkey[in] A remote key handle.

  • raddr[in] A remote memory address within the memory area described by the rkey.

  • addr_p[out] A pointer that can be used for direct access to the remote memory.

Returns

Error code as defined by ucs_status_t if the remote memory cannot be accessed directly or the remote memory address is not valid.

Note

This routine can return a valid pointer only for the endpoints that are reachable via shared memory.

void ucp_rkey_destroy(ucp_rkey_h rkey)

Destroy the remote key.

This routine destroys the RKEY object and the memory that was allocated using the ucp_ep_rkey_unpack() routine. This routine also releases any resources that are associated with the RKEY object.

Parameters

rkey[in] Remote key to destroy.

Warning

  • Once the RKEY object is released an access to the memory will cause an undefined failure.

  • If the RKEY object was not created using ucp_ep_rkey_unpack() routine the behavior of this routine is undefined.

  • The RKEY object must be destroyed after all outstanding operations which are using it are flushed, and before the endpoint on which it was unpacked is destroyed.

ucs_status_t ucp_worker_set_am_handler(ucp_worker_h worker, uint16_t id, ucp_am_callback_t cb, void *arg, uint32_t flags)

Add user defined callback for Active Message.

This routine installs a user defined callback to handle incoming Active Messages with a specific id. This callback is called whenever an Active Message that was sent from the remote peer by ucp_am_send_nb is received on this worker.

Parameters
  • worker[in] UCP worker on which to set the Active Message handler.

  • id[in] Active Message id.

  • cb[in] Active Message callback. NULL to clear.

  • arg[in] Active Message argument, which will be passed in to every invocation of the callback as the arg argument.

  • flags[in] Dictates how an Active Message is handled on the remote endpoint. Currently only UCP_AM_FLAG_WHOLE_MSG is supported, which indicates the callback will not be invoked until all data has arrived.

Returns

error code if the worker does not support Active Messages or requested callback flags.

ucs_status_t ucp_worker_set_am_recv_handler(ucp_worker_h worker, const ucp_am_handler_param_t *param)

Add user defined callback for Active Message.

This routine installs a user defined callback to handle incoming Active Messages with a specific id. This callback is called whenever an Active Message that was sent from the remote peer by ucp_am_send_nbx is received on this worker.

Parameters
  • worker[in] UCP worker on which to set the Active Message handler.

  • param[in] Active Message handler parameters, as defined by ucp_am_handler_param_t.

Returns

error code if the worker does not support Active Messages or requested callback flags.

Warning

Handlers set by this function are not compatible with ucp_am_send_nb routine.

ucs_status_ptr_t ucp_am_send_nb(ucp_ep_h ep, uint16_t id, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_send_callback_t cb, unsigned flags)

Send Active Message.

This routine sends an Active Message to an ep. It does not support CUDA memory.

Parameters
  • ep[in] UCP endpoint where the Active Message will be run.

  • id[in] Active Message id. Specifies which registered callback to run.

  • buffer[in] Pointer to the data to be sent to the target node of the Active Message.

  • count[in] Number of elements to send.

  • datatype[in] Datatype descriptor for the elements in the buffer.

  • cb[in] Callback that is invoked upon completion of the data transfer if it is not completed immediately.

  • flags[in] Operation flags as defined by ucp_send_am_flags.

Returns

NULL Active Message was sent immediately.

Returns

UCS_PTR_IS_ERR(_ptr) Error sending Active Message.

Returns

otherwise Pointer to request, and Active Message is known to be completed after cb is run.

ucs_status_ptr_t ucp_am_send_nbx(ucp_ep_h ep, unsigned id, const void *header, size_t header_length, const void *buffer, size_t count, const ucp_request_param_t *param)

Send Active Message.

This routine sends an Active Message to an ep. If the operation completes immediately, then the routine returns NULL and the callback function is ignored, even if specified. Otherwise, if no error is reported and a callback is requested (i.e. the UCP_OP_ATTR_FIELD_CALLBACK flag is set in the op_attr_mask field of param), then the UCP library will schedule invocation of the callback routine param->cb.send upon completion of the operation.

Parameters
  • ep[in] UCP endpoint where the Active Message will be run.

  • id[in] Active Message id. Specifies which registered callback to run.

  • header[in] User defined Active Message header. NULL value is allowed if no header needed. In this case header_length should be set to 0.

  • header_length[in] Active message header length in bytes.

  • buffer[in] Pointer to the data to be sent to the target node of the Active Message.

  • count[in] Number of elements to send.

  • param[in] Operation parameters, see ucp_request_param_t.

Returns

NULL - Active Message was sent immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - Error sending Active Message.

Returns

otherwise - Operation was scheduled for send and can be completed at any point in time. The request handle is returned to the application in order to track progress of the message. If user request was not provided in param->request, the application is responsible for releasing the handle using ucp_request_free routine.

Note

If UCP_OP_ATTR_FLAG_NO_IMM_CMPL flag is set in the op_attr_mask field of param, then the operation will return a request handle, even if it completes immediately.

Note

This operation supports specific flags, which can be passed in param by ucp_request_param_t::flags. The exact set of flags is defined by ucp_send_am_flags.

Note

Sending only header without actual data is allowed and is recommended for transferring a latency-critical amount of data.

Note

The maximum allowed header size can be obtained by querying worker attributes by the ucp_worker_query routine.

ucs_status_ptr_t ucp_am_recv_data_nbx(ucp_worker_h worker, void *data_desc, void *buffer, size_t count, const ucp_request_param_t *param)

Receive Active Message as defined by provided data descriptor.

This routine receives a message that is described by the data descriptor data_desc, local address buffer, size count and param parameters on the worker. The routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. If the receive operation cannot be started the routine returns an error.

Parameters
  • worker[in] Worker that is used for the receive operation.

  • data_desc[in] Data descriptor, provided in ucp_am_recv_callback_t routine.

  • buffer[in] Pointer to the buffer to receive the data.

  • count[in] Number of elements to receive into buffer.

  • param[in] Operation parameters, see ucp_request_param_t.

Returns

NULL - The receive operation was completed immediately. In this case, if param->recv_info.length is specified in the param, the value to which it points is updated with the size of the received message.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Receive operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track operation progress. If user request was not provided in param->request, the application is responsible for releasing the handle using ucp_request_free routine.

Note

This routine can be performed on any valid data descriptor delivered in ucp_am_recv_callback_t. Data descriptor is considered to be valid if:

  • It is a rendezvous request (UCP_AM_RECV_ATTR_FLAG_RNDV is set in ucp_am_recv_param_t::recv_attr) or

  • It is a persistent data pointer (UCP_AM_RECV_ATTR_FLAG_DATA is set in ucp_am_recv_param_t::recv_attr). In this case receive operation may be needed to unpack data to device memory (for example GPU device) or some specific datatype.

Note

After this call UCP takes ownership of data_desc descriptor, so there is no need to release it even if the operation fails. The routine returns a request handle instead, which can be used for tracking operation progress.

void ucp_am_data_release(ucp_worker_h worker, void *data)

Releases Active Message data.

This routine releases data that persisted through an Active Message callback because that callback returned UCS_INPROGRESS.

Parameters
  • worker[in] Worker which received the Active Message.

  • data[in] Pointer to data that was passed into the Active Message callback as the data parameter.

ucs_status_ptr_t ucp_stream_send_nb(ucp_ep_h ep, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_send_callback_t cb, unsigned flags)

Non-blocking stream send operation.

This routine sends data that is described by the local address buffer, size count, and datatype object to the destination endpoint ep. The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine returns UCS_OK and the callback function cb is not invoked. If the operation is not completed immediately and no error reported, then the UCP library will schedule invocation of the callback cb upon completion of the send operation. In other words, the completion of the operation will be signaled either by the return code or by the callback.

Parameters
  • ep[in] Destination endpoint handle.

  • buffer[in] Pointer to the message buffer (payload).

  • count[in] Number of elements to send.

  • datatype[in] Datatype descriptor for the elements in the buffer.

  • cb[in] Callback function that is invoked whenever the send operation is completed. It is important to note that the callback is only invoked in the event that the operation cannot be completed in place.

  • flags[in] Reserved for future use.

Returns

NULL - The send operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message. The application is responsible for releasing the handle using ucp_request_free routine.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_ptr_t ucp_stream_send_nbx(ucp_ep_h ep, const void *buffer, size_t count, const ucp_request_param_t *param)

Non-blocking stream send operation.

This routine sends data that is described by the local address buffer, size count object to the destination endpoint ep. The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine returns UCS_OK.

Parameters
  • ep[in] Destination endpoint handle.

  • buffer[in] Pointer to the message buffer (payload).

  • count[in] Number of elements to send.

  • param[in] Operation parameters, see ucp_request_param_t.

Returns

NULL - The send operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed at any point in time. The request handle is returned to the application in order to track progress of the message.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_ptr_t ucp_tag_send_nb(ucp_ep_h ep, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, ucp_send_callback_t cb)

Non-blocking tagged-send operations.

This routine sends a messages that is described by the local address buffer, size count, and datatype object to the destination endpoint ep. Each message is associated with a tag value that is used for message matching on the receiver. The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine return UCS_OK and the call-back function cb is not invoked. If the operation is not completed immediately and no error reported then the UCP library will schedule to invoke the call-back cb whenever the send operation will be completed. In other words, the completion of a message can be signaled by the return code or the call-back.

Parameters
  • ep[in] Destination endpoint handle.

  • buffer[in] Pointer to the message buffer (payload).

  • count[in] Number of elements to send

  • datatype[in] Datatype descriptor for the elements in the buffer.

  • tag[in] Message tag.

  • cb[in] Callback function that is invoked whenever the send operation is completed. It is important to note that the call-back is only invoked in a case when the operation cannot be completed in place.

Returns

NULL - The send operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_t ucp_tag_send_nbr(ucp_ep_h ep, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, void *req)

Non-blocking tagged-send operations with user provided request.

This routine provides a convenient and efficient way to implement a blocking send pattern. It also completes requests faster than ucp_tag_send_nb() because:

  • it always uses eager protocol to send data up to the rendezvous threshold.

  • its rendezvous threshold is higher than the one used by the ucp_tag_send_nb(). The threshold is controlled by the UCX_SEND_NBR_RNDV_THRESH environment variable.

  • its request handling is simpler. There is no callback and no need to allocate and free requests. In fact request can be allocated by caller on the stack.

This routine sends a messages that is described by the local address buffer, size count, and datatype object to the destination endpoint ep. Each message is associated with a tag value that is used for message matching on the receiver.

The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine returns UCS_OK.

If the operation is not completed immediately and no error reported then the UCP library will fill a user provided req and return UCS_INPROGRESS status. In order to monitor completion of the operation ucp_request_check_status() should be used.

Following pseudo code implements a blocking send function:

MPI_send(...)
{
    char *request;
    ucs_status_t status;

    // allocate request on the stack
    // ucp_context_query() was used to get ucp_request_size
    request = alloca(ucp_request_size);

    // note: make sure that there is enough memory before the
    // request handle
    status = ucp_tag_send_nbr(ep, ..., request + ucp_request_size);
    if (status != UCS_INPROGRESS) {
        return status;
    }

    do {
        ucp_worker_progress(worker);
        status = ucp_request_check_status(request + ucp_request_size);
    } while (status == UCS_INPROGRESS);

    return status;
}

Parameters
  • ep[in] Destination endpoint handle.

  • buffer[in] Pointer to the message buffer (payload).

  • count[in] Number of elements to send

  • datatype[in] Datatype descriptor for the elements in the buffer.

  • tag[in] Message tag.

  • req[in] Request handle allocated by the user. There should be at least UCP request size bytes of available space before the req. The size of UCP request can be obtained by ucp_context_query function.

Returns

UCS_OK - The send operation was completed immediately.

Returns

UCS_INPROGRESS - The send was not completed and is in progress. ucp_request_check_status() should be used to monitor req status.

Returns

Error code as defined by ucs_status_t

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_ptr_t ucp_tag_send_sync_nb(ucp_ep_h ep, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, ucp_send_callback_t cb)

Non-blocking synchronous tagged-send operation.

Same as ucp_tag_send_nb, except the request completes only after there is a remote tag match on the message (which does not always mean the remote receive has been completed). This function never completes “in-place”, and always returns a request handle.

Parameters
  • ep[in] Destination endpoint handle.

  • buffer[in] Pointer to the message buffer (payload).

  • count[in] Number of elements to send

  • datatype[in] Datatype descriptor for the elements in the buffer.

  • tag[in] Message tag.

  • cb[in] Callback function that is invoked whenever the send operation is completed.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

Note

Returns UCS_ERR_UNSUPPORTED if UCP_ERR_HANDLING_MODE_PEER is enabled. This is a temporary implementation-related constraint that will be addressed in future releases.

ucs_status_ptr_t ucp_tag_send_nbx(ucp_ep_h ep, const void *buffer, size_t count, ucp_tag_t tag, const ucp_request_param_t *param)

Non-blocking tagged-send operation.

This routine sends a messages that is described by the local address buffer, size count object to the destination endpoint ep. Each message is associated with a tag value that is used for message matching on the ucp_tag_recv_nb or receiver. The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine returns UCS_OK and the call-back function is not invoked. If the operation is not completed immediately and no error reported then the UCP library will schedule to invoke the call-back whenever the send operation is completed. In other words, the completion of a message can be signaled by the return code or the call-back. Immediate completion signals can be fine-tuned via the ucp_request_param_t::op_attr_mask field in the ucp_request_param_t structure. The values of this field are a bit-wise OR of the ucp_op_attr_t enumeration.

Parameters
  • ep[in] Destination endpoint handle.

  • buffer[in] Pointer to the message buffer (payload).

  • count[in] Number of elements to send

  • tag[in] Message tag.

  • param[in] Operation parameters, see ucp_request_param_t

Returns

UCS_OK - The send operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_ptr_t ucp_tag_send_sync_nbx(ucp_ep_h ep, const void *buffer, size_t count, ucp_tag_t tag, const ucp_request_param_t *param)

Non-blocking synchronous tagged-send operation.

Same as ucp_tag_send_nbx, except the request completes only after there is a remote tag match on the message (which does not always mean the remote receive has been completed). This function never completes “in-place”, and always returns a request handle.

Parameters
  • ep[in] Destination endpoint handle.

  • buffer[in] Pointer to the message buffer (payload).

  • count[in] Number of elements to send

  • tag[in] Message tag.

  • param[in] Operation parameters, see ucp_request_param_t

Returns

UCS_OK - The send operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

Note

Returns UCS_ERR_UNSUPPORTED if UCP_ERR_HANDLING_MODE_PEER is enabled. This is a temporary implementation-related constraint that will be addressed in future releases.

ucs_status_ptr_t ucp_stream_recv_nb(ucp_ep_h ep, void *buffer, size_t count, ucp_datatype_t datatype, ucp_stream_recv_callback_t cb, size_t *length, unsigned flags)

Non-blocking stream receive operation of structured data into a user-supplied buffer.

This routine receives data that is described by the local address buffer, size count, and datatype object on the endpoint ep. The routine is non-blocking and therefore returns immediately. The receive operation is considered complete when the message is delivered to the buffer. If data is not immediately available, the operation will be scheduled for receive and a request handle will be returned. In order to notify the application about completion of a scheduled receive operation, the UCP library will invoke the call-back cb when data is in the receive buffer and ready for application access. If the receive operation cannot be started, the routine returns an error.

Parameters
  • ep[in] UCP endpoint that is used for the receive operation.

  • buffer[in] Pointer to the buffer to receive the data.

  • count[in] Number of elements to receive into buffer.

  • datatype[in] Datatype descriptor for the elements in the buffer.

  • cb[in] Callback function that is invoked whenever the receive operation is completed and the data is ready in the receive buffer. It is important to note that the call-back is only invoked in a case when the operation cannot be completed immediately.

  • length[out] Size of the received data in bytes. The value is valid only if return code is UCS_OK.

  • flags[in] Flags defined in ucp_stream_recv_flags_t.

Returns

NULL - The receive operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. A request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle by calling the ucp_request_free routine.

Note

The amount of data received, in bytes, is always an integral multiple of the datatype size.

ucs_status_ptr_t ucp_stream_recv_nbx(ucp_ep_h ep, void *buffer, size_t count, size_t *length, const ucp_request_param_t *param)

Non-blocking stream receive operation of structured data into a user-supplied buffer.

This routine receives data that is described by the local address buffer, size count object on the endpoint ep. The routine is non-blocking and therefore returns immediately. The receive operation is considered complete when the message is delivered to the buffer. If the receive operation cannot be started, the routine returns an error.

Parameters
  • ep[in] UCP endpoint that is used for the receive operation.

  • buffer[in] Pointer to the buffer that will receive the data.

  • count[in] Number of elements to receive into buffer.

  • length[out] Size of the received data in bytes. The value is valid only if return code is NULL.

  • param[in] Operation parameters, see ucp_request_param_t. This operation supports specific flags, which can be passed in param by ucp_request_param_t::flags. The exact set of flags is defined by ucp_stream_recv_flags_t.

Returns

NULL - The receive operation was completed immediately. In this case the value pointed by length is updated by the size of received data. Note param->recv_info is not relevant for this function.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. A request handle is returned to the application in order to track progress of the operation.

Note

The amount of data received, in bytes, is always an integral multiple of the datatype size.

ucs_status_ptr_t ucp_stream_recv_data_nb(ucp_ep_h ep, size_t *length)

Non-blocking stream receive operation of unstructured data into a UCP-supplied buffer.

This routine receives any available data from endpoint ep. Unlike ucp_stream_recv_nb, the returned data is unstructured and is treated as an array of bytes. If data is immediately available, UCS_STATUS_PTR(_ptr) is returned as a pointer to the data, and length is set to the size of the returned data buffer. The routine is non-blocking and therefore returns immediately.

Parameters
  • ep[in] UCP endpoint that is used for the receive operation.

  • length[out] Length of received data.

Returns

NULL - No received data available on the ep.

Returns

UCS_PTR_IS_ERR(_ptr) - the receive operation failed and UCS_PTR_STATUS(_ptr) indicates an error.

Returns

otherwise - The pointer to the data UCS_STATUS_PTR(_ptr) is returned to the application. After the data is processed, the application is responsible for releasing the data buffer by calling the ucp_stream_data_release routine.

Note

This function returns packed data (equivalent to ucp_dt_make_contig(1)).

Note

This function returns a pointer to a UCP-supplied buffer, whereas ucp_stream_recv_nb places the data into a user-provided buffer. In some cases, receiving data directly into a UCP-supplied buffer can be more optimal, for example by processing the incoming data in-place and thus avoiding extra memory copy operations.

ucs_status_ptr_t ucp_tag_recv_nb(ucp_worker_h worker, void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, ucp_tag_t tag_mask, ucp_tag_recv_callback_t cb)

Non-blocking tagged-receive operation.

This routine receives a message that is described by the local address buffer, size count, and datatype object on the worker. The tag value of the receive message has to match the tag and tag_mask values, where the tag_mask indicates which bits of the tag have to be matched. The routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to notify the application about completion of the receive operation the UCP library will invoke the call-back cb when the received message is in the receive buffer and ready for application access. If the receive operation cannot be stated the routine returns an error.

Parameters
  • worker[in] UCP worker that is used for the receive operation.

  • buffer[in] Pointer to the buffer to receive the data.

  • count[in] Number of elements to receive

  • datatype[in] Datatype descriptor for the elements in the buffer.

  • tag[in] Message tag to expect.

  • tag_mask[in] Bit mask that indicates the bits that are used for the matching of the incoming tag against the expected tag.

  • cb[in] Callback function that is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

This routine cannot return UCS_OK. It always returns a request handle or an error.

ucs_status_t ucp_tag_recv_nbr(ucp_worker_h worker, void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, ucp_tag_t tag_mask, void *req)

Non-blocking tagged-receive operation.

This routine receives a message that is described by the local address buffer, size count, and datatype object on the worker. The tag value of the receive message has to match the tag and tag_mask values, where the tag_mask indicates which bits of the tag have to be matched. The routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to monitor completion of the operation ucp_request_check_status or ucp_tag_recv_request_test should be used.

Parameters
  • worker[in] UCP worker that is used for the receive operation.

  • buffer[in] Pointer to the buffer to receive the data.

  • count[in] Number of elements to receive

  • datatype[in] Datatype descriptor for the elements in the buffer.

  • tag[in] Message tag to expect.

  • tag_mask[in] Bit mask that indicates the bits that are used for the matching of the incoming tag against the expected tag.

  • req[in] Request handle allocated by the user. There should be at least UCP request size bytes of available space before the req. The size of UCP request can be obtained by ucp_context_query function.

Returns

Error code as defined by ucs_status_t

ucs_status_ptr_t ucp_tag_recv_nbx(ucp_worker_h worker, void *buffer, size_t count, ucp_tag_t tag, ucp_tag_t tag_mask, const ucp_request_param_t *param)

Non-blocking tagged-receive operation.

This routine receives a message that is described by the local address buffer, size count, and info object on the worker. The tag value of the receive message has to match the tag and tag_mask values, where the tag_mask indicates what bits of the tag have to be matched. The routine is a non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to notify the application about completion of the receive operation the UCP library will invoke the call-back cb when the received message is in the receive buffer and ready for application access. If the receive operation cannot be stated the routine returns an error.

Parameters
  • worker[in] UCP worker that is used for the receive operation.

  • buffer[in] Pointer to the buffer to receive the data.

  • count[in] Number of elements to receive

  • tag[in] Message tag to expect.

  • tag_mask[in] Bit mask that indicates the bits that are used for the matching of the incoming tag against the expected tag.

  • param[in] Operation parameters, see ucp_request_param_t

Returns

NULL - The receive operation was completed immediately. In this case, if param->recv_info.tag_info is specified in the param, the value to which it points is updated with the information about the received message.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

ucp_tag_message_h ucp_tag_probe_nb(ucp_worker_h worker, ucp_tag_t tag, ucp_tag_t tag_mask, int remove, ucp_tag_recv_info_t *info)

Non-blocking probe and return a message.

This routine probes (checks) if a messages described by the tag and tag_mask was received (fully or partially) on the worker. The tag value of the received message has to match the tag and tag_mask values, where the tag_mask indicates what bits of the tag have to be matched. The function returns immediately and if the message is matched it returns a handle for the message.

Parameters
  • worker[in] UCP worker that is used for the probe operation.

  • tag[in] Message tag to probe for.

  • tag_mask[in] Bit mask that indicates the bits that are used for the matching of the incoming tag against the expected tag.

  • remove[in] The flag indicates if the matched message has to be removed from UCP library. If true (1), the message handle is removed from the UCP library and the application is responsible to call ucp_tag_msg_recv_nb() in order to receive the data and release the resources associated with the message handle. If false (0), the return value is merely an indication to whether a matching message is present, and it cannot be used in any other way, and in particular it cannot be passed to ucp_tag_msg_recv_nb().

  • info[out] If the matching message is found the descriptor is filled with the details about the message.

Returns

NULL - No match found.

Returns

Message handle (not NULL) - If message is matched the message handle is returned.

Note

This function does not advance the communication state of the network. If this routine is used in busy-poll mode, need to make sure ucp_worker_progress() is called periodically to extract messages from the transport.

ucs_status_ptr_t ucp_tag_msg_recv_nb(ucp_worker_h worker, void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_message_h message, ucp_tag_recv_callback_t cb)

Non-blocking receive operation for a probed message.

This routine receives a message that is described by the local address buffer, size count, message handle, and datatype object on the worker. The message handle can be obtained by calling the ucp_tag_probe_nb() routine. The ucp_tag_msg_recv_nb() routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to notify the application about completion of the receive operation the UCP library will invoke the call-back cb when the received message is in the receive buffer and ready for application access. If the receive operation cannot be started the routine returns an error.

Parameters
  • worker[in] UCP worker that is used for the receive operation.

  • buffer[in] Pointer to the buffer that will receive the data.

  • count[in] Number of elements to receive

  • datatype[in] Datatype descriptor for the elements in the buffer.

  • message[in] Message handle.

  • cb[in] Callback function that is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

ucs_status_ptr_t ucp_tag_msg_recv_nbx(ucp_worker_h worker, void *buffer, size_t count, ucp_tag_message_h message, const ucp_request_param_t *param)

Non-blocking receive operation for a probed message.

This routine receives a message that is described by the local address buffer, size count, and message handle on the worker. The message handle can be obtained by calling the ucp_tag_probe_nb() routine. The ucp_tag_msg_recv_nbx() routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to notify the application about completion of the receive operation the UCP library will invoke the call-back cb when the received message is in the receive buffer and ready for application access. If the receive operation cannot be started the routine returns an error.

Parameters
  • worker[in] UCP worker that is used for the receive operation.

  • buffer[in] Pointer to the buffer that will receive the data.

  • count[in] Number of elements to receive

  • message[in] Message handle.

  • param[in] Operation parameters, see ucp_request_param_t

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

ucs_status_t ucp_put_nbi(ucp_ep_h ep, const void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey)

Non-blocking implicit remote memory put operation.

This routine initiates a storage of contiguous block of data that is described by the local address buffer in the remote contiguous memory region described by remote_addr address and the memoryhandle” rkey. The routine returns immediately and does not guarantee re-usability of the source address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user.

Parameters
  • ep[in] Remote endpoint handle.

  • buffer[in] Pointer to the local source address.

  • length[in] Length of the data (in bytes) stored under the source address.

  • remote_addr[in] Pointer to the destination remote memory address to write to.

  • rkey[in] Remote memory key associated with the remote memory address.

Returns

Error code as defined by ucs_status_t

Note

A user can use ucp_worker_flush_nb() in order to guarantee re-usability of the source address buffer.

ucs_status_ptr_t ucp_put_nb(ucp_ep_h ep, const void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey, ucp_send_callback_t cb)

Non-blocking remote memory put operation.

This routine initiates a storage of contiguous block of data that is described by the local address buffer in the remote contiguous memory region described by remote_addr address and the memoryhandle” rkey. The routine returns immediately and does not guarantee re-usability of the source address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user. If the put operation completes immediately, the routine returns UCS_OK and the call-back routine cb is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine cb upon completion of the put operation. In other words, the completion of a put operation can be signaled by the return code or execution of the call-back.

Parameters
  • ep[in] Remote endpoint handle.

  • buffer[in] Pointer to the local source address.

  • length[in] Length of the data (in bytes) stored under the source address.

  • remote_addr[in] Pointer to the destination remote memory address to write to.

  • rkey[in] Remote memory key associated with the remote memory address.

  • cb[in] Call-back function that is invoked whenever the put operation is completed and the local buffer can be modified. Does not guarantee remote completion.

Returns

NULL - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

A user can use ucp_worker_flush_nb() in order to guarantee re-usability of the source address buffer.

ucs_status_ptr_t ucp_put_nbx(ucp_ep_h ep, const void *buffer, size_t count, uint64_t remote_addr, ucp_rkey_h rkey, const ucp_request_param_t *param)

Non-blocking remote memory put operation.

This routine initiates a storage of contiguous block of data that is described by the local address buffer in the remote contiguous memory region described by remote_addr address and the memoryhandle” rkey. The routine returns immediately and does not guarantee re-usability of the source address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user. If the put operation completes immediately, the routine returns UCS_OK and the call-back routine param.cb.send is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine param.cb.send upon completion of the put operation. In other words, the completion of a put operation can be signaled by the return code or execution of the call-back. Immediate completion signals can be fine-tuned via the ucp_request_param_t::op_attr_mask field in the ucp_request_param_t structure. The values of this field are a bit-wise OR of the ucp_op_attr_t enumeration.

Parameters
  • ep[in] Remote endpoint handle.

  • buffer[in] Pointer to the local source address.

  • count[in] Number of elements of type ucp_request_param_t::datatype to put. If ucp_request_param_t::datatype is not specified, the type defaults to ucp_dt_make_contig(1), which corresponds to byte elements.

  • remote_addr[in] Pointer to the destination remote memory address to write to.

  • rkey[in] Remote memory key associated with the remote memory address.

  • param[in] Operation parameters, see ucp_request_param_t

Returns

UCS_OK - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

A user can use ucp_worker_flush_nb() in order to guarantee re-usability of the source address buffer.

Note

Only the datatype ucp_dt_make_contig(1) is supported for param->datatype, see ucp_dt_make_contig.

ucs_status_t ucp_get_nbi(ucp_ep_h ep, void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey)

Non-blocking implicit remote memory get operation.

This routine initiate a load of contiguous block of data that is described by the remote memory address remote_addr and the memory handle rkey in the local contiguous memory region described by buffer address. The routine returns immediately and does not guarantee that remote data is loaded and stored under the local address buffer.

Parameters
  • ep[in] Remote endpoint handle.

  • buffer[in] Pointer to the local destination address.

  • length[in] Length of the data (in bytes) stored under the destination address.

  • remote_addr[in] Pointer to the source remote memory address to read from.

  • rkey[in] Remote memory key associated with the remote memory address.

Returns

Error code as defined by ucs_status_t

Note

A user can use ucp_worker_flush_nb() in order guarantee that remote data is loaded and stored under the local address buffer.

ucs_status_ptr_t ucp_get_nb(ucp_ep_h ep, void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey, ucp_send_callback_t cb)

Non-blocking remote memory get operation.

This routine initiates a load of a contiguous block of data that is described by the remote memory address remote_addr and the memory handle rkey in the local contiguous memory region described by buffer address. The routine returns immediately and does not guarantee that remote data is loaded and stored under the local address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user. If the get operation completes immediately, the routine returns UCS_OK and the call-back routine cb is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine cb upon completion of the get operation. In other words, the completion of a get operation can be signaled by the return code or execution of the call-back.

Parameters
  • ep[in] Remote endpoint handle.

  • buffer[in] Pointer to the local destination address.

  • length[in] Length of the data (in bytes) stored under the destination address.

  • remote_addr[in] Pointer to the source remote memory address to read from.

  • rkey[in] Remote memory key associated with the remote memory address.

  • cb[in] Call-back function that is invoked whenever the get operation is completed and the data is visible to the local process.

Returns

NULL - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

A user can use ucp_worker_flush_nb() in order to guarantee re-usability of the source address buffer.

ucs_status_ptr_t ucp_get_nbx(ucp_ep_h ep, void *buffer, size_t count, uint64_t remote_addr, ucp_rkey_h rkey, const ucp_request_param_t *param)

Non-blocking remote memory get operation.

This routine initiates a load of a contiguous block of data that is described by the remote memory address remote_addr and the memory handle rkey in the local contiguous memory region described by buffer address. The routine returns immediately and does not guarantee that remote data is loaded and stored under the local address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user. If the get operation completes immediately, the routine returns UCS_OK and the call-back routine param.cb.send is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine param.cb.send upon completion of the get operation. In other words, the completion of a get operation can be signaled by the return code or execution of the call-back.

Parameters
  • ep[in] Remote endpoint handle.

  • buffer[in] Pointer to the local destination address.

  • count[in] Number of elements of type ucp_request_param_t::datatype to put. If ucp_request_param_t::datatype is not specified, the type defaults to ucp_dt_make_contig(1), which corresponds to byte elements.

  • remote_addr[in] Pointer to the source remote memory address to read from.

  • rkey[in] Remote memory key associated with the remote memory address.

  • param[in] Operation parameters, see ucp_request_param_t.

Returns

UCS_OK - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

A user can use ucp_worker_flush_nb() in order to guarantee re-usability of the source address buffer.

Note

Only the datatype ucp_dt_make_contig(1) is supported for param->datatype, see ucp_dt_make_contig.

ucs_status_t ucp_atomic_post(ucp_ep_h ep, ucp_atomic_post_op_t opcode, uint64_t value, size_t op_size, uint64_t remote_addr, ucp_rkey_h rkey)

Post an atomic memory operation.

This routine posts an atomic memory operation to a remote value. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. Return from the function does not guarantee completion. A user must call ucp_ep_flush_nb or ucp_worker_flush_nb to guarantee that the remote value has been updated.

Parameters
  • ep[in] UCP endpoint.

  • opcode[in] One of ucp_atomic_post_op_t.

  • value[in] Source operand for the atomic operation.

  • op_size[in] Size of value in bytes

  • remote_addr[in] Remote address to operate on.

  • rkey[in] Remote key handle for the remote memory address.

Returns

Error code as defined by ucs_status_t

ucs_status_ptr_t ucp_atomic_fetch_nb(ucp_ep_h ep, ucp_atomic_fetch_op_t opcode, uint64_t value, void *result, size_t op_size, uint64_t remote_addr, ucp_rkey_h rkey, ucp_send_callback_t cb)

Post an atomic fetch operation.

This routine will post an atomic fetch operation to remote memory. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The routine is non-blocking and therefore returns immediately. However the actual atomic operation may be delayed. The atomic operation is not considered complete until the values in remote and local memory are completed. If the atomic operation completes immediately, the routine returns UCS_OK and the call-back routine cb is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine cb upon completion of the atomic operation. In other words, the completion of an atomic operation can be signaled by the return code or execution of the call-back.

Parameters
  • ep[in] UCP endpoint.

  • opcode[in] One of ucp_atomic_fetch_op_t.

  • value[in] Source operand for atomic operation. In the case of CSWAP this is the conditional for the swap. For SWAP this is the value to be placed in remote memory.

  • result[inout] Local memory address to store resulting fetch to. In the case of CSWAP the value in result will be swapped into the remote_addr if the condition is true.

  • op_size[in] Size of value in bytes and pointer type for result

  • remote_addr[in] Remote address to operate on.

  • rkey[in] Remote key handle for the remote memory address.

  • cb[in] Call-back function that is invoked whenever the send operation is completed. It is important to note that the call-back function is only invoked in a case when the operation cannot be completed in place.

Returns

NULL - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

The user should not modify any part of the result after this operation is called, until the operation completes.

ucs_status_ptr_t ucp_atomic_op_nbx(ucp_ep_h ep, ucp_atomic_op_t opcode, const void *buffer, size_t count, uint64_t remote_addr, ucp_rkey_h rkey, const ucp_request_param_t *param)

Post an atomic memory operation.

This routine will post an atomic operation to remote memory. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The routine is non-blocking and therefore returns immediately. However, the actual atomic operation may be delayed. In order to enable fetching semantics for atomic operations user has to specify param.reply_buffer. Please see table below for more details.

Atomic Operation

Pseudo code

X

Y

Z

Result

UCP_ATOMIC_OP_ADD

Result=Y; Y+=X

buffer

remote_addr

-

param.reply_buffer(optional)

UCP_ATOMIC_OP_SWAP

Result=Y; Y=X

buffer

remote_addr

-

param.reply_buffer

UCP_ATOMIC_OP_CSWAP

Result=Y; if (X==Y) then Y=Z

buffer

remote_addr

param.reply_buffer

param.reply_buffer

UCP_ATOMIC_OP_AND

Result=Y; Y&=X

buffer

remote_addr

-

param.reply_buffer(optional)

UCP_ATOMIC_OP_OR

Result=Y; Y|=X

buffer

remote_addr

-

param.reply_buffer(optional)

UCP_ATOMIC_OP_XOR

Result=Y; Y^=X

buffer

remote_addr

-

param.reply_buffer(optional)

Parameters
  • ep[in] UCP endpoint.

  • opcode[in] One of ucp_atomic_op_t.

  • buffer[in] Address of operand for the atomic operation. See Atomic Operations Semantic table for exact usage by different atomic operations.

  • count[in] Number of elements in buffer and result. The size of each element is specified by ucp_request_param_t::datatype

  • remote_addr[in] Remote address to operate on.

  • rkey[in] Remote key handle for the remote memory address.

  • param[in] Operation parameters, see ucp_request_param_t.

Returns

NULL - The operation completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at some time in the future. The request handle is returned to the application in order to track progress of the operation.

Note

The user should not modify any part of the buffer (or also param->reply_buffer for fetch operations), until the operation completes.

Note

Only ucp_dt_make_config(4) and ucp_dt_make_contig(8) are supported in param->datatype, see ucp_dt_make_contig. Also, currently atomic operations can handle one element only. Thus, count argument must be set to 1.

ucs_status_t ucp_request_check_status(void *request)

Check the status of non-blocking request.

This routine checks the state of the request and returns its current status. Any value different from UCS_INPROGRESS means that request is in a completed state.

Parameters

request[in] Non-blocking request to check.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_tag_recv_request_test(void *request, ucp_tag_recv_info_t *info)

Check the status and currently available state of non-blocking request returned from ucp_tag_recv_nb routine.

This routine checks the state and returns current status of the request returned from ucp_tag_recv_nb routine or the user allocated request for ucp_tag_recv_nbr. Any value different from UCS_INPROGRESS means that the request is in a completed state.

Parameters
  • request[in] Non-blocking request to check.

  • info[out] It is filled with the details about the message available at the moment of calling.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_stream_recv_request_test(void *request, size_t *length_p)

Check the status and currently available state of non-blocking request returned from ucp_stream_recv_nb routine.

This routine checks the state and returns current status of the request returned from ucp_stream_recv_nb routine. Any value different from UCS_INPROGRESS means that the request is in a completed state.

Parameters
  • request[in] Non-blocking request to check.

  • length_p[out] The size of the received data in bytes. This value is only valid if the status is UCS_OK. If valid, it is always an integral multiple of the datatype size associated with the request.

Returns

Error code as defined by ucs_status_t

void ucp_request_cancel(ucp_worker_h worker, void *request)

Cancel an outstanding communications request.

This routine tries to cancels an outstanding communication request. After calling this routine, the request will be in completed or canceled (but not both) state regardless of the status of the target endpoint associated with the communication request. If the request is completed successfully, the send or receive completion callbacks (based on the type of the request) will be called with the status argument of the callback set to UCS_OK, and in a case it is canceled the status argument is set to UCS_ERR_CANCELED. It is important to note that in order to release the request back to the library the application is responsible for calling ucp_request_free().

Parameters
  • worker[in] UCP worker.

  • request[in] Non-blocking request to cancel.

void ucp_stream_data_release(ucp_ep_h ep, void *data)

Release UCP data buffer returned by ucp_stream_recv_data_nb.

This routine releases internal UCP data buffer returned by ucp_stream_recv_data_nb when data is processed, the application can’t use this buffer after calling this function.

Parameters
  • ep[in] Endpoint data received from.

  • data[in] Data pointer to release, which was returned from ucp_stream_recv_data_nb.

void ucp_request_free(void *request)

Release a communications request.

This routine releases the non-blocking request back to the library, regardless of its current state. Communications operations associated with this request will make progress internally, however no further notifications or callbacks will be invoked for this request.

Parameters

request[in] Non-blocking request to release.

void *ucp_request_alloc(ucp_worker_h worker)

Create an empty communications request.

This routine creates request which may be used in functions ucp_tag_send_nbx, ucp_tag_recv_nbx, etc. The application is responsible for releasing the handle using the ucp_request_free routine

Parameters

worker[in] UCP worker.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_dt_create_generic(const ucp_generic_dt_ops_t *ops, void *context, ucp_datatype_t *datatype_p)

Create a generic datatype.

This routine create a generic datatype object. The generic datatype is described by the ops object which provides a table of routines defining the operations for generic datatype manipulation. Typically, generic datatypes are used for integration with datatype engines provided with MPI implementations (MPICH, Open MPI, etc). The application is responsible for releasing the datatype_p object using ucp_dt_destroy() routine.

Parameters
  • ops[in] Generic datatype function table as defined by ucp_generic_dt_ops_t .

  • context[in] Application defined context passed to this routine. The context is passed as a parameter to the routines in the ops table.

  • datatype_p[out] A pointer to datatype object.

Returns

Error code as defined by ucs_status_t

void ucp_dt_destroy(ucp_datatype_t datatype)

Destroy a datatype and release its resources.

This routine destroys the datatype object and releases any resources that are associated with the object. The datatype object must be allocated using ucp_dt_create_generic() routine.

Parameters

datatype[in] Datatype object to destroy.

Warning

  • Once the datatype object is released an access to this object may cause an undefined failure.

ucs_status_t ucp_worker_fence(ucp_worker_h worker)

Assures ordering between non-blocking operations.

This routine ensures ordering of non-blocking communication operations on the UCP worker. Communication operations issued on the worker prior to this call are guaranteed to be completed before any subsequent communication operations to the same worker which follow the call to fence.

Parameters

worker[in] UCP worker.

Returns

Error code as defined by ucs_status_t

Note

The primary difference between ucp_worker_fence() and the ucp_worker_flush_nb() is the fact the fence routine does not guarantee completion of the operations on the call return but only ensures the order between communication operations. The flush operation on return guarantees that all operations are completed and corresponding memory regions were updated.

ucs_status_ptr_t ucp_worker_flush_nb(ucp_worker_h worker, unsigned flags, ucp_send_callback_t cb)

Flush outstanding AMO and RMA operations on the worker.

This routine flushes all outstanding AMO and RMA communications on the worker. All the AMO and RMA operations issued on the worker prior to this call are completed both at the origin and at the target when this call returns.

Parameters
  • worker[in] UCP worker.

  • flags[in] Flags for flush operation. Reserved for future use.

  • cb[in] Callback which will be called when the flush operation completes.

Returns

NULL - The flush operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The flush operation failed.

Returns

otherwise - Flush operation was scheduled and can be completed in any point in time. The request handle is returned to the application in order to track progress. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

For description of the differences between flush and fence operations please see ucp_worker_fence()

ucs_status_ptr_t ucp_worker_flush_nbx(ucp_worker_h worker, const ucp_request_param_t *param)

Flush outstanding AMO and RMA operations on the worker.

This routine flushes all outstanding AMO and RMA communications on the worker. All the AMO and RMA operations issued on the worker prior to this call are completed both at the origin and at the target when this call returns.

Parameters
Returns

NULL - The flush operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The flush operation failed.

Returns

otherwise - Flush operation was scheduled and can be completed in any point in time. The request handle is returned to the application in order to track progress.

Note

For description of the differences between flush and fence operations please see ucp_worker_fence()

ucs_status_t ucp_ep_query(ucp_ep_h ep, ucp_ep_attr_t *attr)

Get attributes of a given endpoint.

This routine fetches information about the endpoint.

Parameters
  • ep[in] Endpoint object to query.

  • attr[out] Filled with attributes of the endpoint.

Returns

Error code as defined by ucs_status_t

struct ucp_dt_iov
#include <ucp.h>

Structure for scatter-gather I/O.

This structure is used to specify a list of buffers which can be used within a single data transfer function call. This list should remain valid until the data transfer request is completed.

Note

If length is zero, the memory pointed to by buffer will not be accessed. Otherwise, buffer must point to valid memory.

Public Members

void *buffer

Pointer to a data buffer

size_t length

Length of the buffer in bytes

struct ucp_generic_dt_ops
#include <ucp.h>

UCP generic data type descriptor.

This structure provides a generic datatype descriptor that is used for definition of application defined datatypes.

Typically, the descriptor is used for an integration with datatype engines implemented within MPI and SHMEM implementations.

Note

In case of partial receive, any amount of received data is acceptable which matches buffer size.

Public Members

void *(*start_pack)(void *context, const void *buffer, size_t count)

Start a packing request.

The pointer refers to application defined start-to-pack routine. It will be called from the ucp_tag_send_nb routine.

Parameters
  • context[in] User-defined context.

  • buffer[in] Buffer to pack.

  • count[in] Number of elements to pack into the buffer.

Returns

A custom state that is passed to the following pack() routine.

void *(*start_unpack)(void *context, void *buffer, size_t count)

Start an unpacking request.

The pointer refers to application defined start-to-unpack routine. It will be called from the ucp_tag_recv_nb routine.

Parameters
  • context[in] User-defined context.

  • buffer[in] Buffer to unpack to.

  • count[in] Number of elements to unpack in the buffer.

Returns

A custom state that is passed later to the following unpack() routine.

size_t (*packed_size)(void *state)

Get the total size of packed data.

The pointer refers to user defined routine that returns the size of data in a packed format.

Parameters

state[in] State as returned by start_pack() routine.

Returns

The size of the data in a packed form.

size_t (*pack)(void *state, size_t offset, void *dest, size_t max_length)

Pack data.

The pointer refers to application defined pack routine.

Parameters
  • state[in] State as returned by start_pack() routine.

  • offset[in] Virtual offset in the output stream.

  • dest[in] Destination buffer to pack the data.

  • max_length[in] Maximum length to pack.

Returns

The size of the data that was written to the destination buffer. Must be less than or equal to max_length.

ucs_status_t (*unpack)(void *state, size_t offset, const void *src, size_t length)

Unpack data.

The pointer refers to application defined unpack routine.

Parameters
  • state[in] State as returned by start_unpack() routine.

  • offset[in] Virtual offset in the input stream.

  • src[in] Source to unpack the data from.

  • length[in] Length to unpack.

Returns

UCS_OK or an error if unpacking failed.

void (*finish)(void *state)

Finish packing/unpacking.

The pointer refers to application defined finish routine.

Parameters

state[in] State as returned by start_pack() and start_unpack() routines.

struct ucp_params
#include <ucp.h>

Tuning parameters for UCP library.

The structure defines the parameters that are used for UCP library tuning during UCP library initialization.

Note

UCP library implementation uses the features parameter to optimize the library functionality that minimize memory footprint. For example, if the application does not require send/receive semantics UCP library may avoid allocation of expensive resources associated with send/receive queues.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_params_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

uint64_t features

UCP features that are used for library initialization. It is recommended for applications only to request the features that are required for an optimal functionality This field must be specified.

size_t request_size

The size of a reserved space in a non-blocking requests. Typically applications use this space for caching own structures in order to avoid costly memory allocations, pointer dereferences, and cache misses. For example, MPI implementation can use this memory for caching MPI descriptors This field defaults to 0 if not specified.

ucp_request_init_callback_t request_init

Pointer to a routine that is used for the request initialization. This function will be called only on the very first time a request memory is initialized, and may not be called again if a request is reused. If a request should be reset before the next reuse, it can be done before calling ucp_request_free.

NULL can be used if no such is function required, which is also the default if this field is not specified by field_mask.

ucp_request_cleanup_callback_t request_cleanup

Pointer to a routine that is responsible for final cleanup of the memory associated with the request. This routine may not be called every time a request is released. For some implementations, the cleanup call may be delayed and only invoked at ucp_worker_destroy.

NULL can be used if no such function is required, which is also the default if this field is not specified by field_mask.

uint64_t tag_sender_mask

Mask which specifies particular bits of the tag which can uniquely identify the sender (UCP endpoint) in tagged operations. This field defaults to 0 if not specified.

int mt_workers_shared

This flag indicates if this context is shared by multiple workers from different threads. If so, this context needs thread safety support; otherwise, the context does not need to provide thread safety. For example, if the context is used by single worker, and that worker is shared by multiple threads, this context does not need thread safety; if the context is used by worker 1 and worker 2, and worker 1 is used by thread 1 and worker 2 is used by thread 2, then this context needs thread safety. Note that actual thread mode may be different from mode passed to ucp_init. To get actual thread mode use ucp_context_query.

size_t estimated_num_eps

An optimization hint of how many endpoints will be created on this context. For example, when used from MPI or SHMEM libraries, this number will specify the number of ranks (or processing elements) in the job. Does not affect semantics, but only transport selection criteria and the resulting performance. The value can be also set by UCX_NUM_EPS environment variable. In such case it will override the number of endpoints set by estimated_num_eps

size_t estimated_num_ppn

An optimization hint for a single node. For example, when used from MPI or OpenSHMEM libraries, this number will specify the number of Processes Per Node (PPN) in the job. Does not affect semantics, only transport selection criteria and the resulting performance. The value can be also set by the UCX_NUM_PPN environment variable, which will override the number of endpoints set by estimated_num_ppn

const char *name

Tracing and analysis tools can identify the context using this name. To retrieve the context’s name, use ucp_context_query, as the name you supply may be changed by UCX under some circumstances, e.g. a name conflict. This field is only assigned if you set UCP_PARAM_FIELD_NAME in the field mask. If not, then a default unique name will be created for you.

struct ucp_lib_attr
#include <ucp.h>

Lib attributes.

The structure defines the attributes that characterize the Library.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_lib_attr_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

ucs_thread_mode_t max_thread_level

Maximum level of thread support of the library, which is permanent throughout the lifetime of the library. Accordingly, the user can call ucp_worker_create with appropriate ucp_worker_params_t::thread_mode. For supported thread levels please see ucs_thread_mode_t.

struct ucp_context_attr
#include <ucp.h>

Context attributes.

The structure defines the attributes that characterize the particular context.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_context_attr_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

size_t request_size

Size of UCP non-blocking request. When pre-allocated request is used (e.g. in ucp_tag_recv_nbr) it should have enough space to fit UCP request data, which is defined by this value.

ucs_thread_mode_t thread_mode

Thread safe level of the context. For supported thread levels please see ucs_thread_mode_t.

uint64_t memory_types

Mask of which memory types are supported, for supported memory types please see ucs_memory_type_t.

char name[UCP_ENTITY_NAME_MAX]

Tracing and analysis tools can use name to identify this UCX context.

struct ucp_worker_attr
#include <ucp.h>

UCP worker attributes.

The structure defines the attributes which characterize the particular worker.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_worker_attr_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

ucs_thread_mode_t thread_mode

Thread safe level of the worker.

uint32_t address_flags

Flags indicating requested details of the worker address. If UCP_WORKER_ATTR_FIELD_ADDRESS_FLAGS bit is set in the field_mask, this value should be set as well. Possible flags are specified in ucp_worker_address_flags_t.

Note

This is an input attribute.

ucp_address_t *address

Worker address, which can be passed to remote instances of the UCP library in order to connect to this worker. The memory for the address handle is allocated by ucp_worker_query() routine, and must be released by using ucp_worker_release_address() routine.

size_t address_length

Size of worker address in bytes.

size_t max_am_header

Maximum allowed header size for ucp_am_send_nbx routine.

char name[UCP_ENTITY_NAME_MAX]

Tracing and analysis tools can identify the worker using this name.

size_t max_debug_string

Maximum debug string size that can be filled with ucp_request_query.

struct ucp_worker_params
#include <ucp.h>

Tuning parameters for the UCP worker.

The structure defines the parameters that are used for the UCP worker tuning during the UCP worker creation.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_worker_params_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

ucs_thread_mode_t thread_mode

The parameter thread_mode suggests the thread safety mode which worker and the associated resources should be created with. This is an optional parameter. The default value is UCS_THREAD_MODE_SINGLE and it is used when the value of the parameter is not set. When this parameter along with its corresponding bit in the field_mask - UCP_WORKER_PARAM_FIELD_THREAD_MODE is set, the ucp_worker_create attempts to create worker with this thread mode. The thread mode with which worker is created can differ from the suggested mode. The actual thread mode of the worker should be obtained using the query interface ucp_worker_query.

ucs_cpu_set_t cpu_mask

Mask of which CPUs worker resources should preferably be allocated on. This value is optional. If it’s not set (along with its corresponding bit in the field_mask - UCP_WORKER_PARAM_FIELD_CPU_MASK), resources are allocated according to system’s default policy.

unsigned events

Mask of events (ucp_wakeup_event_t) which are expected on wakeup. This value is optional. If it’s not set (along with its corresponding bit in the field_mask - UCP_WORKER_PARAM_FIELD_EVENTS), all types of events will trigger on wakeup.

void *user_data

User data associated with the current worker. This value is optional. If it’s not set (along with its corresponding bit in the field_mask - UCP_WORKER_PARAM_FIELD_USER_DATA), it will default to NULL.

int event_fd

External event file descriptor. This value is optional. If UCP_WORKER_PARAM_FIELD_EVENT_FD is set in the field_mask, events on the worker will be reported on the provided event file descriptor. In this case, calling ucp_worker_get_efd will result in an error. The provided file descriptor must be capable of aggregating notifications for arbitrary events, for example epoll(7) on Linux systems. user_data will be used as the event user-data on systems which support it. For example, on Linux, it will be placed in epoll_data_t::ptr, when returned from epoll_wait(2).

Otherwise, events will be reported to the event file descriptor returned from ucp_worker_get_efd().

uint64_t flags

Worker flags. This value is optional. If UCP_WORKER_PARAM_FIELD_FLAGS is not set in the field_mask, the value of this field will default to 0.

const char *name

Tracing and analysis tools can identify the worker using this name. To retrieve the worker’s name, use ucp_worker_query, as the name you supply may be changed by UCX under some circumstances, e.g. a name conflict. This field is only assigned if you set UCP_WORKER_PARAM_FIELD_NAME in the field mask. If not, then a default unique name will be created for you.

size_t am_alignment

Minimal address alignment of the active message data pointer as passed in argument data to the active message handler, defined as ucp_am_recv_callback_t.

uint64_t client_id

Client id that is sent as part of the connection request payload when connecting to a remote socket address. On the remote side, this value can be obtained from ucp_conn_request_h using ucp_conn_request_query.

struct ucp_worker_address_attr
#include <ucp.h>

UCP worker address attributes.

The structure defines the attributes of the particular worker address.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_worker_address_attr_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

uint64_t worker_uid

Unique id of the worker this address belongs to.

struct ucp_ep_evaluate_perf_param_t
#include <ucp.h>

UCP endpoint performance evaluation request attributes.

The structure defines the attributes which characterize the request for performance estimation of a particular endpoint.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_ep_perf_param_field_t. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

size_t message_size

Message size to use for determining performance. This field must be initialized by the caller.

struct ucp_ep_evaluate_perf_attr_t
#include <ucp.h>

UCP endpoint performance evaluation result attributes.

The structure defines the attributes which characterize the result of performance estimation of a particular endpoint.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_ep_perf_attr_field_t. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

double estimated_time

Estimated time (in seconds) required to send a message of a given size on this endpoint. This field is set by the ucp_ep_evaluate_perf function.

struct ucp_listener_attr
#include <ucp.h>

UCP listener attributes.

The structure defines the attributes which characterize the particular listener.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_listener_attr_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

struct sockaddr_storage sockaddr

Sockaddr on which this listener is listening for incoming connection requests.

struct ucp_conn_request_attr
#include <ucp.h>

UCP listener’s connection request attributes.

The structure defines the attributes that characterize the particular connection request received on the server side.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_conn_request_attr_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

struct sockaddr_storage client_address

The address of the remote client that sent the connection request to the server.

uint64_t client_id

Remote client id if remote endpoint’s flag UCP_EP_PARAMS_FLAGS_SEND_CLIENT_ID is set.

struct ucp_listener_params
#include <ucp.h>

Parameters for a UCP listener object.

This structure defines parameters for ucp_listener_create, which is used to listen for incoming client/server connections.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_listener_params_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

ucs_sock_addr_t sockaddr

An address in the form of a sockaddr. This field is mandatory for filling (along with its corresponding bit in the field_mask - UCP_LISTENER_PARAM_FIELD_SOCK_ADDR). The ucp_listener_create routine will return with an error if sockaddr is not specified.

ucp_listener_accept_handler_t accept_handler

Handler to endpoint creation in a client-server connection flow. In order for the callback inside this handler to be invoked, the UCP_LISTENER_PARAM_FIELD_ACCEPT_HANDLER needs to be set in the field_mask.

ucp_listener_conn_handler_t conn_handler

Handler of an incoming connection request in a client-server connection flow. In order for the callback inside this handler to be invoked, the UCP_LISTENER_PARAM_FIELD_CONN_HANDLER needs to be set in the field_mask.

Note

User is expected to call ucp_ep_create with set UCP_EP_PARAM_FIELD_CONN_REQUEST flag to ucp_ep_params_t::field_mask and ucp_ep_params_t::conn_request in order to be able to receive communications.

struct ucp_stream_poll_ep
#include <ucp.h>

Output parameter of ucp_stream_worker_poll function.

The structure defines the endpoint and its user data.

Public Members

ucp_ep_h ep

Endpoint handle.

void *user_data

User data associated with an endpoint passed in ucp_ep_params_t::user_data.

unsigned flags

Reserved for future use.

uint8_t reserved[16]

Reserved for future use.

struct ucp_mem_map_params
#include <ucp.h>

Tuning parameters for the UCP memory mapping.

The structure defines the parameters that are used for the UCP memory mapping tuning during the ucp_mem_map routine.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_mem_map_params_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

void *address

If the address is not NULL, the routine maps (registers) the memory segment pointed to by this address. If the pointer is NULL, the library allocates mapped (registered) memory segment and returns its address in this argument. Therefore, this value is optional. If it’s not set (along with its corresponding bit in the field_mask - UCP_MEM_MAP_PARAM_FIELD_ADDRESS), the ucp_mem_map routine will consider address as set to NULL and will allocate memory.

size_t length

Length (in bytes) to allocate or map (register). This field is mandatory for filling (along with its corresponding bit in the field_mask - UCP_MEM_MAP_PARAM_FIELD_LENGTH). The ucp_mem_map routine will return with an error if the length isn’t specified.

unsigned flags

Allocation flags, e.g. UCP_MEM_MAP_NONBLOCK. This value is optional. If it’s not set (along with its corresponding bit in the field_mask - UCP_MEM_MAP_PARAM_FIELD_FLAGS), the ucp_mem_map routine will consider the flags as set to zero.

unsigned prot

Memory protection mode, e.g. UCP_MEM_MAP_PROT_LOCAL_READ. This value is optional. If it’s not set, the ucp_mem_map routine will consider the flags as set to UCP_MEM_MAP_PROT_LOCAL_READ|UCP_MEM_MAP_PROT_LOCAL_WRITE| UCP_MEM_MAP_PROT_REMOTE_READ|UCP_MEM_MAP_PROT_REMOTE_WRITE.

ucs_memory_type_t memory_type
struct ucp_tag_recv_info
#include <ucp.h>

UCP receive information descriptor.

The UCP receive information descriptor is allocated by application and filled in with the information about the received message by ucp_tag_probe_nb or ucp_tag_recv_request_test routines or ucp_tag_recv_callback_t callback argument.

Public Members

ucp_tag_t sender_tag

Sender tag

size_t length

The size of the received data

struct ucp_request_param_t
#include <ucp.h>

Operation parameters passed to ucp_tag_send_nbx, ucp_tag_send_sync_nbx, ucp_tag_recv_nbx, ucp_put_nbx, ucp_get_nbx, ucp_am_send_nbx and ucp_am_recv_data_nbx.

The structure ucp_request_param_t is used to specify datatype of operation, provide user request in case the external request is used, set completion callback and custom user data passed to this callback.

Example: implementation of function to send contiguous buffer to ep and invoke callback function at operation completion. If the operation completed immediately (status == UCS_OK) then callback is not called.

ucs_status_ptr_t send_data(ucp_ep_h ep, void *buffer, size_t length,
                           ucp_tag_t tag, void *request)
{
    ucp_request_param_t param = {
        .op_attr_mask               = UCP_OP_ATTR_FIELD_CALLBACK |
                                      UCP_OP_ATTR_FIELD_REQUEST,
        .request                    = request,
        .cb.send                    = custom_send_callback_f,
        .user_data                  = pointer_to_user_context_passed_to_cb
    };

    ucs_status_ptr_t status;

    status = ucp_tag_send_nbx(ep, buffer, length, tag, &param);
    if (UCS_PTR_IS_ERR(status)) {
        handle_error(status);
    } else if (status == UCS_OK) {
        // operation is completed
    }

    return status;
}

Public Members

uint32_t op_attr_mask

Mask of valid fields in this structure and operation flags, using bits from ucp_op_attr_t. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

uint32_t flags
void *request

Request handle allocated by the user. There should be at least UCP request size bytes of available space before the request. The size of the UCP request can be obtained by ucp_context_query function.

union ucp_request_param_t::[anonymous] cb

Callback function that is invoked whenever the send or receive operation is completed.

ucp_datatype_t datatype

Datatype descriptor for the elements in the buffer. In case the op_attr_mask & UCP_OP_ATTR_FIELD_DATATYPE bit is not set, then use default datatype ucp_dt_make_contig(1)

void *user_data

Pointer to user data passed to callback function.

void *reply_buffer

Reply buffer. Can be used for storing operation result, for example by ucp_atomic_op_nbx.

ucs_memory_type_t memory_type

Memory type of the buffer. see ucs_memory_type_t for possible memory types. An optimization hint to avoid memory type detection for request buffer. If this value is not set (along with its corresponding bit in the op_attr_mask - UCP_OP_ATTR_FIELD_MEMORY_TYPE), then use default UCS_MEMORY_TYPE_UNKNOWN which means the memory type will be detected internally.

union ucp_request_param_t::[anonymous] recv_info

Pointer to the information where received data details are stored in case of an immediate completion of receive operation. The user has to provide a pointer to valid memory/variable which will be updated on function return.

struct ucp_request_attr_t
#include <ucp.h>

Attributes of a particular request.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_req_attr_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

char *debug_string

Pointer to allocated string of size debug_string_size that will be filled with debug information about transports and protocols that were selected to complete the request.

size_t debug_string_size

Size of the debug_string. String will be filled up to this size. Maximum possible size debug string can be obtained by querying the worker via ucp_worker_query.

ucs_status_t status

Status of the request. The same as ucp_request_check_status.

ucs_memory_type_t mem_type

Detected memory type of the buffer passed to the operation.

struct ucp_am_handler_param
#include <ucp.h>

Active Message handler parameters passed to ucp_worker_set_am_recv_handler routine.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_am_handler_param_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

unsigned id

Active Message id.

uint32_t flags

Handler flags as defined by ucp_am_cb_flags.

ucp_am_recv_callback_t cb

Active Message callback. To clear the already set callback, this value should be set to NULL.

void *arg

Active Message argument, which will be passed in to every invocation of ucp_am_recv_callback_t function as the arg argument.

struct ucp_am_recv_param
#include <ucp.h>

Operation parameters provided in ucp_am_recv_callback_t callback.

Public Members

uint64_t recv_attr

Mask of valid fields in this structure and receive operation flags, using bits from ucp_am_recv_attr_t. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

ucp_ep_h reply_ep

Endpoint, which can be used for the reply to this message.

struct ucp_mem_advise_params
#include <ucp.h>

Tuning parameters for the UCP memory advice.

This structure defines the parameters that are used for the UCP memory advice tuning during the ucp_mem_advise routine.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_mem_advise_params_field. All fields are mandatory. Provides ABI compatibility with respect to adding new fields.

void *address

Memory base address.

size_t length

Length (in bytes) to allocate or map (register).

ucp_mem_advice_t advice

Memory use advice ucp_mem_advice

struct ucp_ep_attr
#include <ucp.h>

UCP endpoint attributes.

The structure defines the attributes that characterize the particular endpoint.

Public Members

uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_ep_attr_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

char name[UCP_ENTITY_NAME_MAX]

Endpoint name. Tracing and analysis tools can identify the endpoint using this name.

struct sockaddr_storage local_sockaddr

Local socket address for this endpoint. Valid only for endpoints created by connecting to a socket address. If this field is specified for an endpoint not connected to a socket address, UCS_ERR_NOT_CONNECTED will be returned.

struct sockaddr_storage remote_sockaddr

Remote socket address this endpoint is connected to. Valid only for endpoints created by connecting to a socket address. If this field is specified for an endpoint not connected to a socket address, UCS_ERR_NOT_CONNECTED will be returned.

ucp_request_param_t.cb

Callback function that is invoked whenever the send or receive operation is completed.

ucp_request_param_t.recv_info

Pointer to the information where received data details are stored in case of an immediate completion of receive operation. The user has to provide a pointer to valid memory/variable which will be updated on function return.

Public Members

size_t *length
ucp_tag_recv_info_t *tag_info