File ucp_compat.h¶

Typedefs

typedef struct ucp_listener_accept_handler ucp_listener_accept_handler_t

Deprecated:: Replaced by ucp_listener_conn_handler_t.

Enums

enum ucp_atomic_post_op_t

Atomic operation requested for ucp_atomic_post.

Deprecated:: Use ucp_atomic_op_nbx and ucp_atomic_op_t instead.

This enumeration defines which atomic memory operation should be performed by the ucp_atomic_post family of functions. All of these are non-fetching atomics and will not result in a request handle.

Values:

enumerator UCP_ATOMIC_POST_OP_ADD: Atomic add

enumerator UCP_ATOMIC_POST_OP_AND: Atomic and

enumerator UCP_ATOMIC_POST_OP_OR: Atomic or

enumerator UCP_ATOMIC_POST_OP_XOR: Atomic xor

enumerator UCP_ATOMIC_POST_OP_LAST

enum ucp_atomic_fetch_op_t

Atomic operation requested for ucp_atomic_fetch.

Deprecated:: Use ucp_atomic_op_nbx and ucp_atomic_op_t instead.

This enumeration defines which atomic memory operation should be performed by the ucp_atomic_fetch family of functions. All of these functions will fetch data from the remote node.

Values:

enumerator UCP_ATOMIC_FETCH_OP_FADD: Atomic Fetch and add

enumerator UCP_ATOMIC_FETCH_OP_SWAP: Atomic swap

enumerator UCP_ATOMIC_FETCH_OP_CSWAP: Atomic conditional swap

enumerator UCP_ATOMIC_FETCH_OP_FAND: Atomic Fetch and and

enumerator UCP_ATOMIC_FETCH_OP_FOR: Atomic Fetch and or

enumerator UCP_ATOMIC_FETCH_OP_FXOR: Atomic Fetch and xor

enumerator UCP_ATOMIC_FETCH_OP_LAST

enum ucp_ep_close_mode

Close UCP endpoint modes.

Deprecated:: Use ucp_ep_close_nbx and ucp_ep_close_flags_t instead.

The enumeration is used to specify the behavior of ucp_ep_close_nb.

Values:

enumerator UCP_EP_CLOSE_MODE_FORCE: ucp_ep_close_nb releases the endpoint without any confirmation from the peer. All outstanding requests will be completed with UCS_ERR_CANCELED error.

Note

This mode may cause transport level errors on remote side, so it requires set UCP_ERR_HANDLING_MODE_PEER for all endpoints created on both (local and remote) sides to avoid undefined behavior.

enumerator UCP_EP_CLOSE_MODE_FLUSH: ucp_ep_close_nb schedules flushes on all outstanding operations.

Functions

int ucp_request_is_completed(void *request)

Deprecated:: Replaced by ucp_request_test.

void ucp_request_release(void *request)

Deprecated:: Replaced by ucp_request_free.

void ucp_ep_destroy(ucp_ep_h ep)

Deprecated:: Replaced by ucp_ep_close_nb.

ucs_status_ptr_t ucp_disconnect_nb(ucp_ep_h ep)

Deprecated:: Replaced by ucp_ep_close_nb.

ucs_status_t ucp_request_test(void *request, ucp_tag_recv_info_t *info)

Deprecated:: Replaced by ucp_tag_recv_request_test and ucp_request_check_status depends on use case.

Note

Please use ucp_request_check_status for cases that only need to check the completion status of an outstanding request. ucp_request_check_status can be used for any type of request. ucp_tag_recv_request_test should only be used for requests returned by ucp_tag_recv_nb (or request allocated by user for ucp_tag_recv_nbr) for which additional information (returned via the info pointer) is needed.

ucs_status_t ucp_rkey_pack(ucp_context_h context, ucp_mem_h memh, void **rkey_buffer_p, size_t *size_p)

Pack memory region remote access key.

Deprecated:: Replaced by ucp_memh_pack().

This routine allocates a memory buffer and packs a remote access key (RKEY) object into it. RKEY is an opaque object that provides the information that is necessary for remote memory access. This routine packs the RKEY object in a portable format such that the object can be unpacked on any platform supported by the UCP library. In order to release the memory buffer allocated by this routine, the application is responsible for calling the ucp_rkey_buffer_release() routine.

Parameters

context – [in] Application context which was used to allocate/map the memory.
memh – [in] Handle to the memory region.
rkey_buffer_p – [out] Memory buffer allocated by the library. The buffer contains the packed RKEY.
size_p – [out] Size (in bytes) of the packed RKEY.

Returns

Error code as defined by ucs_status_t

Note

RKEYs for InfiniBand and Cray Aries networks typically include the InfiniBand and Aries key.
In order to enable remote direct memory access to the memory associated with the memory handle, the application is responsible for sharing the RKEY with the peers that will initiate the access.

void ucp_rkey_buffer_release(void *rkey_buffer)

Release packed remote key buffer.

Deprecated:: Replaced by ucp_memh_buffer_release().

This routine releases the buffer that was allocated using ucp_rkey_pack().

Parameters: rkey_buffer – [in] Buffer to release.

Warning

Once memory is released, an access to the memory may cause undefined behavior.
If the input memory address was not allocated using ucp_rkey_pack() routine, the behavior of this routine is undefined.

ucs_status_t ucp_ep_flush(ucp_ep_h ep)

Deprecated:: Replaced by ucp_ep_flush_nb.

ucs_status_t ucp_worker_flush(ucp_worker_h worker)

Flush outstanding AMO and RMA operations on the worker.

Deprecated:: Replaced by ucp_worker_flush_nb. The following example implements the same functionality using ucp_worker_flush_nb :

ucs_status_t worker_flush(ucp_worker_h worker)
{
    void *request = ucp_worker_flush_nb(worker);
    if (request == NULL) {
        return UCS_OK;
    } else if (UCS_PTR_IS_ERR(request)) {
        return UCS_PTR_STATUS(request);
    } else {
        ucs_status_t status;
        do {
            ucp_worker_progress(worker);
            status = ucp_request_check_status(request);
        } while (status == UCS_INPROGRESS);
        ucp_request_release(request);
        return status;
    }
}

This routine flushes all outstanding AMO and RMA communications on the worker. All the AMO and RMA operations issued on the worker prior to this call are completed both at the origin and at the target when this call returns.

Parameters: worker – [in] UCP worker.
Returns: Error code as defined by ucs_status_t

Note

For description of the differences between flush and fence operations please see ucp_worker_fence()

ucs_status_t ucp_put(ucp_ep_h ep, const void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey)

Blocking remote memory put operation.

Deprecated:: Replaced by ucp_put_nb. The following example implements the same functionality using ucp_put_nb :

void empty_callback(void *request, ucs_status_t status)
{
}

ucs_status_t put(ucp_ep_h ep, const void *buffer, size_t length,
                  uint64_t remote_addr, ucp_rkey_h rkey)
{
    void *request = ucp_put_nb(ep, buffer, length, remote_addr, rkey,
                               empty_callback),
    if (request == NULL) {
        return UCS_OK;
    } else if (UCS_PTR_IS_ERR(request)) {
        return UCS_PTR_STATUS(request);
    } else {
        ucs_status_t status;
        do {
            ucp_worker_progress(worker);
            status = ucp_request_check_status(request);
        } while (status == UCS_INPROGRESS);
        ucp_request_release(request);
        return status;
    }
}

This routine stores contiguous block of data that is described by the local address buffer in the remote contiguous memory region described by remote_addr address and the memory handle rkey. The routine returns when it is safe to reuse the source address buffer.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local source address.
length – [in] Length of the data (in bytes) stored under the source address.
remote_addr – [in] Pointer to the destination remote address to write to.
rkey – [in] Remote memory key associated with the remote address.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_get(ucp_ep_h ep, void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey)

Blocking remote memory get operation.

Deprecated:: Replaced by ucp_get_nb.

This routine loads contiguous block of data that is described by the remote address remote_addr and the memory handle rkey in the local contiguous memory region described by buffer address. The routine returns when remote data is loaded and stored under the local address buffer.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local source address.
length – [in] Length of the data (in bytes) stored under the source address.
remote_addr – [in] Pointer to the destination remote address to write to.
rkey – [in] Remote memory key associated with the remote address.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_atomic_add32(ucp_ep_h ep, uint32_t add, uint64_t remote_addr, ucp_rkey_h rkey)

Blocking atomic add operation for 32 bit integers.

Deprecated:: Replaced by ucp_atomic_post with opcode UCP_ATOMIC_POST_OP_ADD.

This routine performs an add operation on a 32 bit integer value atomically. The remote integer value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The add value is the value that is used for the add operation. When the operation completes the sum of the original remote value and the operand value (add) is stored in remote memory. The call to the routine returns immediately, independent of operation completion.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
add – [in] Value to add.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 32 bit.

ucs_status_t ucp_atomic_add64(ucp_ep_h ep, uint64_t add, uint64_t remote_addr, ucp_rkey_h rkey)

Blocking atomic add operation for 64 bit integers.

Deprecated:: Replaced by ucp_atomic_post with opcode UCP_ATOMIC_POST_OP_ADD.

This routine performs an add operation on a 64 bit integer value atomically. The remote integer value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The add value is the value that is used for the add operation. When the operation completes the sum of the original remote value and the operand value (add) is stored in remote memory. The call to the routine returns immediately, independent of operation completion.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
add – [in] Value to add.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 64 bit.

ucs_status_t ucp_atomic_fadd32(ucp_ep_h ep, uint32_t add, uint64_t remote_addr, ucp_rkey_h rkey, uint32_t *result)

Blocking atomic fetch and add operation for 32 bit integers.

Deprecated:: Replaced by ucp_atomic_fetch_nb with opcode UCP_ATOMIC_FETCH_OP_FADD.

This routine performs an add operation on a 32 bit integer value atomically. The remote integer value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The add value is the value that is used for the add operation. When the operation completes, the original remote value is stored in the local memory result, and the sum of the original remote value and the operand value is stored in remote memory. The call to the routine returns when the operation is completed and the result value is updated.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
add – [in] Value to add.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.
result – [out] Pointer to the address that is used to store the previous value of the atomic variable described by the remote_addr

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 32 bit.

ucs_status_t ucp_atomic_fadd64(ucp_ep_h ep, uint64_t add, uint64_t remote_addr, ucp_rkey_h rkey, uint64_t *result)

Blocking atomic fetch and add operation for 64 bit integers.

Deprecated:: Replaced by ucp_atomic_fetch_nb with opcode UCP_ATOMIC_FETCH_OP_FADD.

This routine performs an add operation on a 64 bit integer value atomically. The remote integer value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The add value is the value that is used for the add operation. When the operation completes, the original remote value is stored in the local memory result, and the sum of the original remote value and the operand value is stored in remote memory. The call to the routine returns when the operation is completed and the result value is updated.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
add – [in] Value to add.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.
result – [out] Pointer to the address that is used to store the previous value of the atomic variable described by the remote_addr

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 64 bit.

ucs_status_t ucp_atomic_swap32(ucp_ep_h ep, uint32_t swap, uint64_t remote_addr, ucp_rkey_h rkey, uint32_t *result)

Blocking atomic swap operation for 32 bit values.

Deprecated:: Replaced by ucp_atomic_fetch_nb with opcode UCP_ATOMIC_FETCH_OP_SWAP.

This routine swaps a 32 bit value between local and remote memory. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The swap value is the value that is used for the swap operation. When the operation completes, the remote value is stored in the local memory result, and the operand value (swap) is stored in remote memory. The call to the routine returns when the operation is completed and the result value is updated.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
swap – [in] Value to swap.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.
result – [out] Pointer to the address that is used to store the previous value of the atomic variable described by the remote_addr

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 32 bit.

ucs_status_t ucp_atomic_swap64(ucp_ep_h ep, uint64_t swap, uint64_t remote_addr, ucp_rkey_h rkey, uint64_t *result)

Blocking atomic swap operation for 64 bit values.

Deprecated:: Replaced by ucp_atomic_fetch_nb with opcode UCP_ATOMIC_FETCH_OP_SWAP.

This routine swaps a 64 bit value between local and remote memory. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The swap value is the value that is used for the swap operation. When the operation completes, the remote value is stored in the local memory result, and the operand value (swap) is stored in remote memory. The call to the routine returns when the operation is completed and the result value is updated.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
swap – [in] Value to swap.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.
result – [out] Pointer to the address that is used to store the previous value of the atomic variable described by the remote_addr

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 64 bit.

ucs_status_t ucp_atomic_cswap32(ucp_ep_h ep, uint32_t compare, uint32_t swap, uint64_t remote_addr, ucp_rkey_h rkey, uint32_t *result)

Blocking atomic conditional swap (cswap) operation for 32 bit values.

Deprecated:: Replaced by ucp_atomic_fetch_nb with opcode UCP_ATOMIC_FETCH_OP_CSWAP.

This routine conditionally swaps a 32 bit value between local and remote memory. The swap occurs only if the condition value (continue) is equal to the remote value, otherwise the remote memory is not modified. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The swap value is the value that is used to update the remote memory if the condition is true. The call to the routine returns when the operation is completed and the result value is updated.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
compare – [in] Value to compare to.
swap – [in] Value to swap.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.
result – [out] Pointer to the address that is used to store the previous value of the atomic variable described by the remote_addr

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 32 bit.

ucs_status_t ucp_atomic_cswap64(ucp_ep_h ep, uint64_t compare, uint64_t swap, uint64_t remote_addr, ucp_rkey_h rkey, uint64_t *result)

Blocking atomic conditional swap (cswap) operation for 64 bit values.

Deprecated:: Replaced by ucp_atomic_fetch_nb with opcode UCP_ATOMIC_FETCH_OP_CSWAP.

This routine conditionally swaps a 64 bit value between local and remote memory. The swap occurs only if the condition value (continue) is equal to the remote value, otherwise the remote memory is not modified. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The swap value is the value that is used to update the remote memory if the condition is true. The call to the routine returns when the operation is completed and the result value is updated.

See: ucp_put.

Parameters

ep – [in] Remote endpoint handle.
compare – [in] Value to compare to.
swap – [in] Value to swap.
remote_addr – [in] Pointer to the destination remote address of the atomic variable.
rkey – [in] Remote memory key associated with the remote address.
result – [out] Pointer to the address that is used to store the previous value of the atomic variable described by the remote_addr

Returns

Error code as defined by ucs_status_t

Note

The remote address must be aligned to 64 bit.

ucs_status_ptr_t ucp_ep_modify_nb(ucp_ep_h ep, const ucp_ep_params_t *params)

Modify endpoint parameters.

Deprecated:: Use ucp_listener_conn_handler_t instead of ucp_listener_accept_handler_t, if you have other use case please submit an issue on https://github.com/openucx/ucx or report to ucx-group@elist.ornl.gov

This routine modifies endpoint created by ucp_ep_create or ucp_listener_accept_callback_t. For example, this API can be used to setup custom parameters like ucp_ep_params_t::user_data or ucp_ep_params_t::err_handler to endpoint created by ucp_listener_accept_callback_t.

Parameters

ep – [in] A handle to the endpoint.
params – [in] User defined ucp_ep_params_t configurations for the UCP endpoint.

Returns

NULL - The endpoint is modified successfully.

Returns

UCS_PTR_IS_ERR(_ptr) - The reconfiguration failed and an error code indicates the status. However, the endpoint is not modified and can be used further.

Returns

otherwise - The reconfiguration process is started, and can be completed at any point in time. A request handle is returned to the application in order to track progress of the endpoint modification. The application is responsible for releasing the handle using the ucp_request_free routine.

Note

See the documentation of ucp_ep_params_t for details, only some of the parameters can be modified.

ucs_status_t ucp_worker_get_address(ucp_worker_h worker, ucp_address_t **address_p, size_t *address_length_p)

Get the address of the worker object.

Deprecated:: Use ucp_worker_query with the flag UCP_WORKER_ATTR_FIELD_ADDRESS in order to obtain the worker address.

This routine returns the address of the worker object. This address can be passed to remote instances of the UCP library in order to connect to this worker. The memory for the address handle is allocated by this function, and must be released by using ucp_worker_release_address() routine.

Parameters

worker – [in] Worker object whose address to return.
address_p – [out] A pointer to the worker address.
address_length_p – [out] The size in bytes of the address.

Returns

Error code as defined by ucs_status_t

ucs_status_ptr_t ucp_ep_close_nb(ucp_ep_h ep, unsigned mode)

Non-blocking endpoint closure.

Deprecated:: Use ucp_ep_close_nbx instead.

This routine releases the endpoint. The endpoint closure process depends on the selected mode.

Parameters

ep – [in] Handle to the endpoint to close.
mode – [in] One from ucp_ep_close_mode value.

Returns

UCS_OK - The endpoint is closed successfully.

Returns

UCS_PTR_IS_ERR(_ptr) - The closure failed and an error code indicates the transport level status. However, resources are released and the endpoint can no longer be used.

Returns

otherwise - The closure process is started, and can be completed at any point in time. A request handle is returned to the application in order to track progress of the endpoint closure. The application is responsible for releasing the handle using the ucp_request_free routine.

Note

ucp_ep_close_nb replaces deprecated ucp_disconnect_nb and ucp_ep_destroy

ucs_status_ptr_t ucp_ep_flush_nb(ucp_ep_h ep, unsigned flags, ucp_send_callback_t cb)

Non-blocking flush of outstanding AMO and RMA operations on the endpoint.

Deprecated:: Use ucp_ep_flush_nbx instead.

This routine flushes all outstanding AMO and RMA communications on the endpoint. All the AMO and RMA operations issued on the ep prior to this call are completed both at the origin and at the target endpoint when this call returns.

Parameters

ep – [in] UCP endpoint.
flags – [in] Flags for flush operation. Reserved for future use.
cb – [in] Callback which will be called when the flush operation completes.

Returns

NULL - The flush operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The flush operation failed.

Returns

otherwise - Flush operation was scheduled and can be completed in any point in time. The request handle is returned to the application in order to track progress. The application is responsible for releasing the handle using ucp_request_free() routine.

ucs_status_t ucp_worker_set_am_handler(ucp_worker_h worker, uint16_t id, ucp_am_callback_t cb, void *arg, uint32_t flags)

Add user defined callback for Active Message.

Deprecated:: Use ucp_worker_set_am_recv_handler instead.

This routine installs a user defined callback to handle incoming Active Messages with a specific id. This callback is called whenever an Active Message that was sent from the remote peer by ucp_am_send_nb is received on this worker.

Parameters

worker – [in] UCP worker on which to set the Active Message handler.
id – [in] Active Message id.
cb – [in] Active Message callback. NULL to clear.
arg – [in] Active Message argument, which will be passed in to every invocation of the callback as the arg argument.
flags – [in] Dictates how an Active Message is handled on the remote endpoint. Currently only UCP_AM_FLAG_WHOLE_MSG is supported, which indicates the callback will not be invoked until all data has arrived.

Returns

error code if the worker does not support Active Messages or requested callback flags.

ucs_status_ptr_t ucp_am_send_nb(ucp_ep_h ep, uint16_t id, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_send_callback_t cb, unsigned flags)

Send Active Message.

Deprecated:: Use ucp_am_send_nbx instead.

This routine sends an Active Message to an ep. It does not support CUDA memory.

Parameters

ep – [in] UCP endpoint where the Active Message will be run.
id – [in] Active Message id. Specifies which registered callback to run.
buffer – [in] Pointer to the data to be sent to the target node of the Active Message.
count – [in] Number of elements to send.
datatype – [in] Datatype descriptor for the elements in the buffer.
cb – [in] Callback that is invoked upon completion of the data transfer if it is not completed immediately.
flags – [in] Operation flags as defined by ucp_send_am_flags.

Returns

NULL Active Message was sent immediately.

Returns

UCS_PTR_IS_ERR(_ptr) Error sending Active Message.

Returns

otherwise Pointer to request, and Active Message is known to be completed after cb is run.

ucs_status_ptr_t ucp_stream_send_nb(ucp_ep_h ep, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_send_callback_t cb, unsigned flags)

Non-blocking stream send operation.

Deprecated:: Use ucp_stream_send_nbx instead.

This routine sends data that is described by the local address buffer, size count, and datatype object to the destination endpoint ep. The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine returns UCS_OK and the callback function cb is not invoked. If the operation is not completed immediately and no error reported, then the UCP library will schedule invocation of the callback cb upon completion of the send operation. In other words, the completion of the operation will be signaled either by the return code or by the callback.

Parameters

ep – [in] Destination endpoint handle.
buffer – [in] Pointer to the message buffer (payload).
count – [in] Number of elements to send.
datatype – [in] Datatype descriptor for the elements in the buffer.
cb – [in] Callback function that is invoked whenever the send operation is completed. It is important to note that the callback is only invoked in the event that the operation cannot be completed in place.
flags – [in] Reserved for future use.

Returns

NULL - The send operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message. The application is responsible for releasing the handle using ucp_request_free routine.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_ptr_t ucp_stream_recv_nb(ucp_ep_h ep, void *buffer, size_t count, ucp_datatype_t datatype, ucp_stream_recv_callback_t cb, size_t *length, unsigned flags)

Non-blocking stream receive operation of structured data into a user-supplied buffer.

Deprecated:: Use ucp_stream_recv_nbx instead.

This routine receives data that is described by the local address buffer, size count, and datatype object on the endpoint ep. The routine is non-blocking and therefore returns immediately. The receive operation is considered complete when the message is delivered to the buffer. If data is not immediately available, the operation will be scheduled for receive and a request handle will be returned. In order to notify the application about completion of a scheduled receive operation, the UCP library will invoke the call-back cb when data is in the receive buffer and ready for application access. If the receive operation cannot be started, the routine returns an error.

Parameters

ep – [in] UCP endpoint that is used for the receive operation.
buffer – [in] Pointer to the buffer to receive the data.
count – [in] Number of elements to receive into buffer.
datatype – [in] Datatype descriptor for the elements in the buffer.
cb – [in] Callback function that is invoked whenever the receive operation is completed and the data is ready in the receive buffer. It is important to note that the call-back is only invoked in a case when the operation cannot be completed immediately.
length – [out] Size of the received data in bytes. The value is valid only if return code is UCS_OK.
flags – [in] Flags defined in ucp_stream_recv_flags_t.

Returns

NULL - The receive operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. A request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle by calling the ucp_request_free routine.

Note

The amount of data received, in bytes, is always an integral multiple of the datatype size.

ucs_status_ptr_t ucp_tag_send_nb(ucp_ep_h ep, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, ucp_send_callback_t cb)

Non-blocking tagged-send operations.

Deprecated:: Use ucp_tag_send_nbx instead.

This routine sends a messages that is described by the local address buffer, size count, and datatype object to the destination endpoint ep. Each message is associated with a tag value that is used for message matching on the receiver. The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine return UCS_OK and the call-back function cb is not invoked. If the operation is not completed immediately and no error reported then the UCP library will schedule to invoke the call-back cb whenever the send operation will be completed. In other words, the completion of a message can be signaled by the return code or the call-back.

Parameters

ep – [in] Destination endpoint handle.
buffer – [in] Pointer to the message buffer (payload).
count – [in] Number of elements to send
datatype – [in] Datatype descriptor for the elements in the buffer.
tag – [in] Message tag.
cb – [in] Callback function that is invoked whenever the send operation is completed. It is important to note that the call-back is only invoked in a case when the operation cannot be completed in place.

Returns

NULL - The send operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_t ucp_tag_send_nbr(ucp_ep_h ep, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, void *req)

Non-blocking tagged-send operations with user provided request.

Deprecated:: Use ucp_tag_send_nbx with the flag UCP_OP_ATTR_FIELD_REQUEST instead.

This routine provides a convenient and efficient way to implement a blocking send pattern. It also completes requests faster than ucp_tag_send_nb() because:

it always uses eager protocol to send data up to the rendezvous threshold.
its rendezvous threshold is higher than the one used by the ucp_tag_send_nb(). The threshold is controlled by the UCX_SEND_NBR_RNDV_THRESH environment variable.
its request handling is simpler. There is no callback and no need to allocate and free requests. In fact request can be allocated by caller on the stack.

This routine sends a messages that is described by the local address buffer, size count, and datatype object to the destination endpoint ep. Each message is associated with a tag value that is used for message matching on the receiver.

The routine is non-blocking and therefore returns immediately, however the actual send operation may be delayed. The send operation is considered completed when it is safe to reuse the source buffer. If the send operation is completed immediately the routine returns UCS_OK.

If the operation is not completed immediately and no error reported then the UCP library will fill a user provided req and return UCS_INPROGRESS status. In order to monitor completion of the operation ucp_request_check_status() should be used.

Following pseudo code implements a blocking send function:

MPI_send(...)
{
    char *request;
    ucs_status_t status;

    // allocate request on the stack
    // ucp_context_query() was used to get ucp_request_size
    request = alloca(ucp_request_size);

    // note: make sure that there is enough memory before the
    // request handle
    status = ucp_tag_send_nbr(ep, ..., request + ucp_request_size);
    if (status != UCS_INPROGRESS) {
        return status;
    }

    do {
        ucp_worker_progress(worker);
        status = ucp_request_check_status(request + ucp_request_size);
    } while (status == UCS_INPROGRESS);

    return status;
}

Parameters

ep – [in] Destination endpoint handle.
buffer – [in] Pointer to the message buffer (payload).
count – [in] Number of elements to send
datatype – [in] Datatype descriptor for the elements in the buffer.
tag – [in] Message tag.
req – [in] Request handle allocated by the user. There should be at least UCP request size bytes of available space before the req. The size of UCP request can be obtained by ucp_context_query function.

Returns

UCS_OK - The send operation was completed immediately.

Returns

UCS_INPROGRESS - The send was not completed and is in progress. ucp_request_check_status() should be used to monitor req status.

Returns

Error code as defined by ucs_status_t

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

ucs_status_ptr_t ucp_tag_send_sync_nb(ucp_ep_h ep, const void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, ucp_send_callback_t cb)

Non-blocking synchronous tagged-send operation.

Deprecated:: Use ucp_tag_send_sync_nbx instead.

Same as ucp_tag_send_nb, except the request completes only after there is a remote tag match on the message (which does not always mean the remote receive has been completed). This function never completes “in-place”, and always returns a request handle.

Parameters

ep – [in] Destination endpoint handle.
buffer – [in] Pointer to the message buffer (payload).
count – [in] Number of elements to send
datatype – [in] Datatype descriptor for the elements in the buffer.
tag – [in] Message tag.
cb – [in] Callback function that is invoked whenever the send operation is completed.

Returns

UCS_PTR_IS_ERR(_ptr) - The send operation failed.

Returns

otherwise - Operation was scheduled for send and can be completed in any point in time. The request handle is returned to the application in order to track progress of the message. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

The user should not modify any part of the buffer after this operation is called, until the operation completes.

Note

Returns UCS_ERR_UNSUPPORTED if UCP_ERR_HANDLING_MODE_PEER is enabled. This is a temporary implementation-related constraint that will be addressed in future releases.

ucs_status_ptr_t ucp_tag_recv_nb(ucp_worker_h worker, void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, ucp_tag_t tag_mask, ucp_tag_recv_callback_t cb)

Non-blocking tagged-receive operation.

Deprecated:: Use ucp_tag_recv_nbx instead.

This routine receives a message that is described by the local address buffer, size count, and datatype object on the worker. The tag value of the receive message has to match the tag and tag_mask values, where the tag_mask indicates which bits of the tag have to be matched. The routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to notify the application about completion of the receive operation the UCP library will invoke the call-back cb when the received message is in the receive buffer and ready for application access. If the receive operation cannot be stated the routine returns an error.

Parameters

worker – [in] UCP worker that is used for the receive operation.
buffer – [in] Pointer to the buffer to receive the data.
count – [in] Number of elements to receive
datatype – [in] Datatype descriptor for the elements in the buffer.
tag – [in] Message tag to expect.
tag_mask – [in] Bit mask that indicates the bits that are used for the matching of the incoming tag against the expected tag.
cb – [in] Callback function that is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

This routine cannot return UCS_OK. It always returns a request handle or an error.

ucs_status_t ucp_tag_recv_nbr(ucp_worker_h worker, void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_t tag, ucp_tag_t tag_mask, void *req)

Non-blocking tagged-receive operation.

Deprecated:: Use ucp_tag_recv_nbx with the flag UCP_OP_ATTR_FIELD_REQUEST instead.

This routine receives a message that is described by the local address buffer, size count, and datatype object on the worker. The tag value of the receive message has to match the tag and tag_mask values, where the tag_mask indicates which bits of the tag have to be matched. The routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to monitor completion of the operation ucp_request_check_status or ucp_tag_recv_request_test should be used.

Parameters

worker – [in] UCP worker that is used for the receive operation.
buffer – [in] Pointer to the buffer to receive the data.
count – [in] Number of elements to receive
datatype – [in] Datatype descriptor for the elements in the buffer.
tag – [in] Message tag to expect.
tag_mask – [in] Bit mask that indicates the bits that are used for the matching of the incoming tag against the expected tag.
req – [in] Request handle allocated by the user. There should be at least UCP request size bytes of available space before the req. The size of UCP request can be obtained by ucp_context_query function.

Returns

Error code as defined by ucs_status_t

ucs_status_ptr_t ucp_tag_msg_recv_nb(ucp_worker_h worker, void *buffer, size_t count, ucp_datatype_t datatype, ucp_tag_message_h message, ucp_tag_recv_callback_t cb)

Non-blocking receive operation for a probed message.

Deprecated:: Use ucp_tag_recv_nbx instead.

This routine receives a message that is described by the local address buffer, size count, message handle, and datatype object on the worker. The message handle can be obtained by calling the ucp_tag_probe_nb() routine. The ucp_tag_msg_recv_nb() routine is non-blocking and therefore returns immediately. The receive operation is considered completed when the message is delivered to the buffer. In order to notify the application about completion of the receive operation the UCP library will invoke the call-back cb when the received message is in the receive buffer and ready for application access. If the receive operation cannot be started the routine returns an error.

Parameters

worker – [in] UCP worker that is used for the receive operation.
buffer – [in] Pointer to the buffer that will receive the data.
count – [in] Number of elements to receive
datatype – [in] Datatype descriptor for the elements in the buffer.
message – [in] Message handle.
cb – [in] Callback function that is invoked whenever the receive operation is completed and the data is ready in the receive buffer.

Returns

UCS_PTR_IS_ERR(_ptr) - The receive operation failed.

Returns

otherwise - Operation was scheduled for receive. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

ucs_status_t ucp_put_nbi(ucp_ep_h ep, const void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey)

Non-blocking implicit remote memory put operation.

Deprecated:: Use ucp_put_nbx without passing the flag UCP_OP_ATTR_FIELD_CALLBACK instead. If a request pointer is returned, release it immediately by ucp_request_free.

This routine initiates a storage of contiguous block of data that is described by the local address buffer in the remote contiguous memory region described by remote_addr address and the memoryhandle” rkey. The routine returns immediately and does not guarantee re-usability of the source address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local source address.
length – [in] Length of the data (in bytes) stored under the source address.
remote_addr – [in] Pointer to the destination remote memory address to write to.
rkey – [in] Remote memory key associated with the remote memory address.

Returns

Error code as defined by ucs_status_t

Note

A user can use ucp_worker_flush_nb() in order to guarantee re-usability of the source address buffer.

ucs_status_ptr_t ucp_put_nb(ucp_ep_h ep, const void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey, ucp_send_callback_t cb)

Non-blocking remote memory put operation.

Deprecated:: Use ucp_put_nbx instead.

This routine initiates a storage of contiguous block of data that is described by the local address buffer in the remote contiguous memory region described by remote_addr address and the memoryhandle” rkey. The routine returns immediately and does not guarantee re-usability of the source address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user. If the put operation completes immediately, the routine returns UCS_OK and the call-back routine cb is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine cb upon completion of the put operation. In other words, the completion of a put operation can be signaled by the return code or execution of the call-back.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local source address.
length – [in] Length of the data (in bytes) stored under the source address.
remote_addr – [in] Pointer to the destination remote memory address to write to.
rkey – [in] Remote memory key associated with the remote memory address.
cb – [in] Call-back function that is invoked whenever the put operation is completed and the local buffer can be modified. Does not guarantee remote completion.

Returns

NULL - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

A user can use ucp_worker_flush_nb() in order to guarantee re-usability of the source address buffer.

ucs_status_t ucp_get_nbi(ucp_ep_h ep, void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey)

Non-blocking implicit remote memory get operation.

Deprecated:: Use ucp_get_nbx without passing the flag UCP_OP_ATTR_FIELD_CALLBACK instead. If a request pointer is returned, release it immediately by ucp_request_free.

This routine initiate a load of contiguous block of data that is described by the remote memory address remote_addr and the memory handle rkey in the local contiguous memory region described by buffer address. The routine returns immediately and does not guarantee that remote data is loaded and stored under the local address buffer.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local destination address.
length – [in] Length of the data (in bytes) stored under the destination address.
remote_addr – [in] Pointer to the source remote memory address to read from.
rkey – [in] Remote memory key associated with the remote memory address.

Returns

Error code as defined by ucs_status_t

Note

A user can use ucp_worker_flush_nb() in order guarantee that remote data is loaded and stored under the local address buffer.

ucs_status_ptr_t ucp_get_nb(ucp_ep_h ep, void *buffer, size_t length, uint64_t remote_addr, ucp_rkey_h rkey, ucp_send_callback_t cb)

Non-blocking remote memory get operation.

Deprecated:: Use ucp_get_nbx instead.

This routine initiates a load of a contiguous block of data that is described by the remote memory address remote_addr and the memory handle rkey in the local contiguous memory region described by buffer address. The routine returns immediately and does not guarantee that remote data is loaded and stored under the local address buffer. If the operation is completed immediately the routine return UCS_OK, otherwise UCS_INPROGRESS or an error is returned to user. If the get operation completes immediately, the routine returns UCS_OK and the call-back routine cb is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine cb upon completion of the get operation. In other words, the completion of a get operation can be signaled by the return code or execution of the call-back.

Parameters

ep – [in] Remote endpoint handle.
buffer – [in] Pointer to the local destination address.
length – [in] Length of the data (in bytes) stored under the destination address.
remote_addr – [in] Pointer to the source remote memory address to read from.
rkey – [in] Remote memory key associated with the remote memory address.
cb – [in] Call-back function that is invoked whenever the get operation is completed and the data is visible to the local process.

Returns

NULL - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

A user can use ucp_worker_flush_nb() in order to guarantee re-usability of the source address buffer.

ucs_status_t ucp_atomic_post(ucp_ep_h ep, ucp_atomic_post_op_t opcode, uint64_t value, size_t op_size, uint64_t remote_addr, ucp_rkey_h rkey)

Post an atomic memory operation.

Deprecated:: Use ucp_atomic_op_nbx without the flag UCP_OP_ATTR_FIELD_REPLY_BUFFER instead.

This routine posts an atomic memory operation to a remote value. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. Return from the function does not guarantee completion. A user must call ucp_ep_flush_nb or ucp_worker_flush_nb to guarantee that the remote value has been updated.

Parameters

ep – [in] UCP endpoint.
opcode – [in] One of ucp_atomic_post_op_t.
value – [in] Source operand for the atomic operation.
op_size – [in] Size of value in bytes
remote_addr – [in] Remote address to operate on.
rkey – [in] Remote key handle for the remote memory address.

Returns

Error code as defined by ucs_status_t

ucs_status_ptr_t ucp_atomic_fetch_nb(ucp_ep_h ep, ucp_atomic_fetch_op_t opcode, uint64_t value, void *result, size_t op_size, uint64_t remote_addr, ucp_rkey_h rkey, ucp_send_callback_t cb)

Post an atomic fetch operation.

Deprecated:: Use ucp_atomic_op_nbx with the flag UCP_OP_ATTR_FIELD_REPLY_BUFFER instead.

This routine will post an atomic fetch operation to remote memory. The remote value is described by the combination of the remote memory address remote_addr and the remote memory handle rkey. The routine is non-blocking and therefore returns immediately. However the actual atomic operation may be delayed. The atomic operation is not considered complete until the values in remote and local memory are completed. If the atomic operation completes immediately, the routine returns UCS_OK and the call-back routine cb is not invoked. If the operation is not completed immediately and no error is reported, then the UCP library will schedule invocation of the call-back routine cb upon completion of the atomic operation. In other words, the completion of an atomic operation can be signaled by the return code or execution of the call-back.

Parameters

ep – [in] UCP endpoint.
opcode – [in] One of ucp_atomic_fetch_op_t.
value – [in] Source operand for atomic operation. In the case of CSWAP this is the conditional for the swap. For SWAP this is the value to be placed in remote memory.
result – [inout] Local memory address to store resulting fetch to. In the case of CSWAP the value in result will be swapped into the remote_addr if the condition is true.
op_size – [in] Size of value in bytes and pointer type for result
remote_addr – [in] Remote address to operate on.
rkey – [in] Remote key handle for the remote memory address.
cb – [in] Call-back function that is invoked whenever the send operation is completed. It is important to note that the call-back function is only invoked in a case when the operation cannot be completed in place.

Returns

NULL - The operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The operation failed.

Returns

otherwise - Operation was scheduled and can be completed at any point in time. The request handle is returned to the application in order to track progress of the operation. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

The user should not modify any part of the result after this operation is called, until the operation completes.

ucs_status_ptr_t ucp_worker_flush_nb(ucp_worker_h worker, unsigned flags, ucp_send_callback_t cb)

Flush outstanding AMO and RMA operations on the worker.

Deprecated:: Use ucp_worker_flush_nbx instead.

This routine flushes all outstanding AMO and RMA communications on the worker. All the AMO and RMA operations issued on the worker prior to this call are completed both at the origin and at the target when this call returns.

Parameters

worker – [in] UCP worker.
flags – [in] Flags for flush operation. Reserved for future use.
cb – [in] Callback which will be called when the flush operation completes.

Returns

NULL - The flush operation was completed immediately.

Returns

UCS_PTR_IS_ERR(_ptr) - The flush operation failed.

Returns

otherwise - Flush operation was scheduled and can be completed in any point in time. The request handle is returned to the application in order to track progress. The application is responsible for releasing the handle using ucp_request_free() routine.

Note

For description of the differences between flush and fence operations please see ucp_worker_fence()

struct ucp_listener_accept_handler

#include <ucp_compat.h>

Deprecated:: Replaced by ucp_listener_conn_handler_t.

Public Members

ucp_listener_accept_callback_t cb¶: Endpoint creation callback

void *arg¶: User defined argument for the callback