Updated: 2022/Sep/29

Please read Privacy Policy. It's for your privacy.

RUMPUSER(3)                Library Functions Manual                RUMPUSER(3)

     rumpuser - rump kernel hypercall interface

     rump User Library (librumpuser, -lrumpuser)

     #include <rump/rumpuser.h>

     The rumpuser hypercall interfaces allow a rump kernel to access host
     resources.  A hypervisor implementation must implement the routines
     described in this document to allow a rump kernel to run on the host.
     The implementation included in NetBSD is for POSIX-like hosts (*BSD,
     Linux, etc.).  This document is divided into sections based on the
     functionality group of each hypercall.

     Since the hypercall interface is a C function interface, both the rump
     kernel and the hypervisor must conform to the same ABI.  The interface
     itself attempts to assume as little as possible from the type systems,
     and for example off_t is passed as int64_t and enums are passed as ints.
     It is recommended that the hypervisor converts these to the native types
     before starting to process the hypercall, for example by assigning the
     ints back to enums.

     A hypercall is always entered with the calling thread scheduled in the
     rump kernel.  In case the hypercall intends to block while waiting for an
     event, the hypervisor must first release the rump kernel scheduling
     context.  In other words, the rump kernel context is a resource and
     holding on to it while waiting for a rump kernel event/resource may lead
     to a deadlock.  Even when there is no possibility of deadlock in the
     strict sense of the term, holding on to the rump kernel context while
     performing a slow hypercall such as reading a device will prevent other
     threads (including the clock interrupt) from using that rump kernel

     Releasing the context is done by calling the hyp_backend_unschedule()
     upcall which the hypervisor received from rump kernel as a parameter for
     rumpuser_init().  Before a hypercall returns back to the rump kernel, the
     returning thread must carry a rump kernel context.  In case the hypercall
     unscheduled itself, it must reschedule itself by calling

     int rumpuser_init(int version, struct rump_hyperup *hyp)

     Initialize the hypervisor.

     version          hypercall interface version number that the kernel
                      expects to be used.  In case the hypervisor cannot
                      provide an exact match, this routine must return a non-
                      zero value.

     hyp              pointer to a set of upcalls the hypervisor can make into
                      the rump kernel

   Memory allocation
     int rumpuser_malloc(size_t len, int alignment, void **memp)

     len              amount of memory to allocate

     alignment        size the returned memory must be aligned to.  For
                      example, if the value passed is 4096, the returned
                      memory must be aligned to a 4k boundary.

     memp             return pointer for allocated memory

     void rumpuser_free(void *mem, size_t len)

     mem              memory to free

     len              length of allocation.  This is always equal to the
                      amount the caller requested from the rumpuser_malloc()
                      which returned mem.

   Files and I/O
     int rumpuser_open(const char *name, int mode, int *fdp)

     Open name for I/O and associate a file descriptor with it.  Notably,
     there needs to be no mapping between name and the host's file system
     namespace.  For example, it is possible to associate the file descriptor
     with device I/O registers for special values of name.

     name             the identifier of the file to open for I/O

     mode             combination of the following:

                      RUMPUSER_OPEN_RDONLY   open only for reading

                      RUMPUSER_OPEN_WRONLY   open only for writing

                      RUMPUSER_OPEN_RDWR     open for reading and writing

                      RUMPUSER_OPEN_CREATE   do not treat missing name as an

                      RUMPUSER_OPEN_EXCL     combined with
                                             RUMPUSER_OPEN_CREATE, flag an
                                             error if name already exists

                      RUMPUSER_OPEN_BIO      the caller will use this file for
                                             block I/O, usually used in
                                             conjunction with accessing file
                                             system media.  The hypervisor
                                             should treat this flag as
                                             advisory and possibly enable some
                                             optimizations for *fdp based on
                      Notably, the permissions of the created file are left up
                      to the hypervisor implementation.

     fdp              An integer value denoting the open file is returned

     int rumpuser_close(int fd)

     Close a previously opened file descriptor.

     int rumpuser_getfileinfo(const char *name, uint64_t *size, int *type)

     name             file for which information is returned.  The namespace
                      is equal to that of rumpuser_open().

     size             If non-NULL, size of the file is returned here.

     type             If non-NULL, type of the file is returned here.  The
                      options are RUMPUSER_FT_DIR, RUMPUSER_FT_REG,
                      for directory, regular file, block device, character
                      device or unknown, respectively.

     void rumpuser_bio(int fd, int op, void *data, size_t dlen, int64_t off,
     rump_biodone_fn biodone, void *donearg)

     Initiate block I/O and return immediately.

     fd               perform I/O on this file descriptor.  The file
                      descriptor must have been opened with RUMPUSER_OPEN_BIO.

     op               Transfer data from the file descriptor with
                      RUMPUSER_BIO_READ and transfer data to the file
                      descriptor with RUMPUSER_BIO_WRITE.  Unless
                      RUMPUSER_BIO_SYNC is specified, the hypervisor may cache
                      a write instead of committing it to permanent storage.

     data             memory address to transfer data to/from

     dlen             length of I/O.  The length is guaranteed to be a
                      multiple of 512.

     off              offset into fd where I/O is performed

     biodone          To be called when the I/O is complete.  Accessing data
                      is not legal after the call is made.

     donearg          opaque arg that must be passed to biodone.

     int rumpuser_iovread(int fd, struct rumpuser_iovec *ruiov, size_t iovlen,
     int64_t off, size_t *retv)

     int rumpuser_iovwrite(int fd, struct rumpuser_iovec *ruiov,
     size_t iovlen, int64_t off, size_t *retv)

     These routines perform scatter-gather I/O which is not block I/O by
     nature and therefore cannot be handled by rumpuser_bio().

     fd               file descriptor to perform I/O on

     ruiov            an array of I/O descriptors.  It is defined as follows:
                            struct rumpuser_iovec {
                                    void *iov_base;
                                    size_t iov_len;

     iovlen           number of elements in ruiov

     off              offset of fd to perform I/O on.  This can either be a
                      non-negative value or RUMPUSER_IOV_NOSEEK.  The latter
                      denotes that no attempt to change the underlying objects
                      offset should be made.  Using both types of offsets on a
                      single instance of fd results in undefined behavior.

     retv             number of bytes successfully transferred is returned

     int rumpuser_syncfd(int fd, int flags, uint64_t start, uint64_t len)

     Synchronizes fd with respect to backing storage.  The other arguments

     flags            controls how synchronization happens.  It must contain
                      one of the following:

                      RUMPUSER_SYNCFD_READ      Make sure that the next read
                                                sees writes from all other
                                                parties.  This is useful for
                                                example in the case that fd
                                                represents memory to write a
                                                DMA read is being performed.

                      RUMPUSER_SYNCFD_WRITE     Flush cached writes.

                      The following additional parameters may be passed in

                      RUMPUSER_SYNCFD_BARRIER   Issue a barrier.  Outstanding
                                                I/O operations which were
                                                started before the barrier
                                                complete before any operations
                                                after the barrier are

                      RUMPUSER_SYNCFD_SYNC      Wait for the synchronization
                                                operation to fully complete
                                                before returning.  For
                                                example, this could mean that
                                                the data to be written to a
                                                disk has hit either the disk
                                                or non-volatile memory.

     start            offset into the object.

     len              the number of bytes to synchronize.  The value 0 denotes
                      until the end of the object.

     The hypervisor should support two clocks, one for wall time and one for
     monotonically increasing time, the latter of which may be based on some
     arbitrary time (e.g. system boot time).  If this is not possible, the
     hypervisor must make a reasonable effort to retain semantics.

     int rumpuser_clock_gettime(int enum_rumpclock, int64_t *sec, long *nsec)

     enum_rumpclock   specifies the clock type.  In case of
                      RUMPUSER_CLOCK_RELWALL the wall time should be returned.
                      In case of RUMPUSER_CLOCK_ABSMONO the time of a
                      monotonic clock should be returned.

     sec              return value for seconds

     nsec             return value for nanoseconds

     int rumpuser_clock_sleep(int enum_rumpclock, int64_t sec, long nsec)

     enum_rumpclock   In case of RUMPUSER_CLOCK_RELWALL, the sleep should last
                      at least as long as specified.  In case of
                      RUMPUSER_CLOCK_ABSMONO, the sleep should last until the
                      hypervisor monotonic clock hits the specified absolute

     sec              sleep duration, seconds.  exact semantics depend on clk.

     nsec             sleep duration, nanoseconds.  exact semantics depend on

   Parameter retrieval
     int rumpuser_getparam(const char *name, void *buf, size_t buflen)

     Retrieve a configuration parameter from the hypervisor.  It is up to the
     hypervisor to decide how the parameters can be set.

     name             name of the parameter.  If the name starts with an
                      underscore, it means a mandatory parameter.  The
                      mandatory parameters are RUMPUSER_PARAM_NCPU which
                      specifies the amount of virtual CPUs bootstrapped by the
                      rump kernel and RUMPUSER_PARAM_HOSTNAME which returns a
                      preferably unique instance name for the rump kernel.

     buf              buffer to return the data in as a string

     buflen           length of buffer

     void rumpuser_exit(int value)

     Terminate the rump kernel with exit value value.  If value is
     RUMPUSER_PANIC the hypervisor should attempt to provide something akin to
     a core dump.

   Console output
     Console output is divided into two routines: a per-character one and
     printf-like one.  The former is used e.g. by the rump kernel's internal
     printf routine.  The latter can be used for direct debug prints e.g. very
     early on in the rump kernel's bootstrap or when using the in-kernel
     routine causes too much skew in the debug print results (the hypercall
     runs outside of the rump kernel and therefore does not cause any locking
     or scheduling events inside the rump kernel).

     void rumpuser_putchar(int ch)

     Output ch on the console.

     void rumpuser_dprintf(const char *fmt, ...)

     Do output based on printf-like parameters.

     A rump kernel should be able to send signals to client programs due to
     some standard interfaces including signal delivery in their
     specifications.  Examples of these interfaces include setitimer(2) and
     write(2).  The rumpuser_kill() function advises the hypercall
     implementation to raise a signal for the process containing the rump

     int rumpuser_kill(int64_t pid, int sig)

     pid              The pid of the rump kernel process that the signal is
                      directed to.  This value may be used as the hypervisor
                      as a hint on how to deliver the signal.  The value
                      RUMPUSER_PID_SELF may also be specified to indicate no
                      hint.  This value will be removed in a future version of
                      the hypercall interface.

     sig              Number of signal to raise.  The value is in NetBSD
                      signal number namespace.  In case the host has a native
                      representation for signals, the value should be
                      translated before the signal is raised.  In case there
                      is no mapping between sig and native signals (if any),
                      the behavior is implementation-defined.

     A rump kernel will ignore the return value of this hypercall.  The only
     implication of not implementing rumpuser_kill() is that some application
     programs may not experience expected behavior for standard interfaces.

     As an aside,the rump_sp(7) protocol provides equivalent functionality for
     remote clients.

   Random pool
     int rumpuser_getrandom(void *buf, size_t buflen, int flags, size_t *retp)

     buf              buffer that the randomness is written to

     buflen           number of bytes of randomness requested

     flags            The value 0 or a combination of RUMPUSER_RANDOM_HARD
                      (return true randomness instead of something from a
                      PRNG) and RUMPUSER_RANDOM_NOWAIT (do not block in case
                      the requested amount of bytes is not available).

     retp             The number of random bytes written into buf.

     int rumpuser_thread_create(void *(*fun)(void *), void *arg,
     const char *thrname, int mustjoin, int priority, int cpuidx,
     void **cookie)

     Create a schedulable host thread context.  The rump kernel will call this
     interface when it creates a kernel thread.  The scheduling policy for the
     new thread is defined by the hypervisor.  In case the hypervisor wants to
     optimize the scheduling of the threads, it can perform heuristics on the
     thrname, priority and cpuidx parameters.

     fun              function that the new thread must call.  This call will
                      never return.

     arg              argument to be passed to fun

     thrname          Name of the new thread.

     mustjoin         If 1, the thread will be waited for by
                      rumpuser_thread_join() when the thread exits.

     priority         The priority that the kernel requested the thread to be
                      created at.  Higher values mean higher priority.  The
                      exact kernel semantics for each value are not available
                      through this interface.

     cpuidx           The index of the virtual CPU that the thread is bound
                      to, or -1 if the thread is not bound.  The mapping
                      between the virtual CPUs and physical CPUs, if any, is
                      hypervisor implementation specific.

     cookie           In case mustjoin is set, the value returned in cookie
                      will be passed to rumpuser_thread_join().

     void rumpuser_thread_exit(void)

     Called when a thread created with rumpuser_thread_create() exits.

     int rumpuser_thread_join(void *cookie)

     Wait for a joinable thread to exit.  The cookie matches the value from

     void rumpuser_curlwpop(int enum_rumplwpop, struct lwp *l)

     Manipulate the hypervisor's thread context database.  The possible
     operations are create, destroy, and set as specified by enum_rumplwpop:

     RUMPUSER_LWP_CREATE    Inform the hypervisor that l is now a valid thread
                            context which may be set.  A currently valid value
                            of l may not be specified.  This operation is
                            informational and does not mandate any action from
                            the hypervisor.

     RUMPUSER_LWP_DESTROY   Inform the hypervisor that l is no longer a valid
                            thread context.  This means that it may no longer
                            be set as the current context.  A currently set
                            context or an invalid one may not be destroyed.
                            This operation is informational and does not
                            mandate any action from the hypervisor.

     RUMPUSER_LWP_SET       Set l as the current host thread's rump kernel
                            context.  A previous context must not exist.

     RUMPUSER_LWP_CLEAR     Clear the context previous set by
                            RUMPUSER_LWP_SET.  The value passed in l is the
                            current thread and is never NULL.

     struct lwp * rumpuser_curlwp(void)

     Retrieve the rump kernel thread context associated with the current host
     thread, as set by rumpuser_curlwpop().  This routine may be called when a
     context is not set and the routine must return NULL in that case.  This
     interface is expected to be called very often.  Any optimizations
     pertaining to the execution speed of this routine should be done in

     void rumpuser_seterrno(int errno)

     Set an errno value in the calling thread's TLS.  Note: this is used only
     if rump kernel clients make rump system calls.

   Mutexes, rwlocks and condition variables
     The locking interfaces have standard semantics, so we will not discuss
     each one in detail.  The data types struct rumpuser_mtx, struct
     rumpuser_rw and struct rumpuser_cv used by these interfaces are opaque to
     the rump kernel, i.e. the hypervisor has complete freedom over them.

     Most of these interfaces will (and must) relinquish the rump kernel CPU
     context in case they block (or intend to block).  The exceptions are the
     "nowrap" variants of the interfaces which may not relinquish rump kernel

     void rumpuser_mutex_init(struct rumpuser_mtx **mtxp, int flags)

     void rumpuser_mutex_enter(struct rumpuser_mtx *mtx)

     void rumpuser_mutex_enter_nowrap(struct rumpuser_mtx *mtx)

     int rumpuser_mutex_tryenter(struct rumpuser_mtx *mtx)

     void rumpuser_mutex_exit(struct rumpuser_mtx *mtx)

     void rumpuser_mutex_destroy(struct rumpuser_mtx *mtx)

     void rumpuser_mutex_owner(struct rumpuser_mtx *mtx, struct lwp **lp)

     Mutexes provide mutually exclusive locking.  The flags, of which at least
     one must be given, are as follows:

     RUMPUSER_MTX_SPIN     Create a spin mutex.  Locking this type of mutex
                           must not relinquish rump kernel context even when
                           rumpuser_mutex_enter() is used.

     RUMPUSER_MTX_KMUTEX   The mutex must track and be able to return the rump
                           kernel thread that owns the mutex (if any).  If
                           this flag is not specified, rumpuser_mutex_owner()
                           will never be called for that particular mutex.

     void rumpuser_rw_init(struct rumpuser_rw **rwp)

     void rumpuser_rw_enter(int enum_rumprwlock, struct rumpuser_rw *rw)

     int rumpuser_rw_tryenter(int enum_rumprwlock, struct rumpuser_rw *rw)

     int rumpuser_rw_tryupgrade(struct rumpuser_rw *rw)

     void rumpuser_rw_downgrade(struct rumpuser_rw *rw)

     void rumpuser_rw_exit(struct rumpuser_rw *rw)

     void rumpuser_rw_destroy(struct rumpuser_rw *rw)

     void rumpuser_rw_held(int enum_rumprwlock, struct rumpuser_rw *rw,
     int *heldp)

     Read/write locks provide either shared or exclusive locking.  The
     possible values for lk are RUMPUSER_RW_READER and RUMPUSER_RW_WRITER.
     Upgrading means trying to migrate from an already owned shared lock to an
     exclusive lock and downgrading means migrating from an already owned
     exclusive lock to a shared lock.

     void rumpuser_cv_init(struct rumpuser_cv **cvp)

     void rumpuser_cv_destroy(struct rumpuser_cv *cv)

     void rumpuser_cv_wait(struct rumpuser_cv *cv, struct rumpuser_mtx *mtx)

     void rumpuser_cv_wait_nowrap(struct rumpuser_cv *cv, struct rumpuser_mtx

     int rumpuser_cv_timedwait(struct rumpuser_cv *cv,
     struct rumpuser_mtx *mtx, int64_t sec, int64_t nsec)

     void rumpuser_cv_signal(struct rumpuser_cv *cv)

     void rumpuser_cv_broadcast(struct rumpuser_cv *cv)

     void rumpuser_cv_has_waiters(struct rumpuser_cv *cv, int *waitersp)

     Condition variables wait for an event.  The mtx interlock eliminates a
     race between checking the predicate and sleeping on the condition
     variable; the mutex should be released for the duration of the sleep in
     the normal atomic manner.  The timedwait variant takes a specifier
     indicating a relative sleep duration after which the routine will return
     with ETIMEDOUT.  If a timedwait is signaled before the timeout expires,
     the routine will return 0.

     The order in which the hypervisor reacquires the rump kernel context and
     interlock mutex before returning into the rump kernel is as follows.  In
     case the interlock mutex was initialized with both RUMPUSER_MTX_SPIN and
     RUMPUSER_MTX_KMUTEX, the rump kernel context is scheduled before the
     mutex is reacquired.  In case of a purely RUMPUSER_MTX_SPIN mutex, the
     mutex is acquired first.  In the final case the order is implementation-

     All routines which return an integer return an errno value.  The
     hypervisor must translate the value to the native errno namespace used by
     the rump kernel.  Routines which do not return an integer may never fail.


     Antti Kantee, "Flexible Operating System Internals: The Design and
     Implementation of the Anykernel and Rump Kernels", Aalto University
     Doctoral Dissertations, 2012, Section 2.3.2: The Hypercall Interface.

     For a list of all known implementations of the rumpuser interface, see

     The rump kernel hypercall API was first introduced in NetBSD 5.0.  The
     API described above first appeared in NetBSD 7.0.

NetBSD 9.99                     August 24, 2014                    NetBSD 9.99