NAME

SYNOPSIS

T
atomic_load_relaxed(const volatile T *p);

T
atomic_load_acquire(const volatile T *p);

T
atomic_load_consume(const volatile T *p);

void
atomic_store_relaxed(volatile T *p, T v);

void
atomic_store_release(volatile T *p, T v);

DESCRIPTION

Atomic means that the memory operations cannot be fused or torn:

• Fusing is combining multiple memory operations on a single object into one memory operation, such as replacing

          *p = v; 
  	x = *p;

  by

  	*p = v; 
  	x = v;

  since the compiler can prove that *p will yield v after *p = v. For atomic memory operations, the implementation will not assume that
  • consecutive loads of the same object will return the same value, or
  • a store followed by a load of the same object will return the value stored, or
  • consecutive stores of the same object are redundant.
  Thus, the implementation will not replace two consecutive atomic loads by one, will not elide an atomic load following a store, and will not combine two consecutive atomic stores into one.

  For example,

  	atomic_store_relaxed(&flag, 1); 
  	while (atomic_load_relaxed(&flag)) 
  		continue;

  may be used to set a flag and then busy-wait until another thread clears it, whereas

  	flag = 1; 
  	while (flag) 
  		continue;

  may be transformed into the infinite loop

  	flag = 1; 
  	while (1) 
  		continue;

• Tearing is implementing a memory operation on a large data unit such as a 32-bit word by issuing multiple memory operations on smaller data units such as 8-bit bytes. The implementation will not tear atomic loads or stores into smaller ones. Thus, as far as any interrupt, other thread, or other CPU can tell, an atomic memory operation is issued either all at once or not at all.

  For example, if a 32-bit word w is written with atomic_store_relaxed(&w, 0x00010002), then an interrupt, other thread, or other CPU reading it with atomic_load_relaxed(&w) will never witness it partially written, whereas w = 0x00010002 might be compiled into a pair of separate byte-sized store instructions instead of one single word-sized store instruction, in which case other threads may see the intermediate state with only one of the bytes written.

Atomic operations on any single object occur in a total order shared by all interrupts, threads, and CPUs, which is consistent with the program order in every interrupt, thread, and CPU. A single program without interruption or other threads or CPUs will always observe its own loads and stores in program order, but another program in an interrupt handler, in another thread, or on another CPU may issue loads that return values as if the first program's stores occurred out of program order, and vice versa. Two different threads might each observe a third thread's memory operations in different orders.

The memory ordering constraints make limited guarantees of ordering relative to memory operations on other objects as witnessed by interrupts, other threads, or other CPUs, and have the following meanings:

relaxed

No ordering relative to memory operations on any other objects is guaranteed. Relaxed ordering is the default for ordinary non-atomic memory operations like *p and *p = v.

Atomic operations with relaxed ordering are cheap: they are not read/modify/write atomic operations, and they do not involve any kind of inter-CPU ordering barriers.

acquire

This memory operation happens before all subsequent memory operations in program order. However, prior memory operations in program order may be reordered to happen after this one. For example, assuming no aliasing between the pointers, the implementation is allowed to treat

	int x = *p; 
	if (atomic_load_acquire(q)) { 
		int y = *r; 
		*s = x + y; 
		return 1; 
	}

as if it were

	if (atomic_load_acquire(q)) { 
		int x = *p; 
		int y = *r; 
		*s = x + y; 
		return 1; 
	}

but not as if it were

	int x = *p; 
	int y = *r; 
	*s = x + y; 
	if (atomic_load_acquire(q)) { 
		return 1; 
	}

consume

This memory operation happens before all memory operations on objects at addresses that are computed from the value returned by this one. Otherwise, no ordering relative to memory operations on other objects is implied.

For example, the implementation is allowed to treat

	struct foo *foo0, *foo1; 
 
	struct foo *f0 = atomic_load_consume(&foo0); 
	struct foo *f1 = atomic_load_consume(&foo1); 
	int x = f0->x; 
	int y = f1->y;

as if it were

	struct foo *foo0, *foo1; 
 
	struct foo *f1 = atomic_load_consume(&foo1); 
	struct foo *f0 = atomic_load_consume(&foo0); 
	int y = f1->y; 
	int x = f0->x;

but loading f0->x is guaranteed to happen after loading foo0 even if the CPU had a cached value for the address that f0->x happened to be at, and likewise for f1->y and foo1.

atomic_load_consume() functions like atomic_load_acquire() as long as the memory operations that must happen after it are limited to addresses that depend on the value returned by it, but it is almost always as cheap as atomic_load_relaxed(). See ACQUIRE OR CONSUME? below for more details.

release

All prior memory operations in program order happen before this one. However, subsequent memory operations in program order may be reordered to happen before this one too. For example, assuming no aliasing between the pointers, the implementation is allowed to treat

	int x = *p; 
	*q = x; 
	atomic_store_release(r, 0); 
	int y = *s; 
	return x + y;

as if it were

	int y = *s; 
	int x = *p; 
	*q = x; 
	atomic_store_release(r, 0); 
	return x + y;

but not as if it were

	atomic_store_release(r, 0); 
	int x = *p; 
	int y = *s; 
	*q = x; 
	return x + y;

	struct gadget { 
		int ready, x; 
	} the_gadget; 
 
	/* Producer */ 
	the_gadget.x = 42; 
	atomic_store_release(&the_gadget.ready, 1); 
 
	/* Consumer */ 
	if (atomic_load_acquire(&the_gadget.ready) == 0) 
		return EWOULDBLOCK; 
	int x = the_gadget.x;

	struct gizmo { 
		int x, y, z; 
	}; 
	struct gizmo null_gizmo; 
	struct gizmo *the_gizmo = &null_gizmo; 
 
	/* Producer */ 
	struct gizmo *g = kmem_alloc(sizeof(*g), KM_SLEEP); 
	g->x = 12; 
	g->y = 34; 
	g->z = 56; 
	atomic_store_release(&the_gizmo, g); 
 
	/* Consumer */ 
	struct gizmo *g = atomic_load_consume(&the_gizmo); 
	int y = g->y;

	int x[2]; 
	bool b; 
 
	/* Producer */ 
	x[0] = 42; 
	atomic_store_release(&b, 0); 
 
	/* Consumer 1 */ 
	int y = atomic_load_???(&b) ? x[0] : x[1]; 
 
	/* Consumer 2 */ 
	int y = x[atomic_load_???(&b) ? 0 : 1]; 
 
	/* Consumer 3 */ 
	int y = x[atomic_load_???(&b) ^ 1];

	struct foo { 
		kmutex_t	f_lock; 
		uint32_t	f_refcnt; 
		uint64_t	f_ticket; 
	}; 
 
	if (atomic_load_relaxed(&foo->f_refcnt) == 0) 
		return 123; 
#ifdef __HAVE_ATOMIC64_LOADSTORE 
	if (atomic_load_relaxed(&foo->f_ticket) == ticket) 
		return 123; 
#endif 
	mutex_enter(&foo->f_lock); 
	if (foo->f_refcnt == 0 || foo->f_ticket == ticket) 
		ret = 123; 
	... 
#ifdef __HAVE_ATOMIC64_LOADSTORE 
	atomic_store_relaxed(&foo->f_ticket, foo->f_ticket + 1); 
#else 
	foo->f_ticket++; 
#endif 
	... 
	mutex_exit(&foo->f_lock);

C11 COMPATIBILITY

LINUX ANALOGUES

• They do not require the address &x to be aligned.
• They do not require sizeof(x) to be at most the largest size of available atomic loads and stores on the host architecture.

EXAMPLES

	unsigned count; 
 
	void 
	record_event(void) 
	{ 
		atomic_store_relaxed(&count, 
		    1 + atomic_load_relaxed(&count)); 
	} 
 
	unsigned 
	read_event_count(void) 
	{ 
 
		return atomic_load_relaxed(&count); 
	}

Initialization barrier.

	int ready; 
	struct data d; 
 
	void 
	setup_and_notify(void) 
	{ 
 
		setup_data(&d.things); 
		atomic_store_release(&ready, 1); 
	} 
 
	void 
	try_if_ready(void) 
	{ 
 
		if (atomic_load_acquire(&ready)) 
			do_stuff(d.things); 
	}

Publishing a pointer to the current snapshot of data. (Caller must arrange that only one call to take_snapshot happens at any given time; generally this should be done in coordination with pserialize(9) or similar to enable resource reclamation.)

	struct data *current_d; 
 
	void 
	take_snapshot(void) 
        { 
		struct data *d = kmem_alloc(sizeof(*d)); 
 
		d->things = ...; 
 
		atomic_store_release(&current_d, d); 
	} 
 
	struct data * 
	get_snapshot(void) 
	{ 
 
		return atomic_load_consume(&current_d); 
	}

CODE REFERENCES

SEE ALSO

HISTORY

CAVEATS

Instead, we implement atomic_load_consume() as atomic_load_relaxed() followed by membar_datadep_consumer(3), which is equivalent to membar_consumer(3) on DEC Alpha and __insn_barrier(3) elsewhere.

BUGS