Macros
#define	M_mem_str(haystack, haystack_len, needle) M_mem_mem(haystack,haystack_len,needle,M_str_len(needle) )

#define	M_mem_strpos(haystack, haystack_len, needle, idx) M_mem_mempos(haystack,haystack_len,needle,M_str_len(needle) ,idx)

Typedefs
typedef M_bool(*	M_malloc_error_cb) (void)

Functions
M_bool	M_malloc_register_errorcb (M_malloc_error_cb cb)

M_bool	M_malloc_deregister_errorcb (M_malloc_error_cb cb)

void	M_malloc_clear_errorcb (void)

void *	M_malloc (size_t size) M_ALLOC_SIZE(1) M_WARN_UNUSED_RESULT M_MALLOC

void *	M_malloc_zero (size_t size) M_ALLOC_SIZE(1) M_WARN_UNUSED_RESULT M_MALLOC

void	M_free (void *ptr) M_FREE(1)

void *	M_realloc (void *ptr, size_t size) M_ALLOC_SIZE(2) M_WARN_UNUSED_RESULT

void *	M_realloc_zero (void *ptr, size_t size) M_ALLOC_SIZE(2) M_WARN_UNUSED_RESULT

void *	M_memdup (const void *src, size_t size) M_ALLOC_SIZE(2) M_WARN_UNUSED_RESULT M_MALLOC

void *	M_memdup_max (const void *src, size_t size, size_t min_alloc_size) M_ALLOC_SIZE(2) M_WARN_UNUSED_RESULT M_MALLOC

void *	M_mem_set (void *s, int c, size_t n)

void *	M_mem_move (void dst, const void src, size_t size)

void *	M_mem_copy (void dst, const void src, size_t size)

M_bool	M_mem_eq (const void m1, const void m2, size_t size)

int	M_mem_cmpsort (const void m1, size_t size1, const void m2, size_t size2) M_WARN_UNUSED_RESULT

void *	M_mem_chr (const void *s, M_uint8 b, size_t n) M_WARN_UNUSED_RESULT

M_bool	M_mem_contains (const void haystack, size_t haystack_len, const void needle, size_t needle_len) M_WARN_UNUSED_RESULT

void *	M_mem_mem (const void haystack, size_t haystack_len, const void needle, size_t needle_len) M_WARN_UNUSED_RESULT

void *	M_mem_rmem (const void haystack, size_t haystack_len, const void needle, size_t needle_len) M_WARN_UNUSED_RESULT

void *	M_mem_str (const void haystack, size_t haystack_len, const char needle) M_WARN_UNUSED_RESULT

M_bool	M_mem_mempos (const void haystack, size_t haystack_len, const void needle, size_t needle_len, size_t *idx) M_WARN_UNUSED_RESULT

size_t	M_mem_count (const void *s, size_t s_len, M_uint8 b) M_WARN_UNUSED_RESULT

unsigned char	M_mem_calc_lrc (const void *s, size_t s_len)

M_uint8	M_mem_calc_crc8_ccitt (const void *s, size_t s_len)

M_uint16	M_mem_calc_crc16_ccitt (const void *s, size_t s_len)

M_uint32	M_mem_calc_crc32 (const void *s, size_t s_len)

M_bool	M_mem_swap_bytes (M_uint8 *s, size_t s_len, size_t idx1, size_t idx2)

Detailed Description

Memory manipulation.

Hardening

To aid in hardening M_malloc wraps the system malloc and stores the length of allocated memory. M_free uses this length and zeros the memory before calling the system free. The length is prepended to the memory segment and the pointer to the memory after the size is returned. The size is then read by M_free so the full allocated memory segment can be zeroed.

Zeroing memory is performed to combat memory scan attacks. It's not possible to know what data in memory is sensitive so all data is considered sensitive. Zeroing limits the amount of time data such as credit card numbers or encryption keys are available. This way data is available only as long as needed.

M_malloc and M_free are not replacements for the system provided malloc and free functions. They work on top of the system functions. This is so a hardened system malloc will not have it's functionality disrupted. Further, a replacement malloc implementation (such as jemalloc) can still be used.

Some system mallocs already zero memory but many do not. Mstdlib's M_malloc brings this to systems that do not implement this security feature. This is a case where security trumps performance.

Memory allocated using M_malloc must never be passed directly to the system free due to the length offset prefix, the caller would not be passing the base of the block and therefore cause undefined behavior (probably a segfault).

Memory allocated by malloc (not M_malloc) should never be passed to M_free. This will result in undefined behavior (probably a segfault) as well.

All mstdlib functions that allocate memory use mstdlib's M_malloc. Thus any memory that needs to be freed using free that is returned by an mstdlib function must be passed to M_free.

Hardening that doesn't work

There are a few "hardening" features that are available on Linux and other Unix platforms that were evaluated. These were determined to not be usable.

mlock

mlock prevents a memory segment from being written to on disk swap space.

The issue with mlock is limits set by the OS. RLIMIT_MEMLOCK (ulimit -l) limits the amount of memory that can be locked. munlock must be used (before or after, testing showed it didn't matter) to reduce the locked memory amount. munmap should implicitly unlock the memory as well but in testing a simple free did not cause the memory to be unlocked.

munlock is not enough to avoid hitting the limit. In simple / small applications or test cases, it would function fine. However, a larger application which uses more memory will fail. Once the lock limit is reached an out of memory error will be returned.

On Ubuntu 14.04.2 the default RLIMIT_MEMLOCK is 64K. On some versions of Debian is was found to be 32K. This limit will quickly be reached by a non-trivial application.

Configuring the system to have a larger limit or making the limit unlimited may not alleviate this issue. For example, FreeBSD allows mlock use to be restricted to the user-user only.

Further, Requiring system configuration to use a general purpose library is unacceptable. Especially when the configuration is non-obvious. Also if mlock is limited to super-user only then mstdlib would be unusable as user level application.

madvise with MADV_DONTDUMP

This is used to prevent marked memory from being in a core dump.

On Linux madvise requires the memory to be page-aligned. If the memory is not page-aligned madvise will return a failure with errno EINVAL. Page-alignment can easily cause the application to run out of address space.

For example you could use an allocation like:

void *ptr;
posix_memalign(&ptr, sysconf(_SC_PAGESIZE), size);

Getting the page size on the command line (which is the size of _SC_PAGESIZE):

$ getconf PAGESIZE
4096

In this (and many) cases we have a 4096 byte boundary. Meaning the address of the allocated data must be the address of a page boundary. There is 4K between each boundary. A large amount of data can be allocated there but if a small amount of data is allocated then there is a large amount of unusable space due to the next allocation needing to also be on a 4K boundary.

Take the following allocations:

8 bytes page-aligned.
4 bytes page-aligned.

Assuming One and Two are allocated next to each other. One allocates 8 bytes. Two will be aligned to the 4K boundary after One. A total of 8K of memory is reserved due to this. Only 12 bytes are actually needed but 8K is reserved. Since memory is now aligned in 4K blocks the total available memory space is greatly reduced. Not the amount of memory but the amount of allocations.

On a 32bit system only ~2GB of memory is available to a process. With 4K page-alignment allocations the amount usable memory is greatly reduced. This might be okay on a 64 bit system but will still be wasteful.

Also since Linux, since 3.18, has made madvise optional which severely limits its use.

Conclusion

Neither mlock nor madvise can be used on every malloc. It may be okay to use this selectively but in a general purpose library there is no way to truly know what is sensitive. For example M_list_str and M_hash_dict duplicate the strings they are given. There is no way for them to know that a particular string needs to be securely allocated.

Additional External Security

One option to add additional security is to create an encrypting wrapper around a list or hashtable:

Use a different key for each hashtable.
Limit the life of a hashtable in order to rotate the key often.
Insert, wrapper will take the key and value. Encrypt them and store them in the hashtable.
Get, wrapper will take the key, encrypt it. Use that to look up the value. Decrypt the value. Return the value.

This option further limits the amount of time sensitive data is stored in the clear in memory because the value in the hashtable is encrypted. The plain text data is only exposed as long as it is being actively used. This will further protect against memory scrapers.

It also, reduces the concern of swap and core dumps because the data is stored encrypted. Granted the key as well as the encrypted value could be stored on disk. However, it will still be difficult to determine what data is the key, and what set of data the key belongs to.

Macro Definition Documentation

◆ M_mem_str

#define M_mem_str	(	haystack,
		haystack_len,
		needle
	)	M_mem_mem(haystack,haystack_len,needle,M_str_len(needle) )

◆ M_mem_strpos

#define M_mem_strpos	(	haystack,
		haystack_len,
		needle,
		idx
	)	M_mem_mempos(haystack,haystack_len,needle,M_str_len(needle) ,idx)

Typedef Documentation

◆ M_malloc_error_cb

typedef M_bool(* M_malloc_error_cb) (void)

Error callback for handling malloc failure.

Can return M_TRUE to retry malloc.

Function Documentation

◆ M_malloc_register_errorcb()

M_bool M_malloc_register_errorcb ( M_malloc_error_cb cb )

Register a callback to be called when M_malloc()/M_realloc() returns a failure.

Up to 12 callbacks can be registered. They will be called from newest to oldest. If a callback returns M_TRUE callback processing will stop and malloc will be retried. If malloc fails again the callbacks processing will resume. Each callback will be run until either one returns success or all have returned failure.

Typically this will be used for external error reporting, or (more) graceful shutdown scenarios.

Parameters

[in] cb Callback to be called. This should not ever try to allocate memory as it will most likely fail.

Returns: M_TRUE on success, M_FALSE on failure. The only failure reason currently is if the maximum number of registered callbacks has been reached.

◆ M_malloc_deregister_errorcb()

M_bool M_malloc_deregister_errorcb ( M_malloc_error_cb cb )

Deregister an allocation error callback

Parameters

[in] cb The callback to remove.

Returns: M_TRUE if the callback was removed otherwise M_FALSE. M_FALSE means the callback is not currently registered.

◆ M_malloc_clear_errorcb()

void M_malloc_clear_errorcb ( void )

Clears all user registered callbacks. The default abort callback is not cleared.

◆ M_malloc()

void * M_malloc ( size_t size )

Allocate size bytes and returns pointer to allocated memory.

Retains information about the size of the allocation and must be released using M_free().

On failure registered error callbacks will be called and malloc will be repleted if any error callback return M_TRUE indicating malloc should be retried. If no callbacks return retry the application will abort. The callbacks will be run in reverse order they were registered.

Parameters

[in] size Number of bytes of memory to allocate.

Returns: Pointer to the newly allocated memory or NULL if the requested memory is unavailable. Memory must be released using M_free().

See also: M_free

◆ M_malloc_zero()

void * M_malloc_zero ( size_t size )

Allocate size bytes and returns pointer to allocated memory and fills the memory with 0's.

Retains information about the size of the allocation and must be released using M_free().

Parameters

[in] size Number of bytes of memory to allocate.

Returns: Pointer to the newly allocated memory or NULL if the requested memory is unavailable. Memory must be released using M_free().

See also: M_free

◆ M_free()

void M_free ( void * ptr )

Release allocated memory.

Like libc free, but works with memory allocated by M_malloc class of functions to free allocated memory. Before being released, each byte of ptr is first set to zero.

Parameters

[in] ptr A pointer to a memory location to release returned by M_malloc like functions.

See also: M_malloc; M_malloc_zero; M_realloc; M_memdup; M_memdup_max

◆ M_realloc()

void * M_realloc	(	void *	ptr,
		size_t	size
	)

Resize an allocated memory block.

Like libc realloc, but works with memory allocated by M_malloc like functions. If ptr is unable to be resized, before being released, each byte of ptr is first set to zero.

Parameters

[in]	ptr	A pointer to a memory location to release/resize returned by M_malloc.
[in]	size	Number of bytes of memory to allocate.

Returns: Pointer to the newly allocated memory or NULL if the requested memory is zero in size or unavailable. Memory must be released using M_free().

See also: M_free

◆ M_realloc_zero()

void * M_realloc_zero	(	void *	ptr,
		size_t	size
	)

Resize an allocated memory block and fill any extended allocated memory with 0's.

Like libc realloc, but works with memory allocated by M_malloc like functions. If ptr is unable to be resized, before being released, each byte of ptr is first set to zero.

Parameters

[in]	ptr	A pointer to a memory location to release/resize returned by M_malloc.
[in]	size	Number of bytes of memory to allocate.

Returns: Pointer to the newly allocated memory or NULL if the requested memory is zero in size or unavailable. Memory must be released using M_free().

See also: M_free

◆ M_memdup()

void * M_memdup	(	const void *	src,
		size_t	size
	)

Allocate and copy size bytes from src to the newly allocated space.

src should be at least size memory area.

Parameters

[in]	src	Memory area to copy.
[in]	size	Number of bytes of memory to allocate and copy from src.

Returns: Pointer to the newly allocated memory or NULL if the requested memory is unavailable. Memory must be released with M_free().

See also: M_free

◆ M_memdup_max()

void * M_memdup_max	(	const void *	src,
		size_t	size,
		size_t	min_alloc_size
	)

Allocate at minimum min_alloc_size bytes, but copy no more than size bytes from ptr to the newly allocated space.

If size is larger than min_alloc_size, then size bytes will be allocated. src should be at least size memory area or NULL is returned.

This function behaves like M_malloc(size) when called M_memdup_max(NULL,0,size).

Parameters

[in]	src	Memory area to copy.
[in]	size	Number of bytes of memory to allocate and copy from src.
[in]	min_alloc_size	The minimum size of the returned allocation.

Returns: Pointer to the newly allocated memory or NULL if the requested memory is unavailable or if src is NULL but has positive * size. Memory must be released with M_free().

See also: M_free

◆ M_mem_set()

void * M_mem_set	(	void *	s,
		int	c,
		size_t	n
	)

Set memory.

Parameters

[in,out]	s	The memory to set.
[in]	c	The value to set.
[in]	n	The length of the memory segment.

Returns: A pointer to s.

◆ M_mem_move()

void * M_mem_move	(	void *	dst,
		const void *	src,
		size_t	size
	)

Copy memory area.

This function behaves like memcpy, but handles NULL gracefully.

Parameters

[in,out]	dst	Memory location to copy to.
[in]	src	Memory location to copy from.
[in]	size	Number of bytes to copy.

Returns: A pointer to dst.

◆ M_mem_copy()

void * M_mem_copy	(	void *	dst,
		const void *	src,
		size_t	size
	)

Copy memory area.

This function behaves like memcpy, but handles NULL gracefully.

Parameters

[in,out]	dst	Memory location to copy to.
[in]	src	Memory location to copy from.
[in]	size	Number of bytes to copy.

Returns: A pointer to dst.

◆ M_mem_eq()

M_bool M_mem_eq	(	const void *	m1,
		const void *	m2,
		size_t	size
	)

Compare memory segments.

This is done in a constant-time manner to prevent against timing related attacks.

Parameters

[in]	m1	Memory address.
[in]	m2	Memory address.
[in]	size	Length of memory to check.

Returns: M_TRUE if equal, M_FALSE if not.

◆ M_mem_cmpsort()

int M_mem_cmpsort	(	const void *	m1,
		size_t	size1,
		const void *	m2,
		size_t	size2
	)

A wrapper around memcmp that is NULL safe.

NOTE: this is not a constant-time comparison and thus should ONLY be used for sorting such as within qsort()!

Parameters

[in]	m1	Memory address.
[in]	size1	Size of m1.
[in]	m2	Memory address.
[in]	size2	Size of m2.

Returns: an integer less than, equal to, or greater than zero if m1 is less than, equal, or greater than m2 respectively

◆ M_mem_chr()

void * M_mem_chr	(	const void *	s,
		M_uint8	b,
		size_t	n
	)

Find first occurrence of b in s.

Parameters

[in]	s	The memory area to search.
[in]	b	The byte to search the memory area for.
[in]	n	The size of the memory area to search.

Returns: Pointer to the first occurence of b in s or NULL if not found or s is NULL or is 0.

◆ M_mem_contains()

M_bool M_mem_contains	(	const void *	haystack,
		size_t	haystack_len,
		const void *	needle,
		size_t	needle_len
	)

Determine if needle exists in haystack.

Parameters

[in]	haystack	Memory to search in.
[in]	haystack_len	The size in bytes of haystack.
[in]	needle	Memory to search for.
[in]	needle_len	The size in bytes of needle.

Returns: M_TRUE if needle exists in haystack or needle_len is 0, M_FALSE otherwise.

◆ M_mem_mem()

void * M_mem_mem	(	const void *	haystack,
		size_t	haystack_len,
		const void *	needle,
		size_t	needle_len
	)

Find first occurring bytes needle of length needle_len in haystack.

Parameters

[in]	haystack	Memory to search in.
[in]	haystack_len	The size in bytes of haystack.
[in]	needle	Memory to search for.
[in]	needle_len	The size in bytes of needle.

Returns: Pointer to first occurrence of needle in haystack or NULL if not found or haystack is NULL or haystack_len is 0.

◆ M_mem_rmem()

void * M_mem_rmem	(	const void *	haystack,
		size_t	haystack_len,
		const void *	needle,
		size_t	needle_len
	)

Find last occurring bytes needle of length needle_len in haystack.

Parameters

[in]	haystack	Memory to search in.
[in]	haystack_len	The size in bytes of haystack.
[in]	needle	Memory to search for.
[in]	needle_len	The size in bytes of needle.

Returns: Pointer to beginning of the last occurrence of needle in haystack. NULL if not found or haystack is NULL or haystack_len is 0.

◆ M_mem_str()

void * M_mem_str	(	const void *	haystack,
		size_t	haystack_len,
		const char *	needle
	)

Find first occurring string needle in haystack.

Parameters

[in]	haystack	Memory to search in.
[in]	haystack_len	The size in bytes of haystack.
[in]	needle	Memory to search for.

Returns: Pointer to first occurrence of needle in haystack or NULL if not found or haystack is NULL.

◆ M_mem_mempos()

M_bool M_mem_mempos	(	const void *	haystack,
		size_t	haystack_len,
		const void *	needle,
		size_t	needle_len,
		size_t *	idx
	)

Find index of first occurring bytes needle of length needle_len in haystack.

Parameters

[in]	haystack	Memory to search in.
[in]	haystack_len	The size in bytes of haystack.
[in]	needle	Memory to search for.
[in]	needle_len	The size in bytes of needle.
[out]	idx	The index of first occurrence of needle in haystack. Optional, pass NULL if not needed.

Returns: M_TRUE if found, M_FALSE otherwise.

◆ M_mem_count()

size_t M_mem_count	(	const void *	s,
		size_t	s_len,
		M_uint8	b
	)

Count the number of occurrences of byte b in memory area s of length s_len

Parameters

[in]	s	Pointer to the memory area to search.
[in]	s_len	The size of the memory area s.
[in]	b	The byte value to count occurrences of.

◆ M_mem_calc_lrc()

unsigned char M_mem_calc_lrc	(	const void *	s,
		size_t	s_len
	)

Calculate an LRC.

Parameters

[in]	s	Pointer to the memory area to search.
[in]	s_len	The size of the memory area s.

Returns: LRC

◆ M_mem_calc_crc8_ccitt()

M_uint8 M_mem_calc_crc8_ccitt	(	const void *	s,
		size_t	s_len
	)

Calculate a CRC (CRC-8/CCITT).

This is an 8-bit cyclic redundancy check (CRC), using the CCITT standard polynomial: x^8 + x^2 + x + 1. It's calculated using an initial value of zero.

Implementation is based on public-domain code that can be found here: https://www.3dbrew.org/wiki/CRC-8-CCITT

Parameters

[in]	s	Pointer to data to perform check on.
[in]	s_len	Size of memory area s.

Returns: CRC-8/CCITT value.

◆ M_mem_calc_crc16_ccitt()

M_uint16 M_mem_calc_crc16_ccitt	(	const void *	s,
		size_t	s_len
	)

Calculate a CRC (CRC-16/CCITT).

This is a 16-bit cyclic redundancy check (CRC), using the CCITT standard polynomial: x^16 + x^12 + x^5 + 1. It's calculated using an initial value of 0xFFFF, with no final XOR of the output CRC.

Implementation is based on public-domain code that can be found here: http://automationwiki.com/index.php/CRC-16-CCITT

Parameters

[in]	s	Pointer to data to perform check on.
[in]	s_len	Size of memory area s.

Returns: CRC-16/CCITT value.

◆ M_mem_calc_crc32()

M_uint32 M_mem_calc_crc32	(	const void *	s,
		size_t	s_len
	)

Calculate a CRC (CRC-32).

This is a 32-bit cyclic redundancy check (CRC), using the most common specification for 32-bit CRC's (ISO 3309 / ITU-T V.42). This particular CRC-32 variant is used all over the place - it's used in PNG, SATA, MPEG-2, Gzip, Bzip2, Posix cksum, and many others.

This spec uses the following polynomial: x^32 + x^26 + x^23 + x^22 + x^16 + x^12 + x^11 + x^10 + x^8 + x^7 + x^5 + x^4 + x^2 + x + 1

It's calculated using an initial value of 0xFFFFFFFF, and XOR'ing the CRC value by 0xFFFFFFFF after the computation is done to invert it.

Parameters

[in]	s	Pointer to data to perform check on.
[in]	s_len	Size of memory area s.

Returns: CRC-32 value.

◆ M_mem_swap_bytes()

M_bool M_mem_swap_bytes	(	M_uint8 *	s,
		size_t	s_len,
		size_t	idx1,
		size_t	idx2
	)

Swap byes between positions.

Parameters

[in,out]	s	Buffer with data to swap.
[in]	s_len	size of s.
[in]	idx1	Index to swap.
[in]	idx2	Index to swap.

Returns: M_TRUE on success. Otherwise M_FALSE.

Macros

Typedefs

Functions

Detailed Description

Hardening

Hardening that doesn't work

mlock

madvise with MADV_DONTDUMP

Conclusion

Additional External Security

Macro Definition Documentation

◆ M_mem_str

◆ M_mem_strpos

Typedef Documentation

◆ M_malloc_error_cb

Function Documentation

◆ M_malloc_register_errorcb()

◆ M_malloc_deregister_errorcb()

◆ M_malloc_clear_errorcb()

◆ M_malloc()

◆ M_malloc_zero()

◆ M_free()

◆ M_realloc()

◆ M_realloc_zero()

◆ M_memdup()

◆ M_memdup_max()

◆ M_mem_set()

◆ M_mem_move()

◆ M_mem_copy()

◆ M_mem_eq()

◆ M_mem_cmpsort()

◆ M_mem_chr()

◆ M_mem_contains()

◆ M_mem_mem()

◆ M_mem_rmem()

◆ M_mem_str()

◆ M_mem_mempos()

◆ M_mem_count()

◆ M_mem_calc_lrc()

◆ M_mem_calc_crc8_ccitt()

◆ M_mem_calc_crc16_ccitt()

◆ M_mem_calc_crc32()

◆ M_mem_swap_bytes()