From f4ba41ebdd6a9c3e7c2dd8ccf2940b724b333f99 Mon Sep 17 00:00:00 2001 From: Nicolas Williams Date: Thu, 29 Dec 2011 01:29:26 -0600 Subject: [PATCH] Pluggable libheimbase interface for DBs and misc libheimbase enhancements MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [Code reviewed by Love Hörnquist Åstrand ] Added heim_db_*() entry points for dealing with databases, and make krb5_aname_to_localname() use it. The following enhancements to libheimbase are included: - Add heim_data_t and heim_string_t "reference" variants to avoid memory copies of potentially large data/strings. See heim_data_ref_create() and heim_string_ref_create(). - Added enhancements to heim_array_t to allow their use for queues and stacks, and to improve performance. See heim_array_insert_value(). - Added XPath-like accessors for heim_object_t. See heim_path_get(), heim_path_copy(), heim_path_create(), and heim_path_delete(). These are used extensively in the DB framework's generic composition of ACID support and in the test_base program - Made libheimbase more consistent with Core Foundation naming conventions. See heim_{dict, array}_{get, copy}_value() and heim_path_{get, copy}(). - Added functionality to and fixed bugs in base/json.c: - heim_serialize(); - depth limit for JSON parsing (for DoS protection); - pretty-printing; - JSON compliance (see below); - flag options for parsing and serializing; these are needed because of impedance mismatches between heim_object_t and JSON (e.g., heim_dict_t allows non-string keys, but JSON does not; heimbase supports binary data, while JSON does not). - Added heim_error_enomem(). - Enhanced the test_base program to test new functionality and to use heim_path*() to better test JSON encoding. This includes some fuzz testing of JSON parsing, and running the test under valgrind. - Started to add doxygen documentation for libheimbase (but doc build for libheimbase is still incomplete). Note that there's still some incomplete JSON support: - JSON string quoting is not fully implemented; - libheimbase lacks support for real numbers, while JSON has it -- otherwise libheimbase is a superset of JSON, specifically in that any heim_object_t can be a key for an associative array. The following DB backends are supported natively: - "sorted-text", a binary search of sorted (in C locale), flat text files; - "json", a backend that stores DB contents serialized as JSON (this is intended for configuration-like contents). The DB framework supports: - multiple key/value tables per-DB - ACID transactions The DB framework also natively implements ACID transactions for any DB backends that a) do not provide transactions natively, b) do provide lock/unlock/sync methods (even on Windows). This includes autocommit of DB updates outside transactions. Future DB enhancements may include: - add backends for various DB types (BDB, CDB, MDB, ...); - make libhdb use heim_db_t; - add a command-line tool for interfacing to databases via libheimbase (e.g., to get/set/delete values, create/copy/ backup DBs, inspect history, check integrity); - framework-level transaction logging (with redo and undo logging), for generic incremental replication; - framework-level DB integrity checking. We could store a MAC of the XOR of a hash function applied to {key, value} for every entry in the DB, then use this to check DB integrity incrementally during incremental replication, as well as for the whole DB. --- Makefile.am | 2 +- appl/dbutils/bsearch.c | 10 +- base/Makefile.am | 21 +- base/NTMakefile | 30 +- base/array.c | 222 ++++- base/baselocl.h | 10 + base/bsearch.c | 127 ++- base/data.c | 36 +- base/db.c | 1709 +++++++++++++++++++++++++++++++++ base/dict.c | 22 +- base/error.c | 30 +- base/heimbase.c | 404 +++++++- base/heimbase.h | 215 ++++- base/heimbasepriv.h | 18 +- base/json.c | 426 +++++++- base/roken_rename.h | 9 +- base/string.c | 71 ++ base/test_base.c | 687 ++++++++++++- base/version-script.map | 48 +- doc/Makefile.am | 11 +- doc/base.din | 15 + doc/base.hhp | 8 + kdc/kdc-tester.c | 2 +- lib/krb5/Makefile.am | 6 +- lib/krb5/NTMakefile | 10 +- lib/krb5/aname_to_localname.c | 186 ++-- lib/krb5/db_plugin.c | 31 + lib/krb5/db_plugin.h | 68 ++ lib/krb5/krb5-plugin.7 | 35 +- lib/roken/NTMakefile | 1 + 30 files changed, 4211 insertions(+), 259 deletions(-) create mode 100644 base/db.c create mode 100644 doc/base.din create mode 100644 doc/base.hhp create mode 100644 lib/krb5/db_plugin.c create mode 100644 lib/krb5/db_plugin.h diff --git a/Makefile.am b/Makefile.am index 250809631..8d875cb9b 100644 --- a/Makefile.am +++ b/Makefile.am @@ -6,7 +6,7 @@ if KCM kcm_dir = kcm endif -SUBDIRS= include base lib kuser kdc admin kadmin kpasswd +SUBDIRS= include lib/roken base lib kuser kdc admin kadmin kpasswd SUBDIRS+= $(kcm_dir) appl tools tests packages etc po if HEIMDAL_DOCUMENTATION diff --git a/appl/dbutils/bsearch.c b/appl/dbutils/bsearch.c index b6750994d..da37251d3 100644 --- a/appl/dbutils/bsearch.c +++ b/appl/dbutils/bsearch.c @@ -140,13 +140,13 @@ main(int argc, char **argv) argc--; argv++; - ret = __bsearch_file_open(fname, max_size, block_size, &bfh, &reads); + ret = _bsearch_file_open(fname, max_size, block_size, &bfh, &reads); if (ret != 0) { perror("bsearch_file_open"); return 1; } - __bsearch_file_info(bfh, &block_size, &max_size, &blockwise); + _bsearch_file_info(bfh, &block_size, &max_size, &blockwise); if (verbose_flag && blockwise) { fprintf(stderr, "Using block-wise method with block size %lu and " "cache size %lu\n", @@ -172,11 +172,11 @@ main(int argc, char **argv) if (!*key) continue; } - ret = __bsearch_file(bfh, key, &value, &loc, &loops, &reads); + ret = _bsearch_file(bfh, key, &value, &loc, &loops, &reads); if (ret != 0) { if (ret > 0) { fprintf(stderr, "Error: %s\n", strerror(ret)); - __bsearch_file_close(&bfh); + _bsearch_file_close(&bfh); return 1; } if (verbose_flag) @@ -200,6 +200,6 @@ main(int argc, char **argv) } if (failures) return 2; - __bsearch_file_close(&bfh); + _bsearch_file_close(&bfh); return 0; } diff --git a/base/Makefile.am b/base/Makefile.am index 7d22ed325..ecd5b2240 100644 --- a/base/Makefile.am +++ b/base/Makefile.am @@ -1,6 +1,12 @@ include $(top_srcdir)/Makefile.am.common +if do_roken_rename +ES = base64.c +endif + +AM_CPPFLAGS += $(ROKEN_RENAME) + lib_LTLIBRARIES = libheimbase.la check_PROGRAMS = test_base @@ -20,6 +26,7 @@ dist_libheimbase_la_SOURCES = \ bsearch.c \ bool.c \ data.c \ + db.c \ dict.c \ error.c \ heimbase.c \ @@ -28,10 +35,22 @@ dist_libheimbase_la_SOURCES = \ json.c \ null.c \ number.c \ + roken_rename.h \ string.c +nodist_libsl_la_SOURCES = $(ES) + +# install these? + libheimbase_la_DEPENDENCIES = version-script.map -test_base_LDADD = libheimbase.la +test_base_LDADD = libheimbase.la $(LIB_roken) + +CLEANFILES = base64.c EXTRA_DIST = NTMakefile version-script.map + +base64.c: + $(RM) base64.c + $(LN_S) $(srcdir)/../roken/base64.c . + diff --git a/base/NTMakefile b/base/NTMakefile index 6a389d50d..87abf3b93 100644 --- a/base/NTMakefile +++ b/base/NTMakefile @@ -31,25 +31,51 @@ RELDIR=base +intcflags=-I$(SRCDIR) -I$(OBJ) + !include ../windows/NTMakefile.w32 INCFILES=$(INCDIR)\heimbase.h +test_binaries = $(OBJ)\test_base.exe + libheimbase_OBJS = \ $(OBJ)\array.obj \ - $(OBJ)\bsearch.obj \ $(OBJ)\bool.obj \ + $(OBJ)\bsearch.obj \ + $(OBJ)\data.obj \ + $(OBJ)\db.obj \ $(OBJ)\dict.obj \ $(OBJ)\error.obj \ $(OBJ)\heimbase.obj \ + $(OBJ)\json.obj \ $(OBJ)\null.obj \ $(OBJ)\number.obj \ $(OBJ)\string.obj $(LIBHEIMBASE): $(libheimbase_OBJS) - $(LIBCON) + $(LIBCON_C) -OUT:$@ $(LIBROKEN) @<< +$(libheimbase_OBJS: = +) +<< + +test:: test-binaries test-run + +test-run: + cd $(OBJ) + test_base.exe + cd $(SRCDIR) all:: $(INCFILES) $(LIBHEIMBASE) clean:: -$(RM) $(INCFILES) + +test-binaries: $(test_binaries) + +$(test_binaries): $$(@R).obj $(LIBHEIMBASE) $(LIBVERS) $(LIBROKEN) + $(EXECONLINK) + $(EXEPREP_NODIST) + +$(test_binaries:.exe=.obj): $$(@B).c + $(C2OBJ_C) -Fo$@ -Fd$(@D)\ $** -DBlah diff --git a/base/array.c b/base/array.c index ddd63f713..a513077c1 100644 --- a/base/array.c +++ b/base/array.c @@ -42,6 +42,8 @@ struct heim_array_data { size_t len; heim_object_t *val; + size_t allocated_len; + heim_object_t *allocated; }; static void @@ -51,7 +53,7 @@ array_dealloc(heim_object_t ptr) size_t n; for (n = 0; n < array->len; n++) heim_release(array->val[n]); - free(array->val); + free(array->allocated); } struct heim_type_data array_object = { @@ -79,6 +81,8 @@ heim_array_create(void) if (array == NULL) return NULL; + array->allocated = NULL; + array->allocated_len = 0; array->val = NULL; array->len = 0; @@ -110,16 +114,134 @@ int heim_array_append_value(heim_array_t array, heim_object_t object) { heim_object_t *ptr; + size_t leading = array->val - array->allocated; /* unused leading slots */ + size_t trailing = array->allocated_len - array->len - leading; + size_t new_len; - ptr = realloc(array->val, (array->len + 1) * sizeof(array->val[0])); + if (trailing > 0) { + /* We have pre-allocated space; use it */ + array->val[array->len++] = heim_retain(object); + return 0; + } + + if (leading > (array->len + 1)) { + /* + * We must have appending to, and deleting at index 0 from this + * array a lot; don't want to grow forever! + */ + (void) memmove(&array->allocated[0], &array->val[0], + array->len * sizeof(array->val[0])); + array->val = array->allocated; + + /* We have pre-allocated space; use it */ + array->val[array->len++] = heim_retain(object); + return 0; + } + + /* Pre-allocate extra .5 times number of used slots */ + new_len = leading + array->len + 1 + (array->len >> 1); + ptr = realloc(array->allocated, new_len * sizeof(array->val[0])); if (ptr == NULL) return ENOMEM; - array->val = ptr; + array->allocated = ptr; + array->allocated_len = new_len; + array->val = &ptr[leading]; array->val[array->len++] = heim_retain(object); return 0; } +/* + * Internal function to insert at index 0, taking care to optimize the + * case where we're always inserting at index 0, particularly the case + * where we insert at index 0 and delete from the right end. + */ +static int +heim_array_prepend_value(heim_array_t array, heim_object_t object) +{ + heim_object_t *ptr; + size_t leading = array->val - array->allocated; /* unused leading slots */ + size_t trailing = array->allocated_len - array->len - leading; + size_t new_len; + + if (leading > 0) { + /* We have pre-allocated space; use it */ + array->val--; + array->val[0] = heim_retain(object); + array->len++; + return 0; + } + if (trailing > (array->len + 1)) { + /* + * We must have prepending to, and deleting at index + * array->len - 1 from this array a lot; don't want to grow + * forever! + */ + (void) memmove(&array->allocated[array->len], &array->val[0], + array->len * sizeof(array->val[0])); + array->val = &array->allocated[array->len]; + + /* We have pre-allocated space; use it */ + array->val--; + array->val[0] = heim_retain(object); + array->len++; + return 0; + } + /* Pre-allocate extra .5 times number of used slots */ + new_len = array->len + 1 + trailing + (array->len >> 1); + ptr = realloc(array->allocated, new_len * sizeof(array->val[0])); + if (ptr == NULL) + return ENOMEM; + (void) memmove(&ptr[1], &ptr[0], array->len * sizeof (array->val[0])); + array->allocated = ptr; + array->allocated_len = new_len; + array->val = &ptr[0]; + array->val[0] = heim_retain(object); + array->len++; + + return 0; +} + +/** + * Insert an object at a given index in an array + * + * @param array array to add too + * @param idx index where to add element (-1 == append, -2 next to last, ...) + * @param object the object to add + * + * @return zero if added, errno otherwise + */ + +int +heim_array_insert_value(heim_array_t array, size_t idx, heim_object_t object) +{ + int ret; + + if (idx == 0) + return heim_array_prepend_value(array, object); + else if (idx > array->len) + heim_abort("index too large"); + + /* + * We cheat: append this element then rotate elements around so we + * have this new element at the desired location, unless we're truly + * appending the new element. This means reusing array growth in + * heim_array_append_value() instead of duplicating that here. + */ + ret = heim_array_append_value(array, object); + if (ret != 0 || idx == (array->len - 1)) + return ret; + /* + * Shift to the right by one all the elements after idx, then set + * [idx] to the new object. + */ + (void) memmove(&array->val[idx + 1], &array->val[idx], + (array->len - idx - 1) * sizeof(array->val[0])); + array->val[idx] = heim_retain(object); + + return 0; +} + /** * Iterate over all objects in array * @@ -153,6 +275,39 @@ heim_array_iterate(heim_array_t array, void (^fn)(heim_object_t)) } #endif +/** + * Iterate over all objects in array, backwards + * + * @param array array to iterate over + * @param ctx context passed to fn + * @param fn function to call on each object + */ + +void +heim_array_iterate_reverse_f(heim_array_t array, void *ctx, heim_array_iterator_f_t fn) +{ + size_t n; + for (n = array->len; n > 0; n--) + fn(array->val[n - 1], ctx); +} + +#ifdef __BLOCKS__ +/** + * Iterate over all objects in array, backwards + * + * @param array array to iterate over + * @param fn block to call on each object + */ + +void +heim_array_iterate_reverse(heim_array_t array, void (^fn)(heim_object_t)) +{ + size_t n; + for (n = array->len; n > 0; n--) + fn(array->val[n - 1]); +} +#endif + /** * Get length of array * @@ -168,7 +323,25 @@ heim_array_get_length(heim_array_t array) } /** - * Copy value of array + * Get value of element at array index + * + * @param array array copy object from + * @param idx index of object, 0 based, must be smaller then + * heim_array_get_length() + * + * @return a not-retained copy of the object + */ + +heim_object_t +heim_array_get_value(heim_array_t array, size_t idx) +{ + if (idx >= array->len) + heim_abort("index too large"); + return array->val[idx]; +} + +/** + * Get value of element at array index * * @param array array copy object from * @param idx index of object, 0 based, must be smaller then @@ -178,11 +351,30 @@ heim_array_get_length(heim_array_t array) */ heim_object_t -heim_array_get_value(heim_array_t array, size_t idx) +heim_array_copy_value(heim_array_t array, size_t idx) { if (idx >= array->len) heim_abort("index too large"); - return array->val[idx]; + return heim_retain(array->val[idx]); +} + +/** + * Set value at array index + * + * @param array array copy object from + * @param idx index of object, 0 based, must be smaller then + * heim_array_get_length() + * @param value value to set + * + */ + +void +heim_array_set_value(heim_array_t array, size_t idx, heim_object_t value) +{ + if (idx >= array->len) + heim_abort("index too large"); + heim_release(array->val[idx]); + array->val[idx] = heim_retain(value); } /** @@ -202,9 +394,21 @@ heim_array_delete_value(heim_array_t array, size_t idx) array->len--; - if (idx < array->len) - memmove(&array->val[idx], &array->val[idx + 1], - (array->len - idx) * sizeof(array->val[0])); + /* + * Deleting the first or last elements is cheap, as we leave + * allocated space for opportunistic reuse later; no realloc(), no + * memmove(). All others require a memmove(). + * + * If we ever need to optimize deletion of non-last/ non-first + * element we can use a tagged object type to signify "deleted + * value" so we can leave holes in the array, avoid memmove()s on + * delete, and opportunistically re-use those holes on insert. + */ + if (idx == 0) + array->val++; + else if (idx < array->len) + (void) memmove(&array->val[idx], &array->val[idx + 1], + (array->len - idx) * sizeof(array->val[0])); heim_release(obj); } diff --git a/base/baselocl.h b/base/baselocl.h index a038d2016..ce9b37d28 100644 --- a/base/baselocl.h +++ b/base/baselocl.h @@ -52,6 +52,16 @@ #include #endif +#ifdef LIBINTL +#include +#define N_(x,y) dgettext(HEIMDAL_TEXTDOMAIN, x) +#else +#define N_(x,y) (x) +#define bindtextdomain(package, localedir) +#endif + +#include + #include "heimqueue.h" #include "heim_threads.h" #include "heimbase.h" diff --git a/base/bsearch.c b/base/bsearch.c index 513d57f1e..443408c45 100644 --- a/base/bsearch.c +++ b/base/bsearch.c @@ -33,6 +33,9 @@ #include #include +#ifdef HAVE_IO_H +#include +#endif #ifdef HAVE_UNISTD_H #include #endif @@ -75,8 +78,8 @@ * * bsearch_common() contains the common text block binary search code. * - * __bsearch_text() is the interface for searching in-core text. - * __bsearch_file() is the interface for block-wise searching files. + * _bsearch_text() is the interface for searching in-core text. + * _bsearch_file() is the interface for block-wise searching files. */ struct bsearch_file_handle { @@ -104,7 +107,7 @@ find_line(const char *buf, size_t i, size_t right) return NULL; } -/** +/* * Common routine for binary searching text in core. * * Perform a binary search of a char array containing a block from a @@ -150,7 +153,7 @@ bsearch_common(const char *buf, size_t sz, const char *key, const char *linep; size_t key_start, key_len; /* key string in buf */ size_t val_start, val_len; /* value string in buf */ - int key_cmp; + int key_cmp = -1; size_t k; size_t l; /* left side of buffer for binary search */ size_t r; /* right side of buffer for binary search */ @@ -289,7 +292,7 @@ bsearch_common(const char *buf, size_t sz, const char *key, return ret; } -/** +/* * Binary search a char array containing sorted text records separated * by new-lines (or CRLF). Each record consists of a key and an * optional value following the key, separated from the key by unquoted @@ -315,7 +318,7 @@ bsearch_common(const char *buf, size_t sz, const char *key, * needed for the search (useful for benchmarking) */ int -__bsearch_text(const char *buf, size_t buf_sz, const char *key, +_bsearch_text(const char *buf, size_t buf_sz, const char *key, char **value, size_t *location, size_t *loops) { return bsearch_common(buf, buf_sz, key, 1, value, location, NULL, loops); @@ -323,7 +326,7 @@ __bsearch_text(const char *buf, size_t buf_sz, const char *key, #define MAX_BLOCK_SIZE (1024 * 1024) #define DEFAULT_MAX_FILE_SIZE (1024 * 1024) -/** +/* * Open a file for binary searching. The file will be read in entirely * if it is smaller than @max_sz, else a cache of @max_sz bytes will be * allocated. @@ -339,14 +342,14 @@ __bsearch_text(const char *buf, size_t buf_sz, const char *key, * * Outputs: * - * @bfh Handle for use with __bsearch_file() and __bsearch_file_close() + * @bfh Handle for use with _bsearch_file() and _bsearch_file_close() * @reads Number of reads performed */ int -__bsearch_file_open(const char *fname, size_t max_sz, size_t page_sz, +_bsearch_file_open(const char *fname, size_t max_sz, size_t page_sz, bsearch_file_handle *bfh, size_t *reads) { - bsearch_file_handle new_bfh; + bsearch_file_handle new_bfh = NULL; struct stat st; size_t i; int fd; @@ -469,12 +472,12 @@ err: return ret; } -/** +/* * Indicate whether the given binary search file handle will be searched * with block-wise method. */ void -__bsearch_file_info(bsearch_file_handle bfh, +_bsearch_file_info(bsearch_file_handle bfh, size_t *page_sz, size_t *max_sz, int *blockwise) { if (page_sz) @@ -485,7 +488,7 @@ __bsearch_file_info(bsearch_file_handle bfh, *blockwise = (bfh->file_sz != bfh->cache_sz); } -/** +/* * Close the given binary file search handle. * * Inputs: @@ -493,7 +496,7 @@ __bsearch_file_info(bsearch_file_handle bfh, * @bfh Pointer to variable containing handle to close. */ void -__bsearch_file_close(bsearch_file_handle *bfh) +_bsearch_file_close(bsearch_file_handle *bfh) { if (!*bfh) return; @@ -507,7 +510,7 @@ __bsearch_file_close(bsearch_file_handle *bfh) *bfh = NULL; } -/** +/* * Private function to get a page from a cache. The cache is a char * array of 2^n - 1 double-size page worth of bytes, where n is the * number of tree levels that the cache stores. The cache can be @@ -567,7 +570,7 @@ get_page_from_cache(bsearch_file_handle bfh, size_t level, size_t page_idx, return 1; } -/** +/* * Private function to read a page of @page_sz from @fd at offset @off * into @buf, outputing the number of bytes read, which will be the same * as @page_sz unless the page being read is the last page, in which @@ -646,7 +649,7 @@ read_page(bsearch_file_handle bfh, size_t level, size_t page_idx, size_t page, return 0; } -/** +/* * Perform a binary search of a file where each line is a record (LF and * CRLF supported). Each record consists of a key followed by an * optional value separated from the key by whitespace. Whitespace can @@ -685,7 +688,7 @@ read_page(bsearch_file_handle bfh, size_t level, size_t page_idx, size_t page, * (useful for confirming logarithmic performance) */ int -__bsearch_file(bsearch_file_handle bfh, const char *key, +_bsearch_file(bsearch_file_handle bfh, const char *key, char **value, size_t *location, size_t *loops, size_t *reads) { int ret; @@ -707,7 +710,7 @@ __bsearch_file(bsearch_file_handle bfh, const char *key, /* If whole file is in memory then search that and we're done */ if (bfh->file_sz == bfh->cache_sz) - return __bsearch_text(bfh->cache, bfh->cache_sz, key, value, location, loops); + return _bsearch_text(bfh->cache, bfh->cache_sz, key, value, location, loops); /* Else block-wise binary search */ @@ -794,3 +797,89 @@ __bsearch_file(bsearch_file_handle bfh, const char *key, return -1; } + +static int +stdb_open(void *plug, const char *dbtype, const char *dbname, + heim_dict_t options, void **db, heim_error_t *error) +{ + bsearch_file_handle bfh; + char *p; + int ret; + + if (error) + *error = NULL; + if (dbname == NULL || *dbname == '\0') { + if (error) + *error = heim_error_create(EINVAL, + N_("DB name required for sorted-text DB " + "plugin", "")); + return EINVAL; + } + p = strrchr(dbname, '.'); + if (p == NULL || strcmp(p, ".txt") != 0) { + if (error) + *error = heim_error_create(ENOTSUP, + N_("Text file (name ending in .txt) " + "required for sorted-text DB plugin", + "")); + return ENOTSUP; + } + + ret = _bsearch_file_open(dbname, 0, 0, &bfh, NULL); + if (ret) + return ret; + + *db = bfh; + return 0; +} + +static int +stdb_close(void *db, heim_error_t *error) +{ + bsearch_file_handle bfh = db; + + if (error) + *error = NULL; + _bsearch_file_close(&bfh); + return 0; +} + +static heim_data_t +stdb_copy_value(void *db, heim_string_t table, heim_data_t key, + heim_error_t *error) +{ + bsearch_file_handle bfh = db; + const char *k; + char *v; + heim_data_t value; + int ret; + + if (error) + *error = NULL; + + if (table == NULL) + table = HSTR(""); + + if (table != HSTR("")) + return NULL; + + if (heim_get_tid(key) == HEIM_TID_STRING) + k = heim_string_get_utf8((heim_string_t)key); + else + k = (const char *)heim_data_get_ptr(key); + ret = _bsearch_file(bfh, k, &v, NULL, NULL, NULL); + if (ret != 0) { + if (ret > 0 && error) + *error = heim_error_create(ret, "%s", strerror(ret)); + return NULL; + } + value = heim_data_create(v, strlen(v)); + free(v); + /* XXX Handle ENOMEM */ + return value; +} + +struct heim_db_type heim_sorted_text_file_dbtype = { + 1, stdb_open, NULL, stdb_close, NULL, NULL, NULL, NULL, NULL, NULL, + stdb_copy_value, NULL, NULL, NULL +}; diff --git a/base/data.c b/base/data.c index fc1c4422e..a9f836d98 100644 --- a/base/data.c +++ b/base/data.c @@ -37,6 +37,19 @@ static void data_dealloc(void *ptr) { + heim_data_t d = ptr; + heim_octet_string *os = (heim_octet_string *)d; + heim_data_free_f_t *deallocp; + heim_data_free_f_t dealloc; + + if (os->data == NULL) + return; + + /* Possible string ref */ + deallocp = _heim_get_isaextra(os, 0); + dealloc = *deallocp; + if (dealloc != NULL) + dealloc(os->data); } static int @@ -92,10 +105,28 @@ heim_data_create(const void *data, size_t length) return (heim_data_t)os; } +heim_data_t +heim_data_ref_create(const void *data, size_t length, + heim_data_free_f_t dealloc) +{ + heim_octet_string *os; + heim_data_free_f_t *deallocp; + + os = _heim_alloc_object(&_heim_data_object, sizeof(*os) + length); + if (os) { + os->data = (void *)data; + os->length = length; + deallocp = _heim_get_isaextra(os, 0); + *deallocp = dealloc; + } + return (heim_data_t)os; +} + + /** * Return the type ID of data objects * - * @return type id of string objects + * @return type id of data objects */ heim_tid_t @@ -115,16 +146,19 @@ heim_data_get_type_id(void) const heim_octet_string * heim_data_get_data(heim_data_t data) { + /* Note that this works for data and data_ref objects */ return (const heim_octet_string *)data; } const void * heim_data_get_ptr(heim_data_t data) { + /* Note that this works for data and data_ref objects */ return ((const heim_octet_string *)data)->data; } size_t heim_data_get_length(heim_data_t data) { + /* Note that this works for data and data_ref objects */ return ((const heim_octet_string *)data)->length; } diff --git a/base/db.c b/base/db.c new file mode 100644 index 000000000..0bc769f1d --- /dev/null +++ b/base/db.c @@ -0,0 +1,1709 @@ +/* + * Copyright (c) 2011, Secure Endpoints Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/* + * This is a pluggable simple DB abstraction, with a simple get/set/ + * delete key/value pair interface. + * + * Plugins may provide any of the following optional features: + * + * - tables -- multiple attribute/value tables in one DB + * - locking + * - transactions (i.e., allow any heim_object_t as key or value) + * - transcoding of values + * + * Stackable plugins that provide missing optional features are + * possible. + * + * Any plugin that provides locking will also provide transactions, but + * those transactions will not be atomic in the face of failures (a + * memory-based rollback log is used). + */ + +#include +#include +#include +#include +#include +#include +#ifndef WIN32 +#include +#endif +#ifdef HAVE_IO_H +#include +#endif +#ifdef HAVE_UNISTD_H +#include +#endif +#include + +#include "baselocl.h" +#include + +#define HEIM_ENOMEM(ep) \ + (((ep) && !*(ep)) ? \ + heim_error_get_code((*(ep) = heim_error_enomem())) : ENOMEM) + +#define HEIM_ERROR_HELPER(ep, ec, args) \ + (((ep) && !*(ep)) ? \ + heim_error_get_code((*(ep) = heim_error_create args)) : (ec)) + +#define HEIM_ERROR(ep, ec, args) \ + (ec == ENOMEM) ? HEIM_ENOMEM(ep) : HEIM_ERROR_HELPER(ep, ec, args); + +static heim_string_t to_base64(heim_data_t, heim_error_t *); +static heim_data_t from_base64(heim_string_t, heim_error_t *); + +static int open_file(const char *, int , int, int *, heim_error_t *); +static int read_json(const char *, heim_object_t *, heim_error_t *); +static struct heim_db_type json_dbt; + +static void db_dealloc(void *ptr); + +struct heim_type_data db_object = { + HEIM_TID_DB, + "db-object", + NULL, + db_dealloc, + NULL, + NULL, + NULL +}; + + +static heim_base_once_t db_plugin_init_once = HEIM_BASE_ONCE_INIT; + +static heim_dict_t db_plugins; + +typedef struct db_plugin { + heim_string_t name; + heim_db_plug_open_f_t openf; + heim_db_plug_clone_f_t clonef; + heim_db_plug_close_f_t closef; + heim_db_plug_lock_f_t lockf; + heim_db_plug_unlock_f_t unlockf; + heim_db_plug_sync_f_t syncf; + heim_db_plug_begin_f_t beginf; + heim_db_plug_commit_f_t commitf; + heim_db_plug_rollback_f_t rollbackf; + heim_db_plug_copy_value_f_t copyf; + heim_db_plug_set_value_f_t setf; + heim_db_plug_del_key_f_t delf; + heim_db_plug_iter_f_t iterf; + void *data; +} db_plugin_desc, *db_plugin; + +struct heim_db_data { + db_plugin plug; + heim_string_t dbtype; + heim_string_t dbname; + heim_dict_t options; + void *db_data; + heim_data_t to_release; + heim_error_t error; + int ret; + unsigned int in_transaction:1; + unsigned int ro:1; + unsigned int ro_tx:1; + heim_dict_t set_keys; + heim_dict_t del_keys; + heim_string_t current_table; +}; + +static int +db_do_log_actions(heim_db_t db, heim_error_t *error); +static int +db_replay_log(heim_db_t db, heim_error_t *error); + +static HEIMDAL_MUTEX db_type_mutex = HEIMDAL_MUTEX_INITIALIZER; + +static void +db_init_plugins_once(void *arg) +{ + db_plugins = heim_retain(arg); +} + +static void +plugin_dealloc(void *arg) +{ + db_plugin plug = arg; + + heim_release(plug->name); +} + +/** heim_db_register + * @brief Registers a DB type for use with heim_db_create(). + * + * @param dbtype Name of DB type + * @param data Private data argument to the dbtype's openf method + * @param plugin Structure with DB type methods (function pointers) + * + * Backends that provide begin/commit/rollback methods must provide ACID + * semantics. + * + * The registered DB type will have ACID semantics for backends that do + * not provide begin/commit/rollback methods but do provide lock/unlock + * and rdjournal/wrjournal methods (using a replay log journalling + * scheme). + * + * If the registered DB type does not natively provide read vs. write + * transaction isolation but does provide a lock method then the DB will + * provide read/write transaction isolation. + * + * @return ENOMEM on failure, else 0. + * + * @addtogroup heimbase + */ +int +heim_db_register(const char *dbtype, + void *data, + struct heim_db_type *plugin) +{ + heim_dict_t plugins; + heim_string_t s; + db_plugin plug, plug2; + int ret = 0; + + if ((plugin->beginf != NULL && plugin->commitf == NULL) || + (plugin->beginf != NULL && plugin->rollbackf == NULL) || + (plugin->lockf != NULL && plugin->unlockf == NULL) || + plugin->copyf == NULL) + heim_abort("Invalid DB plugin; make sure methods are paired"); + + /* Initialize */ + plugins = heim_dict_create(11); + if (plugins == NULL) + return ENOMEM; + heim_base_once_f(&db_plugin_init_once, plugins, db_init_plugins_once); + heim_release(plugins); + heim_assert(db_plugins != NULL, "heim_db plugin table initialized"); + + s = heim_string_create(dbtype); + if (s == NULL) + return ENOMEM; + + plug = heim_alloc(sizeof (*plug), "db_plug", plugin_dealloc); + if (plug == NULL) { + heim_release(s); + return ENOMEM; + } + + plug->name = heim_retain(s); + plug->openf = plugin->openf; + plug->clonef = plugin->clonef; + plug->closef = plugin->closef; + plug->lockf = plugin->lockf; + plug->unlockf = plugin->unlockf; + plug->syncf = plugin->syncf; + plug->beginf = plugin->beginf; + plug->commitf = plugin->commitf; + plug->rollbackf = plugin->rollbackf; + plug->copyf = plugin->copyf; + plug->setf = plugin->setf; + plug->delf = plugin->delf; + plug->iterf = plugin->iterf; + plug->data = data; + + HEIMDAL_MUTEX_lock(&db_type_mutex); + plug2 = heim_dict_get_value(db_plugins, s); + if (plug2 == NULL) + ret = heim_dict_set_value(db_plugins, s, plug); + HEIMDAL_MUTEX_unlock(&db_type_mutex); + heim_release(plug); + heim_release(s); + + return ret; +} + +static void +db_dealloc(void *arg) +{ + heim_db_t db = arg; + heim_assert(!db->in_transaction, + "rollback or commit heim_db_t before releasing it"); + if (db->db_data) + (void) db->plug->closef(db->db_data, NULL); + heim_release(db->to_release); + heim_release(db->dbtype); + heim_release(db->dbname); + heim_release(db->options); + heim_release(db->set_keys); + heim_release(db->del_keys); + heim_release(db->error); +} + +struct dbtype_iter { + heim_db_t db; + const char *dbname; + heim_dict_t options; + heim_error_t *error; +}; + +/* + * Helper to create a DB handle with the first registered DB type that + * can open the given DB. This is useful when the app doesn't know the + * DB type a priori. This assumes that DB types can "taste" DBs, either + * from the filename extension or from the actual file contents. + */ +static void +dbtype_iter2create_f(heim_object_t dbtype, heim_object_t junk, void *arg) +{ + struct dbtype_iter *iter_ctx = arg; + + if (iter_ctx->db != NULL) + return; + iter_ctx->db = heim_db_create(heim_string_get_utf8(dbtype), + iter_ctx->dbname, iter_ctx->options, + iter_ctx->error); +} + +/** + * Open a database of the given dbtype. + * + * Database type names can be composed of one or more pseudo-DB types + * and one concrete DB type joined with a '+' between each. For + * example: "transaction+bdb" might be a Berkeley DB with a layer above + * that provides transactions. + * + * Options may be provided via a dict (an associative array). Existing + * options include: + * + * - "create", with any value (create if DB doesn't exist) + * - "exclusive", with any value (exclusive create) + * - "truncate", with any value (truncate the DB) + * - "read-only", with any value (disallow writes) + * - "sync", with any value (make transactions durable) + * - "journal-name", with a string value naming a journal file name + * + * @param dbtype Name of DB type + * @param dbname Name of DB (likely a file path) + * @param options Options dict + * @param db Output open DB handle + * @param error Output error object + * + * @return a DB handle + * + * @addtogroup heimbase + */ +heim_db_t +heim_db_create(const char *dbtype, const char *dbname, + heim_dict_t options, heim_error_t *error) +{ + heim_string_t s; + char *p; + db_plugin plug; + heim_db_t db; + int ret = 0; + + if (options == NULL) { + options = heim_dict_create(11); + if (options == NULL) { + if (error) + *error = heim_error_enomem(); + return NULL; + } + } else { + (void) heim_retain(options); + } + + if (db_plugins == NULL) { + heim_release(options); + return NULL; + } + + if (dbtype == NULL || *dbtype == '\0') { + struct dbtype_iter iter_ctx = { NULL, dbname, options, error}; + + /* Try all dbtypes */ + heim_dict_iterate_f(db_plugins, &iter_ctx, dbtype_iter2create_f); + heim_release(options); + return iter_ctx.db; + } else if (strstr(dbtype, "json")) { + (void) heim_db_register(dbtype, NULL, &json_dbt); + } + + /* + * Allow for dbtypes that are composed from pseudo-dbtypes chained + * to a real DB type with '+'. For example a pseudo-dbtype might + * add locking, transactions, transcoding of values, ... + */ + p = strchr(dbtype, '+'); + if (p != NULL) + s = heim_string_create_with_bytes(dbtype, p - dbtype); + else + s = heim_string_create(dbtype); + if (s == NULL) { + heim_release(options); + return NULL; + } + + HEIMDAL_MUTEX_lock(&db_type_mutex); + plug = heim_dict_get_value(db_plugins, s); + HEIMDAL_MUTEX_unlock(&db_type_mutex); + heim_release(s); + if (plug == NULL) { + if (error) + *error = heim_error_create(ENOENT, + N_("Heimdal DB plugin not found: %s", ""), + dbtype); + heim_release(options); + return NULL; + } + + db = _heim_alloc_object(&db_object, sizeof(*db)); + if (db == NULL) { + heim_release(options); + return NULL; + } + + db->in_transaction = 0; + db->ro_tx = 0; + db->set_keys = NULL; + db->del_keys = NULL; + db->plug = plug; + db->options = options; + + ret = plug->openf(plug->data, dbtype, dbname, options, &db->db_data, error); + if (ret) { + heim_release(db); + if (error && *error == NULL) + *error = heim_error_create(ENOENT, + N_("Heimdal DB could not be opened: %s", ""), + dbname); + return NULL; + } + + ret = db_replay_log(db, error); + if (ret) { + heim_release(db); + return NULL; + } + + if (plug->clonef == NULL) { + db->dbtype = heim_string_create(dbtype); + db->dbname = heim_string_create(dbname); + + if (!db->dbtype || ! db->dbname) { + heim_release(db); + if (error) + *error = heim_error_enomem(); + return NULL; + } + } + + return db; +} + +/** + * Clone (duplicate) an open DB handle. + * + * This is useful for multi-threaded applications. Applications must + * synchronize access to any given DB handle. + * + * Returns EBUSY if there is an open transaction for the input db. + * + * @param db Open DB handle + * @param error Output error object + * + * @return a DB handle + * + * @addtogroup heimbase + */ +heim_db_t +heim_db_clone(heim_db_t db, heim_error_t *error) +{ + heim_db_t clone; + int ret; + + if (heim_get_tid(db) != HEIM_TID_DB) + heim_abort("Expected a database"); + if (db->in_transaction) + heim_abort("DB handle is busy"); + + if (db->plug->clonef == NULL) { + return heim_db_create(heim_string_get_utf8(db->dbtype), + heim_string_get_utf8(db->dbname), + db->options, error); + } + + clone = _heim_alloc_object(&db_object, sizeof(*clone)); + if (clone == NULL) { + if (error) + *error = heim_error_enomem(); + return NULL; + } + + clone->set_keys = NULL; + clone->del_keys = NULL; + ret = db->plug->clonef(db->db_data, &clone->db_data, error); + if (ret) { + heim_release(clone); + if (error && !*error) + *error = heim_error_create(ENOENT, + N_("Could not re-open DB while cloning", "")); + return NULL; + } + db->db_data = NULL; + return clone; +} + +/** + * Open a transaction on the given db. + * + * @param db Open DB handle + * @param error Output error object + * + * @return 0 on success, system error otherwise + * + * @addtogroup heimbase + */ +int +heim_db_begin(heim_db_t db, int read_only, heim_error_t *error) +{ + int ret; + + if (heim_get_tid(db) != HEIM_TID_DB) + return EINVAL; + + if (db->in_transaction && (read_only || !db->ro_tx || (!read_only && !db->ro_tx))) + heim_abort("DB already in transaction"); + + if (db->plug->setf == NULL || db->plug->delf == NULL) + return EINVAL; + + if (db->plug->beginf) { + ret = db->plug->beginf(db->db_data, read_only, error); + } else if (!db->in_transaction) { + /* Try to emulate transactions */ + + if (db->plug->lockf == NULL) + return EINVAL; /* can't lock? -> no transactions */ + + /* Assume unlock provides sync/durability */ + ret = db->plug->lockf(db->db_data, read_only, error); + if (ret) + return ret; + + ret = db_replay_log(db, error); + if (ret) { + ret = db->plug->unlockf(db->db_data, error); + return ret; + } + + db->set_keys = heim_dict_create(11); + if (db->set_keys == NULL) + return ENOMEM; + db->del_keys = heim_dict_create(11); + if (db->del_keys == NULL) { + heim_release(db->set_keys); + db->set_keys = NULL; + return ENOMEM; + } + } else { + heim_assert(read_only == 0, "Internal error"); + ret = db->plug->lockf(db->db_data, 0, error); + if (ret) + return ret; + } + db->in_transaction = 1; + db->ro_tx = !!read_only; + return 0; +} + +/** + * Commit an open transaction on the given db. + * + * @param db Open DB handle + * @param error Output error object + * + * @return 0 on success, system error otherwise + * + * @addtogroup heimbase + */ +int +heim_db_commit(heim_db_t db, heim_error_t *error) +{ + int ret, ret2; + heim_string_t journal_fname = NULL; + + if (heim_get_tid(db) != HEIM_TID_DB) + return EINVAL; + if (!db->in_transaction) + return 0; + if (db->plug->commitf == NULL && db->plug->lockf == NULL) + return EINVAL; + + if (db->plug->commitf != NULL) { + ret = db->plug->commitf(db->db_data, error); + if (ret) + (void) db->plug->rollbackf(db->db_data, error); + + db->in_transaction = 0; + db->ro_tx = 0; + return ret; + } + + if (db->ro_tx) { + ret = 0; + goto done; + } + + if (db->options == NULL) + journal_fname = heim_dict_get_value(db->options, HSTR("journal-filename")); + + if (journal_fname != NULL) { + heim_array_t a; + heim_string_t journal_contents; + size_t len, bytes; + int save_errno; + + /* Create contents for replay log */ + ret = ENOMEM; + a = heim_array_create(); + if (a == NULL) + goto err; + ret = heim_array_append_value(a, db->set_keys); + if (ret) { + heim_release(a); + goto err; + } + ret = heim_array_append_value(a, db->del_keys); + if (ret) { + heim_release(a); + goto err; + } + journal_contents = heim_serialize(a, 0, error); + heim_release(a); + + /* Write replay log */ + if (journal_fname != NULL) { + int fd; + + ret = open_file(heim_string_get_utf8(journal_fname), 1, 0, &fd, error); + if (ret) { + heim_release(journal_contents); + goto err; + } + len = strlen(heim_string_get_utf8(journal_contents)); + bytes = write(fd, heim_string_get_utf8(journal_contents), len); + save_errno = errno; + heim_release(journal_contents); + ret = close(fd); + if (bytes != len) { + /* Truncate replay log */ + (void) open_file(heim_string_get_utf8(journal_fname), 1, 0, NULL, error); + ret = save_errno; + goto err; + } + if (ret) + goto err; + } + } + + /* Apply logged actions */ + ret = db_do_log_actions(db, error); + if (ret) + return ret; + + if (db->plug->syncf != NULL) { + /* fsync() or whatever */ + ret = db->plug->syncf(db->db_data, error); + if (ret) + return ret; + } + + /* Truncate replay log and we're done */ + if (journal_fname != NULL) { + int fd; + + ret2 = open_file(heim_string_get_utf8(journal_fname), 1, 0, &fd, error); + if (ret2 == 0) + (void) close(fd); + } + + /* + * Clean up; if we failed to remore the replay log that's OK, we'll + * handle that again in heim_db_commit() + */ +done: + heim_release(db->set_keys); + heim_release(db->del_keys); + db->set_keys = NULL; + db->del_keys = NULL; + db->in_transaction = 0; + db->ro_tx = 0; + + ret2 = db->plug->unlockf(db->db_data, error); + if (ret == 0) + ret = ret2; + + return ret; + +err: + return HEIM_ERROR(error, ret, + (ret, N_("Error while committing transaction: %s", ""), + strerror(ret))); +} + +/** + * Rollback an open transaction on the given db. + * + * @param db Open DB handle + * @param error Output error object + * + * @return 0 on success, system error otherwise + * + * @addtogroup heimbase + */ +int +heim_db_rollback(heim_db_t db, heim_error_t *error) +{ + int ret = 0; + + if (heim_get_tid(db) != HEIM_TID_DB) + return EINVAL; + if (!db->in_transaction) + return 0; + + if (db->plug->rollbackf != NULL) + ret = db->plug->rollbackf(db->db_data, error); + else if (db->plug->unlockf != NULL) + ret = db->plug->unlockf(db->db_data, error); + + heim_release(db->set_keys); + heim_release(db->del_keys); + db->set_keys = NULL; + db->del_keys = NULL; + db->in_transaction = 0; + db->ro_tx = 0; + + return ret; +} + +/** + * Get type ID of heim_db_t objects. + * + * @addtogroup heimbase + */ +heim_tid_t +heim_db_get_type_id(void) +{ + return HEIM_TID_DB; +} + +heim_data_t +_heim_db_get_value(heim_db_t db, heim_string_t table, heim_data_t key, + heim_error_t *error) +{ + heim_release(db->to_release); + db->to_release = heim_db_copy_value(db, table, key, error); + return db->to_release; +} + +/** + * Lookup a key's value in the DB. + * + * Returns 0 on success, -1 if the key does not exist in the DB, or a + * system error number on failure. + * + * @param db Open DB handle + * @param key Key + * @param error Output error object + * + * @return the value (retained), if there is one for the given key + * + * @addtogroup heimbase + */ +heim_data_t +heim_db_copy_value(heim_db_t db, heim_string_t table, heim_data_t key, + heim_error_t *error) +{ + heim_object_t v; + heim_data_t result; + + if (heim_get_tid(db) != HEIM_TID_DB) + return NULL; + + if (error != NULL) + *error = NULL; + + if (table == NULL) + table = HSTR(""); + + if (db->in_transaction) { + heim_string_t key64; + + key64 = to_base64(key, error); + if (key64 == NULL) { + if (error) + *error = heim_error_enomem(); + return NULL; + } + + v = heim_path_copy(db->set_keys, error, table, key64, NULL); + if (v != NULL) { + heim_release(key64); + return v; + } + v = heim_path_copy(db->del_keys, error, table, key64, NULL); /* can't be NULL */ + heim_release(key64); + if (v != NULL) + return NULL; + } + + result = db->plug->copyf(db->db_data, table, key, error); + + return result; +} + +/** + * Set a key's value in the DB. + * + * @param db Open DB handle + * @param key Key + * @param value Value (if NULL the key will be deleted, but empty is OK) + * @param error Output error object + * + * @return 0 on success, system error otherwise + * + * @addtogroup heimbase + */ +int +heim_db_set_value(heim_db_t db, heim_string_t table, + heim_data_t key, heim_data_t value, heim_error_t *error) +{ + heim_string_t key64 = NULL; + int ret; + + if (error != NULL) + *error = NULL; + + if (table == NULL) + table = HSTR(""); + + if (value == NULL) + /* Use heim_null_t instead of NULL */ + return heim_db_delete_key(db, table, key, error); + + if (heim_get_tid(db) != HEIM_TID_DB) + return EINVAL; + + if (heim_get_tid(key) != HEIM_TID_DATA) + return HEIM_ERROR(error, EINVAL, + (EINVAL, N_("DB keys must be data", ""))); + + if (db->plug->setf == NULL) + return EBADF; + + if (!db->in_transaction) { + ret = heim_db_begin(db, 0, error); + if (ret) + goto err; + heim_assert(db->in_transaction, "Internal error"); + ret = heim_db_set_value(db, table, key, value, error); + if (ret) { + (void) heim_db_rollback(db, NULL); + return ret; + } + return heim_db_commit(db, error); + } + + /* Transaction emulation */ + heim_assert(db->set_keys != NULL, "Internal error"); + key64 = to_base64(key, error); + if (key64 == NULL) + return HEIM_ENOMEM(error); + + if (db->ro_tx) { + ret = heim_db_begin(db, 0, error); + if (ret) + goto err; + } + ret = heim_path_create(db->set_keys, 29, value, error, table, key64, NULL); + if (ret) + goto err; + heim_path_delete(db->del_keys, error, table, key64, NULL); + heim_release(key64); + + return 0; + +err: + heim_release(key64); + return HEIM_ERROR(error, ret, + (ret, N_("Could not set a dict value while while " + "setting a DB value", ""))); +} + +/** + * Delete a key and its value from the DB + * + * + * @param db Open DB handle + * @param key Key + * @param error Output error object + * + * @return 0 on success, system error otherwise + * + * @addtogroup heimbase + */ +int +heim_db_delete_key(heim_db_t db, heim_string_t table, heim_data_t key, + heim_error_t *error) +{ + heim_string_t key64 = NULL; + int ret; + + if (error != NULL) + *error = NULL; + + if (table == NULL) + table = HSTR(""); + + if (heim_get_tid(db) != HEIM_TID_DB) + return EINVAL; + + if (db->plug->delf == NULL) + return EBADF; + + if (!db->in_transaction) { + ret = heim_db_begin(db, 0, error); + if (ret) + goto err; + heim_assert(db->in_transaction, "Internal error"); + ret = heim_db_delete_key(db, table, key, error); + if (ret) { + (void) heim_db_rollback(db, NULL); + return ret; + } + return heim_db_commit(db, error); + } + + /* Transaction emulation */ + heim_assert(db->set_keys != NULL, "Internal error"); + key64 = to_base64(key, error); + if (key64 == NULL) + return HEIM_ENOMEM(error); + if (db->ro_tx) { + ret = heim_db_begin(db, 0, error); + if (ret) + goto err; + } + ret = heim_path_create(db->del_keys, 29, heim_number_create(1), error, table, key64, NULL); + if (ret) + goto err; + heim_path_delete(db->set_keys, error, table, key64, NULL); + heim_release(key64); + + return 0; + +err: + heim_release(key64); + return HEIM_ERROR(error, ret, + (ret, N_("Could not set a dict value while while " + "deleting a DB value", ""))); +} + +/** + * Iterate a callback function over keys and values from a DB. + * + * @param db Open DB handle + * @param iter_data Callback function's private data + * @param iter_f Callback function, called once per-key/value pair + * @param error Output error object + * + * @addtogroup heimbase + */ +void +heim_db_iterate_f(heim_db_t db, heim_string_t table, void *iter_data, + heim_db_iterator_f_t iter_f, heim_error_t *error) +{ + if (error != NULL) + *error = NULL; + + if (heim_get_tid(db) != HEIM_TID_DB) + return; + + if (!db->in_transaction) + db->plug->iterf(db->db_data, table, iter_data, iter_f, error); +} + +static void +db_replay_log_table_set_keys_iter(heim_object_t key, heim_object_t value, + void *arg) +{ + heim_db_t db = arg; + heim_data_t k, v; + + if (db->ret) + return; + + k = from_base64((heim_string_t)key, &db->error); + if (k == NULL) { + db->ret = ENOMEM; + return; + } + v = (heim_data_t)value; + + db->ret = db->plug->setf(db->db_data, db->current_table, k, v, &db->error); + heim_release(k); +} + +static void +db_replay_log_table_del_keys_iter(heim_object_t key, heim_object_t value, + void *arg) +{ + heim_db_t db = arg; + heim_data_t k; + + if (db->ret) { + db->ret = ENOMEM; + return; + } + + k = from_base64((heim_string_t)key, &db->error); + if (k == NULL) + return; + + k = (heim_data_t)key; + + db->ret = db->plug->delf(db->db_data, db->current_table, k, &db->error); + heim_release(k); +} + +static void +db_replay_log_set_keys_iter(heim_object_t table, heim_object_t table_dict, + void *arg) +{ + heim_db_t db = arg; + + if (db->ret) + return; + + db->current_table = table; + heim_dict_iterate_f(table_dict, db, db_replay_log_table_set_keys_iter); +} + +static void +db_replay_log_del_keys_iter(heim_object_t table, heim_object_t table_dict, + void *arg) +{ + heim_db_t db = arg; + + if (db->ret) + return; + + db->current_table = table; + heim_dict_iterate_f(table_dict, db, db_replay_log_table_del_keys_iter); +} + +static int +db_do_log_actions(heim_db_t db, heim_error_t *error) +{ + int ret; + + if (error) + *error = NULL; + + db->ret = 0; + db->error = NULL; + if (db->set_keys != NULL) + heim_dict_iterate_f(db->set_keys, db, db_replay_log_set_keys_iter); + if (db->del_keys != NULL) + heim_dict_iterate_f(db->del_keys, db, db_replay_log_del_keys_iter); + + ret = db->ret; + db->ret = 0; + if (error && db->error) { + *error = db->error; + db->error = NULL; + } else { + heim_release(db->error); + db->error = NULL; + } + return ret; +} + +static int +db_replay_log(heim_db_t db, heim_error_t *error) +{ + int ret; + heim_string_t journal_fname = NULL; + heim_object_t journal; + size_t len; + + heim_assert(!db->in_transaction, "DB transaction not open"); + heim_assert(db->set_keys == NULL && db->set_keys == NULL, "DB transaction not open"); + + if (error) + *error = NULL; + + if (db->options == NULL) + return 0; + + journal_fname = heim_dict_get_value(db->options, HSTR("journal-filename")); + if (journal_fname == NULL) + return 0; + + ret = read_json(heim_string_get_utf8(journal_fname), &journal, error); + if (ret == ENOENT) + return 0; + if (ret == 0 && journal == NULL) + return 0; + if (ret != 0) + return ret; + + if (heim_get_tid(journal) != HEIM_TID_ARRAY) + return HEIM_ERROR(error, EINVAL, + (ret, N_("Invalid journal contents; delete journal", + ""))); + + len = heim_array_get_length(journal); + + if (len > 0) + db->set_keys = heim_array_get_value(journal, 0); + if (len > 1) + db->del_keys = heim_array_get_value(journal, 1); + ret = db_do_log_actions(db, error); + if (ret) + return ret; + + /* Truncate replay log and we're done */ + ret = open_file(heim_string_get_utf8(journal_fname), 1, 0, NULL, error); + if (ret) + return ret; + heim_release(db->set_keys); + heim_release(db->del_keys); + db->set_keys = NULL; + db->del_keys = NULL; + + return 0; +} + +static +heim_string_t to_base64(heim_data_t data, heim_error_t *error) +{ + char *b64 = NULL; + heim_string_t s = NULL; + const heim_octet_string *d; + int ret; + + d = heim_data_get_data(data); + ret = base64_encode(d->data, d->length, &b64); + if (ret < 0 || b64 == NULL) + goto enomem; + s = heim_string_ref_create(b64, free); + if (s == NULL) + goto enomem; + return s; + +enomem: + free(b64); + if (error) + *error = heim_error_enomem(); + return NULL; +} + +static +heim_data_t from_base64(heim_string_t s, heim_error_t *error) +{ + void *buf; + size_t len; + heim_data_t d; + + buf = malloc(strlen(heim_string_get_utf8(s))); + if (buf == NULL) + goto enomem; + + len = base64_decode(heim_string_get_utf8(s), buf); + d = heim_data_ref_create(buf, len, free); + if (d == NULL) + goto enomem; + return d; + +enomem: + free(buf); + if (error) + *error = heim_error_enomem(); + return NULL; +} + + +static int +open_file(const char *dbname, int for_write, int excl, int *fd_out, heim_error_t *error) +{ +#ifdef WIN32 + HANDLE hFile; + int ret = 0; + + if (fd_out) + *fd_out = -1; + + if (for_write) + hFile = CreateFile(dbname, GENERIC_WRITE | GENERIC_READ, 0, + NULL, /* we'll close as soon as we read */ + CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL); + else + hFile = CreateFile(dbname, GENERIC_READ, FILE_SHARE_READ, + NULL, /* we'll close as soon as we read */ + OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL); + if (hFile == INVALID_HANDLE_VALUE) { + ret = GetLastError(); + _set_errno(ret); /* CreateFile() does not set errno */ + goto err; + } + if (fd_out == NULL) { + (void) CloseHandle(hFile); + return 0; + } + + *fd_out = _open_osfhandle((intptr_t) hFile, 0); + if (*fd_out < 0) { + ret = errno; + (void) CloseHandle(hFile); + goto err; + } + + /* No need to lock given share deny mode */ + return 0; + +err: + if (error != NULL) { + char *s = NULL; + FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_ALLOCATE_BUFFER, + 0, ret, 0, (LPTSTR) &s, 0, NULL); + *error = heim_error_create(ret, N_("Could not open JSON file %s: %s", ""), + dbname, s ? s : ""); + LocalFree(s); + } + return ret; +#else + int ret = 0; + int fd; + + if (fd_out) + *fd_out = -1; + + if (for_write && excl) + fd = open(dbname, O_CREAT | O_EXCL | O_WRONLY, 0600); + else if (for_write) + fd = open(dbname, O_CREAT | O_TRUNC | O_WRONLY, 0600); + else + fd = open(dbname, O_RDONLY); + if (fd < 0) { + if (error != NULL) + *error = heim_error_create(ret, N_("Could not open JSON file %s: %s", ""), + dbname, strerror(errno)); + return errno; + } + + if (fd_out == NULL) { + (void) close(fd); + return 0; + } + + ret = flock(fd, for_write ? LOCK_EX : LOCK_SH); + if (ret == -1) { + /* Note that we if O_EXCL we're leaving the [lock] file around */ + (void) close(fd); + return HEIM_ERROR(error, errno, + (errno, N_("Could not lock JSON file %s: %s", ""), + dbname, strerror(errno))); + } + + *fd_out = fd; + + return 0; +#endif +} + +static int +read_json(const char *dbname, heim_object_t *out, heim_error_t *error) +{ + struct stat st; + char *str = NULL; + int ret; + int fd = -1; + ssize_t bytes; + + *out = NULL; + ret = open_file(dbname, 0, 0, &fd, error); + if (ret) + return ret; + + ret = fstat(fd, &st); + if (ret == -1) { + (void) close(fd); + return HEIM_ERROR(error, errno, + (ret, N_("Could not stat JSON DB %s: %s", ""), + dbname, strerror(errno))); + } + + if (st.st_size == 0) { + (void) close(fd); + return 0; + } + + str = malloc(st.st_size + 1); + if (str == NULL) { + (void) close(fd); + return HEIM_ENOMEM(error); + } + + bytes = read(fd, str, st.st_size); + (void) close(fd); + if (bytes != st.st_size) { + free(str); + if (bytes >= 0) + errno = EINVAL; /* ?? */ + return HEIM_ERROR(error, errno, + (ret, N_("Could not read JSON DB %s: %s", ""), + dbname, strerror(errno))); + } + str[st.st_size] = '\0'; + *out = heim_json_create(str, 10, 0, error); + free(str); + if (*out == NULL) + return (error && *error) ? heim_error_get_code(*error) : EINVAL; + return 0; +} + +typedef struct json_db { + heim_dict_t dict; + heim_string_t dbname; + heim_string_t bkpname; + int fd; + time_t last_read_time; + unsigned int read_only:1; + unsigned int locked:1; + unsigned int locked_needs_unlink:1; +} *json_db_t; + +static int +json_db_open(void *plug, const char *dbtype, const char *dbname, + heim_dict_t options, void **db, heim_error_t *error) +{ + json_db_t jsondb; + heim_dict_t contents = NULL; + heim_string_t dbname_s = NULL; + heim_string_t bkpname_s = NULL; + + if (error) + *error = NULL; + if (dbtype && *dbtype && strcmp(dbtype, "json")) + return HEIM_ERROR(error, EINVAL, (EINVAL, N_("Wrong DB type", ""))); + if (dbname && *dbname && strcmp(dbname, "MEMORY") != 0) { + char *ext = strrchr(dbname, '.'); + char *bkpname; + size_t len; + int ret; + + if (ext == NULL || strcmp(ext, ".json") != 0) + return HEIM_ERROR(error, EINVAL, + (EINVAL, N_("JSON DB files must end in .json", + ""))); + + if (options) { + heim_object_t vc, ve, vt; + + vc = heim_dict_get_value(options, HSTR("create")); + ve = heim_dict_get_value(options, HSTR("exclusive")); + vt = heim_dict_get_value(options, HSTR("truncate")); + if (vc && vt) { + ret = open_file(dbname, 1, ve ? 1 : 0, NULL, error); + if (ret) + return ret; + } else if (vc || ve || vt) { + return HEIM_ERROR(error, EINVAL, + (EINVAL, N_("Invalid JSON DB open options", + ""))); + } + /* + * We don't want cloned handles to truncate the DB, eh? + * + * We should really just create a copy of the options dict + * rather than modify the caller's! But for that it'd be + * nicer to have copy utilities in heimbase, something like + * this: + * + * heim_object_t heim_copy(heim_object_t src, int depth, + * heim_error_t *error); + * + * so that options = heim_copy(options, 1); means copy the + * dict but nothing else (whereas depth == 0 would mean + * heim_retain(), and depth > 1 would be copy that many + * levels). + */ + heim_dict_delete_key(options, HSTR("create")); + heim_dict_delete_key(options, HSTR("exclusive")); + heim_dict_delete_key(options, HSTR("truncate")); + } + dbname_s = heim_string_create(dbname); + if (dbname_s == NULL) + return HEIM_ENOMEM(error); + + len = snprintf(NULL, 0, "%s~", dbname); + bkpname = malloc(len + 2); + if (bkpname == NULL) { + heim_release(dbname_s); + return HEIM_ENOMEM(error); + } + (void) snprintf(bkpname, len + 1, "%s~", dbname); + bkpname_s = heim_string_create(bkpname); + free(bkpname); + if (bkpname_s == NULL) { + heim_release(dbname_s); + return HEIM_ENOMEM(error); + } + + ret = read_json(dbname, (heim_object_t *)&contents, error); + if (ret) + return ret; + + if (contents != NULL && heim_get_tid(contents) != HEIM_TID_DICT) + return HEIM_ERROR(error, EINVAL, + (EINVAL, N_("JSON DB contents not valid JSON", + ""))); + } + + jsondb = heim_alloc(sizeof (*jsondb), "json_db", NULL); + if (jsondb == NULL) { + heim_release(contents); + heim_release(dbname_s); + return ENOMEM; + } + + jsondb->last_read_time = time(NULL); + jsondb->fd = -1; + jsondb->dbname = dbname_s; + jsondb->bkpname = bkpname_s; + jsondb->read_only = 0; + + if (contents != NULL) + jsondb->dict = contents; + else { + jsondb->dict = heim_dict_create(29); + if (jsondb->dict == NULL) { + heim_release(jsondb); + return ENOMEM; + } + } + + *db = jsondb; + return 0; +} + +static int +json_db_close(void *db, heim_error_t *error) +{ + json_db_t jsondb = db; + + if (error) + *error = NULL; + if (jsondb->fd > -1) + (void) close(jsondb->fd); + jsondb->fd = -1; + heim_release(jsondb->dbname); + heim_release(jsondb->bkpname); + heim_release(jsondb->dict); + heim_release(jsondb); + return 0; +} + +static int +json_db_lock(void *db, int read_only, heim_error_t *error) +{ + json_db_t jsondb = db; + int ret; + + heim_assert(jsondb->fd == -1 || (jsondb->read_only && !read_only), + "DB locks are not recursive"); + + jsondb->read_only = read_only ? 1 : 0; + if (jsondb->fd > -1) + return 0; + + ret = open_file(heim_string_get_utf8(jsondb->bkpname), 1, 1, &jsondb->fd, error); + if (ret == 0) { + jsondb->locked_needs_unlink = 1; + jsondb->locked = 1; + } + return ret; +} + +static int +json_db_unlock(void *db, heim_error_t *error) +{ + json_db_t jsondb = db; + int ret = 0; + + heim_assert(jsondb->locked, "DB not locked when unlock attempted"); + if (jsondb->fd > -1) + ret = close(jsondb->fd); + jsondb->fd = -1; + jsondb->read_only = 0; + jsondb->locked = 0; + if (jsondb->locked_needs_unlink) + unlink(heim_string_get_utf8(jsondb->bkpname)); + jsondb->locked_needs_unlink = 0; + return ret; +} + +static int +json_db_sync(void *db, heim_error_t *error) +{ + json_db_t jsondb = db; + size_t len, bytes; + heim_error_t e; + heim_string_t json; + const char *json_text = NULL; + int ret = 0; + int fd = -1; +#ifdef WIN32 + int tries = 3; +#endif + + heim_assert(jsondb->fd > -1, "DB not locked when sync attempted"); + + json = heim_serialize(jsondb->dict, 0, &e); + if (json == NULL) { + if (error) + *error = e; + else + heim_release(e); + return heim_error_get_code(e); + } + + json_text = heim_string_get_utf8(json); + len = strlen(json_text); + errno = 0; + +#ifdef WIN32 + while (tries--) { + ret = open_file(heim_string_get_utf8(jsondb->dbname), 1, 0, &fd, error); + if (ret == 0) + break; + sleep(1); + } + if (ret) { + heim_release(json); + return ret; + } +#else + fd = jsondb->fd; +#endif /* WIN32 */ + + bytes = write(fd, json_text, len); + heim_release(json); + if (bytes != len) + return errno ? errno : EIO; + ret = fsync(fd); + if (ret) + return ret; + +#ifdef WIN32 + ret = close(fd); + if (ret) + return GetLastError(); +#else + ret = rename(heim_string_get_utf8(jsondb->bkpname), heim_string_get_utf8(jsondb->dbname)); + if (ret == 0) { + jsondb->locked_needs_unlink = 0; + return 0; + } +#endif /* WIN32 */ + + return errno; +} + +static heim_data_t +json_db_copy_value(void *db, heim_string_t table, heim_data_t key, + heim_error_t *error) +{ + json_db_t jsondb = db; + heim_string_t key_string; + const heim_octet_string *key_data = heim_data_get_data(key); + struct stat st; + heim_data_t ret; + + if (error) + *error = NULL; + + if (strnlen(key_data->data, key_data->length) != key_data->length) { + HEIM_ERROR(error, EINVAL, + (EINVAL, N_("JSON DB requires keys that are actually " + "strings", ""))); + return NULL; + } + + if (stat(heim_string_get_utf8(jsondb->dbname), &st) == -1) { + HEIM_ERROR(error, errno, + (errno, N_("Could not stat JSON DB file", ""))); + return NULL; + } + + if (st.st_mtime > jsondb->last_read_time || + st.st_ctime > jsondb->last_read_time) { + heim_dict_t contents = NULL; + int ret; + + /* Ignore file is gone (ENOENT) */ + ret = read_json(heim_string_get_utf8(jsondb->dbname), + (heim_object_t *)&contents, error); + if (ret) + return NULL; + if (contents == NULL) + contents = heim_dict_create(29); + heim_release(jsondb->dict); + jsondb->dict = contents; + jsondb->last_read_time = time(NULL); + } + + key_string = heim_string_create_with_bytes(key_data->data, + key_data->length); + if (key_string == NULL) { + (void) HEIM_ENOMEM(error); + return NULL; + } + + ret = heim_path_copy(jsondb->dict, error, table, key_string, NULL); + heim_release(key_string); + return ret; +} + +static int +json_db_set_value(void *db, heim_string_t table, + heim_data_t key, heim_data_t value, heim_error_t *error) +{ + json_db_t jsondb = db; + heim_string_t key_string; + const heim_octet_string *key_data = heim_data_get_data(key); + int ret; + + if (error) + *error = NULL; + + if (strnlen(key_data->data, key_data->length) != key_data->length) + return HEIM_ERROR(error, EINVAL, + (EINVAL, + N_("JSON DB requires keys that are actually strings", + ""))); + + key_string = heim_string_create_with_bytes(key_data->data, + key_data->length); + if (key_string == NULL) + return HEIM_ENOMEM(error); + + if (table == NULL) + table = HSTR(""); + + ret = heim_path_create(jsondb->dict, 29, value, error, table, key_string, NULL); + heim_release(key_string); + return ret; +} + +static int +json_db_del_key(void *db, heim_string_t table, heim_data_t key, + heim_error_t *error) +{ + json_db_t jsondb = db; + heim_string_t key_string; + const heim_octet_string *key_data = heim_data_get_data(key); + + if (error) + *error = NULL; + + if (strnlen(key_data->data, key_data->length) != key_data->length) + return HEIM_ERROR(error, EINVAL, + (EINVAL, + N_("JSON DB requires keys that are actually strings", + ""))); + + key_string = heim_string_create_with_bytes(key_data->data, + key_data->length); + if (key_string == NULL) + return HEIM_ENOMEM(error); + + if (table == NULL) + table = HSTR(""); + + heim_path_delete(jsondb->dict, error, table, key_string, NULL); + heim_release(key_string); + return 0; +} + +struct json_db_iter_ctx { + heim_db_iterator_f_t iter_f; + void *iter_ctx; +}; + +static void json_db_iter_f(heim_object_t key, heim_object_t value, void *arg) +{ + struct json_db_iter_ctx *ctx = arg; + const char *key_string; + heim_data_t key_data; + + key_string = heim_string_get_utf8((heim_string_t)key); + key_data = heim_data_ref_create(key_string, strlen(key_string), NULL); + ctx->iter_f(key_data, (heim_object_t)value, ctx->iter_ctx); + heim_release(key_data); +} + +static void +json_db_iter(void *db, heim_string_t table, void *iter_data, + heim_db_iterator_f_t iter_f, heim_error_t *error) +{ + json_db_t jsondb = db; + struct json_db_iter_ctx ctx; + heim_dict_t table_dict; + + if (error) + *error = NULL; + + if (table == NULL) + table = HSTR(""); + + table_dict = heim_dict_get_value(jsondb->dict, table); + if (table_dict == NULL) + return; + + ctx.iter_ctx = iter_data; + ctx.iter_f = iter_f; + + heim_dict_iterate_f(table_dict, &ctx, json_db_iter_f); +} + +static struct heim_db_type json_dbt = { + 1, json_db_open, NULL, json_db_close, + json_db_lock, json_db_unlock, json_db_sync, + NULL, NULL, NULL, + json_db_copy_value, json_db_set_value, + json_db_del_key, json_db_iter +}; + diff --git a/base/dict.c b/base/dict.c index 37f4b95bd..3893dfb9d 100644 --- a/base/dict.c +++ b/base/dict.c @@ -164,11 +164,31 @@ _search(heim_dict_t dict, heim_object_t ptr) * @value dict the dict to search in * @value key the key to search for * - * @return a retained copy of the value for key or NULL if not found + * @return a not-retained copy of the value for key or NULL if not found */ heim_object_t heim_dict_get_value(heim_dict_t dict, heim_object_t key) +{ + struct hashentry *p; + p = _search(dict, key); + if (p == NULL) + return NULL; + + return p->value; +} + +/** + * Search for element in hash table + * + * @value dict the dict to search in + * @value key the key to search for + * + * @return a retained copy of the value for key or NULL if not found + */ + +heim_object_t +heim_dict_copy_value(heim_dict_t dict, heim_object_t key) { struct hashentry *p; p = _search(dict, key); diff --git a/base/error.c b/base/error.c index 91c660be8..1b2413d68 100644 --- a/base/error.c +++ b/base/error.c @@ -75,6 +75,13 @@ struct heim_type_data _heim_error_object = { error_hash }; +heim_error_t +heim_error_enomem(void) +{ + /* This is an immediate object; see heim_number_create() */ + return (heim_error_t)heim_number_create(ENOMEM); +} + heim_error_t heim_error_create(int error_code, const char *fmt, ...) { @@ -94,14 +101,17 @@ heim_error_createv(int error_code, const char *fmt, va_list ap) heim_error_t e; char *str; int len; + int save_errno = errno; str = malloc(1024); + errno = save_errno; if (str == NULL) - return NULL; + return heim_error_enomem(); len = vsnprintf(str, 1024, fmt, ap); + errno = save_errno; if (len < 0) { free(str); - return NULL; + return NULL; /* XXX We should have a special heim_error_t for this */ } e = _heim_alloc_object(&_heim_error_object, sizeof(struct heim_error)); @@ -111,12 +121,18 @@ heim_error_createv(int error_code, const char *fmt, va_list ap) } free(str); + errno = save_errno; return e; } heim_string_t heim_error_copy_string(heim_error_t error) { + if (heim_get_tid(error) != HEIM_TID_ERROR) { + if (heim_get_tid(error) == heim_number_get_type_id()) + return __heim_string_constant(strerror(heim_number_get_int((heim_number_t)error))); + heim_abort("invalid heim_error_t"); + } /* XXX concat all strings */ return heim_retain(error->msg); } @@ -124,12 +140,22 @@ heim_error_copy_string(heim_error_t error) int heim_error_get_code(heim_error_t error) { + if (heim_get_tid(error) != HEIM_TID_ERROR) { + if (heim_get_tid(error) == heim_number_get_type_id()) + return heim_number_get_int((heim_number_t)error); + heim_abort("invalid heim_error_t"); + } return error->error_code; } heim_error_t heim_error_append(heim_error_t top, heim_error_t append) { + if (heim_get_tid(top) != HEIM_TID_ERROR) { + if (heim_get_tid(top) == heim_number_get_type_id()) + return top; + heim_abort("invalid heim_error_t"); + } if (top->next) heim_release(top->next); top->next = heim_retain(append); diff --git a/base/heimbase.c b/base/heimbase.c index 1c02e8b56..1ae2b551c 100644 --- a/base/heimbase.c +++ b/base/heimbase.c @@ -76,7 +76,7 @@ struct heim_auto_release { /** - * Retain object + * Retain object (i.e., take a reference) * * @param object to be released, NULL is ok * @@ -100,7 +100,7 @@ heim_retain(void *ptr) } /** - * Release object, free is reference count reaches zero + * Release object, free if reference count reaches zero * * @param object to be released */ @@ -257,6 +257,18 @@ struct heim_type_data memory_object = { NULL }; +/** + * Allocate memory for an object of anonymous type + * + * @param size size of object to be allocated + * @param name name of ad-hoc type + * @param dealloc destructor function + * + * Objects allocated with this interface do not serialize. + * + * @return allocated object + */ + void * heim_alloc(size_t size, const char *name, heim_type_dealloc dealloc) { @@ -310,6 +322,18 @@ _heim_alloc_object(heim_type_t type, size_t size) return BASE2PTR(p); } +void * +_heim_get_isaextra(heim_object_t ptr, size_t idx) +{ + struct heim_base *p = (struct heim_base *)PTR2BASE(ptr); + + heim_assert(ptr != NULL, "internal error"); + if (p->isa == &memory_object) + return NULL; + heim_assert(idx < 3, "invalid private heim_base extra data index"); + return &p->isaextra[idx]; +} + heim_tid_t _heim_type_get_tid(heim_type_t type) { @@ -489,7 +513,11 @@ static struct heim_type_data _heim_autorel_object = { }; /** + * Create thread-specific object auto-release pool * + * Objects placed on the per-thread auto-release pool (with + * heim_auto_release()) can be released in one fell swoop by calling + * heim_auto_release_drain(). */ heim_auto_release_t @@ -515,7 +543,9 @@ heim_auto_release_create(void) } /** - * Mark the current object as a + * Place the current object on the thread's auto-release pool + * + * @param ptr object */ void @@ -546,7 +576,7 @@ heim_auto_release(heim_object_t ptr) } /** - * + * Release all objects on the given auto-release pool */ void @@ -565,3 +595,369 @@ heim_auto_release_drain(heim_auto_release_t autorel) } HEIMDAL_MUTEX_unlock(&autorel->pool_mutex); } + +/* + * Helper for heim_path_vget() and heim_path_delete(). On success + * outputs the node named by the path and the parent node and key + * (useful for heim_path_delete()). + */ + +static heim_object_t +heim_path_vget2(heim_object_t ptr, heim_object_t *parent, heim_object_t *key, + heim_error_t *error, va_list ap) +{ + heim_object_t path_element; + heim_object_t node, next_node; + heim_tid_t node_type; + + *parent = NULL; + *key = NULL; + if (ptr == NULL) + return NULL; + + for (node = ptr; node != NULL; ) { + path_element = va_arg(ap, heim_object_t); + if (path_element == NULL) { + *parent = node; + *key = path_element; + return node; + } + + node_type = heim_get_tid(node); + switch (node_type) { + case HEIM_TID_ARRAY: + case HEIM_TID_DICT: + case HEIM_TID_DB: + break; + default: + if (node == ptr) + heim_abort("heim_path_get() only operates on container types"); + return NULL; + } + + if (node_type == HEIM_TID_DICT) { + next_node = heim_dict_get_value(node, path_element); + } else if (node_type == HEIM_TID_DB) { + next_node = _heim_db_get_value(node, NULL, path_element, NULL); + } else if (node_type == HEIM_TID_ARRAY) { + int idx = -1; + + if (heim_get_tid(path_element) == HEIM_TID_NUMBER) + idx = heim_number_get_int(path_element); + if (idx < 0) { + if (error) + *error = heim_error_create(EINVAL, + "heim_path_get() path elements " + "for array nodes must be " + "numeric and positive"); + return NULL; + } + next_node = heim_array_get_value(node, idx); + } else { + if (error) + *error = heim_error_create(EINVAL, + "heim_path_get() node in path " + "not a container type"); + return NULL; + } + node = next_node; + } + return NULL; +} + +/** + * Get a node in a heim_object tree by path + * + * @param ptr tree + * @param error error (output) + * @param ap NULL-terminated va_list of heim_object_ts that form a path + * + * @return object (not retained) if found + * + * @addtogroup heimbase + */ + +heim_object_t +heim_path_vget(heim_object_t ptr, heim_error_t *error, va_list ap) +{ + heim_object_t p, k; + + return heim_path_vget2(ptr, &p, &k, error, ap); +} + +/** + * Get a node in a tree by path, with retained reference + * + * @param ptr tree + * @param error error (output) + * @param ap NULL-terminated va_list of heim_object_ts that form a path + * + * @return retained object if found + * + * @addtogroup heimbase + */ + +heim_object_t +heim_path_vcopy(heim_object_t ptr, heim_error_t *error, va_list ap) +{ + heim_object_t p, k; + + return heim_retain(heim_path_vget2(ptr, &p, &k, error, ap)); +} + +/** + * Get a node in a tree by path + * + * @param ptr tree + * @param error error (output) + * @param ... NULL-terminated va_list of heim_object_ts that form a path + * + * @return object (not retained) if found + * + * @addtogroup heimbase + */ + +heim_object_t +heim_path_get(heim_object_t ptr, heim_error_t *error, ...) +{ + heim_object_t o; + heim_object_t p, k; + va_list ap; + + if (ptr == NULL) + return NULL; + + va_start(ap, error); + o = heim_path_vget2(ptr, &p, &k, error, ap); + va_end(ap); + return o; +} + +/** + * Get a node in a tree by path, with retained reference + * + * @param ptr tree + * @param error error (output) + * @param ... NULL-terminated va_list of heim_object_ts that form a path + * + * @return retained object if found + * + * @addtogroup heimbase + */ + +heim_object_t +heim_path_copy(heim_object_t ptr, heim_error_t *error, ...) +{ + heim_object_t o; + heim_object_t p, k; + va_list ap; + + if (ptr == NULL) + return NULL; + + va_start(ap, error); + o = heim_retain(heim_path_vget2(ptr, &p, &k, error, ap)); + va_end(ap); + return o; +} + +/** + * Create a path in a heim_object_t tree + * + * @param ptr the tree + * @param size the size of the heim_dict_t nodes to be created + * @param leaf leaf node to be added, if any + * @param error error (output) + * @param ap NULL-terminated of path component objects + * + * Create a path of heim_dict_t interior nodes in a given heim_object_t + * tree, as necessary, and set/replace a leaf, if given (if leaf is NULL + * then the leaf is not deleted). + * + * @return 0 on success, else a system error + * + * @addtogroup heimbase + */ + +int +heim_path_vcreate(heim_object_t ptr, size_t size, heim_object_t leaf, + heim_error_t *error, va_list ap) +{ + heim_object_t path_element = va_arg(ap, heim_object_t); + heim_object_t next_path_element = NULL; + heim_object_t node = ptr; + heim_object_t next_node = NULL; + heim_tid_t node_type; + int ret; + + if (ptr == NULL) + heim_abort("heim_path_vcreate() does not create root nodes"); + + while (path_element != NULL) { + next_path_element = va_arg(ap, heim_object_t); + node_type = heim_get_tid(node); + + if (node_type == HEIM_TID_DICT) { + next_node = heim_dict_get_value(node, path_element); + } else if (node_type == HEIM_TID_ARRAY) { + int idx = -1; + + if (heim_get_tid(path_element) == HEIM_TID_NUMBER) + idx = heim_number_get_int(path_element); + if (idx < 0) { + if (error) + *error = heim_error_create(EINVAL, + "heim_path() path elements for " + "array nodes must be numeric " + "and positive"); + return EINVAL; + } + if (idx < heim_array_get_length(node)) + next_node = heim_array_get_value(node, idx); + else + next_node = NULL; + } else if (node_type == HEIM_TID_DB && next_path_element != NULL) { + if (error) + *error = heim_error_create(EINVAL, "Interior node is a DB"); + return EINVAL; + } + + if (next_path_element == NULL) + break; + + /* Create missing interior node */ + if (next_node == NULL) { + next_node = heim_dict_create(size); /* no arrays or DBs, just dicts */ + if (next_node == NULL) { + ret = ENOMEM; + goto err; + } + + if (node_type == HEIM_TID_DICT) { + ret = heim_dict_set_value(node, path_element, next_node); + } else if (node_type == HEIM_TID_ARRAY && + heim_number_get_int(path_element) <= heim_array_get_length(node)) { + ret = heim_array_insert_value(node, + heim_number_get_int(path_element), + next_node); + } else { + ret = EINVAL; + if (error) + *error = heim_error_create(ret, "Node in path not a " + "container"); + goto err; + } + heim_release(next_node); + if (ret) + goto err; + } + + path_element = next_path_element; + node = next_node; + next_node = NULL; + } + + if (path_element == NULL) + goto err; + + /* Add the leaf */ + if (leaf != NULL) { + if (node_type == HEIM_TID_DICT) + ret = heim_dict_set_value(node, path_element, leaf); + else + ret = heim_array_insert_value(node, + heim_number_get_int(path_element), + leaf); + } + return 0; + +err: + if (error && !*error) { + if (ret == ENOMEM) + *error = heim_error_enomem(); + else + *error = heim_error_create(ret, "Could not set " + "dict value"); + } + return ret; +} + +/** + * Create a path in a heim_object_t tree + * + * @param ptr the tree + * @param size the size of the heim_dict_t nodes to be created + * @param leaf leaf node to be added, if any + * @param error error (output) + * @param ... NULL-terminated list of path component objects + * + * Create a path of heim_dict_t interior nodes in a given heim_object_t + * tree, as necessary, and set/replace a leaf, if given (if leaf is NULL + * then the leaf is not deleted). + * + * @return 0 on success, else a system error + * + * @addtogroup heimbase + */ + +int +heim_path_create(heim_object_t ptr, size_t size, heim_object_t leaf, + heim_error_t *error, ...) +{ + va_list ap; + int ret; + + va_start(ap, error); + ret = heim_path_vcreate(ptr, size, leaf, error, ap); + va_end(ap); + return ret; +} + +/** + * Delete leaf node named by a path in a heim_object_t tree + * + * @param ptr the tree + * @param error error (output) + * @param ap NULL-terminated list of path component objects + * + * @addtogroup heimbase + */ + +void +heim_path_vdelete(heim_object_t ptr, heim_error_t *error, va_list ap) +{ + heim_object_t parent, key, child; + + child = heim_path_vget2(ptr, &parent, &key, error, ap); + if (child != NULL) { + if (heim_get_tid(parent) == HEIM_TID_DICT) + heim_dict_delete_key(parent, key); + else if (heim_get_tid(parent) == HEIM_TID_DB) + heim_db_delete_key(parent, NULL, key, error); + else if (heim_get_tid(parent) == HEIM_TID_ARRAY) + heim_array_delete_value(parent, heim_number_get_int(key)); + heim_release(child); + } +} + +/** + * Delete leaf node named by a path in a heim_object_t tree + * + * @param ptr the tree + * @param error error (output) + * @param ap NULL-terminated list of path component objects + * + * @addtogroup heimbase + */ + +void +heim_path_delete(heim_object_t ptr, heim_error_t *error, ...) +{ + va_list ap; + + va_start(ap, error); + heim_path_vdelete(ptr, error, ap); + va_end(ap); + return; +} + diff --git a/base/heimbase.h b/base/heimbase.h index da35bb93e..c5172eced 100644 --- a/base/heimbase.h +++ b/base/heimbase.h @@ -124,13 +124,19 @@ heim_tid_t heim_array_get_type_id(void); typedef void (*heim_array_iterator_f_t)(heim_object_t, void *); int heim_array_append_value(heim_array_t, heim_object_t); +int heim_array_insert_value(heim_array_t, size_t idx, heim_object_t); void heim_array_iterate_f(heim_array_t, void *, heim_array_iterator_f_t); +void heim_array_iterate_reverse_f(heim_array_t, void *, heim_array_iterator_f_t); #ifdef __BLOCKS__ void heim_array_iterate(heim_array_t, void (^)(heim_object_t)); +void heim_array_iterate_reverse(heim_array_t, void (^)(heim_object_t)); #endif size_t heim_array_get_length(heim_array_t); heim_object_t heim_array_get_value(heim_array_t, size_t); +heim_object_t + heim_array_copy_value(heim_array_t, size_t); +void heim_array_set_value(heim_array_t, size_t, heim_object_t); void heim_array_delete_value(heim_array_t, size_t); #ifdef __BLOCKS__ void heim_array_filter(heim_array_t, int (^)(heim_object_t)); @@ -155,6 +161,8 @@ void heim_dict_iterate(heim_dict_t, void (^)(heim_object_t, heim_object_t)); heim_object_t heim_dict_get_value(heim_dict_t, heim_object_t); +heim_object_t + heim_dict_copy_value(heim_dict_t, heim_object_t); void heim_dict_delete_key(heim_dict_t, heim_object_t); /* @@ -162,15 +170,154 @@ void heim_dict_delete_key(heim_dict_t, heim_object_t); */ typedef struct heim_string_data *heim_string_t; +typedef void (*heim_string_free_f_t)(void *); heim_string_t heim_string_create(const char *); +heim_string_t heim_string_ref_create(const char *, heim_string_free_f_t); heim_string_t heim_string_create_with_bytes(const void *, size_t); +heim_string_t heim_string_ref_create_with_bytes(const void *, size_t, + heim_string_free_f_t); heim_tid_t heim_string_get_type_id(void); const char * heim_string_get_utf8(heim_string_t); #define HSTR(_str) (__heim_string_constant("" _str "")) heim_string_t __heim_string_constant(const char *); +/* + * Errors + */ + +typedef struct heim_error * heim_error_t; +heim_error_t heim_error_enomem(void); + +heim_error_t heim_error_create(int, const char *, ...) + HEIMDAL_PRINTF_ATTRIBUTE((printf, 2, 3)); + +heim_error_t heim_error_createv(int, const char *, va_list) + HEIMDAL_PRINTF_ATTRIBUTE((printf, 2, 0)); + +heim_string_t heim_error_copy_string(heim_error_t); +int heim_error_get_code(heim_error_t); + +heim_error_t heim_error_append(heim_error_t, heim_error_t); + +/* + * Path + */ + +heim_object_t heim_path_get(heim_object_t ptr, heim_error_t *error, ...); +heim_object_t heim_path_copy(heim_object_t ptr, heim_error_t *error, ...); +heim_object_t heim_path_vget(heim_object_t ptr, heim_error_t *error, + va_list ap); +heim_object_t heim_path_vcopy(heim_object_t ptr, heim_error_t *error, + va_list ap); + +int heim_path_vcreate(heim_object_t ptr, size_t size, heim_object_t leaf, + heim_error_t *error, va_list ap); +int heim_path_create(heim_object_t ptr, size_t size, heim_object_t leaf, + heim_error_t *error, ...); + +void heim_path_vdelete(heim_object_t ptr, heim_error_t *error, va_list ap); +void heim_path_delete(heim_object_t ptr, heim_error_t *error, ...); + +/* + * Data (octet strings) + */ + +#ifndef __HEIM_OCTET_STRING__ +#define __HEIM_OCTET_STRING__ +typedef struct heim_octet_string { + size_t length; + void *data; +} heim_octet_string; +#endif + +typedef struct heim_data * heim_data_t; +typedef void (*heim_data_free_f_t)(void *); + +heim_data_t heim_data_create(const void *, size_t); +heim_data_t heim_data_ref_create(const void *, size_t, heim_data_free_f_t); +heim_tid_t heim_data_get_type_id(void); +const heim_octet_string * + heim_data_get_data(heim_data_t); +const void * heim_data_get_ptr(heim_data_t); +size_t heim_data_get_length(heim_data_t); + +/* + * DB + */ + +typedef struct heim_db_data *heim_db_t; + +typedef void (*heim_db_iterator_f_t)(heim_data_t, heim_data_t, void *); + +typedef int (*heim_db_plug_open_f_t)(void *, const char *, const char *, + heim_dict_t, void **, heim_error_t *); +typedef int (*heim_db_plug_clone_f_t)(void *, void **, heim_error_t *); +typedef int (*heim_db_plug_close_f_t)(void *, heim_error_t *); +typedef int (*heim_db_plug_lock_f_t)(void *, int, heim_error_t *); +typedef int (*heim_db_plug_unlock_f_t)(void *, heim_error_t *); +typedef int (*heim_db_plug_sync_f_t)(void *, heim_error_t *); +typedef int (*heim_db_plug_begin_f_t)(void *, int, heim_error_t *); +typedef int (*heim_db_plug_commit_f_t)(void *, heim_error_t *); +typedef int (*heim_db_plug_rollback_f_t)(void *, heim_error_t *); +typedef heim_data_t (*heim_db_plug_copy_value_f_t)(void *, heim_string_t, + heim_data_t, + heim_error_t *); +typedef int (*heim_db_plug_set_value_f_t)(void *, heim_string_t, heim_data_t, + heim_data_t, heim_error_t *); +typedef int (*heim_db_plug_del_key_f_t)(void *, heim_string_t, heim_data_t, + heim_error_t *); +typedef void (*heim_db_plug_iter_f_t)(void *, heim_string_t, void *, + heim_db_iterator_f_t, heim_error_t *); + +struct heim_db_type { + int version; + heim_db_plug_open_f_t openf; + heim_db_plug_clone_f_t clonef; + heim_db_plug_close_f_t closef; + heim_db_plug_lock_f_t lockf; + heim_db_plug_unlock_f_t unlockf; + heim_db_plug_sync_f_t syncf; + heim_db_plug_begin_f_t beginf; + heim_db_plug_commit_f_t commitf; + heim_db_plug_rollback_f_t rollbackf; + heim_db_plug_copy_value_f_t copyf; + heim_db_plug_set_value_f_t setf; + heim_db_plug_del_key_f_t delf; + heim_db_plug_iter_f_t iterf; +}; + +extern struct heim_db_type heim_sorted_text_file_dbtype; + +#define HEIM_DB_TYPE_VERSION_01 1 + +int heim_db_register(const char *dbtype, + void *data, + struct heim_db_type *plugin); + +heim_db_t heim_db_create(const char *dbtype, const char *dbname, + heim_dict_t options, heim_error_t *error); +heim_db_t heim_db_clone(heim_db_t, heim_error_t *); +int heim_db_begin(heim_db_t, int, heim_error_t *); +int heim_db_commit(heim_db_t, heim_error_t *); +int heim_db_rollback(heim_db_t, heim_error_t *); +heim_tid_t heim_db_get_type_id(void); + +int heim_db_set_value(heim_db_t, heim_string_t, heim_data_t, heim_data_t, + heim_error_t *); +heim_data_t heim_db_copy_value(heim_db_t, heim_string_t, heim_data_t, + heim_error_t *); +int heim_db_delete_key(heim_db_t, heim_string_t, heim_data_t, + heim_error_t *); +void heim_db_iterate_f(heim_db_t, heim_string_t, void *, + heim_db_iterator_f_t, heim_error_t *); +#ifdef __BLOCKS__ +void heim_db_iterate(heim_db_t, heim_string_t, + void (^)(heim_data_t, heim_data_t), heim_error_t *); +#endif + + /* * Number */ @@ -191,50 +338,28 @@ heim_auto_release_t heim_auto_release_create(void); void heim_auto_release_drain(heim_auto_release_t); void heim_auto_release(heim_object_t); - -/* - * - */ - -typedef struct heim_error * heim_error_t; - -heim_error_t heim_error_create(int, const char *, ...) - HEIMDAL_PRINTF_ATTRIBUTE((printf, 2, 3)); - -heim_error_t heim_error_createv(int, const char *, va_list) - HEIMDAL_PRINTF_ATTRIBUTE((printf, 2, 0)); - -heim_string_t heim_error_copy_string(heim_error_t); -int heim_error_get_code(heim_error_t); - -heim_error_t heim_error_append(heim_error_t, heim_error_t); - /* * JSON */ -heim_object_t heim_json_create(const char *, heim_error_t *); -heim_object_t heim_json_create_with_bytes(const void *, size_t, heim_error_t *); +typedef enum heim_json_flags { + HEIM_JSON_F_NO_C_NULL = 1, + HEIM_JSON_F_STRICT_STRINGS = 2, + HEIM_JSON_F_NO_DATA = 4, + HEIM_JSON_F_NO_DATA_DICT = 8, + HEIM_JSON_F_STRICT_DICT = 16, + HEIM_JSON_F_STRICT = 31, + HEIM_JSON_F_CNULL2JSNULL = 32, + HEIM_JSON_F_TRY_DECODE_DATA = 64, + HEIM_JSON_F_ONE_LINE = 128 +} heim_json_flags_t; -/* - * - */ - -#ifndef __HEIM_OCTET_STRING__ -#define __HEIM_OCTET_STRING__ -typedef struct heim_octet_string { - size_t length; - void *data; -} heim_octet_string; -#endif - -typedef struct heim_data * heim_data_t; - -heim_data_t heim_data_create(const void *, size_t); -heim_tid_t heim_data_get_type_id(void); -const heim_octet_string * - heim_data_get_data(heim_data_t); -const void * heim_data_get_ptr(heim_data_t); -size_t heim_data_get_length(heim_data_t); +heim_object_t heim_json_create(const char *, size_t, heim_json_flags_t, + heim_error_t *); +heim_object_t heim_json_create_with_bytes(const void *, size_t, size_t, + heim_json_flags_t, + heim_error_t *); +heim_string_t heim_serialize(heim_object_t, heim_json_flags_t flags, + heim_error_t *); /* @@ -243,14 +368,14 @@ size_t heim_data_get_length(heim_data_t); * Note: these are private until integrated into the heimbase object system. */ typedef struct bsearch_file_handle *bsearch_file_handle; -int __bsearch_text(const char *buf, size_t buf_sz, const char *key, +int _bsearch_text(const char *buf, size_t buf_sz, const char *key, char **value, size_t *location, size_t *loops); -int __bsearch_file_open(const char *fname, size_t max_sz, size_t page_sz, +int _bsearch_file_open(const char *fname, size_t max_sz, size_t page_sz, bsearch_file_handle *bfh, size_t *reads); -int __bsearch_file(bsearch_file_handle bfh, const char *key, char **value, +int _bsearch_file(bsearch_file_handle bfh, const char *key, char **value, size_t *location, size_t *loops, size_t *reads); -void __bsearch_file_info(bsearch_file_handle bfh, size_t *page_sz, +void _bsearch_file_info(bsearch_file_handle bfh, size_t *page_sz, size_t *max_sz, int *blockwise); -void __bsearch_file_close(bsearch_file_handle *bfh); +void _bsearch_file_close(bsearch_file_handle *bfh); #endif /* HEIM_BASE_H */ diff --git a/base/heimbasepriv.h b/base/heimbasepriv.h index b06b010ad..e20734292 100644 --- a/base/heimbasepriv.h +++ b/base/heimbasepriv.h @@ -44,11 +44,11 @@ enum { HEIM_TID_NUMBER = 0, HEIM_TID_NULL = 1, HEIM_TID_BOOL = 2, - HEIM_TID_TAGGED_UNUSED2 = 3, - HEIM_TID_TAGGED_UNUSED3 = 4, - HEIM_TID_TAGGED_UNUSED4 = 5, - HEIM_TID_TAGGED_UNUSED5 = 6, - HEIM_TID_TAGGED_UNUSED6 = 7, + HEIM_TID_TAGGED_UNUSED2 = 3, /* reserved for tagged object types */ + HEIM_TID_TAGGED_UNUSED3 = 4, /* reserved for tagged object types */ + HEIM_TID_TAGGED_UNUSED4 = 5, /* reserved for tagged object types */ + HEIM_TID_TAGGED_UNUSED5 = 6, /* reserved for tagged object types */ + HEIM_TID_TAGGED_UNUSED6 = 7, /* reserved for tagged object types */ HEIM_TID_MEMORY = 128, HEIM_TID_ARRAY = 129, HEIM_TID_DICT = 130, @@ -56,6 +56,7 @@ enum { HEIM_TID_AUTORELEASE = 132, HEIM_TID_ERROR = 133, HEIM_TID_DATA = 134, + HEIM_TID_DB = 135, HEIM_TID_USER = 255 }; @@ -83,12 +84,19 @@ _heim_create_type(const char *name, heim_object_t _heim_alloc_object(heim_type_t type, size_t size); +void * +_heim_get_isaextra(heim_object_t o, size_t idx); + heim_tid_t _heim_type_get_tid(heim_type_t type); void _heim_make_permanent(heim_object_t ptr); +heim_data_t +_heim_db_get_value(heim_db_t, heim_string_t, heim_data_t, heim_error_t *); + + /* tagged tid */ extern struct heim_type_data _heim_null_object; extern struct heim_type_data _heim_bool_object; diff --git a/base/json.c b/base/json.c index 35fe7f540..6cf04733c 100644 --- a/base/json.c +++ b/base/json.c @@ -35,11 +35,34 @@ #include "baselocl.h" #include +#include + +static heim_base_once_t heim_json_once = HEIM_BASE_ONCE_INIT; +static heim_string_t heim_tid_data_uuid_key = NULL; +static const char base64_chars[] = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + +static void +json_init_once(void *arg) +{ + heim_tid_data_uuid_key = __heim_string_constant("heimdal-type-data-76d7fca2-d0da-4b20-a126-1a10f8a0eae6"); +} struct twojson { void *ctx; - size_t indent; void (*out)(void *, const char *); + size_t indent; + heim_json_flags_t flags; + int ret; + int first; +}; + +struct strbuf { + char *str; + size_t len; + size_t alloced; + int enomem; + heim_json_flags_t flags; }; static int @@ -49,6 +72,8 @@ static void indent(struct twojson *j) { size_t indent = j->indent; + if (j->flags & HEIM_JSON_F_ONE_LINE) + return; while (indent--) j->out(j->ctx, "\t"); } @@ -57,31 +82,56 @@ static void array2json(heim_object_t value, void *ctx) { struct twojson *j = ctx; - indent(j); - base2json(value, j); - j->out(j->ctx, ",\n"); - j->indent--; + if (j->ret) + return; + if (j->first) { + j->first = 0; + } else { + j->out(j->ctx, NULL); /* eat previous '\n' if possible */ + j->out(j->ctx, ",\n"); + } + j->ret = base2json(value, j); } static void dict2json(heim_object_t key, heim_object_t value, void *ctx) { struct twojson *j = ctx; - indent(j); - base2json(key, j); - j->out(j->ctx, " = "); - base2json(value, j); - j->out(j->ctx, ",\n"); + if (j->ret) + return; + if (j->first) { + j->first = 0; + } else { + j->out(j->ctx, NULL); /* eat previous '\n' if possible */ + j->out(j->ctx, ",\n"); + } + j->ret = base2json(key, j); + if (j->ret) + return; + j->out(j->ctx, " : \n"); + j->indent++; + j->ret = base2json(value, j); + if (j->ret) + return; + j->indent--; } static int base2json(heim_object_t obj, struct twojson *j) { heim_tid_t type; + int first = 0; if (obj == NULL) { - indent(j); - j->out(j->ctx, "\n"); + if (j->flags & HEIM_JSON_F_CNULL2JSNULL) { + obj = heim_null_create(); + } else if (j->flags & HEIM_JSON_F_NO_C_NULL) { + return EINVAL; + } else { + indent(j); + j->out(j->ctx, "\n"); /* This is NOT valid JSON! */ + return 0; + } } type = heim_get_tid(obj); @@ -90,20 +140,30 @@ base2json(heim_object_t obj, struct twojson *j) indent(j); j->out(j->ctx, "[\n"); j->indent++; + first = j->first; + j->first = 1; heim_array_iterate_f(obj, j, array2json); j->indent--; + if (!j->first) + j->out(j->ctx, "\n"); indent(j); j->out(j->ctx, "]\n"); + j->first = first; break; case HEIM_TID_DICT: indent(j); j->out(j->ctx, "{\n"); j->indent++; + first = j->first; + j->first = 1; heim_dict_iterate_f(obj, j, dict2json); j->indent--; + if (!j->first) + j->out(j->ctx, "\n"); indent(j); j->out(j->ctx, "}\n"); + j->first = first; break; case HEIM_TID_STRING: @@ -113,10 +173,65 @@ base2json(heim_object_t obj, struct twojson *j) j->out(j->ctx, "\""); break; + case HEIM_TID_DATA: { + heim_dict_t d; + heim_string_t v; + const heim_octet_string *data; + char *b64 = NULL; + int ret; + + if (j->flags & HEIM_JSON_F_NO_DATA) + return EINVAL; /* JSON doesn't do binary */ + + data = heim_data_get_data(obj); + ret = base64_encode(data->data, data->length, &b64); + if (ret < 0 || b64 == NULL) + return ENOMEM; + + if (j->flags & HEIM_JSON_F_NO_DATA_DICT) { + indent(j); + j->out(j->ctx, "\""); + j->out(j->ctx, b64); /* base64-encode; hope there's no aliasing */ + j->out(j->ctx, "\""); + free(b64); + } else { + /* + * JSON has no way to represent binary data, therefore the + * following is a Heimdal-specific convention. + * + * We encode binary data as a dict with a single very magic + * key with a base64-encoded value. The magic key includes + * a uuid, so we're not likely to alias accidentally. + */ + d = heim_dict_create(2); + if (d == NULL) { + free(b64); + return ENOMEM; + } + v = heim_string_ref_create(b64, free); + if (v == NULL) { + free(b64); + heim_release(d); + return ENOMEM; + } + ret = heim_dict_set_value(d, heim_tid_data_uuid_key, v); + heim_release(v); + if (ret) { + heim_release(d); + return ENOMEM; + } + ret = base2json(d, j); + heim_release(d); + if (ret) + return ret; + } + break; + } + case HEIM_TID_NUMBER: { - char num[16]; + char num[32]; indent(j); - snprintf(num, sizeof(num), "%d", heim_number_get_int(obj)); + snprintf(num, sizeof (num), "%d", heim_number_get_int(obj)); j->out(j->ctx, num); break; } @@ -135,14 +250,22 @@ base2json(heim_object_t obj, struct twojson *j) } static int -heim_base2json(heim_object_t obj, void *ctx, +heim_base2json(heim_object_t obj, void *ctx, heim_json_flags_t flags, void (*out)(void *, const char *)) { struct twojson j; + if (flags & HEIM_JSON_F_STRICT_STRINGS) + return ENOTSUP; /* Sorry, not yet! */ + + heim_base_once_f(&heim_json_once, NULL, json_init_once); + j.indent = 0; j.ctx = ctx; j.out = out; + j.flags = flags; + j.ret = 0; + j.first = 1; return base2json(obj, &j); } @@ -158,12 +281,18 @@ struct parse_ctx { const uint8_t *pstart; const uint8_t *pend; heim_error_t error; + size_t depth; + heim_json_flags_t flags; }; static heim_object_t parse_value(struct parse_ctx *ctx); +/* + * This function eats whitespace, but, critically, it also succeeds + * only if there's anything left to parse. + */ static int white_spaces(struct parse_ctx *ctx) { @@ -195,6 +324,8 @@ parse_number(struct parse_ctx *ctx) return NULL; if (*ctx->p == '-') { + if (ctx->p + 1 >= ctx->pend) + return NULL; neg = -1; ctx->p += 1; } @@ -217,7 +348,18 @@ parse_string(struct parse_ctx *ctx) const uint8_t *start; int quote = 0; - heim_assert(*ctx->p == '"', "string doesnt' start with \""); + if (ctx->flags & HEIM_JSON_F_STRICT_STRINGS) { + ctx->error = heim_error_create(EINVAL, "Strict JSON string encoding " + "not yet supported"); + return NULL; + } + + if (*ctx->p != '"') { + ctx->error = heim_error_create(EINVAL, "Expected a JSON string but " + "found something else at line %lu", + ctx->lineno); + return NULL; + } start = ++ctx->p; while (ctx->p < ctx->pend) { @@ -226,7 +368,7 @@ parse_string(struct parse_ctx *ctx) } else if (*ctx->p == '\\') { if (ctx->p + 1 == ctx->pend) goto out; - ctx->p += 1; + ctx->p++; quote = 1; } else if (*ctx->p == '"') { heim_object_t o; @@ -239,7 +381,7 @@ parse_string(struct parse_ctx *ctx) while (start < ctx->p) { if (*start == '\\') { start++; - /* XXX validate qouted char */ + /* XXX validate quoted char */ } *p++ = *start++; } @@ -247,11 +389,41 @@ parse_string(struct parse_ctx *ctx) free(p0); } else { o = heim_string_create_with_bytes(start, ctx->p - start); + if (o == NULL) { + ctx->error = heim_error_enomem(); + return NULL; + } + + /* If we can decode as base64, then let's */ + if (ctx->flags & HEIM_JSON_F_TRY_DECODE_DATA) { + void *buf; + size_t len; + const char *s; + + s = heim_string_get_utf8(o); + len = strlen(s); + + if (len >= 4 && strspn(s, base64_chars) >= len - 2) { + buf = malloc(len); + if (buf == NULL) { + heim_release(o); + ctx->error = heim_error_enomem(); + return NULL; + } + len = base64_decode(s, buf); + if (len == -1) { + free(buf); + return o; + } + heim_release(o); + o = heim_data_ref_create(buf, len, free); + } + } } ctx->p += 1; return o; - } + } ctx->p += 1; } out: @@ -268,22 +440,32 @@ parse_pair(heim_dict_t dict, struct parse_ctx *ctx) if (white_spaces(ctx)) return -1; - if (*ctx->p == '}') + if (*ctx->p == '}') { + ctx->p++; return 0; + } - key = parse_string(ctx); + if (ctx->flags & HEIM_JSON_F_STRICT_DICT) + /* JSON allows only string keys */ + key = parse_string(ctx); + else + /* heim_dict_t allows any heim_object_t as key */ + key = parse_value(ctx); if (key == NULL) + /* Even heim_dict_t does not allow C NULLs as keys though! */ return -1; - if (white_spaces(ctx)) + if (white_spaces(ctx)) { + heim_release(key); return -1; + } if (*ctx->p != ':') { heim_release(key); return -1; } - ctx->p += 1; + ctx->p += 1; /* safe because we call white_spaces() next */ if (white_spaces(ctx)) { heim_release(key); @@ -291,7 +473,10 @@ parse_pair(heim_dict_t dict, struct parse_ctx *ctx) } value = parse_value(ctx); - if (value == NULL) { + if (value == NULL && + (ctx->error != NULL || (ctx->flags & HEIM_JSON_F_NO_C_NULL))) { + if (ctx->error == NULL) + ctx->error = heim_error_create(EINVAL, "Invalid JSON encoding"); heim_release(key); return -1; } @@ -303,8 +488,11 @@ parse_pair(heim_dict_t dict, struct parse_ctx *ctx) return -1; if (*ctx->p == '}') { - ctx->p++; - return 0; + /* + * Return 1 but don't consume the '}' so we can count the one + * pair in a one-pair dict + */ + return 1; } else if (*ctx->p == ',') { ctx->p++; return 1; @@ -315,18 +503,54 @@ parse_pair(heim_dict_t dict, struct parse_ctx *ctx) static heim_dict_t parse_dict(struct parse_ctx *ctx) { - heim_dict_t dict = heim_dict_create(11); + heim_dict_t dict; + size_t count = 0; int ret; heim_assert(*ctx->p == '{', "string doesn't start with {"); - ctx->p += 1; + + dict = heim_dict_create(11); + if (dict == NULL) { + ctx->error = heim_error_enomem(); + return NULL; + } + + ctx->p += 1; /* safe because parse_pair() calls white_spaces() first */ while ((ret = parse_pair(dict, ctx)) > 0) - ; + count++; if (ret < 0) { heim_release(dict); return NULL; } + if (count == 1 && !(ctx->flags & HEIM_JSON_F_NO_DATA_DICT)) { + heim_object_t v = heim_dict_copy_value(dict, heim_tid_data_uuid_key); + + /* + * Binary data encoded as a dict with a single magic key with + * base64-encoded value? Decode as heim_data_t. + */ + if (v != NULL && heim_get_tid(v) == HEIM_TID_STRING) { + void *buf; + size_t len; + + buf = malloc(strlen(heim_string_get_utf8(v))); + if (buf == NULL) { + heim_release(dict); + heim_release(v); + ctx->error = heim_error_enomem(); + return NULL; + } + len = base64_decode(heim_string_get_utf8(v), buf); + heim_release(v); + if (len == -1) { + free(buf); + return dict; /* assume aliasing accident */ + } + heim_release(dict); + return (heim_dict_t)heim_data_ref_create(buf, len, free); + } + } return dict; } @@ -338,11 +562,14 @@ parse_item(heim_array_t array, struct parse_ctx *ctx) if (white_spaces(ctx)) return -1; - if (*ctx->p == ']') + if (*ctx->p == ']') { + ctx->p++; /* safe because parse_value() calls white_spaces() first */ return 0; + } value = parse_value(ctx); - if (value == NULL) + if (value == NULL && + (ctx->error || (ctx->flags & HEIM_JSON_F_NO_C_NULL))) return -1; heim_array_append_value(array, value); @@ -383,6 +610,7 @@ static heim_object_t parse_value(struct parse_ctx *ctx) { size_t len; + heim_object_t o; if (white_spaces(ctx)) return NULL; @@ -390,16 +618,32 @@ parse_value(struct parse_ctx *ctx) if (*ctx->p == '"') { return parse_string(ctx); } else if (*ctx->p == '{') { - return parse_dict(ctx); + if (ctx->depth-- == 1) { + ctx->error = heim_error_create(EINVAL, "JSON object too deep"); + return NULL; + } + o = parse_dict(ctx); + ctx->depth++; + return o; } else if (*ctx->p == '[') { - return parse_array(ctx); + if (ctx->depth-- == 1) { + ctx->error = heim_error_create(EINVAL, "JSON object too deep"); + return NULL; + } + o = parse_array(ctx); + ctx->depth++; + return o; } else if (is_number(*ctx->p) || *ctx->p == '-') { return parse_number(ctx); } len = ctx->pend - ctx->p; - if (len >= 4 && memcmp(ctx->p, "null", 4) == 0) { + if ((ctx->flags & HEIM_JSON_F_NO_C_NULL) == 0 && + len >= 6 && memcmp(ctx->p, "", 6) == 0) { + ctx->p += 6; + return heim_null_create(); + } else if (len >= 4 && memcmp(ctx->p, "null", 4) == 0) { ctx->p += 4; return heim_null_create(); } else if (len >= 4 && strncasecmp((char *)ctx->p, "true", 4) == 0) { @@ -419,22 +663,29 @@ parse_value(struct parse_ctx *ctx) heim_object_t -heim_json_create(const char *string, heim_error_t *error) +heim_json_create(const char *string, size_t max_depth, heim_json_flags_t flags, + heim_error_t *error) { - return heim_json_create_with_bytes(string, strlen(string), error); + return heim_json_create_with_bytes(string, strlen(string), max_depth, flags, + error); } heim_object_t -heim_json_create_with_bytes(const void *data, size_t length, heim_error_t *error) +heim_json_create_with_bytes(const void *data, size_t length, size_t max_depth, + heim_json_flags_t flags, heim_error_t *error) { struct parse_ctx ctx; heim_object_t o; + heim_base_once_f(&heim_json_once, NULL, json_init_once); + ctx.lineno = 1; ctx.p = data; ctx.pstart = data; ctx.pend = ((uint8_t *)data) + length; ctx.error = NULL; + ctx.flags = flags; + ctx.depth = max_depth; o = parse_value(&ctx); @@ -451,11 +702,110 @@ heim_json_create_with_bytes(const void *data, size_t length, heim_error_t *error static void show_printf(void *ctx, const char *str) { + if (str == NULL) + return; fprintf(ctx, "%s", str); } +/** + * Dump a heimbase object to stderr (useful from the debugger!) + * + * @param obj object to dump using JSON or JSON-like format + * + * @addtogroup heimbase + */ void heim_show(heim_object_t obj) { - heim_base2json(obj, stderr, show_printf); + heim_base2json(obj, stderr, HEIM_JSON_F_NO_DATA_DICT, show_printf); +} + +static void +strbuf_add(void *ctx, const char *str) +{ + struct strbuf *strbuf = ctx; + size_t len; + + if (strbuf->enomem) + return; + + if (str == NULL) { + /* + * Eat the last '\n'; this is used when formatting dict pairs + * and array items so that the ',' separating them is never + * preceded by a '\n'. + */ + if (strbuf->len > 0 && strbuf->str[strbuf->len - 1] == '\n') + strbuf->len--; + return; + } + + len = strlen(str); + if ((len + 1) > (strbuf->alloced - strbuf->len)) { + size_t new_len = strbuf->alloced + (strbuf->alloced >> 2) + len + 1; + char *s; + + s = realloc(strbuf->str, new_len); + if (s == NULL) { + strbuf->enomem = 1; + return; + } + strbuf->str = s; + strbuf->alloced = new_len; + } + /* +1 so we copy the NUL */ + (void) memcpy(strbuf->str + strbuf->len, str, len + 1); + strbuf->len += len; + if (strbuf->str[strbuf->len - 1] == '\n' && + strbuf->flags & HEIM_JSON_F_ONE_LINE) + strbuf->len--; +} + +#define STRBUF_INIT_SZ 64 + +heim_string_t +heim_serialize(heim_object_t obj, heim_json_flags_t flags, heim_error_t *error) +{ + heim_string_t str; + struct strbuf strbuf; + int ret; + + if (error) + *error = NULL; + + memset(&strbuf, 0, sizeof (strbuf)); + strbuf.str = malloc(STRBUF_INIT_SZ); + if (strbuf.str == NULL) { + if (error) + *error = heim_error_enomem(); + return NULL; + } + strbuf.len = 0; + strbuf.alloced = STRBUF_INIT_SZ; + strbuf.str[0] = '\0'; + strbuf.flags = flags; + + ret = heim_base2json(obj, &strbuf, flags, strbuf_add); + if (ret || strbuf.enomem) { + if (error) { + if (strbuf.enomem || ret == ENOMEM) + *error = heim_error_enomem(); + else + *error = heim_error_create(1, "Impossible to JSON-encode " + "object"); + } + free(strbuf.str); + return NULL; + } + if (flags & HEIM_JSON_F_ONE_LINE) { + strbuf.flags &= ~HEIM_JSON_F_ONE_LINE; + strbuf_add(&strbuf, "\n"); + } + str = heim_string_ref_create(strbuf.str, free); + if (str == NULL) { + if (error) + *error = heim_error_enomem(); + free(strbuf.str); + } + return str; } diff --git a/base/roken_rename.h b/base/roken_rename.h index 07c009263..ea7209893 100644 --- a/base/roken_rename.h +++ b/base/roken_rename.h @@ -36,9 +36,6 @@ #ifndef __heimbase_roken_rename_h__ #define __heimbase_roken_rename_h__ -#ifndef HAVE_SNPRINTF -#define rk_snprintf heimbase_snprintf -#endif #ifndef HAVE_VSNPRINTF #define rk_vsnprintf heimbase_vsnprintf #endif @@ -54,5 +51,11 @@ #ifndef HAVE_VASNPRINTF #define rk_vasnprintf heimbase_vasnprintf #endif +#ifndef HAVE_STRDUP +#define rk_strdup heimbase_strdup +#endif +#ifndef HAVE_STRNDUP +#define rk_strndup heimbase_strndup +#endif #endif /* __heimbase_roken_rename_h__ */ diff --git a/base/string.c b/base/string.c index 633d5a568..49779fbac 100644 --- a/base/string.c +++ b/base/string.c @@ -39,11 +39,37 @@ static void string_dealloc(void *ptr) { + heim_string_t s = ptr; + heim_string_free_f_t *deallocp; + heim_string_free_f_t dealloc; + + if (*(const char *)ptr != '\0') + return; + + /* Possible string ref */ + deallocp = _heim_get_isaextra(s, 0); + dealloc = *deallocp; + if (dealloc != NULL) { + char **strp = _heim_get_isaextra(s, 1); + dealloc(*strp); + } } static int string_cmp(void *a, void *b) { + if (*(char *)a == '\0') { + char **strp = _heim_get_isaextra(a, 1); + + if (*strp != NULL) + a = *strp; /* a is a string ref */ + } + if (*(char *)b == '\0') { + char **strp = _heim_get_isaextra(b, 1); + + if (*strp != NULL) + b = *strp; /* b is a string ref */ + } return strcmp(a, b); } @@ -82,6 +108,43 @@ heim_string_create(const char *string) return heim_string_create_with_bytes(string, strlen(string)); } +/** + * Create a string object without copying the source. + * + * @param string the string to referenced, must be UTF-8 + * @param dealloc the function to use to release the referece to the string + * + * @return string object + */ + +heim_string_t +heim_string_ref_create(const char *string, heim_string_free_f_t dealloc) +{ + heim_string_t s; + heim_string_free_f_t *deallocp; + + s = _heim_alloc_object(&_heim_string_object, 1); + if (s) { + const char **strp; + + ((char *)s)[0] = '\0'; + deallocp = _heim_get_isaextra(s, 0); + *deallocp = dealloc; + strp = _heim_get_isaextra(s, 1); + *strp = string; + } + return s; +} + +/** + * Create a string object + * + * @param string the string to create, must be an utf8 string + * @param len the length of the string + * + * @return string object + */ + heim_string_t heim_string_create_with_bytes(const void *data, size_t len) { @@ -118,6 +181,14 @@ heim_string_get_type_id(void) const char * heim_string_get_utf8(heim_string_t string) { + if (*(const char *)string == '\0') { + const char **strp; + + /* String ref */ + strp = _heim_get_isaextra(string, 1); + if (*strp != NULL) + return *strp; + } return (const char *)string; } diff --git a/base/test_base.c b/base/test_base.c index 7228e4488..78a1c4f4c 100644 --- a/base/test_base.c +++ b/base/test_base.c @@ -33,12 +33,35 @@ * SUCH DAMAGE. */ +/* + * This is a test of libheimbase functionality. If you make any changes + * to libheimbase or to this test you should run it under valgrind with + * the following options: + * + * -v --track-fds=yes --num-callers=30 --leak-check=full + * + * and make sure that there are no leaks that don't have + * __heim_string_constant() or heim_db_register() in their stack trace. + */ + +#include #include #include #include +#include +#include +#ifndef WIN32 +#include +#endif +#ifdef HAVE_IO_H +#include +#endif +#ifdef HAVE_UNISTD_H +#include +#endif +#include -#include "heimbase.h" -#include "heimbasepriv.h" +#include "baselocl.h" static void memory_free(heim_object_t obj) @@ -160,72 +183,702 @@ test_error(void) static int test_json(void) { + static char *j[] = { + "{ \"k1\" : \"s1\", \"k2\" : \"s2\" }", + "{ \"k1\" : [\"s1\", \"s2\", \"s3\"], \"k2\" : \"s3\" }", + "{ \"k1\" : {\"k2\":\"s1\",\"k3\":\"s2\",\"k4\":\"s3\"}, \"k5\" : \"s4\" }", + "[ \"v1\", \"v2\", [\"v3\",\"v4\",[\"v 5\",\" v 7 \"]], -123456789, " + "null, true, false, 123456789, \"\"]", + " -1" + }; + char *s; + size_t i, k; heim_object_t o, o2; heim_string_t k1 = heim_string_create("k1"); - o = heim_json_create("\"string\"", NULL); + o = heim_json_create("\"string\"", 10, 0, NULL); heim_assert(o != NULL, "string"); heim_assert(heim_get_tid(o) == heim_string_get_type_id(), "string-tid"); heim_assert(strcmp("string", heim_string_get_utf8(o)) == 0, "wrong string"); heim_release(o); - o = heim_json_create(" \"foo\\\"bar\" ]", NULL); + o = heim_json_create(" \"foo\\\"bar\" ]", 10, 0, NULL); heim_assert(o != NULL, "string"); heim_assert(heim_get_tid(o) == heim_string_get_type_id(), "string-tid"); heim_assert(strcmp("foo\"bar", heim_string_get_utf8(o)) == 0, "wrong string"); heim_release(o); - o = heim_json_create(" { \"key\" : \"value\" }", NULL); + o = heim_json_create(" { \"key\" : \"value\" }", 10, 0, NULL); heim_assert(o != NULL, "dict"); heim_assert(heim_get_tid(o) == heim_dict_get_type_id(), "dict-tid"); heim_release(o); - o = heim_json_create(" { \"k1\" : \"s1\", \"k2\" : \"s2\" }", NULL); + o = heim_json_create("{ { \"k1\" : \"s1\", \"k2\" : \"s2\" } : \"s3\", " + "{ \"k3\" : \"s4\" } : -1 }", 10, 0, NULL); heim_assert(o != NULL, "dict"); heim_assert(heim_get_tid(o) == heim_dict_get_type_id(), "dict-tid"); - o2 = heim_dict_get_value(o, k1); + heim_release(o); + + o = heim_json_create("{ { \"k1\" : \"s1\", \"k2\" : \"s2\" } : \"s3\", " + "{ \"k3\" : \"s4\" } : -1 }", 10, + HEIM_JSON_F_STRICT_DICT, NULL); + heim_assert(o == NULL, "dict"); + + o = heim_json_create(" { \"k1\" : \"s1\", \"k2\" : \"s2\" }", 10, 0, NULL); + heim_assert(o != NULL, "dict"); + heim_assert(heim_get_tid(o) == heim_dict_get_type_id(), "dict-tid"); + o2 = heim_dict_copy_value(o, k1); heim_assert(heim_get_tid(o2) == heim_string_get_type_id(), "string-tid"); + heim_release(o2); heim_release(o); - o = heim_json_create(" { \"k1\" : { \"k2\" : \"s2\" } }", NULL); + o = heim_json_create(" { \"k1\" : { \"k2\" : \"s2\" } }", 10, 0, NULL); heim_assert(o != NULL, "dict"); heim_assert(heim_get_tid(o) == heim_dict_get_type_id(), "dict-tid"); - o2 = heim_dict_get_value(o, k1); + o2 = heim_dict_copy_value(o, k1); heim_assert(heim_get_tid(o2) == heim_dict_get_type_id(), "dict-tid"); + heim_release(o2); heim_release(o); - o = heim_json_create("{ \"k1\" : 1 }", NULL); + o = heim_json_create("{ \"k1\" : 1 }", 10, 0, NULL); heim_assert(o != NULL, "array"); heim_assert(heim_get_tid(o) == heim_dict_get_type_id(), "dict-tid"); - o2 = heim_dict_get_value(o, k1); + o2 = heim_dict_copy_value(o, k1); heim_assert(heim_get_tid(o2) == heim_number_get_type_id(), "number-tid"); + heim_release(o2); heim_release(o); - o = heim_json_create("-10", NULL); + o = heim_json_create("-10", 10, 0, NULL); heim_assert(o != NULL, "number"); heim_assert(heim_get_tid(o) == heim_number_get_type_id(), "number-tid"); heim_release(o); - o = heim_json_create("99", NULL); + o = heim_json_create("99", 10, 0, NULL); heim_assert(o != NULL, "number"); heim_assert(heim_get_tid(o) == heim_number_get_type_id(), "number-tid"); heim_release(o); - o = heim_json_create(" [ 1 ]", NULL); + o = heim_json_create(" [ 1 ]", 10, 0, NULL); heim_assert(o != NULL, "array"); heim_assert(heim_get_tid(o) == heim_array_get_type_id(), "array-tid"); heim_release(o); - o = heim_json_create(" [ -1 ]", NULL); + o = heim_json_create(" [ -1 ]", 10, 0, NULL); heim_assert(o != NULL, "array"); heim_assert(heim_get_tid(o) == heim_array_get_type_id(), "array-tid"); heim_release(o); + for (i = 0; i < (sizeof (j) / sizeof (j[0])); i++) { + o = heim_json_create(j[i], 10, 0, NULL); + if (o == NULL) { + fprintf(stderr, "Failed to parse this JSON: %s\n", j[i]); + return 1; + } + heim_release(o); + /* Simple fuzz test */ + for (k = strlen(j[i]) - 1; k > 0; k--) { + o = heim_json_create_with_bytes(j[i], k, 10, 0, NULL); + if (o != NULL) { + fprintf(stderr, "Invalid JSON parsed: %.*s\n", k, j[i]); + return EINVAL; + } + } + /* Again, but this time make it so valgrind can find invalid accesses */ + for (k = strlen(j[i]) - 1; k > 0; k--) { + s = strndup(j[i], k); + if (s == NULL) + return ENOMEM; + o = heim_json_create(s, 10, 0, NULL); + free(s); + if (o != NULL) { + fprintf(stderr, "Invalid JSON parsed: %s\n", j[i]); + return EINVAL; + } + } + /* Again, but with no NUL termination */ + for (k = strlen(j[i]) - 1; k > 0; k--) { + s = malloc(k); + if (s == NULL) + return ENOMEM; + memcpy(s, j[i], k); + o = heim_json_create_with_bytes(s, k, 10, 0, NULL); + free(s); + if (o != NULL) { + fprintf(stderr, "Invalid JSON parsed: %s\n", j[i]); + return EINVAL; + } + } + } + heim_release(k1); return 0; } +static int +test_path(void) +{ + heim_dict_t dict = heim_dict_create(11); + heim_string_t p1 = heim_string_create("abc"); + heim_string_t p2a = heim_string_create("def"); + heim_string_t p2b = heim_string_create("DEF"); + heim_number_t p3 = heim_number_create(0); + heim_string_t p4a = heim_string_create("ghi"); + heim_string_t p4b = heim_string_create("GHI"); + heim_array_t a = heim_array_create(); + heim_number_t l1 = heim_number_create(42); + heim_number_t l2 = heim_number_create(813); + heim_number_t l3 = heim_number_create(1234); + heim_string_t k1 = heim_string_create("k1"); + heim_string_t k2 = heim_string_create("k2"); + heim_string_t k3 = heim_string_create("k3"); + heim_string_t k2_1 = heim_string_create("k2-1"); + heim_string_t k2_2 = heim_string_create("k2-2"); + heim_string_t k2_3 = heim_string_create("k2-3"); + heim_string_t k2_4 = heim_string_create("k2-4"); + heim_string_t k2_5 = heim_string_create("k2-5"); + heim_string_t k2_5_1 = heim_string_create("k2-5-1"); + heim_object_t o; + heim_object_t neg_num; + int ret; + + if (!dict || !p1 || !p2a || !p2b || !p4a || !p4b) + return ENOMEM; + + ret = heim_path_create(dict, 11, a, NULL, p1, p2a, NULL); + heim_release(a); + if (ret) + return ret; + ret = heim_path_create(dict, 11, l3, NULL, p1, p2b, NULL); + if (ret) + return ret; + o = heim_path_get(dict, NULL, p1, p2b, NULL); + if (o != l3) + return 1; + ret = heim_path_create(dict, 11, NULL, NULL, p1, p2a, p3, NULL); + if (ret) + return ret; + ret = heim_path_create(dict, 11, l1, NULL, p1, p2a, p3, p4a, NULL); + if (ret) + return ret; + ret = heim_path_create(dict, 11, l2, NULL, p1, p2a, p3, p4b, NULL); + if (ret) + return ret; + + o = heim_path_get(dict, NULL, p1, p2a, p3, p4a, NULL); + if (o != l1) + return 1; + o = heim_path_get(dict, NULL, p1, p2a, p3, p4b, NULL); + if (o != l2) + return 1; + + heim_release(dict); + + /* Test that JSON parsing works right by using heim_path_get() */ + dict = heim_json_create("{\"k1\":1," + "\"k2\":{\"k2-1\":21," + "\"k2-2\":null," + "\"k2-3\":true," + "\"k2-4\":false," + "\"k2-5\":[1,2,3,{\"k2-5-1\":-1},-2]}," + "\"k3\":[true,false,0,42]}", 10, 0, NULL); + heim_assert(dict != NULL, "dict"); + o = heim_path_get(dict, NULL, k1, NULL); + if (heim_cmp(o, heim_number_create(1))) return 1; + o = heim_path_get(dict, NULL, k2, NULL); + if (heim_get_tid(o) != heim_dict_get_type_id()) return 1; + o = heim_path_get(dict, NULL, k2, k2_1, NULL); + if (heim_cmp(o, heim_number_create(21))) return 1; + o = heim_path_get(dict, NULL, k2, k2_2, NULL); + if (heim_cmp(o, heim_null_create())) return 1; + o = heim_path_get(dict, NULL, k2, k2_3, NULL); + if (heim_cmp(o, heim_bool_create(1))) return 1; + o = heim_path_get(dict, NULL, k2, k2_4, NULL); + if (heim_cmp(o, heim_bool_create(0))) return 1; + o = heim_path_get(dict, NULL, k2, k2_5, NULL); + if (heim_get_tid(o) != heim_array_get_type_id()) return 1; + o = heim_path_get(dict, NULL, k2, k2_5, heim_number_create(0), NULL); + if (heim_cmp(o, heim_number_create(1))) return 1; + o = heim_path_get(dict, NULL, k2, k2_5, heim_number_create(1), NULL); + if (heim_cmp(o, heim_number_create(2))) return 1; + o = heim_path_get(dict, NULL, k2, k2_5, heim_number_create(3), k2_5_1, NULL); + if (heim_cmp(o, neg_num = heim_number_create(-1))) return 1; + heim_release(neg_num); + o = heim_path_get(dict, NULL, k2, k2_5, heim_number_create(4), NULL); + if (heim_cmp(o, neg_num = heim_number_create(-2))) return 1; + heim_release(neg_num); + o = heim_path_get(dict, NULL, k3, heim_number_create(3), NULL); + if (heim_cmp(o, heim_number_create(42))) return 1; + + heim_release(dict); + heim_release(p1); + heim_release(p2a); + heim_release(p2b); + heim_release(p4a); + heim_release(p4b); + heim_release(k1); + heim_release(k2); + heim_release(k3); + heim_release(k2_1); + heim_release(k2_2); + heim_release(k2_3); + heim_release(k2_4); + heim_release(k2_5); + heim_release(k2_5_1); + + return 0; +} + +typedef struct dict_db { + heim_dict_t dict; + int locked; +} *dict_db_t; + +static int +dict_db_open(void *plug, const char *dbtype, const char *dbname, + heim_dict_t options, void **db, heim_error_t *error) +{ + dict_db_t dictdb; + heim_dict_t contents = NULL; + + if (error) + *error = NULL; + if (dbtype && *dbtype && strcmp(dbtype, "dictdb")) + return EINVAL; + if (dbname && *dbname && strcmp(dbname, "MEMORY") != 0) + return EINVAL; + dictdb = heim_alloc(sizeof (*dictdb), "dict_db", NULL); + if (dictdb == NULL) + return ENOMEM; + + if (contents != NULL) + dictdb->dict = contents; + else { + dictdb->dict = heim_dict_create(29); + if (dictdb->dict == NULL) { + heim_release(dictdb); + return ENOMEM; + } + } + + *db = dictdb; + return 0; +} + +static int +dict_db_close(void *db, heim_error_t *error) +{ + dict_db_t dictdb = db; + + if (error) + *error = NULL; + heim_release(dictdb->dict); + heim_release(dictdb); + return 0; +} + +static int +dict_db_lock(void *db, int read_only, heim_error_t *error) +{ + dict_db_t dictdb = db; + + if (error) + *error = NULL; + if (dictdb->locked) + return EWOULDBLOCK; + dictdb->locked = 1; + return 0; +} + +static int +dict_db_unlock(void *db, heim_error_t *error) +{ + dict_db_t dictdb = db; + + if (error) + *error = NULL; + dictdb->locked = 0; + return 0; +} + +static heim_data_t +dict_db_copy_value(void *db, heim_string_t table, heim_data_t key, + heim_error_t *error) +{ + dict_db_t dictdb = db; + + if (error) + *error = NULL; + + return heim_retain(heim_path_get(dictdb->dict, error, table, key, NULL)); +} + +static int +dict_db_set_value(void *db, heim_string_t table, + heim_data_t key, heim_data_t value, heim_error_t *error) +{ + dict_db_t dictdb = db; + + if (error) + *error = NULL; + + if (table == NULL) + table = HSTR(""); + + return heim_path_create(dictdb->dict, 29, value, error, table, key, NULL); +} + +static int +dict_db_del_key(void *db, heim_string_t table, heim_data_t key, + heim_error_t *error) +{ + dict_db_t dictdb = db; + + if (error) + *error = NULL; + + if (table == NULL) + table = HSTR(""); + + heim_path_delete(dictdb->dict, error, table, key, NULL); + return 0; +} + +struct dict_db_iter_ctx { + heim_db_iterator_f_t iter_f; + void *iter_ctx; +}; + +static void dict_db_iter_f(heim_object_t key, heim_object_t value, void *arg) +{ + struct dict_db_iter_ctx *ctx = arg; + + ctx->iter_f((heim_object_t)key, (heim_object_t)value, ctx->iter_ctx); +} + +static void +dict_db_iter(void *db, heim_string_t table, void *iter_data, + heim_db_iterator_f_t iter_f, heim_error_t *error) +{ + dict_db_t dictdb = db; + struct dict_db_iter_ctx ctx; + heim_dict_t table_dict; + + if (error) + *error = NULL; + + if (table == NULL) + table = HSTR(""); + + table_dict = heim_dict_copy_value(dictdb->dict, table); + if (table_dict == NULL) + return; + + ctx.iter_ctx = iter_data; + ctx.iter_f = iter_f; + + heim_dict_iterate_f(table_dict, &ctx, dict_db_iter_f); + heim_release(table_dict); +} + +static void +test_db_iter(heim_data_t k, heim_data_t v, void *arg) +{ + int *ret = arg; + + heim_assert(heim_get_tid(k) == heim_data_get_type_id(), "..."); + + if (heim_data_get_length(k) == strlen("msg") && strncmp(heim_data_get_ptr(k), "msg", strlen("msg")) == 0 && + heim_data_get_length(v) == strlen("abc") && strncmp(heim_data_get_ptr(v), "abc", strlen("abc")) == 0) + *ret &= ~(1); + else if (heim_data_get_length(k) == strlen("msg2") && strncmp(heim_data_get_ptr(k), "msg2", strlen("msg2")) == 0 && + heim_data_get_length(v) == strlen("FooBar") && strncmp(heim_data_get_ptr(v), "FooBar", strlen("FooBar")) == 0) + *ret &= ~(2); + else + *ret |= 4; +} + +static struct heim_db_type dbt = { + 1, dict_db_open, NULL, dict_db_close, + dict_db_lock, dict_db_unlock, NULL, NULL, NULL, NULL, + dict_db_copy_value, dict_db_set_value, + dict_db_del_key, dict_db_iter +}; + +static int +test_db(const char *dbtype, const char *dbname) +{ + heim_data_t k1, k2, v, v1, v2, v3; + heim_db_t db; + int ret; + + if (dbtype == NULL) { + ret = heim_db_register("dictdb", NULL, &dbt); + heim_assert(!ret, "..."); + db = heim_db_create("dictdb", "foo", NULL, NULL); + heim_assert(!db, "..."); + db = heim_db_create("foobar", "MEMORY", NULL, NULL); + heim_assert(!db, "..."); + db = heim_db_create("dictdb", "MEMORY", NULL, NULL); + heim_assert(db, "..."); + } else { + heim_dict_t options; + + options = heim_dict_create(11); + if (options == NULL) return ENOMEM; + if (heim_dict_set_value(options, HSTR("journal-filename"), + HSTR("json-journal"))) + return ENOMEM; + if (heim_dict_set_value(options, HSTR("create"), heim_null_create())) + return ENOMEM; + if (heim_dict_set_value(options, HSTR("truncate"), heim_null_create())) + return ENOMEM; + db = heim_db_create(dbtype, dbname, options, NULL); + heim_assert(db, "..."); + heim_release(options); + } + + k1 = heim_data_create("msg", strlen("msg")); + k2 = heim_data_create("msg2", strlen("msg2")); + v1 = heim_data_create("Hello world!", strlen("Hello world!")); + v2 = heim_data_create("FooBar", strlen("FooBar")); + v3 = heim_data_create("abc", strlen("abc")); + + ret = heim_db_set_value(db, NULL, k1, v1, NULL); + heim_assert(!ret, "..."); + + v = heim_db_copy_value(db, NULL, k1, NULL); + heim_assert(v && !heim_cmp(v, v1), "..."); + heim_release(v); + + ret = heim_db_set_value(db, NULL, k2, v2, NULL); + heim_assert(!ret, "..."); + + v = heim_db_copy_value(db, NULL, k2, NULL); + heim_assert(v && !heim_cmp(v, v2), "..."); + heim_release(v); + + ret = heim_db_set_value(db, NULL, k1, v3, NULL); + heim_assert(!ret, "..."); + + v = heim_db_copy_value(db, NULL, k1, NULL); + heim_assert(v && !heim_cmp(v, v3), "..."); + heim_release(v); + + ret = 3; + heim_db_iterate_f(db, NULL, &ret, test_db_iter, NULL); + heim_assert(!ret, "..."); + + ret = heim_db_begin(db, 0, NULL); + heim_assert(!ret, "..."); + + ret = heim_db_commit(db, NULL); + heim_assert(!ret, "..."); + + ret = heim_db_begin(db, 0, NULL); + heim_assert(!ret, "..."); + + ret = heim_db_rollback(db, NULL); + heim_assert(!ret, "..."); + + ret = heim_db_begin(db, 0, NULL); + heim_assert(!ret, "..."); + + ret = heim_db_set_value(db, NULL, k1, v1, NULL); + heim_assert(!ret, "..."); + + v = heim_db_copy_value(db, NULL, k1, NULL); + heim_assert(v && !heim_cmp(v, v1), "..."); + heim_release(v); + + ret = heim_db_rollback(db, NULL); + heim_assert(!ret, "..."); + + v = heim_db_copy_value(db, NULL, k1, NULL); + heim_assert(v && !heim_cmp(v, v3), "..."); + heim_release(v); + + ret = heim_db_begin(db, 0, NULL); + heim_assert(!ret, "..."); + + ret = heim_db_set_value(db, NULL, k1, v1, NULL); + heim_assert(!ret, "..."); + + v = heim_db_copy_value(db, NULL, k1, NULL); + heim_assert(v && !heim_cmp(v, v1), "..."); + heim_release(v); + + ret = heim_db_commit(db, NULL); + heim_assert(!ret, "..."); + + v = heim_db_copy_value(db, NULL, k1, NULL); + heim_assert(v && !heim_cmp(v, v1), "..."); + heim_release(v); + + ret = heim_db_begin(db, 0, NULL); + heim_assert(!ret, "..."); + + ret = heim_db_delete_key(db, NULL, k1, NULL); + heim_assert(!ret, "..."); + + v = heim_db_copy_value(db, NULL, k1, NULL); + heim_assert(v == NULL, "..."); + heim_release(v); + + ret = heim_db_rollback(db, NULL); + heim_assert(!ret, "..."); + + v = heim_db_copy_value(db, NULL, k1, NULL); + heim_assert(v && !heim_cmp(v, v1), "..."); + heim_release(v); + + if (dbtype != NULL) { + heim_data_t k3 = heim_data_create("value-is-a-dict", strlen("value-is-a-dict")); + heim_dict_t vdict = heim_dict_create(11); + heim_db_t db2; + + heim_assert(k3 && vdict, "..."); + ret = heim_dict_set_value(vdict, HSTR("vdict-k1"), heim_number_create(11)); + heim_assert(!ret, "..."); + ret = heim_dict_set_value(vdict, HSTR("vdict-k2"), heim_null_create()); + heim_assert(!ret, "..."); + ret = heim_dict_set_value(vdict, HSTR("vdict-k3"), HSTR("a value")); + heim_assert(!ret, "..."); + ret = heim_db_set_value(db, NULL, k3, (heim_data_t)vdict, NULL); + heim_assert(!ret, "..."); + + heim_release(vdict); + + db2 = heim_db_create(dbtype, dbname, NULL, NULL); + heim_assert(db2, "..."); + + vdict = (heim_dict_t)heim_db_copy_value(db2, NULL, k3, NULL); + heim_release(db2); + heim_release(k3); + heim_assert(vdict, "..."); + heim_assert(heim_get_tid(vdict) == heim_dict_get_type_id(), "..."); + + v = heim_dict_copy_value(vdict, HSTR("vdict-k1")); + heim_assert(v && !heim_cmp(v, heim_number_create(11)), "..."); + heim_release(v); + + v = heim_dict_copy_value(vdict, HSTR("vdict-k2")); + heim_assert(v && !heim_cmp(v, heim_null_create()), "..."); + heim_release(v); + + v = heim_dict_copy_value(vdict, HSTR("vdict-k3")); + heim_assert(v && !heim_cmp(v, HSTR("a value")), "..."); + heim_release(v); + + heim_release(vdict); + } + + heim_release(db); + heim_release(k1); + heim_release(k2); + heim_release(v1); + heim_release(v2); + heim_release(v3); + + return 0; +} + +struct test_array_iter_ctx { + char buf[256]; +}; + +static void test_array_iter(heim_object_t elt, void *arg) +{ + struct test_array_iter_ctx *iter_ctx = arg; + + strcat(iter_ctx->buf, heim_string_get_utf8((heim_string_t)elt)); +} + +static int +test_array() +{ + struct test_array_iter_ctx iter_ctx; + heim_string_t s1 = heim_string_create("abc"); + heim_string_t s2 = heim_string_create("def"); + heim_string_t s3 = heim_string_create("ghi"); + heim_string_t s4 = heim_string_create("jkl"); + heim_string_t s5 = heim_string_create("mno"); + heim_string_t s6 = heim_string_create("pqr"); + heim_array_t a = heim_array_create(); + + if (!s1 || !s2 || !s3 || !s4 || !s5 || !s6 || !a) + return ENOMEM; + + heim_array_append_value(a, s4); + heim_array_append_value(a, s5); + heim_array_insert_value(a, 0, s3); + heim_array_insert_value(a, 0, s2); + heim_array_append_value(a, s6); + heim_array_insert_value(a, 0, s1); + + iter_ctx.buf[0] = '\0'; + heim_array_iterate_f(a, &iter_ctx, test_array_iter); + if (strcmp(iter_ctx.buf, "abcdefghijklmnopqr") != 0) + return 1; + + iter_ctx.buf[0] = '\0'; + heim_array_delete_value(a, 2); + heim_array_iterate_f(a, &iter_ctx, test_array_iter); + if (strcmp(iter_ctx.buf, "abcdefjklmnopqr") != 0) + return 1; + + iter_ctx.buf[0] = '\0'; + heim_array_delete_value(a, 2); + heim_array_iterate_f(a, &iter_ctx, test_array_iter); + if (strcmp(iter_ctx.buf, "abcdefmnopqr") != 0) + return 1; + + iter_ctx.buf[0] = '\0'; + heim_array_delete_value(a, 0); + heim_array_iterate_f(a, &iter_ctx, test_array_iter); + if (strcmp(iter_ctx.buf, "defmnopqr") != 0) + return 1; + + iter_ctx.buf[0] = '\0'; + heim_array_delete_value(a, 2); + heim_array_iterate_f(a, &iter_ctx, test_array_iter); + if (strcmp(iter_ctx.buf, "defmno") != 0) + return 1; + + heim_array_insert_value(a, 0, s1); + iter_ctx.buf[0] = '\0'; + heim_array_iterate_f(a, &iter_ctx, test_array_iter); + if (strcmp(iter_ctx.buf, "abcdefmno") != 0) + return 1; + + heim_array_insert_value(a, 0, s2); + iter_ctx.buf[0] = '\0'; + heim_array_iterate_f(a, &iter_ctx, test_array_iter); + if (strcmp(iter_ctx.buf, "defabcdefmno") != 0) + return 1; + + heim_array_append_value(a, s3); + iter_ctx.buf[0] = '\0'; + heim_array_iterate_f(a, &iter_ctx, test_array_iter); + if (strcmp(iter_ctx.buf, "defabcdefmnoghi") != 0) + return 1; + + heim_array_append_value(a, s6); + iter_ctx.buf[0] = '\0'; + heim_array_iterate_f(a, &iter_ctx, test_array_iter); + if (strcmp(iter_ctx.buf, "defabcdefmnoghipqr") != 0) + return 1; + + heim_release(s1); + heim_release(s2); + heim_release(s3); + heim_release(s4); + heim_release(s5); + heim_release(s6); + heim_release(a); + + return 0; +} int main(int argc, char **argv) @@ -238,6 +891,10 @@ main(int argc, char **argv) res |= test_string(); res |= test_error(); res |= test_json(); + res |= test_path(); + res |= test_db(NULL, NULL); + res |= test_db("json", argc > 1 ? argv[1] : "test_db.json"); + res |= test_array(); return res ? 1 : 0; } diff --git a/base/version-script.map b/base/version-script.map index 5157b3021..77848ada7 100644 --- a/base/version-script.map +++ b/base/version-script.map @@ -1,21 +1,26 @@ HEIMDAL_BASE_1.0 { global: - __bsearch_file; - __bsearch_file_close; - __bsearch_file_info; - __bsearch_file_open; - __bsearch_text; + _bsearch_file; + _bsearch_file_close; + _bsearch_file_info; + _bsearch_file_open; + _bsearch_text; __heim_string_constant; heim_abort; + heim_abortv; heim_alloc; heim_array_append_value; + heim_array_copy_value; heim_array_create; heim_array_delete_value; heim_array_get_length; heim_array_get_type_id; heim_array_get_value; heim_array_iterate_f; + heim_array_iterate_reverse_f; + heim_array_insert_value; + heim_array_set_value; heim_auto_release; heim_auto_release_create; heim_auto_release_drain; @@ -23,6 +28,25 @@ HEIMDAL_BASE_1.0 { heim_bool_create; heim_bool_val; heim_cmp; + heim_data_create; + heim_data_ref_create; + heim_data_get_data; + heim_data_get_length; + heim_data_get_ptr; + heim_data_get_type_id; + heim_data_ref_get_type_id; + heim_db_begin; + heim_db_clone; + heim_db_commit; + heim_db_copy_value; + heim_db_delete_key; + heim_db_get_type_id; + heim_db_iterate_f; + heim_db_create; + heim_db_register; + heim_db_rollback; + heim_db_set_value; + heim_dict_copy_value; heim_dict_create; heim_dict_delete_key; heim_dict_get_type_id; @@ -33,20 +57,34 @@ HEIMDAL_BASE_1.0 { heim_error_copy_string; heim_error_create; heim_error_createv; + heim_error_enomem; heim_error_get_code; + heim_get_hash; heim_get_tid; heim_json_create; heim_json_create_with_bytes; + heim_null_create; heim_number_create; heim_number_get_int; heim_number_get_type_id; + heim_path_create; + heim_path_delete; + heim_path_get; + heim_path_copy; + heim_path_vcreate; + heim_path_vdelete; + heim_path_vget; + heim_path_vcopy; heim_release; heim_retain; + heim_serialize; heim_show; + heim_sorted_text_file_dbtype; heim_string_create; heim_string_create_with_bytes; heim_string_get_type_id; heim_string_get_utf8; + heim_string_ref_create; local: *; }; diff --git a/doc/Makefile.am b/doc/Makefile.am index d1c4cadf4..ab8eca4f1 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -24,6 +24,11 @@ hdb.dxy: hdb.din Makefile chmod +x hdb.dxy.tmp mv hdb.dxy.tmp hdb.dxy +base.dxy: base.din Makefile + $(dxy_subst) < $(srcdir)/base.din > base.dxy.tmp + chmod +x base.dxy.tmp + mv base.dxy.tmp base.dxy + hx509.dxy: hx509.din Makefile $(dxy_subst) < $(srcdir)/hx509.din > hx509.dxy.tmp chmod +x hx509.dxy.tmp @@ -57,13 +62,13 @@ vars.texi: vars.tin Makefile chmod +x vars.texi.tmp mv vars.texi.tmp vars.texi -PROJECTS = hdb hx509 gssapi krb5 ntlm wind +PROJECTS = base hdb hx509 gssapi krb5 ntlm wind if !HAVE_OPENSSL PROJECTS += hcrypto endif -doxyout doxygen: hdb.dxy hx509.dxy hcrypto.dxy gssapi.dxy krb5.dxy ntlm.dxy wind.dxy +doxyout doxygen: base.dxy hdb.dxy hx509.dxy hcrypto.dxy gssapi.dxy krb5.dxy ntlm.dxy wind.dxy @find $(srcdir)/doxyout -type d ! -perm -200 -exec chmod u+w {} ';' ; \ rm -rf $(srcdir)/doxyout ; \ mkdir $(srcdir)/doxyout ; \ @@ -127,6 +132,7 @@ EXTRA_DIST = \ hcrypto.din \ header.html \ heimdal.css \ + base.din \ hx509.din \ krb5.din \ ntlm.din \ @@ -139,6 +145,7 @@ EXTRA_DIST = \ CLEANFILES = \ hcrypto.dxy* \ + base.dxy* \ hx509.dxy* \ hdb.dxy* \ gssapi.dxy* \ diff --git a/doc/base.din b/doc/base.din new file mode 100644 index 000000000..35ad6d9de --- /dev/null +++ b/doc/base.din @@ -0,0 +1,15 @@ +# Doxyfile 1.5.3 + +PROJECT_NAME = Heimdal x509 library +PROJECT_NUMBER = @PACKAGE_VERSION@ +OUTPUT_DIRECTORY = @srcdir@/doxyout/heimbase +INPUT = @srcdir@/../base + +WARN_IF_UNDOCUMENTED = YES + +PERL_PATH = /usr/bin/perl + +HTML_HEADER = "@srcdir@/header.html" +HTML_FOOTER = "@srcdir@/footer.html" + +@INCLUDE = "@srcdir@/doxytmpl.dxy" diff --git a/doc/base.hhp b/doc/base.hhp new file mode 100644 index 000000000..e1a3d3cf5 --- /dev/null +++ b/doc/base.hhp @@ -0,0 +1,8 @@ +[OPTIONS] +Compatibility=1.1 or later +Compiled file=heimbase.chm +Contents file=toc.hhc +Default topic=index.html +Display compile progress=No +Language=0x409 English (United States) +Title=Heimdal Base diff --git a/kdc/kdc-tester.c b/kdc/kdc-tester.c index ed58f9775..6f4a1702c 100644 --- a/kdc/kdc-tester.c +++ b/kdc/kdc-tester.c @@ -420,7 +420,7 @@ main(int argc, char **argv) if (rk_undumpdata(argv[0], &buf, &size)) errx(1, "undumpdata: %s", argv[0]); - o = heim_json_create_with_bytes(buf, size, NULL); + o = heim_json_create_with_bytes(buf, size, 10, 0, NULL); free(buf); if (o == NULL) errx(1, "heim_json"); diff --git a/lib/krb5/Makefile.am b/lib/krb5/Makefile.am index e6ecce207..f4bb0c739 100644 --- a/lib/krb5/Makefile.am +++ b/lib/krb5/Makefile.am @@ -53,7 +53,7 @@ LDADD = libkrb5.la \ $(LIB_hcrypto) \ $(top_builddir)/lib/asn1/libasn1.la \ $(top_builddir)/lib/wind/libwind.la \ - $(LIB_roken) + $(LIB_heimbase) $(LIB_roken) if PKINIT LIB_pkinit = ../hx509/libhx509.la @@ -142,6 +142,8 @@ dist_libkrb5_la_SOURCES = \ crypto-rand.c \ doxygen.c \ data.c \ + db_plugin.c \ + db_plugin.h \ deprecated.c \ digest.c \ eai_to_heim_errno.c \ @@ -341,7 +343,7 @@ nodist_include_HEADERS = krb5_err.h heim_err.h k524_err.h # XXX use nobase_include_HEADERS = krb5/locate_plugin.h krb5dir = $(includedir)/krb5 -krb5_HEADERS = locate_plugin.h send_to_kdc_plugin.h ccache_plugin.h an2ln_plugin.h +krb5_HEADERS = locate_plugin.h send_to_kdc_plugin.h ccache_plugin.h an2ln_plugin.h db_plugin.h build_HEADERZ = \ $(krb5_HEADERS) \ diff --git a/lib/krb5/NTMakefile b/lib/krb5/NTMakefile index 07041aa88..cbf5538e4 100644 --- a/lib/krb5/NTMakefile +++ b/lib/krb5/NTMakefile @@ -66,6 +66,7 @@ libkrb5_OBJS = \ $(OBJ)\crypto-pk.obj \ $(OBJ)\crypto-rand.obj \ $(OBJ)\data.obj \ + $(OBJ)\db_plugin.obj \ $(OBJ)\deprecated.obj \ $(OBJ)\digest.obj \ $(OBJ)\dll.obj \ @@ -329,8 +330,13 @@ $(OBJ)\k524_err.c $(OBJ)\k524_err.h: k524_err.et #---------------------------------------------------------------------- # libkrb5 -$(LIBKRB5): $(libkrb5_OBJS) $(libkrb5_gen_OBJS) - $(LIBCON) +$(LIBKRB5): $(libkrb5_OBJS) $(libkrb5_gen_OBJS) + $(LIBCON_C) -OUT:$@ $(LIBHEIMBASE) @<< +$(libkrb5_OBJS: = +) +$(libkrb5_gen_OBJS: = +) +<< all:: $(LIBKRB5) diff --git a/lib/krb5/aname_to_localname.c b/lib/krb5/aname_to_localname.c index 5cde83352..5a47268fc 100644 --- a/lib/krb5/aname_to_localname.c +++ b/lib/krb5/aname_to_localname.c @@ -33,93 +33,16 @@ #include "krb5_locl.h" #include "an2ln_plugin.h" +#include "db_plugin.h" /* Default plugin (DB using binary search of sorted text file) follows */ -static krb5_error_code -an2ln_def_plug_init(krb5_context context, void **ctx) -{ - *ctx = NULL; - return 0; -} +static krb5_error_code an2ln_def_plug_init(krb5_context, void **); +static void an2ln_def_plug_fini(void *); +static krb5_error_code an2ln_def_plug_an2ln(void *, krb5_context, const char *, + krb5_const_principal, set_result_f, + void *); -static void -an2ln_def_plug_fini(void *ctx) -{ -} - -static krb5_error_code -an2ln_def_plug_an2ln(void *plug_ctx, krb5_context context, - const char *rule, - krb5_const_principal aname, - set_result_f set_res_f, void *set_res_ctx) -{ - krb5_error_code ret; - const char *an2ln_db_fname; - const char *ext; - bsearch_file_handle bfh = NULL; - char *unparsed = NULL; - char *value = NULL; - - if (strncmp(rule, "DB:", strlen("DB:") != 0)) - return KRB5_PLUGIN_NO_HANDLE; - - /* - * This plugin implements a binary search of a sorted text file - * (sorted in the C locale). We really need to know that the file - * is text, so we implement a trivial heuristic: the file name must - * end in .txt. - */ - an2ln_db_fname = &rule[strlen("DB:")]; - if (!*an2ln_db_fname) - return KRB5_PLUGIN_NO_HANDLE; - if (strlen(an2ln_db_fname) < (strlen(".txt") + 1)) - return KRB5_PLUGIN_NO_HANDLE; - ext = strrchr(an2ln_db_fname, '.'); - if (!ext || strcmp(ext, ".txt") != 0) - return KRB5_PLUGIN_NO_HANDLE; - - ret = krb5_unparse_name(context, aname, &unparsed); - if (ret) - return ret; - - ret = __bsearch_file_open(an2ln_db_fname, 0, 0, &bfh, NULL); - if (ret) { - krb5_set_error_message(context, ret, - N_("Couldn't open aname2lname-text-db", "")); - ret = KRB5_PLUGIN_NO_HANDLE; - goto cleanup; - } - - /* Binary search; file should be sorted (in C locale) */ - ret = __bsearch_file(bfh, unparsed, &value, NULL, NULL, NULL); - if (ret > 0) { - krb5_set_error_message(context, ret, - N_("Couldn't map principal name to username", "")); - ret = KRB5_PLUGIN_NO_HANDLE; - goto cleanup; - } else if (ret < 0) { - ret = KRB5_PLUGIN_NO_HANDLE; - goto cleanup; - } else { - /* ret == 0 -> found */ - if (!value || !*value) { - krb5_set_error_message(context, ret, - N_("Principal mapped to empty username", "")); - ret = KRB5_NO_LOCALNAME; - goto cleanup; - } - ret = set_res_f(set_res_ctx, value); - } - -cleanup: - if (bfh) - __bsearch_file_close(&bfh); - free(unparsed); - free(value); - return ret; -} - -krb5plugin_an2ln_ftable an2ln_def_plug = { +static krb5plugin_an2ln_ftable an2ln_def_plug = { 0, an2ln_def_plug_init, an2ln_def_plug_fini, @@ -425,3 +348,98 @@ krb5_aname_to_localname(krb5_context context, krb5_config_free_strings(rules); return ret; } + +static krb5_error_code +an2ln_def_plug_init(krb5_context context, void **ctx) +{ + *ctx = NULL; + return 0; +} + +static void +an2ln_def_plug_fini(void *ctx) +{ +} + +static heim_base_once_t sorted_text_db_init_once = HEIM_BASE_ONCE_INIT; + +static void +sorted_text_db_init_f(void *arg) +{ + (void) heim_db_register("sorted-text", NULL, &heim_sorted_text_file_dbtype); +} + +static krb5_error_code +an2ln_def_plug_an2ln(void *plug_ctx, krb5_context context, + const char *rule, + krb5_const_principal aname, + set_result_f set_res_f, void *set_res_ctx) +{ + krb5_error_code ret; + const char *an2ln_db_fname; + heim_db_t dbh = NULL; + heim_dict_t db_options; + heim_data_t k, v; + heim_error_t error; + char *unparsed = NULL; + char *value = NULL; + + _krb5_load_db_plugins(context); + heim_base_once_f(&sorted_text_db_init_once, NULL, sorted_text_db_init_f); + + if (strncmp(rule, "DB:", strlen("DB:") != 0)) + return KRB5_PLUGIN_NO_HANDLE; + + an2ln_db_fname = &rule[strlen("DB:")]; + if (!*an2ln_db_fname) + return KRB5_PLUGIN_NO_HANDLE; + + ret = krb5_unparse_name(context, aname, &unparsed); + if (ret) + return ret; + + db_options = heim_dict_create(11); + if (db_options != NULL) + heim_dict_set_value(db_options, HSTR("read-only"), + heim_number_create(1)); + dbh = heim_db_create(NULL, an2ln_db_fname, db_options, &error); + if (dbh == NULL) { + krb5_set_error_message(context, heim_error_get_code(error), + N_("Couldn't open aname2lname-text-db", "")); + ret = KRB5_PLUGIN_NO_HANDLE; + goto cleanup; + } + + /* Binary search; file should be sorted (in C locale) */ + k = heim_data_ref_create(unparsed, strlen(unparsed), NULL); + if (k == NULL) + return krb5_enomem(context); + v = heim_db_copy_value(dbh, NULL, k, &error); + heim_release(k); + if (v == NULL && error != NULL) { + krb5_set_error_message(context, heim_error_get_code(error), + N_("Lookup in aname2lname-text-db failed", "")); + ret = heim_error_get_code(error); + goto cleanup; + } else if (v == NULL) { + ret = KRB5_PLUGIN_NO_HANDLE; + goto cleanup; + } else { + /* found */ + if (heim_data_get_length(v) == 0) { + krb5_set_error_message(context, ret, + N_("Principal mapped to empty username", "")); + ret = KRB5_NO_LOCALNAME; + goto cleanup; + } + ret = set_res_f(set_res_ctx, heim_data_get_ptr(v)); + heim_release(v); + } + +cleanup: + heim_release(dbh); + free(unparsed); + free(value); + return ret; +} + diff --git a/lib/krb5/db_plugin.c b/lib/krb5/db_plugin.c new file mode 100644 index 000000000..2a47a2c61 --- /dev/null +++ b/lib/krb5/db_plugin.c @@ -0,0 +1,31 @@ +/* + */ + +#include "krb5_locl.h" +#include "db_plugin.h" + +/* Default plugin (DB using binary search of sorted text file) follows */ +static heim_base_once_t db_plugins_once = HEIM_BASE_ONCE_INIT; + +static krb5_error_code KRB5_LIB_CALL +db_plugins_plcallback(krb5_context context, const void *plug, void *plugctx, + void *userctx) +{ + return 0; +} + +static void +db_plugins_init(void *arg) +{ + krb5_context context = arg; + (void)_krb5_plugin_run_f(context, "krb5", KRB5_PLUGIN_DB, + KRB5_PLUGIN_DB_VERSION_0, 0, NULL, + db_plugins_plcallback); +} + +void +_krb5_load_db_plugins(krb5_context context) +{ + heim_base_once_f(&db_plugins_once, context, db_plugins_init); +} + diff --git a/lib/krb5/db_plugin.h b/lib/krb5/db_plugin.h new file mode 100644 index 000000000..cb319a290 --- /dev/null +++ b/lib/krb5/db_plugin.h @@ -0,0 +1,68 @@ +/* + * Copyright (c) 2011, Secure Endpoints Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/* $Id$ */ + +#ifndef HEIMDAL_KRB5_DB_PLUGIN_H +#define HEIMDAL_KRB5_DB_PLUGIN_H 1 + +#define KRB5_PLUGIN_DB "krb5_db_plug" +#define KRB5_PLUGIN_DB_VERSION_0 0 + +/** @struct krb5plugin_db_ftable_desc + * + * @brief Description of the krb5 DB plugin facility. + * + * The krb5_aname_to_lname(3) function's DB rule is pluggable. The + * plugin is named KRB5_PLUGIN_DB ("krb5_db_plug"), with a single minor + * version, KRB5_PLUGIN_DB_VERSION_0 (0). + * + * The plugin consists of a data symbol referencing a structure of type + * krb5plugin_db_ftable_desc, with three fields: + * + * @param init Plugin initialization function (see krb5-plugin(7)) + * + * @param minor_version The plugin minor version number (0) + * + * @param fini Plugin finalization function + * + * The init entry point is expected to call heim_db_register(). The + * fini entry point is expected to do nothing. + * + * @ingroup krb5_support + */ +typedef struct krb5plugin_db_ftable_desc { + int minor_version; + krb5_error_code (*init)(krb5_context, void **); + void (*fini)(void *); +} krb5plugin_db_ftable; + +#endif /* HEIMDAL_KRB5_DB_PLUGIN_H */ + diff --git a/lib/krb5/krb5-plugin.7 b/lib/krb5/krb5-plugin.7 index 549463880..d704e6ee8 100644 --- a/lib/krb5/krb5-plugin.7 +++ b/lib/krb5/krb5-plugin.7 @@ -41,6 +41,7 @@ .In krb5.h .In krb5/an2ln_plugin.h .In krb5/ccache_plugin.h +.In krb5/db_plugin.h .In krb5/kuserok_plugin.h .In krb5/locate_plugin.h .In krb5/send_to_kdc_plugin.h @@ -48,8 +49,8 @@ Heimdal has a plugin interface. Plugins may be statically linked into Heimdal and registered via the .Xr krb5_plugin_register 3 -function, or they may be loaded from shared objects present in the -Heimdal plugins directories. +function, or they may be dynamically loaded from shared objects present +in the Heimdal plugins directories. .Pp Plugins consist of a C struct whose struct name is given in the associated header file, such as, for example, @@ -57,7 +58,7 @@ associated header file, such as, for example, and a pointer to which is either registered via .Xr krb5_plugin_register 3 or found in a shared object via a symbol lookup for the symbol name -defined in the associated header file (e.g., "kuserok-plugin" for the +defined in the associated header file (e.g., "kuserok" for the plugin for .Xr krb5_kuserok 3 ). @@ -81,16 +82,26 @@ be output through the init function's second argument. plugin's context to be destroyed, and returning void. .El .Pp -Each plugin type must add one or more fields to this struct following -the above three. Plugins are typically invoked in no particular order until -one succeeds or fails, or all return a special return value such as -KRB5_PLUGIN_NO_HANDLE to indicate that the plugin was not applicable. Most -plugin types obtain deterministic plugin behavior in spite of the -non-deterministic invokation order by, for example, invoking all plugins for -each "rule" and passing the rule to each plugin with the expectation that just -one plugin will match any given rule. +Each plugin type must add zero or more fields to this struct following +the above three. Plugins are typically invoked in no particular order +until one succeeds or fails, or all return a special return value such +as KRB5_PLUGIN_NO_HANDLE to indicate that the plugin was not applicable. +Most plugin types obtain deterministic plugin behavior in spite of the +non-deterministic invokation order by, for example, invoking all plugins +for each "rule" and passing the rule to each plugin with the expectation +that just one plugin will match any given rule. .Pp -The krb5-kuserok plugin adds a single field to its struct: a pointer to +There is a database plugin system intended for many of the uses of +databases in Heimdal. The plugin is expected to call +.Xr heim_db_register 3 +from its +.Va init +entry point to register a DB type. The DB plugin's +.Va fini +function must do nothing, and the plugin must not provide any other +entry points. +.Pp +The krb5_kuserok plugin adds a single field to its struct: a pointer to a function that implements kuserok functionality with the following form: .Bd -literal -offset indent diff --git a/lib/roken/NTMakefile b/lib/roken/NTMakefile index 0bfa87f4e..90162b5af 100644 --- a/lib/roken/NTMakefile +++ b/lib/roken/NTMakefile @@ -94,6 +94,7 @@ libroken_la_OBJS = \ $(OBJ)\strerror_r.obj \ $(OBJ)\strlcat.obj \ $(OBJ)\strlcpy.obj \ + $(OBJ)\strndup.obj \ $(OBJ)\strpool.obj \ $(OBJ)\strptime.obj \ $(OBJ)\strsep.obj \