Excluded: libtomath and libedit files, most of which appear to be
testing or example code not involved in production, and which are
derived from an upstream that should perhaps have patches submitted
upstream instead.
fix https://github.com/heimdal/heimdal/issues/1111
5398425c introduced support for propagating ASN.1 default values to the emitted
JSON, but it neglected to quote string values, which caused ASN.1 parsing
errors. Correct this.
Record when a CHOICE field is promoted from IMPLICIT to EXPLICIT and convey
this in the ASN.1 compiler's JSON output, so that other tools (e.g. which have
a representation isomorphic to the original ASN.1) may use it.
First, we enlarge Member's val field to int64_t.
Then we warn about skipping 2int, int2, and parse units glue for such
things with too-large members.
And we error out when generating the template for such things with
>UINT32_MAX members.
What about too-negative members? That could be a thing for INTEGER /
ENUMERATED. We'll look at that later.
Prior to this commit only those C structs for SET and SEQUENCE types
could be decorated. Now those for CHOICE types also can be decorated.
We could further extend this to SET OF and SEQUENCE OF types if it
proves useful.
This commits allows `heim_object_t` as a type and causes the generated
code to use the `heim_retain()` and `heim_release()` functions for
copying and releasing values of such types.
Also, now one can have more than one decoration per-type.
If the decorated external type is a void *, don't include the decoration
header in the ASN.1 header, only in the template implementation. This allows
the copy constructor and destructor to be implementation private.
This adds support for asn1_compile --decorate=... variation that causes
decoration of an ASN.1 SET/SEQUENCE type with a field of a non-ASN.1
type.
This means we can now have an ASN.1 type to represent a request that can
then have a "hidden" field -- hidden in that it is neither encoded nor
decoded. This field will be copied and freed when the decoration is of
an ASN.1 type or of a external, C type that comes with copy constructor
and destructor functions. Decoration with a `void *` field which is
neither copied nor freed is also supported.
We may end up using this to, for example, replace the `hdb_entry_ex`
type by decorating `HDB_entry` with a C type that points to the `HDB` in
which the entry was found or to which it should be written.
The generated .x source and .hx header files are plain C source files.
Generate them as .c source files and avoid unnecessary file copying
and special makefile rules.
Change-Id: Ifc4bbe3c46dd357fdd642040ad964c7cfe1d395c
Instead of defining ASN1_LIB in each of the generated ASN1 source
files require that the generated source file be compiled with the
ASN1_LIB symbol defined (on Windows). All of the compiled objects
linked into the same library which might be accessing an ASN1 exported
symbol must have ASN1_LIB defined to ensure that the use of the
exported symbols will not be via an import.
Change-Id: Ie38c105d1d376419717fce0830dc28f5b82547d4
This option, `--decorate=TYPE-NAME:FIELD-TYPE:field-name[?]` allows one to add
a field to any struct generated by the ASN.1 compiler for any SET or SEQUENCE
type such that:
- the field will be freed by the `free_TYPE_NAME()` function
- the field will be copied by the `copy_TYPE_NAME()` function
- the field will not be printed by the `print_TYPE_NAME()` function
- the field will NOT be encoded or decoded
This is useful for internal bookkeeping.
The first use of this may well be for adding an optional field to
`Principal` where information about name attributes will be stored,
which will then allow us to have GSS name attributes for the krb5
mechanism w/o having to refactor the mechanism to use a different
structure for representing `gss_name_t` mechnames than the one currently
used (`Principal`; `krb5_principal` happens to be a typedef alias of
`Principal *`).
So w/o massive rototilling of the GSS krb5 mechanism we can have name
attributes, _and_ we'll also be able to have those in the krb5 API as
well w/o any massive rototilling there either.
Although not required to address bad code generation in
some versions of gcc 9 and 10, a coding style that requires
explicit comparison of the result to zero before use is
both clearer and would have avoided the generation of bad
code.
This change converts all use of cmp function usage from
```
if (strcmp(a, b) || !strcmp(c, d)) ...
```
to
```
if (strcmp(a, b) != 0 || strcmp(c, d)) == 0
```
for all C library cmp functions and related:
- strcmp(), strncmp()
- strcasecmp(), strncasecmp()
- stricmp(), strnicmp()
- memcmp()
Change-Id: Ic60c15e1e3a07e4faaf10648eefe3adae2543188
This reverts commit e27e056b45.
e27e056b45 was needed mainly for ENOTSUP.
ENOTSUP is not available in older C run-times.
Also, lib/roken has wrappers for the CRT allocator, but we don't need
those in lib/asn1 because all the functions generated by the compiler
effectively encapsulate the corresponding DLL's CRT's allocator.
This will be followed by a change to not use ENOTSUP.
roken ensures the correct headers are used for each platform,
ensures availability of non-portable constants (e.g. ENOTSUP),
and on Windows enforces a consistent source for memory management.
Change-Id: I31aa2935d0af9f3d9529166679d9eff35ccedfad
Our asn1_print, like OpenSSL's, just knows how to parse and dump DER.
Ours can attempt to decode OCTET STRING and IMPLICIT-tagged constructed
values as DER, which is very useful.
But _now_ it's even better. Now it knows about all types exported from
all ASN.1 modules in `lib/asn1/` in Heimdal, and if told to print as
some type, it will use the new printing interface to print JSON-like
representations of values:
```
$ ./asn1_print /tmp/t490/ek2.crt Certificate |
jq '.tbsCertificate.extensions[3]._extnValue[]._values'
[
{
"_type": "TPMSpecification",
"family": "2.0",
"level": "0",
"revision": "138"
}
]
[
{
"_type": "TPMSecurityAssertions",
"version": "0",
"fieldUpgradable": true,
"ekGenerationType": "655617",
"ekGenerationLocation": "655616",
"ekCertificateGenerationLocation": "655616",
"ccInfo": {
"_type": "CommonCriteriaMeasures",
"version": "3.1",
"assurancelevel": "4",
"evaluationStatus": "2",
"plus": true,
"strengthOfFunction": null,
"profileOid": null,
"profileUri": null,
"targetOid": null,
"targetUri": null
},
"fipsLevel": {
"_type": "FIPSLevel",
"version": "140-2",
"level": "2",
"plus": false
},
"iso9000Certified": false,
"iso9000Uri": null
}
]
```
Status:
- And it works!
- We have an extensive test based on decoding a rich EK certficate.
This test exercises all of:
- decoding
- encoding with and without decoded open types
- copying of decoded values with decoded open types
- freeing of decoded values with decoded open types
Valgrind finds no memory errors.
- Added a manual page for the compiler.
- rfc2459.asn1 now has all three primary PKIX types that we care about
defined as in RFC5912, with IOS constraints and parameterization:
- `Extension` (embeds open type in an `OCTET STRING`)
- `OtherName` (embeds open type in an `ANY`-like type)
- `SingleAttribute` (embeds open type in an `ANY`-like type)
- `AttributeSet` (embeds open type in a `SET OF ANY`-like type)
All of these use OIDs as the open type type ID field, but integer
open type type ID fields are also supported (and needed, for
Kerberos).
That will cover every typed hole pattern in all our ASN.1 modules.
With this we'll be able to automatically and recursively decode
through all subject DN attributes even when the subject DN is a
directoryName SAN, and subjectDirectoryAttributes, and all
extensions, and all SANs, and all authorization-data elements, and
PA-data, and...
We're not really using `SingleAttribute` and `AttributeSet` yet
because various changes are needed in `lib/hx509` for that.
- `asn1_compile` builds and recognizes the subset of X.681/682/683 that
we need for, and now use in, rfc2459.asn1. It builds the necessary
AST, generates the correct C types, and generates templating for
object sets and open types!
- See READMEs for details.
- Codegen backend not tested; I won't make it implement automatic open
type handling, but it should at least not crash by substituting
`heim_any` for open types not embedded in `OCTET STRING`.
- We're _really_ starting to have problems with the ITU-T ASN.1
grammar and our version of it...
Type names have to start with upper-case, value names with
lower-case, but it's not enough to disambiguate.
The fact the we've allowed value and type names to violate their
respective start-with case rules is causing us trouble now that we're
adding grammar from X.681/682/683, and we're going to have to undo
that.
In preparation for that I'm capitalizing the `heim_any` and
`heim_any_set` types, and doing some additional cleanup, which
requires changes to other parts of Heimdal (all in this same commit
for now).
Problems we have because of this:
- We cannot IMPORT values into modules because we have no idea if a
symbol being imported refers to a value or a type because the only
clue we would have is the symbol's name, so we assume IMPORTed
symbols are for types.
This means we can't import OIDs, for example, which is super
annoying.
One thing we might be able to do here is mark imported symbols as
being of an undetermined-but-not-undefined type, then coerce the
symbol's type the first time it's used in a context where its type
is inferred as type, value, object, object set, or class. (Though
since we don't generate C symbols for objects or classes, we won't
be able to import them, especially since we need to know them at
compile time and cannot defer their handling to link- or
run-time.)
- The `NULL` type name, and the `NULL` value name now cause two
reduce/reduce conflicts via the `FieldSetting` production.
- Various shift/reduce conflicts involving `NULL` values in
non-top-level contexts (in constraints, for example).
- Currently I have a bug where to disambiguate the grammar I have a
CLASS_IDENTIFIER token that is all caps, while TYPE_IDENTIFIER must
start with a capital but not be all caps, but this breaks Kerberos
since all its types are all capitalized -- oof!
To fix this I made it so class names have to be all caps and
start with an underscore (ick).
TBD:
- Check all the XXX comments and address them
- Apply this treatment to Kerberos! Automatic handling of authz-data
sounds useful :)
- Apply this treatment to PKCS#10 (CSRs) and other ASN.1 modules too.
- Replace various bits of code in `lib/hx509/` with uses of this
feature.
- Add JER.
- Enhance `hxtool` and `asn1_print`.
Getting there!
Commit 89389bc7a (asn1: Fix long-standing IMPLICIT tagging brokenness)
was incomplete. Removing the hacks in lib/asn1/cms.asn1 revealed this.
Now the ASN.1 compiler generates enums to indicate what is the class and
tag of each type. This is needed so the decoder functions generated by
the compiler can know what tag to restore.
Now, too, the compiler does handle IMPLICIT tags whose encoded length is
different from that of the underlying type.
However, we now don't handle indefinite BER and non-DER definite lengths
(DCE) following IMPLICIT tags. This would affect only CMS in-tree.
This commit _mostly_ fixes the Heimdal ASN.1 compiler to properly
support IMPLICIT tagging in most if not all the many cases where it
didn't already, as you could see in lib/asn1/canthandle.asn1 prior to
this commit.
This fix is a bit of a hack in that a proper fix would change the
function prototypes of the encode/decode/length functions generated by
the compiler to take an optional IMPLICIT tag to tag with instead of the
type they code. That fix would not be localized to lib/asn1/ however,
and would change the API and ABI of generated code (which is mostly not
an ABI for Heimdal, but still, some external projects would have to make
changes).
Instead, for IMPLICIT tags we currently depend on the IMPLICIT tag and
the sub-type's tag having the same size -- this can be fixed with extra
allocation on the encoder side as we do on the decoder side, but we
might leave it for later.
The issue we're fixing manifested as:
-- The [CONTEXT 0] tag in Bar below was turned into an EXPLICIT tag
-- instead of an IMPLICIT one, netting the DER encoding for the `foo`
-- member as:
-- [CONTEXT 0] [UNIVERSAL Seq] [UNIVERSAL Int] <encoding of i>
-- instead of the correct:
-- [CONTEXT 0] [UNIVERSAL Int] <encoding of i>
Foo ::= SEQUENCE { i INTEGER }
Bar ::= SEQUENCE { foo [0] IMPLICIT Foo }
or
Foo ::= INTEGER
Bar ::= SEQUENCE { foo [0] IMPLICIT Foo } -- tag context 0 marked
-- constructed!
I've reviewed this in part by reviewing the output of the compiler
before and after this change using this procedure:
- Run an earlier version of the ASN.1 compiler output for all
modules in lib/asn1/. Save these in a different location.
- Run this (or later) version of the ASN.1 compiler output for
the same modules, adding --original-order for modules that
have been manually sorted already (e.g., rfc2459.asn1).
- Run clang-format on the saved and newest generated C source
and header files.
- Diff the generated output. Substantial differences will
relate to handling of IMPLICIT tagging. These are
particularly evident in the tcg.asn1 module, which uses a lot
of those.
Later commits add test data (certificates with extensions that use
IMPLICIT tagging) taken from external specifications as well, which
exercise this fix.
Non-urgent brokenness yet to be fixed:
- When the IMPLICIT tag and the tag of the underlying type require
differing numbers of bytes to encode, the encoding and decoding will
fail. The prototypes of generated length_*() functions make it
impossible to do much better.
- SET OF <primitive> still crashes the compiler (not a new bug).
Futures:
- Unwind hackery in cms.asn1 that worked around our lack of proper
IMPLICIT tagging support.
Here are some of the generated code deltas one expects to see around
this commit:
$ git checkout $earlier_version
$ ./autogen.sh
$ mkdir build
$ cd build
$ ../configure ...
$ make -j4
$ make check
$ cd lib/asn1
$ for i in *.c; do
[[ $i = asn1parse.? || $i = lex.? || $i = *.h ]] && continue
clang-format -i $i $i
cmp /tmp/save/$i $i && echo NO DIFFS: $i && continue; echo DIFF: $i
done
NO DIFFS: asn1_cms_asn1.c
NO DIFFS: asn1_digest_asn1.c
NO DIFFS: asn1_err.c
NO DIFFS: asn1_krb5_asn1.c
/tmp/save/asn1_kx509_asn1.c asn1_kx509_asn1.c differ: byte 6433, line 264
DIFF: asn1_kx509_asn1.c
NO DIFFS: asn1_ocsp_asn1.c
NO DIFFS: asn1_pkcs10_asn1.c
/tmp/save/asn1_pkcs12_asn1.c asn1_pkcs12_asn1.c differ: byte 12934, line 455
DIFF: asn1_pkcs12_asn1.c
NO DIFFS: asn1_pkcs8_asn1.c
NO DIFFS: asn1_pkcs9_asn1.c
NO DIFFS: asn1_pkinit_asn1.c
/tmp/save/asn1_rfc2459_asn1.c asn1_rfc2459_asn1.c differ: byte 20193, line 532
DIFF: asn1_rfc2459_asn1.c
NO DIFFS: asn1_rfc4043_asn1.c
/tmp/save/asn1_rfc4108_asn1.c asn1_rfc4108_asn1.c differ: byte 595, line 26
DIFF: asn1_rfc4108_asn1.c
/tmp/save/asn1_tcg_asn1.c asn1_tcg_asn1.c differ: byte 31835, line 1229
DIFF: asn1_tcg_asn1.c
/tmp/save/asn1_test_asn1.c asn1_test_asn1.c differ: byte 384, line 21
DIFF: asn1_test_asn1.c
/tmp/save/test_template_asn1-template.c test_template_asn1-template.c differ: byte 650, line 20
DIFF: test_template_asn1-template.c
$
$ cd ../..
$ git checkout $newer_version
$ make -j4 && make check
$ cd lib/asn1
$ for i in *.[ch]; do
[[ $i = asn1parse.? || $i = lex.? || $i = *.h ]] && continue
clang-format -i $i $i
cmp /tmp/save/$i $i && echo NO DIFFS: $i && continue
diff -ubw /tmp/save/$i $i
done | $PAGER
and one should see deltas such as the following:
- a small enhancement to handling of OPTIONAL members:
(data)->macData = calloc(1, sizeof(*(data)->macData));
if ((data)->macData == NULL)
goto fail;
e = decode_PKCS12_MacData(p, len, (data)->macData, &l);
- if (e) {
+ if (e == ASN1_MISSING_FIELD) {
free((data)->macData);
(data)->macData = NULL;
+ } else if (e) {
+ goto fail;
} else {
p += l;
len -= l;
ret += l;
- more complete handling of DEFAULTed members:
e = decode_FWReceiptVersion(p, len, &(data)->version, &l);
- if (e)
+ if (e == ASN1_MISSING_FIELD) {
+ (data)->version = 1;
+ } else if (e) {
goto fail;
- p += l;
- len -= l;
- ret += l;
+ } else {
+ p += l;
+ len -= l;
+ ret += l;
+ }
{
- replacement of tags with implicit tags (encode side):
/* targetUri */
if ((data)->targetUri) {
size_t Top_tag_oldret HEIMDAL_UNUSED_ATTRIBUTE = ret;
ret = 0;
e = encode_URIReference(p, len, (data)->targetUri, &l);
if (e)
return e;
p -= l;
len -= l;
ret += l;
- e = der_put_length_and_tag(p, len, ret, ASN1_C_CONTEXT, PRIM, 4, &l);
+ e = der_replace_tag(p, len, ASN1_C_CONTEXT, CONS, 4);
if (e)
return e;
p -= l;
len -= l;
ret += l;
ret += Top_tag_oldret;
}
- replacement of tags with implicit tags (decode side):
strengthOfFunction_oldlen = len;
if (strengthOfFunction_datalen > len) {
e = ASN1_OVERRUN;
goto fail;
}
len = strengthOfFunction_datalen;
- e = decode_StrengthOfFunction(p, len, (data)->strengthOfFunction, &l);
- if (e)
- goto fail;
- p += l;
- len -= l;
- ret += l;
+ {
+ unsigned char *pcopy;
+ pcopy = calloc(1, len);
+ if (pcopy == 0) {
+ e = ENOMEM;
+ goto fail;
+ }
+ memcpy(pcopy, p, len);
+ e = der_replace_tag(pcopy, len, ASN1_C_UNIV, PRIM, 0);
+ if (e)
+ goto fail;
+ e = decode_StrengthOfFunction(p, len, (data)->strengthOfFunction, &l);
+ if (e)
+ goto fail;
+ p += l;
+ len -= l;
+ ret += l;
+ free(pcopy);
+ }
len = strengthOfFunction_oldlen - strengthOfFunction_datalen;
}
}
{
size_t profileOid_datalen, profileOid_oldlen;
- correct determination of implicit tag constructed vs no for IMPLICT-
tagged named primitive types:
{
size_t profileUri_datalen, profileUri_oldlen;
Der_type profileUri_type;
e = der_match_tag_and_length(p, len, ASN1_C_CONTEXT, &profileUri_type, 2,
&profileUri_datalen, &l);
- if (e == 0 && profileUri_type != PRIM) {
+ if (e == 0 && profileUri_type != CONS) {
e = ASN1_BAD_ID;
}
if (e) {
(data)->profileUri = NULL;
} else {
(data)->profileUri = calloc(1, sizeof(*(data)->profileUri));
if ((data)->profileUri == NULL) {
e = ENOMEM;
goto fail;
}
- correct determination of length of IMPLICT-tagged OIDs:
if ((data)->profileOid) {
size_t Top_tag_oldret = ret;
ret = 0;
ret += der_length_oid((data)->profileOid);
+ ret += 1 + der_length_len(ret);
ret += Top_tag_oldret;
}
These deltas should be examined with the corresponding ASN.1 module at
hand, cross-referencing the source code to the ASN.1 type definitions
and manually applying X.690 rules to double-check the choices of
primitive vs. constructed tag, and the choices of when to replace tags
and when not.
Many external ASN.1 modules that we have imported over time define types
like this:
Foo ::= SEQUENCE { bar Bar }
Bar ::= SEQUENCE { aMember INTEGER }
and before this change one had to re-order the definitions so that the
one for `Bar` came first. No more.
We can now have out of order definitions in ASN.1 modules and the
compiler will topologically sort output C type declarations so that one
no longer has to manually sort types in ASN.1 modules when importing
them.
Besides that, it is now possible to create circular data types using
OPTIONAL since we generate such fields as pointers (which can then be
pointers to incomplete struct declarations):
Circular ::= SEQUENCE {
name UTF8String,
next Circular OPTIONAL
}
Circular types aren't necessarily useful, but they have been used in the
past. E.g., the rpc.mountd protocol uses a circular type as a linked
list -- it should just have used an array, of course, as that's
semantically equivalent but more space efficient in its encoding, but
the point is that such types exist out there.
C enum labels have to be globally unique. ASN.1 module ENUMERATED and
INTEGER types with named values are not globally unique. This means
that ASN.1 integer type value names and enumerations can cause conflicts
when compiled to C.
This new option allows the user to specify a prefix to apply to such
names. Then this:
Foo ::= ENUMERATED { v1 (0) }
can generate:
typedef enum Foo {
prefix_v1 = 0,
} Foo;
instead of
typedef enum Foo {
v1 = 0,
} Foo;
which is very likely to conflict.
TBD: Add option to use the type name as the prefix?
TRUE/FALSE may not be defined, so emitting those symbols when generating
code for `... BOOLEAN DEFAULT TRUE -- or FALSE` causes the generated
code to fail to compile. We could move the definitions of TRUE/FALSE to
krb5-types.h, or maybe we could have an asn1_compile option to force
inclusion of more than one header file so we can have headers defining
such constants. But the simplest fix is to just emit 1/0 instead of
TRUE/FALSE.
This explains why some BOOLEAN DEFAULT usages in PKIX are made OPTIONAL
in Heimdal.
The code generators were shifting "1LU" by (<< 32) and (<< 63) which
are undefined operations for a 32-bit integer. To ensure the integer
is 64-bit use "1ULL".
Change-Id: I062cae5638139a9fe51563f64b1964f87e2f49e3
This commit adds functions for finding OIDs by symbolic name, meaning by
their symbolic names given in the ASN.1 modules that define them.
TBD:
- Resolve OIDs to names.
- Support a file in /etc for additional OID resolution.
- Add support for resolving OID arc names.
the code generated by asn1_compile.exe includes a large number
of unreferenced local variables. The resulting warnings drown
out other potentially more serious warnings.
This change suppresses the C4101 warnings in the generated
source files.
Change-Id: I17642ff427f457c885b1eb0e62436f3bc9057ee1
This reverts commit cb6f7ea40e.
stdint.h can be included everywhere now that the Windows
platform generates and installs a stdint.h when Visual
Studio does not provide one.
Change-Id: Ia3cab28d7f5806203cd45227765debda54ac7472
Looks like they defined basename() in string.h and ntohs/htonl are
implemented in terms of __bswap16() which is a macro with tmp
variables and so one cannot embed one call to ntohs/htons in another.
Not good but we workaround this limitation in glibc.
