Commit Graph

29373 Commits

Author SHA1 Message Date
Nicolas Williams
db7763ca7b asn1: X.681/682/683 magic handling of open types
Status:

 - And it works!

 - We have an extensive test based on decoding a rich EK certficate.

   This test exercises all of:

    - decoding
    - encoding with and without decoded open types
    - copying of decoded values with decoded open types
    - freeing of decoded values with decoded open types

   Valgrind finds no memory errors.

 - Added a manual page for the compiler.

 - rfc2459.asn1 now has all three primary PKIX types that we care about
   defined as in RFC5912, with IOS constraints and parameterization:

    - `Extension`       (embeds open type in an `OCTET STRING`)
    - `OtherName`       (embeds open type in an        `ANY`-like type)
    - `SingleAttribute` (embeds open type in an        `ANY`-like type)
    - `AttributeSet`    (embeds open type in a  `SET OF ANY`-like type)

   All of these use OIDs as the open type type ID field, but integer
   open type type ID fields are also supported (and needed, for
   Kerberos).

   That will cover every typed hole pattern in all our ASN.1 modules.

   With this we'll be able to automatically and recursively decode
   through all subject DN attributes even when the subject DN is a
   directoryName SAN, and subjectDirectoryAttributes, and all
   extensions, and all SANs, and all authorization-data elements, and
   PA-data, and...

   We're not really using `SingleAttribute` and `AttributeSet` yet
   because various changes are needed in `lib/hx509` for that.

 - `asn1_compile` builds and recognizes the subset of X.681/682/683 that
   we need for, and now use in, rfc2459.asn1.  It builds the necessary
   AST, generates the correct C types, and generates templating for
   object sets and open types!

 - See READMEs for details.

 - Codegen backend not tested; I won't make it implement automatic open
   type handling, but it should at least not crash by substituting
   `heim_any` for open types not embedded in `OCTET STRING`.

 - We're _really_ starting to have problems with the ITU-T ASN.1
   grammar and our version of it...

   Type names have to start with upper-case, value names with
   lower-case, but it's not enough to disambiguate.

   The fact the we've allowed value and type names to violate their
   respective start-with case rules is causing us trouble now that we're
   adding grammar from X.681/682/683, and we're going to have to undo
   that.

   In preparation for that I'm capitalizing the `heim_any` and
   `heim_any_set` types, and doing some additional cleanup, which
   requires changes to other parts of Heimdal (all in this same commit
   for now).

   Problems we have because of this:

    - We cannot IMPORT values into modules because we have no idea if a
      symbol being imported refers to a value or a type because the only
      clue we would have is the symbol's name, so we assume IMPORTed
      symbols are for types.

      This means we can't import OIDs, for example, which is super
      annoying.

      One thing we might be able to do here is mark imported symbols as
      being of an undetermined-but-not-undefined type, then coerce the
      symbol's type the first time it's used in a context where its type
      is inferred as type, value, object, object set, or class.  (Though
      since we don't generate C symbols for objects or classes, we won't
      be able to import them, especially since we need to know them at
      compile time and cannot defer their handling to link- or
      run-time.)

    - The `NULL` type name, and the `NULL` value name now cause two
      reduce/reduce conflicts via the `FieldSetting` production.

    - Various shift/reduce conflicts involving `NULL` values in
      non-top-level contexts (in constraints, for example).

 - Currently I have a bug where to disambiguate the grammar I have a
   CLASS_IDENTIFIER token that is all caps, while TYPE_IDENTIFIER must
   start with a capital but not be all caps, but this breaks Kerberos
   since all its types are all capitalized -- oof!

   To fix this I made it so class names have to be all caps and
   start with an underscore (ick).

TBD:

 - Check all the XXX comments and address them
 - Apply this treatment to Kerberos!  Automatic handling of authz-data
   sounds useful :)
 - Apply this treatment to PKCS#10 (CSRs) and other ASN.1 modules too.
 - Replace various bits of code in `lib/hx509/` with uses of this
   feature.
 - Add JER.
 - Enhance `hxtool` and `asn1_print`.

Getting there!
2021-02-28 18:13:08 -06:00
Nicolas Williams
89f97e8287 asn1: Improve XXX comment 2021-02-25 17:33:18 -06:00
Nicolas Williams
dbb8882971 asn1: Fix some of the primitive comparators
The comparators for BIT STRING and unconstrained INTEGER need help too.
2021-02-25 17:33:18 -06:00
Nicolas Williams
6747e16283 asn1: Use unsigned bitfields for named bitsets
Signed 1-bit bitfields are undefined in C.
2021-02-25 17:33:18 -06:00
Nicolas Williams
d5099a8d3b roken: Take https://github.com/dlfcn-w32/dlfcn-w32
This is a much better and more complete implementation of POSIX RTLD
functions than what we had.  In particular this gets us a more complete
`dladdr()` implementation.
2021-02-22 00:21:45 -06:00
Nicolas Williams
a775937e2e hx509: Let test_nist keep going on failure
Instead of reporting the first failure and exiting, it should report all
failures and then exit(1).
2021-02-22 00:21:45 -06:00
Nicolas Williams
22b8c3d448 asn1: Update README-X681.md 2021-02-10 14:51:38 -06:00
Nicolas Williams
24d26a2ddc asn1: Fix README-X681.md headers 2021-02-05 17:27:45 -06:00
Nicolas Williams
cd1663b13e asn1: Update README-X681.md 2021-02-05 17:21:54 -06:00
Nicolas Williams
6cb0a2d667 asn1: Update README-X681.md 2021-02-05 16:49:32 -06:00
Nicolas Williams
b0bf816ff8 asn1: Update README-X681.md 2021-02-02 17:35:54 -06:00
Nicolas Williams
b440b29394 clang-format w/ Mozilla style for generated code
Mozilla puts function names on column 1.  That's better.
2021-02-01 22:52:18 -06:00
Nicolas Williams
783b632f1f asn1: Teach template backend to DEFAULT 2021-02-01 22:30:33 -06:00
Nicolas Williams
3da24c19ad asn1: Template backend partial support for SETs
This adds enough support for SET { ... } types to the template backend
to allow the X.690 sample test to be run with the template backend.

Limitations:

 - For DER encoding, the SET { ... } members must be manually sorted by
   the module author.

 - Decoding of out-of-order encodings (BER) is not supported at this
   time.

These shortcomings will be addressed later.

Note that because we don't parse IMPORTed modules at this time, we can't
sort SET { ... } members at compile time if any of them out an
outer-most tag that the compiler cannot see without learning to parse
IMPORTed modules.
2021-02-01 22:30:33 -06:00
Nicolas Williams
3b8b9a797c asn1: Update ASN.1 IOS futures README 2021-02-01 22:28:41 -06:00
Nicolas Williams
6aefc255b6 roken: Fix rk_strrasvisx() bug 2021-01-26 22:34:36 -06:00
Nicolas Williams
a8205cacb8 asn1: Add README-X681.md (futures) 2021-01-26 22:34:36 -06:00
Nicolas Williams
cb1ccf50fd Revert "asn1: Don't use heim_any_set"
This reverts commit 7da32e1b68.

Needs more work.
2021-01-26 11:12:41 -06:00
Nicolas Williams
7da32e1b68 asn1: Don't use heim_any_set 2021-01-25 16:28:44 -06:00
Nicolas Williams
8ab671d470 asn1: Update README.temlate 2021-01-25 16:28:44 -06:00
Nicolas Williams
9b54accd4d asn1: Bare template sup. for SET{...} types
The regular ASN.1 compiler does NOT sort SET { ... } types' members by
tag, though it should.  It cannot because if a field is of an untagged
imported type, then the compiler won't know the field's tag because the
compiler does not read and parse IMPORTed modules.  At least the regular
ASN.1 compiler does handle out-of-order encodings on decode.

The template ASN.1 compiler did not even support SET { ... } types at
all.  With this commit the template ASN.1 compiler does, but still it
does not sort members on encode, and it does not decode out-of-
[definition-]order encodings.

A proper fix to these issues will require run-time sorting of SET
members on encode.  An even better fix will require making the compiler
able to read and parse more than one module in one run, that way it can
know all the things about IMPORTed types that it currently leaves to
run-time.
2021-01-25 16:28:44 -06:00
Nicolas Williams
35ad30b2b3 asn1/hx509: Undo workaround for fixed bug
Now that the ASN.1 compiler properly supports IMPLICIT tagging of named
CHOICE types (meaning: treat them as EXPLICIT tags), we can remove one
workaround for that.
2021-01-25 16:28:44 -06:00
Nicolas Williams
17b1e809ba asn1: Teach template compiler about units 2021-01-25 16:28:44 -06:00
Nicolas Williams
8fde460772 asn1: More IMPLICIT tag fixes (both compilers)
The template compiler was applying IMPLICIT tags to CHOICE types.  This
is very wrong, as the tag of a CHOICE's taken choice cannot be replaced
without making it impossible to figure out what the choice was.  An
example of this is GeneralName's directoryName, which is an IMPLICIT-
tagged CHOICE.

Separately, the non-template compiler was requiring inlining of
IMPLICIT-tagged CHOICEs, which also happens in GeneralName's
directoryName case:

```
    205 Name ::= CHOICE {
    206         rdnSequence  RDNSequence
    207 }
    ...
    287 GeneralName ::= CHOICE {
    288         otherName                       [0]     IMPLICIT -- OtherName --
    SEQUENCE {
    289                 type-id    OBJECT IDENTIFIER,
    290                 value      [0] EXPLICIT heim_any
    291         },
    292         rfc822Name                      [1]     IMPLICIT IA5String,
    293         dNSName                         [2]     IMPLICIT IA5String,
    294 --      x400Address                     [3]     IMPLICIT ORAddress,--
--->295         directoryName                   [4]     IMPLICIT -- Name -- CHOICE
    {
    296                 rdnSequence  RDNSequence
    297         },
    298 --      ediPartyName                    [5]     IMPLICIT EDIPartyName, --
    299         uniformResourceIdentifier       [6]     IMPLICIT IA5String,
    300         iPAddress                       [7]     IMPLICIT OCTET STRING,
    301         registeredID                    [8]     IMPLICIT OBJECT IDENTIFIER
    302 }
```

Anyways, that's fixed now, though changing that will require making
corresponding changes to `lib/hx509/`.

We're getting closer to parity between the two compilers.  The template
compiler is still missing support for `SET { ... }` types.  Speaking of
`SET { ... }`, the regular compiler generates code that uses `qsort()`
to sort the encoded values values of the members of such a set, but this
seems silly because the order of members is knowable at compile time, as
for DER and CER the order by the tags of the members, from lowest to
highest (see X.690, section 9.3 and X.680, section 8.6).  As it happens
using `qsort()` on the encodings of the members works, but it would be
be better to sort in `lib/asn1/asn1parse.y` and then not have to bother
anywhere else.  Sorting SETs at definition time will help keep the
tamplate compiler simple.  Not that we _need_ `SET { ... }` for anything
in-tree other than the X.690 sample...

While we're at it, let's note that the core of PKIX from the RFC
2459/3280/5280/5912 consists of *two* ASN.1 modules, one with
default-EXPLICIT tags, and one with default-IMPLICIT tags, and
Heimdal has these merged as a default-EXPLICIT tags module in
`lib/asn1/rfc2459.asn1`, with `IMPLICIT` added in by hand in all the
tags in the default-IMPLICIT tagged module.  This fixes one recently
added type from PKIX that didn't have `IMPLICIT` added in manually!
2021-01-24 20:24:01 -06:00
Nicolas Williams
0729692cc8 asn1: Templates work for IMPLICIT; add build opt
Finally.  We're almost at parity for the template compiler.

Now we have a build option to use templating:

    `./configure --enable-asn1-templating`

Tests fail if you build `rfc2459.asn1` with `--template`.

TBD: Figure out what differences remain between the two compilers, and
     fix the templating compiler accordingly, adding tests along the
     way.

Making IMPLICIT tags work in the templating compiler turned out to be a
simple fix: don't attempt to do anything clever about IMPLICIT tags in
the template generator in the compiler other than denoting them --
instead leave all the smarts about IMPLICIT tags to the interpreter.
This might be a very slight pessimization, but also a great
simplification.

The result is very elegant: when the interpreter finds an IMPLICIT
tag it then recurses to find the template for the body of the type
so-tagged, and evaluates that.  Much more elegant than the code
generated by the non-template compiler, not least for not needing
any additional temporary memory allocation.

With this we finally have parity in basic testing of the template
compiler.  Indeed, for IMPLICIT tags the template compiler and
interpreter might even be better because they support IMPLICIT tags
with BER lengths, whereas the non-template compiler doesn't (mostly
because `der_replace_tag()` needs to be changed to support it.

And, of course, the template compiler is simply superior in that it
produces smaller code and is *much* easier to work with because the
functions to interpret templates are small and simple.  Which means we
can add more functions to deal with other encoding rules fairly
trivially.  It should be possible to add all of these with very little
work, almost all of it localized to `lib/asn1/template.c`:

 - PER  Packed Encoding Rules [X.691]
 - XER  XML Encoding Rules    [X.693]
 - OER  Octet Encoding Rules  [X.696] (intended to replace PER)
 - JER  JSON Encoding Rules   [X.697] (doubles as visual representation)
 - GSER Generic String E.R.s  [RFC3641] (a visual representation)

 - XDR  External Data Repr.   [STD67][RFC4506]

       (XDR is *not* an ASN.1 encoding rules specification, but it's a
        *lot* like PER/OER but with 4-octet alignment, and is specified
        for the syntax equivalent (XDR) of only a subset of ASN.1 syntax
        and semantics.)

All we'd have to do is add variants of `_asn1_{length,encode,decode}()`
for each set of rules, then generate per-type stub functions that call
them (as we already do for DER).

We could then have an encoding rule transliteration program that takes a
`TypeName` and some representation of a value encoded by some encoding
rules, and outputs the same thing encoded by a different set of rules.
This would double as a pretty-printer and parser if we do add support
for JER and/or GSER.  It would find the template for the given type
using `dlsym()` against some shared object (possibly `libasn1` itself).

Whereas generating source code for C (or whatever language) for
additional ERs requires much more work.  Plus, templates are much
smaller, and the interpreter is tiny, which yields much smaller text and
much smaller CPU icache/dcache footprint, which yields better
performance in many cases.

As well, the template system should be much easier to port to other
languages.  Though in the cases of, e.g., Rust, it would require use of
`unsafe` in the interpreter, so in fact the inverse might be true: that
it's easier to generate safe Rust code than to implement a template
interpreter in Rust.  Similarly for Haskell, OCAML, etc.  But wherever
the template interpreter is easy to implement, it's a huge win.

Note that implementing OER and PER using the templates as they are
currently would be a bit of a challenge, as the interpreter would have
to first do a pass of each SEQUENCE/SET to determine the size and
layout of the OER/PER sequence/set preamble by counting the number of
OPTIONAL/DEFAULT members, BOOLEAN members, and extensibility markers
with extensions present.  We could always generate more entries to
encode precomputed preamble metadata.  We would also need to add a
template entry type for extensibility markers, which currently we do
not.
2021-01-23 17:48:12 -06:00
Nicolas Williams
44b56c485e asn1: Fix Makefile.am 2021-01-22 22:06:44 -06:00
Nicolas Williams
1febdcb954 asn1: Teach template interp. about IMPLICIT tags
The earlier fixes to the ASN.1 compiler for IMPLICIT tags did not
include the template interpreter.

TBD:

 - TESTImplicit encoding/decoding still fails due to a bug in the
   template generator.

 - There are missing cases in the template interpreter.  See XXX
   comments.
2021-01-22 13:47:08 -06:00
Nicolas Williams
f09c3a26e8 asn1: Template gen must emit include of asn1_err.h 2021-01-22 13:47:08 -06:00
Nicolas Williams
51d3cb376a asn1: Make templating less fragile: test it more
`lib/asn1/check-gen.c` almost works with templates, and is a pretty
extensive test.  The only thing that fails is everything to do with
IMPLICIT tags (so, `test_implicit()`).

So now we compile `lib/asn1/test.asn1` both, w/ and w/o templating, and
we build two programs from `lib/asn1/check-gen.c`: `check-gen` and
`check-gen-template`, respectively linking with the non-templated and
the templated compilation of `lib/asn1/test.asn1`.

Because the template compiler still doesn't support IMPLICIT tagging
well, we disable testing of IMPLICIT tags in `check-gen-template`.

This will make it much harder to break the template compiler in the
future.
2021-01-22 13:47:08 -06:00
Nicolas Williams
81195acafa asn1: Further IMPLICIT tagging fixes
Commit 89389bc7a (asn1: Fix long-standing IMPLICIT tagging brokenness)
was incomplete.  Removing the hacks in lib/asn1/cms.asn1 revealed this.

Now the ASN.1 compiler generates enums to indicate what is the class and
tag of each type.  This is needed so the decoder functions generated by
the compiler can know what tag to restore.

Now, too, the compiler does handle IMPLICIT tags whose encoded length is
different from that of the underlying type.

However, we now don't handle indefinite BER and non-DER definite lengths
(DCE) following IMPLICIT tags.  This would affect only CMS in-tree.
2021-01-20 21:04:34 -06:00
Nicolas Williams
6d85ba384f clang-format generated code for easier debugging 2021-01-20 18:56:03 -06:00
Nicolas Williams
37ab2a5308 hx509: Add DNSSRV and TCG SAN types and DN attrs
This adds hx509 API and hxtool(1) support for PermanentIdentifier,
HardwareModuleName, and DNSSRV SAN types, as well as for serialNumber,
TPMManufacturer, TPMModel, and TPMVersion DN attributes.
2021-01-15 13:21:18 -06:00
Nicolas Williams
dbb0463ca8 hx509: Add support for cert policies and mappings 2021-01-15 13:21:18 -06:00
Nicolas Williams
b193d75a15 hx509: Revamp name handling ahead of new SAN types 2021-01-15 13:21:18 -06:00
Nicolas Williams
989fd1199b hx509: Add TCG EK and DevID certs to test with
These are sample certificates from the Trusted Computing Group
Endorsement Key Credential Profile For TPM Family 2.0; Level 0.

 - lib/hx509/data/tcg-ek-cp.pem (Endorsement Key certificate)
 - lib/hx509/data/tcg-devid.pem (DevID certificate)

https://trustedcomputinggroup.org/wp-content/uploads/Credential_Profile_EK_V2.0_R14_published.pdf
https://trustedcomputinggroup.org/wp-content/uploads/TCG_IWG_EKCredentialProfile_v2p3_r2_pub.pdf
https://trustedcomputinggroup.org/wp-content/uploads/TCG_IWG_DevID_v1r2_02dec2020.pdf

This certificate came from the Trusted Computing Group Endorsement Key
(EK) Credential Profile [0], Appendix A, page 34.

[0] https://trustedcomputinggroup.org/wp-content/uploads/TCG_IWG_EKCredentialProfile_v2p3_r2_pub.pdf

Note that hxtool at this point both certificates, including all their
extensions, HardwareModule Name SAN, certificate policies, and the new
DN attributes TPMVersion, TPMModel, and TPMManufacturer, as shown below.

The work on the ASN.1 compiler helped.

The goal of this work will be to enable a Heimdal service for device
enrolment using TPMs.  More TCG profiling has to be done, and perhaps
some ECC work as well.  But Heimdal will mostly just be a relying party
and CA, and will not include the client side piece of enrolment.

$ cd build/lib/hx509
$ ./hxtool print --content PEM-FILE:../../../lib/hx509/data/tcg-ek-cp.pem
cert: 0
    friendly name:
    private key: no
    issuer:  "CN=ExampleCA"
    subject: ""
    serial: 01
    keyusage: keyEncipherment
subject name:
issuer name: CN=ExampleCA
Validity:
        notBefore 2014-01-15 15:40:50
        notAfter  2015-01-15 15:40:50
checking extension: authorityInfoAccess
        Critical not set on MUST
        type: 1.3.6.1.5.5.7.48.2
        dirname: URI: http://www.example.com/ExampleCA.crt
checking extension: keyUsage
checking extension: subjectAltName
        Critical set on MUST NOT
        directoryName: TPMVersion=id:00010023,TPMModel=ABCDEF123456,TPMManufacturer=id:54434700
checking extension: basicConstraints
        is NOT a CA
checking extension: cRLDistributionPoints
CRL Distribution Points:
Fullname:
   URI: http://www.example.com/ExampleCA.crl
checking extension: certificatePolicies
        Policy: 1.2.3.4
checking extension: authorityKeyIdentifier
        authority key id: 347767244C44AFE79E2AE0B24C69579524B33DDA
checking extension: extKeyUsage
        eku-0: 2.23.133.8.1
checking extension: subjectDirectoryAttributes
Doesn't have SubjectKeyIdentifier
$
$
$ ./hxtool print --content PEM-FILE:../../../lib/hx509/data/tcg-devid.pem
cert: 0
    friendly name:
    private key: no
    issuer:  "CN=ExampleCA"
    subject: ""
    serial: 01
    keyusage: keyEncipherment
subject name:
issuer name: CN=ExampleCA
Validity:
        notBefore 2014-01-15 15:40:50
        notAfter  2015-01-15 15:40:50
checking extension: authorityInfoAccess
        Critical not set on MUST
        type: 1.3.6.1.5.5.7.48.2
        dirname: URI: http://www.example.com/ExampleCA.crt
checking extension: keyUsage
checking extension: subjectAltName
        Critical set on MUST NOT
        directoryName: TPMVersion=id:00010023,TPMModel=ABCDEF123456,TPMManufacturer=id:54434700
        otherName: 1.3.6.1.5.5.7.8.4 HardwareModuleName 2.23.133.1.2:tpmserialnumber
checking extension: basicConstraints
        is NOT a CA
checking extension: cRLDistributionPoints
CRL Distribution Points:
Fullname:
   URI: http://www.example.com/ExampleCA.crl
checking extension: certificatePolicies
        Policy: 1.2.3.4
checking extension: authorityKeyIdentifier
        authority key id: 347767244C44AFE79E2AE0B24C69579524B33DDA
checking extension: extKeyUsage
        eku-0: 2.23.133.8.1
checking extension: subjectDirectoryAttributes
Doesn't have SubjectKeyIdentifier
$
2021-01-15 13:21:18 -06:00
Nicolas Williams
6aeab13f06 hdb: Use a CHOICE instead of ASN1_BAD_ID
Checking the error code of decoding a blob as an hdb_entry or
hdb_entry_alias to determine which of those the blob is depends on a
detail of the Heimdal ASN.1 compiler and library that shouldn't be
depended on.  Using a CHOICE adds no octets to the encoding:

    HDB-EntryOrAlias ::= CHOICE {
            entry       hdb_entry,
            alias       hdb_entry_alias
    }

since we're adding no additional tags and the two arms of the CHOICE
already differ in tag (hdb_entry's tag is a [UNIVERSAL Sequence] tag,
while hdb_entry_alias's is an [APPLICATION 0] tag).
2021-01-15 13:21:18 -06:00
Nicolas Williams
5cefb340ac roken: Make strpool more efficient
Grow the buffer more than absolutely needed, but not too fast.
2021-01-14 18:24:43 -06:00
Nicolas Williams
333af2051c asn1: Fix leaks in two tests 2021-01-13 20:17:58 -06:00
Nicolas Williams
7f1cfb0396 asn1: Add sample from X.690 Appendix A
This helped find a bug fixed in the preceding commit.

This also depends on the earlier fixes to IMPLICT tagging support, thus
implementing a test of that using a test vector from a standard.
2021-01-13 20:17:58 -06:00
Nicolas Williams
426adfa395 asn1: Handle named subtypes that are tagged
This:

    Foo ::= SEQUENCE { bar Bar }
    Bar ::= [APPLICATION 0] INTEGER

was crashing the compiler.
2021-01-13 20:17:58 -06:00
Nicolas Williams
adaf7c5fb9 asn1: Fix make clean target 2021-01-13 20:17:58 -06:00
Nicolas Williams
89389bc7a0 asn1: Fix long-standing IMPLICIT tagging brokenness
This commit _mostly_ fixes the Heimdal ASN.1 compiler to properly
support IMPLICIT tagging in most if not all the many cases where it
didn't already, as you could see in lib/asn1/canthandle.asn1 prior to
this commit.

This fix is a bit of a hack in that a proper fix would change the
function prototypes of the encode/decode/length functions generated by
the compiler to take an optional IMPLICIT tag to tag with instead of the
type they code.  That fix would not be localized to lib/asn1/ however,
and would change the API and ABI of generated code (which is mostly not
an ABI for Heimdal, but still, some external projects would have to make
changes).

Instead, for IMPLICIT tags we currently depend on the IMPLICIT tag and
the sub-type's tag having the same size -- this can be fixed with extra
allocation on the encoder side as we do on the decoder side, but we
might leave it for later.

The issue we're fixing manifested as:

  -- The [CONTEXT 0] tag in Bar below was turned into an EXPLICIT tag
  -- instead of an IMPLICIT one, netting the DER encoding for the `foo`
  -- member as:
  --    [CONTEXT 0] [UNIVERSAL Seq] [UNIVERSAL Int] <encoding of i>
  -- instead of the correct:
  --    [CONTEXT 0] [UNIVERSAL Int] <encoding of i>
  Foo ::= SEQUENCE { i INTEGER }
  Bar ::= SEQUENCE { foo [0] IMPLICIT Foo }

or

  Foo ::= INTEGER
  Bar ::= SEQUENCE { foo [0] IMPLICIT Foo } -- tag context 0 marked
                                            -- constructed!

I've reviewed this in part by reviewing the output of the compiler
before and after this change using this procedure:

 - Run an earlier version of the ASN.1 compiler output for all
   modules in lib/asn1/.  Save these in a different location.

 - Run this (or later) version of the ASN.1 compiler output for
   the same modules, adding --original-order for modules that
   have been manually sorted already (e.g., rfc2459.asn1).

 - Run clang-format on the saved and newest generated C source
   and header files.

 - Diff the generated output.  Substantial differences will
   relate to handling of IMPLICIT tagging.  These are
   particularly evident in the tcg.asn1 module, which uses a lot
   of those.

Later commits add test data (certificates with extensions that use
IMPLICIT tagging) taken from external specifications as well, which
exercise this fix.

Non-urgent brokenness yet to be fixed:

 - When the IMPLICIT tag and the tag of the underlying type require
   differing numbers of bytes to encode, the encoding and decoding will
   fail.  The prototypes of generated length_*() functions make it
   impossible to do much better.

 - SET OF <primitive> still crashes the compiler (not a new bug).

Futures:

 - Unwind hackery in cms.asn1 that worked around our lack of proper
   IMPLICIT tagging support.

Here are some of the generated code deltas one expects to see around
this commit:

$ git checkout $earlier_version
$ ./autogen.sh
$ mkdir build
$ cd build
$ ../configure ...
$ make -j4
$ make check
$ cd lib/asn1
$ for i in *.c; do
      [[ $i = asn1parse.? || $i = lex.? || $i = *.h ]] && continue
      clang-format -i $i $i
      cmp /tmp/save/$i $i && echo NO DIFFS: $i && continue; echo DIFF: $i
  done
NO DIFFS: asn1_cms_asn1.c
NO DIFFS: asn1_digest_asn1.c
NO DIFFS: asn1_err.c
NO DIFFS: asn1_krb5_asn1.c
/tmp/save/asn1_kx509_asn1.c asn1_kx509_asn1.c differ: byte 6433, line 264
DIFF: asn1_kx509_asn1.c
NO DIFFS: asn1_ocsp_asn1.c
NO DIFFS: asn1_pkcs10_asn1.c
/tmp/save/asn1_pkcs12_asn1.c asn1_pkcs12_asn1.c differ: byte 12934, line 455
DIFF: asn1_pkcs12_asn1.c
NO DIFFS: asn1_pkcs8_asn1.c
NO DIFFS: asn1_pkcs9_asn1.c
NO DIFFS: asn1_pkinit_asn1.c
/tmp/save/asn1_rfc2459_asn1.c asn1_rfc2459_asn1.c differ: byte 20193, line 532
DIFF: asn1_rfc2459_asn1.c
NO DIFFS: asn1_rfc4043_asn1.c
/tmp/save/asn1_rfc4108_asn1.c asn1_rfc4108_asn1.c differ: byte 595, line 26
DIFF: asn1_rfc4108_asn1.c
/tmp/save/asn1_tcg_asn1.c asn1_tcg_asn1.c differ: byte 31835, line 1229
DIFF: asn1_tcg_asn1.c
/tmp/save/asn1_test_asn1.c asn1_test_asn1.c differ: byte 384, line 21
DIFF: asn1_test_asn1.c
/tmp/save/test_template_asn1-template.c test_template_asn1-template.c differ: byte 650, line 20
DIFF: test_template_asn1-template.c
$
$ cd ../..
$ git checkout $newer_version
$ make -j4 && make check
$ cd lib/asn1
$ for i in *.[ch]; do
    [[ $i = asn1parse.? || $i = lex.? || $i = *.h ]] && continue
    clang-format -i $i $i
    cmp /tmp/save/$i $i && echo NO DIFFS: $i && continue
    diff -ubw /tmp/save/$i $i
  done | $PAGER

and one should see deltas such as the following:

 - a small enhancement to handling of OPTIONAL members:

     (data)->macData = calloc(1, sizeof(*(data)->macData));
     if ((data)->macData == NULL)
       goto fail;
     e = decode_PKCS12_MacData(p, len, (data)->macData, &l);
-    if (e) {
+    if (e == ASN1_MISSING_FIELD) {
       free((data)->macData);
       (data)->macData = NULL;
+    } else if (e) {
+      goto fail;
     } else {
       p += l;
       len -= l;
       ret += l;

 - more complete handling of DEFAULTed members:

     e = decode_FWReceiptVersion(p, len, &(data)->version, &l);
-    if (e)
+    if (e == ASN1_MISSING_FIELD) {
+      (data)->version = 1;
+    } else if (e) {
       goto fail;
-    p += l;
-    len -= l;
-    ret += l;
+    } else {
+      p += l;
+      len -= l;
+      ret += l;
+    }
     {

 - replacement of tags with implicit tags (encode side):

   /* targetUri */
   if ((data)->targetUri) {
     size_t Top_tag_oldret HEIMDAL_UNUSED_ATTRIBUTE = ret;
     ret = 0;
     e = encode_URIReference(p, len, (data)->targetUri, &l);
     if (e)
       return e;
     p -= l;
     len -= l;
     ret += l;

-    e = der_put_length_and_tag(p, len, ret, ASN1_C_CONTEXT, PRIM, 4, &l);
+    e = der_replace_tag(p, len, ASN1_C_CONTEXT, CONS, 4);
     if (e)
       return e;
     p -= l;
     len -= l;
     ret += l;

     ret += Top_tag_oldret;
   }

 - replacement of tags with implicit tags (decode side):

         strengthOfFunction_oldlen = len;
         if (strengthOfFunction_datalen > len) {
           e = ASN1_OVERRUN;
           goto fail;
         }
         len = strengthOfFunction_datalen;
-        e = decode_StrengthOfFunction(p, len, (data)->strengthOfFunction, &l);
-        if (e)
-          goto fail;
-        p += l;
-        len -= l;
-        ret += l;
+        {
+          unsigned char *pcopy;
+          pcopy = calloc(1, len);
+          if (pcopy == 0) {
+            e = ENOMEM;
+            goto fail;
+          }
+          memcpy(pcopy, p, len);
+          e = der_replace_tag(pcopy, len, ASN1_C_UNIV, PRIM, 0);
+          if (e)
+            goto fail;
+          e = decode_StrengthOfFunction(p, len, (data)->strengthOfFunction, &l);
+          if (e)
+            goto fail;
+          p += l;
+          len -= l;
+          ret += l;
+          free(pcopy);
+        }
         len = strengthOfFunction_oldlen - strengthOfFunction_datalen;
       }
     }
     {
       size_t profileOid_datalen, profileOid_oldlen;

 - correct determination of implicit tag constructed vs no for IMPLICT-
   tagged named primitive types:

     {
       size_t profileUri_datalen, profileUri_oldlen;
       Der_type profileUri_type;
       e = der_match_tag_and_length(p, len, ASN1_C_CONTEXT, &profileUri_type, 2,
                                    &profileUri_datalen, &l);
-      if (e == 0 && profileUri_type != PRIM) {
+      if (e == 0 && profileUri_type != CONS) {
         e = ASN1_BAD_ID;
       }
       if (e) {
         (data)->profileUri = NULL;
       } else {
         (data)->profileUri = calloc(1, sizeof(*(data)->profileUri));
         if ((data)->profileUri == NULL) {
           e = ENOMEM;
           goto fail;
         }

 - correct determination of length of IMPLICT-tagged OIDs:

   if ((data)->profileOid) {
     size_t Top_tag_oldret = ret;
     ret = 0;
     ret += der_length_oid((data)->profileOid);
+    ret += 1 + der_length_len(ret);
     ret += Top_tag_oldret;
   }

These deltas should be examined with the corresponding ASN.1 module at
hand, cross-referencing the source code to the ASN.1 type definitions
and manually applying X.690 rules to double-check the choices of
primitive vs. constructed tag, and the choices of when to replace tags
and when not.
2021-01-13 20:17:58 -06:00
Nicolas Williams
24543c4a31 asn1: Improve option handling; help msg
- Giving asn1_compile the name of an ASN.1 module w/o the ".asn1" stem
   will cause the compiler to add the ".asn1" stem, and it will cause
   the compiler to look for a ".opt" file as well.

 - The default C module name substring derivation from the .asn1 file
   name is improved.

 - There is now a --gen-name=NAME option for specifying the C module
   name substring.  This is useful for specifying that in a .opt file.

 - More options now have helpful usage messages.

This will allow simplification of lib/asn1/Makefile.am's invocations of
asn1_compile.

We may well end up requiring the automatic .opt file finding feature
when we eventualy add support for parsing multiple modules in a single
invocation for better support of IMPORTs.
2021-01-13 20:17:58 -06:00
Nicolas Williams
83d4c6ddb5 asn1: Circular types and Topo. sort declarations
Many external ASN.1 modules that we have imported over time define types
like this:

  Foo ::= SEQUENCE { bar Bar }
  Bar ::= SEQUENCE { aMember INTEGER }

and before this change one had to re-order the definitions so that the
one for `Bar` came first.  No more.

We can now have out of order definitions in ASN.1 modules and the
compiler will topologically sort output C type declarations so that one
no longer has to manually sort types in ASN.1 modules when importing
them.

Besides that, it is now possible to create circular data types using
OPTIONAL since we generate such fields as pointers (which can then be
pointers to incomplete struct declarations):

  Circular ::= SEQUENCE {
          name UTF8String,
          next Circular OPTIONAL
  }

Circular types aren't necessarily useful, but they have been used in the
past.  E.g., the rpc.mountd protocol uses a circular type as a linked
list -- it should just have used an array, of course, as that's
semantically equivalent but more space efficient in its encoding, but
the point is that such types exist out there.
2021-01-13 20:17:58 -06:00
Nicolas Williams
5b978c7437 asn1: Resolve RFCs 4043 and 4108 OIDs 2021-01-13 20:17:58 -06:00
Nicolas Williams
6923cf7f76 asn1: Handle dashes in OID names 2021-01-13 20:17:58 -06:00
Nicolas Williams
94746b4bf8 asn1: Add CPS bits from RFC5280 2021-01-13 20:17:58 -06:00
Nicolas Williams
727578f7b1 asn1: Add TCG module
This is in preparation for adding support for TPM-related functionality
in lib/hx509 and, eventually, in bx509d.
2021-01-13 20:17:58 -06:00
Nicolas Williams
936d8dd4ee asn1: Add SRVName to PKIX module
This is in preparation to add more support for DNSSRV SANs in lib/hx509.
2021-01-13 20:17:58 -06:00
Nicolas Williams
07d4b1fc74 asn1: Add compiler --enum-prefix=PREFIX option
C enum labels have to be globally unique.  ASN.1 module ENUMERATED and
INTEGER types with named values are not globally unique.  This means
that ASN.1 integer type value names and enumerations can cause conflicts
when compiled to C.

This new option allows the user to specify a prefix to apply to such
names.  Then this:

    Foo ::= ENUMERATED { v1 (0) }

can generate:

    typedef enum Foo {
      prefix_v1 = 0,
    } Foo;

instead of

    typedef enum Foo {
      v1 = 0,
    } Foo;

which is very likely to conflict.

TBD: Add option to use the type name as the prefix?
2021-01-13 20:17:58 -06:00