For now we just add new types that can auto-decode through open types
in CSRs, but we leave the originals as they were to avoid having to
update lib/hx509/req.c for now.
Our asn1_print, like OpenSSL's, just knows how to parse and dump DER.
Ours can attempt to decode OCTET STRING and IMPLICIT-tagged constructed
values as DER, which is very useful.
But _now_ it's even better. Now it knows about all types exported from
all ASN.1 modules in `lib/asn1/` in Heimdal, and if told to print as
some type, it will use the new printing interface to print JSON-like
representations of values:
```
$ ./asn1_print /tmp/t490/ek2.crt Certificate |
jq '.tbsCertificate.extensions[3]._extnValue[]._values'
[
{
"_type": "TPMSpecification",
"family": "2.0",
"level": "0",
"revision": "138"
}
]
[
{
"_type": "TPMSecurityAssertions",
"version": "0",
"fieldUpgradable": true,
"ekGenerationType": "655617",
"ekGenerationLocation": "655616",
"ekCertificateGenerationLocation": "655616",
"ccInfo": {
"_type": "CommonCriteriaMeasures",
"version": "3.1",
"assurancelevel": "4",
"evaluationStatus": "2",
"plus": true,
"strengthOfFunction": null,
"profileOid": null,
"profileUri": null,
"targetOid": null,
"targetUri": null
},
"fipsLevel": {
"_type": "FIPSLevel",
"version": "140-2",
"level": "2",
"plus": false
},
"iso9000Certified": false,
"iso9000Uri": null
}
]
```
Status:
- And it works!
- We have an extensive test based on decoding a rich EK certficate.
This test exercises all of:
- decoding
- encoding with and without decoded open types
- copying of decoded values with decoded open types
- freeing of decoded values with decoded open types
Valgrind finds no memory errors.
- Added a manual page for the compiler.
- rfc2459.asn1 now has all three primary PKIX types that we care about
defined as in RFC5912, with IOS constraints and parameterization:
- `Extension` (embeds open type in an `OCTET STRING`)
- `OtherName` (embeds open type in an `ANY`-like type)
- `SingleAttribute` (embeds open type in an `ANY`-like type)
- `AttributeSet` (embeds open type in a `SET OF ANY`-like type)
All of these use OIDs as the open type type ID field, but integer
open type type ID fields are also supported (and needed, for
Kerberos).
That will cover every typed hole pattern in all our ASN.1 modules.
With this we'll be able to automatically and recursively decode
through all subject DN attributes even when the subject DN is a
directoryName SAN, and subjectDirectoryAttributes, and all
extensions, and all SANs, and all authorization-data elements, and
PA-data, and...
We're not really using `SingleAttribute` and `AttributeSet` yet
because various changes are needed in `lib/hx509` for that.
- `asn1_compile` builds and recognizes the subset of X.681/682/683 that
we need for, and now use in, rfc2459.asn1. It builds the necessary
AST, generates the correct C types, and generates templating for
object sets and open types!
- See READMEs for details.
- Codegen backend not tested; I won't make it implement automatic open
type handling, but it should at least not crash by substituting
`heim_any` for open types not embedded in `OCTET STRING`.
- We're _really_ starting to have problems with the ITU-T ASN.1
grammar and our version of it...
Type names have to start with upper-case, value names with
lower-case, but it's not enough to disambiguate.
The fact the we've allowed value and type names to violate their
respective start-with case rules is causing us trouble now that we're
adding grammar from X.681/682/683, and we're going to have to undo
that.
In preparation for that I'm capitalizing the `heim_any` and
`heim_any_set` types, and doing some additional cleanup, which
requires changes to other parts of Heimdal (all in this same commit
for now).
Problems we have because of this:
- We cannot IMPORT values into modules because we have no idea if a
symbol being imported refers to a value or a type because the only
clue we would have is the symbol's name, so we assume IMPORTed
symbols are for types.
This means we can't import OIDs, for example, which is super
annoying.
One thing we might be able to do here is mark imported symbols as
being of an undetermined-but-not-undefined type, then coerce the
symbol's type the first time it's used in a context where its type
is inferred as type, value, object, object set, or class. (Though
since we don't generate C symbols for objects or classes, we won't
be able to import them, especially since we need to know them at
compile time and cannot defer their handling to link- or
run-time.)
- The `NULL` type name, and the `NULL` value name now cause two
reduce/reduce conflicts via the `FieldSetting` production.
- Various shift/reduce conflicts involving `NULL` values in
non-top-level contexts (in constraints, for example).
- Currently I have a bug where to disambiguate the grammar I have a
CLASS_IDENTIFIER token that is all caps, while TYPE_IDENTIFIER must
start with a capital but not be all caps, but this breaks Kerberos
since all its types are all capitalized -- oof!
To fix this I made it so class names have to be all caps and
start with an underscore (ick).
TBD:
- Check all the XXX comments and address them
- Apply this treatment to Kerberos! Automatic handling of authz-data
sounds useful :)
- Apply this treatment to PKCS#10 (CSRs) and other ASN.1 modules too.
- Replace various bits of code in `lib/hx509/` with uses of this
feature.
- Add JER.
- Enhance `hxtool` and `asn1_print`.
Getting there!
This is a much better and more complete implementation of POSIX RTLD
functions than what we had. In particular this gets us a more complete
`dladdr()` implementation.
This adds enough support for SET { ... } types to the template backend
to allow the X.690 sample test to be run with the template backend.
Limitations:
- For DER encoding, the SET { ... } members must be manually sorted by
the module author.
- Decoding of out-of-order encodings (BER) is not supported at this
time.
These shortcomings will be addressed later.
Note that because we don't parse IMPORTed modules at this time, we can't
sort SET { ... } members at compile time if any of them out an
outer-most tag that the compiler cannot see without learning to parse
IMPORTed modules.
The regular ASN.1 compiler does NOT sort SET { ... } types' members by
tag, though it should. It cannot because if a field is of an untagged
imported type, then the compiler won't know the field's tag because the
compiler does not read and parse IMPORTed modules. At least the regular
ASN.1 compiler does handle out-of-order encodings on decode.
The template ASN.1 compiler did not even support SET { ... } types at
all. With this commit the template ASN.1 compiler does, but still it
does not sort members on encode, and it does not decode out-of-
[definition-]order encodings.
A proper fix to these issues will require run-time sorting of SET
members on encode. An even better fix will require making the compiler
able to read and parse more than one module in one run, that way it can
know all the things about IMPORTed types that it currently leaves to
run-time.
Now that the ASN.1 compiler properly supports IMPLICIT tagging of named
CHOICE types (meaning: treat them as EXPLICIT tags), we can remove one
workaround for that.
The template compiler was applying IMPLICIT tags to CHOICE types. This
is very wrong, as the tag of a CHOICE's taken choice cannot be replaced
without making it impossible to figure out what the choice was. An
example of this is GeneralName's directoryName, which is an IMPLICIT-
tagged CHOICE.
Separately, the non-template compiler was requiring inlining of
IMPLICIT-tagged CHOICEs, which also happens in GeneralName's
directoryName case:
```
205 Name ::= CHOICE {
206 rdnSequence RDNSequence
207 }
...
287 GeneralName ::= CHOICE {
288 otherName [0] IMPLICIT -- OtherName --
SEQUENCE {
289 type-id OBJECT IDENTIFIER,
290 value [0] EXPLICIT heim_any
291 },
292 rfc822Name [1] IMPLICIT IA5String,
293 dNSName [2] IMPLICIT IA5String,
294 -- x400Address [3] IMPLICIT ORAddress,--
--->295 directoryName [4] IMPLICIT -- Name -- CHOICE
{
296 rdnSequence RDNSequence
297 },
298 -- ediPartyName [5] IMPLICIT EDIPartyName, --
299 uniformResourceIdentifier [6] IMPLICIT IA5String,
300 iPAddress [7] IMPLICIT OCTET STRING,
301 registeredID [8] IMPLICIT OBJECT IDENTIFIER
302 }
```
Anyways, that's fixed now, though changing that will require making
corresponding changes to `lib/hx509/`.
We're getting closer to parity between the two compilers. The template
compiler is still missing support for `SET { ... }` types. Speaking of
`SET { ... }`, the regular compiler generates code that uses `qsort()`
to sort the encoded values values of the members of such a set, but this
seems silly because the order of members is knowable at compile time, as
for DER and CER the order by the tags of the members, from lowest to
highest (see X.690, section 9.3 and X.680, section 8.6). As it happens
using `qsort()` on the encodings of the members works, but it would be
be better to sort in `lib/asn1/asn1parse.y` and then not have to bother
anywhere else. Sorting SETs at definition time will help keep the
tamplate compiler simple. Not that we _need_ `SET { ... }` for anything
in-tree other than the X.690 sample...
While we're at it, let's note that the core of PKIX from the RFC
2459/3280/5280/5912 consists of *two* ASN.1 modules, one with
default-EXPLICIT tags, and one with default-IMPLICIT tags, and
Heimdal has these merged as a default-EXPLICIT tags module in
`lib/asn1/rfc2459.asn1`, with `IMPLICIT` added in by hand in all the
tags in the default-IMPLICIT tagged module. This fixes one recently
added type from PKIX that didn't have `IMPLICIT` added in manually!
Finally. We're almost at parity for the template compiler.
Now we have a build option to use templating:
`./configure --enable-asn1-templating`
Tests fail if you build `rfc2459.asn1` with `--template`.
TBD: Figure out what differences remain between the two compilers, and
fix the templating compiler accordingly, adding tests along the
way.
Making IMPLICIT tags work in the templating compiler turned out to be a
simple fix: don't attempt to do anything clever about IMPLICIT tags in
the template generator in the compiler other than denoting them --
instead leave all the smarts about IMPLICIT tags to the interpreter.
This might be a very slight pessimization, but also a great
simplification.
The result is very elegant: when the interpreter finds an IMPLICIT
tag it then recurses to find the template for the body of the type
so-tagged, and evaluates that. Much more elegant than the code
generated by the non-template compiler, not least for not needing
any additional temporary memory allocation.
With this we finally have parity in basic testing of the template
compiler. Indeed, for IMPLICIT tags the template compiler and
interpreter might even be better because they support IMPLICIT tags
with BER lengths, whereas the non-template compiler doesn't (mostly
because `der_replace_tag()` needs to be changed to support it.
And, of course, the template compiler is simply superior in that it
produces smaller code and is *much* easier to work with because the
functions to interpret templates are small and simple. Which means we
can add more functions to deal with other encoding rules fairly
trivially. It should be possible to add all of these with very little
work, almost all of it localized to `lib/asn1/template.c`:
- PER Packed Encoding Rules [X.691]
- XER XML Encoding Rules [X.693]
- OER Octet Encoding Rules [X.696] (intended to replace PER)
- JER JSON Encoding Rules [X.697] (doubles as visual representation)
- GSER Generic String E.R.s [RFC3641] (a visual representation)
- XDR External Data Repr. [STD67][RFC4506]
(XDR is *not* an ASN.1 encoding rules specification, but it's a
*lot* like PER/OER but with 4-octet alignment, and is specified
for the syntax equivalent (XDR) of only a subset of ASN.1 syntax
and semantics.)
All we'd have to do is add variants of `_asn1_{length,encode,decode}()`
for each set of rules, then generate per-type stub functions that call
them (as we already do for DER).
We could then have an encoding rule transliteration program that takes a
`TypeName` and some representation of a value encoded by some encoding
rules, and outputs the same thing encoded by a different set of rules.
This would double as a pretty-printer and parser if we do add support
for JER and/or GSER. It would find the template for the given type
using `dlsym()` against some shared object (possibly `libasn1` itself).
Whereas generating source code for C (or whatever language) for
additional ERs requires much more work. Plus, templates are much
smaller, and the interpreter is tiny, which yields much smaller text and
much smaller CPU icache/dcache footprint, which yields better
performance in many cases.
As well, the template system should be much easier to port to other
languages. Though in the cases of, e.g., Rust, it would require use of
`unsafe` in the interpreter, so in fact the inverse might be true: that
it's easier to generate safe Rust code than to implement a template
interpreter in Rust. Similarly for Haskell, OCAML, etc. But wherever
the template interpreter is easy to implement, it's a huge win.
Note that implementing OER and PER using the templates as they are
currently would be a bit of a challenge, as the interpreter would have
to first do a pass of each SEQUENCE/SET to determine the size and
layout of the OER/PER sequence/set preamble by counting the number of
OPTIONAL/DEFAULT members, BOOLEAN members, and extensibility markers
with extensions present. We could always generate more entries to
encode precomputed preamble metadata. We would also need to add a
template entry type for extensibility markers, which currently we do
not.
The earlier fixes to the ASN.1 compiler for IMPLICIT tags did not
include the template interpreter.
TBD:
- TESTImplicit encoding/decoding still fails due to a bug in the
template generator.
- There are missing cases in the template interpreter. See XXX
comments.
`lib/asn1/check-gen.c` almost works with templates, and is a pretty
extensive test. The only thing that fails is everything to do with
IMPLICIT tags (so, `test_implicit()`).
So now we compile `lib/asn1/test.asn1` both, w/ and w/o templating, and
we build two programs from `lib/asn1/check-gen.c`: `check-gen` and
`check-gen-template`, respectively linking with the non-templated and
the templated compilation of `lib/asn1/test.asn1`.
Because the template compiler still doesn't support IMPLICIT tagging
well, we disable testing of IMPLICIT tags in `check-gen-template`.
This will make it much harder to break the template compiler in the
future.
Commit 89389bc7a (asn1: Fix long-standing IMPLICIT tagging brokenness)
was incomplete. Removing the hacks in lib/asn1/cms.asn1 revealed this.
Now the ASN.1 compiler generates enums to indicate what is the class and
tag of each type. This is needed so the decoder functions generated by
the compiler can know what tag to restore.
Now, too, the compiler does handle IMPLICIT tags whose encoded length is
different from that of the underlying type.
However, we now don't handle indefinite BER and non-DER definite lengths
(DCE) following IMPLICIT tags. This would affect only CMS in-tree.
This adds hx509 API and hxtool(1) support for PermanentIdentifier,
HardwareModuleName, and DNSSRV SAN types, as well as for serialNumber,
TPMManufacturer, TPMModel, and TPMVersion DN attributes.
