The regular ASN.1 compiler does NOT sort SET { ... } types' members by
tag, though it should. It cannot because if a field is of an untagged
imported type, then the compiler won't know the field's tag because the
compiler does not read and parse IMPORTed modules. At least the regular
ASN.1 compiler does handle out-of-order encodings on decode.
The template ASN.1 compiler did not even support SET { ... } types at
all. With this commit the template ASN.1 compiler does, but still it
does not sort members on encode, and it does not decode out-of-
[definition-]order encodings.
A proper fix to these issues will require run-time sorting of SET
members on encode. An even better fix will require making the compiler
able to read and parse more than one module in one run, that way it can
know all the things about IMPORTed types that it currently leaves to
run-time.
The template compiler was applying IMPLICIT tags to CHOICE types. This
is very wrong, as the tag of a CHOICE's taken choice cannot be replaced
without making it impossible to figure out what the choice was. An
example of this is GeneralName's directoryName, which is an IMPLICIT-
tagged CHOICE.
Separately, the non-template compiler was requiring inlining of
IMPLICIT-tagged CHOICEs, which also happens in GeneralName's
directoryName case:
```
205 Name ::= CHOICE {
206 rdnSequence RDNSequence
207 }
...
287 GeneralName ::= CHOICE {
288 otherName [0] IMPLICIT -- OtherName --
SEQUENCE {
289 type-id OBJECT IDENTIFIER,
290 value [0] EXPLICIT heim_any
291 },
292 rfc822Name [1] IMPLICIT IA5String,
293 dNSName [2] IMPLICIT IA5String,
294 -- x400Address [3] IMPLICIT ORAddress,--
--->295 directoryName [4] IMPLICIT -- Name -- CHOICE
{
296 rdnSequence RDNSequence
297 },
298 -- ediPartyName [5] IMPLICIT EDIPartyName, --
299 uniformResourceIdentifier [6] IMPLICIT IA5String,
300 iPAddress [7] IMPLICIT OCTET STRING,
301 registeredID [8] IMPLICIT OBJECT IDENTIFIER
302 }
```
Anyways, that's fixed now, though changing that will require making
corresponding changes to `lib/hx509/`.
We're getting closer to parity between the two compilers. The template
compiler is still missing support for `SET { ... }` types. Speaking of
`SET { ... }`, the regular compiler generates code that uses `qsort()`
to sort the encoded values values of the members of such a set, but this
seems silly because the order of members is knowable at compile time, as
for DER and CER the order by the tags of the members, from lowest to
highest (see X.690, section 9.3 and X.680, section 8.6). As it happens
using `qsort()` on the encodings of the members works, but it would be
be better to sort in `lib/asn1/asn1parse.y` and then not have to bother
anywhere else. Sorting SETs at definition time will help keep the
tamplate compiler simple. Not that we _need_ `SET { ... }` for anything
in-tree other than the X.690 sample...
While we're at it, let's note that the core of PKIX from the RFC
2459/3280/5280/5912 consists of *two* ASN.1 modules, one with
default-EXPLICIT tags, and one with default-IMPLICIT tags, and
Heimdal has these merged as a default-EXPLICIT tags module in
`lib/asn1/rfc2459.asn1`, with `IMPLICIT` added in by hand in all the
tags in the default-IMPLICIT tagged module. This fixes one recently
added type from PKIX that didn't have `IMPLICIT` added in manually!
Finally. We're almost at parity for the template compiler.
Now we have a build option to use templating:
`./configure --enable-asn1-templating`
Tests fail if you build `rfc2459.asn1` with `--template`.
TBD: Figure out what differences remain between the two compilers, and
fix the templating compiler accordingly, adding tests along the
way.
Making IMPLICIT tags work in the templating compiler turned out to be a
simple fix: don't attempt to do anything clever about IMPLICIT tags in
the template generator in the compiler other than denoting them --
instead leave all the smarts about IMPLICIT tags to the interpreter.
This might be a very slight pessimization, but also a great
simplification.
The result is very elegant: when the interpreter finds an IMPLICIT
tag it then recurses to find the template for the body of the type
so-tagged, and evaluates that. Much more elegant than the code
generated by the non-template compiler, not least for not needing
any additional temporary memory allocation.
With this we finally have parity in basic testing of the template
compiler. Indeed, for IMPLICIT tags the template compiler and
interpreter might even be better because they support IMPLICIT tags
with BER lengths, whereas the non-template compiler doesn't (mostly
because `der_replace_tag()` needs to be changed to support it.
And, of course, the template compiler is simply superior in that it
produces smaller code and is *much* easier to work with because the
functions to interpret templates are small and simple. Which means we
can add more functions to deal with other encoding rules fairly
trivially. It should be possible to add all of these with very little
work, almost all of it localized to `lib/asn1/template.c`:
- PER Packed Encoding Rules [X.691]
- XER XML Encoding Rules [X.693]
- OER Octet Encoding Rules [X.696] (intended to replace PER)
- JER JSON Encoding Rules [X.697] (doubles as visual representation)
- GSER Generic String E.R.s [RFC3641] (a visual representation)
- XDR External Data Repr. [STD67][RFC4506]
(XDR is *not* an ASN.1 encoding rules specification, but it's a
*lot* like PER/OER but with 4-octet alignment, and is specified
for the syntax equivalent (XDR) of only a subset of ASN.1 syntax
and semantics.)
All we'd have to do is add variants of `_asn1_{length,encode,decode}()`
for each set of rules, then generate per-type stub functions that call
them (as we already do for DER).
We could then have an encoding rule transliteration program that takes a
`TypeName` and some representation of a value encoded by some encoding
rules, and outputs the same thing encoded by a different set of rules.
This would double as a pretty-printer and parser if we do add support
for JER and/or GSER. It would find the template for the given type
using `dlsym()` against some shared object (possibly `libasn1` itself).
Whereas generating source code for C (or whatever language) for
additional ERs requires much more work. Plus, templates are much
smaller, and the interpreter is tiny, which yields much smaller text and
much smaller CPU icache/dcache footprint, which yields better
performance in many cases.
As well, the template system should be much easier to port to other
languages. Though in the cases of, e.g., Rust, it would require use of
`unsafe` in the interpreter, so in fact the inverse might be true: that
it's easier to generate safe Rust code than to implement a template
interpreter in Rust. Similarly for Haskell, OCAML, etc. But wherever
the template interpreter is easy to implement, it's a huge win.
Note that implementing OER and PER using the templates as they are
currently would be a bit of a challenge, as the interpreter would have
to first do a pass of each SEQUENCE/SET to determine the size and
layout of the OER/PER sequence/set preamble by counting the number of
OPTIONAL/DEFAULT members, BOOLEAN members, and extensibility markers
with extensions present. We could always generate more entries to
encode precomputed preamble metadata. We would also need to add a
template entry type for extensibility markers, which currently we do
not.
`lib/asn1/check-gen.c` almost works with templates, and is a pretty
extensive test. The only thing that fails is everything to do with
IMPLICIT tags (so, `test_implicit()`).
So now we compile `lib/asn1/test.asn1` both, w/ and w/o templating, and
we build two programs from `lib/asn1/check-gen.c`: `check-gen` and
`check-gen-template`, respectively linking with the non-templated and
the templated compilation of `lib/asn1/test.asn1`.
Because the template compiler still doesn't support IMPLICIT tagging
well, we disable testing of IMPLICIT tags in `check-gen-template`.
This will make it much harder to break the template compiler in the
future.
Commit 89389bc7a (asn1: Fix long-standing IMPLICIT tagging brokenness)
was incomplete. Removing the hacks in lib/asn1/cms.asn1 revealed this.
Now the ASN.1 compiler generates enums to indicate what is the class and
tag of each type. This is needed so the decoder functions generated by
the compiler can know what tag to restore.
Now, too, the compiler does handle IMPLICIT tags whose encoded length is
different from that of the underlying type.
However, we now don't handle indefinite BER and non-DER definite lengths
(DCE) following IMPLICIT tags. This would affect only CMS in-tree.
This helped find a bug fixed in the preceding commit.
This also depends on the earlier fixes to IMPLICT tagging support, thus
implementing a test of that using a test vector from a standard.
We turn on a few extra warnings and fix the fallout that occurs
when building with --enable-developer. Note that we get different
warnings on different machines and so this will be a work in
progress. So far, we have built on NetBSD/amd64 5.99.64 (which
uses gcc 4.5.3) and Ubuntu 10.04.3 LTS (which uses gcc 4.4.3).
Notably, we fixed
1. a lot of missing structure initialisers,
2. unchecked return values for functions that glibc
marks as __attribute__((warn-unused-result)),
3. made minor modifications to slc and asn1_compile
which can generate code which generates warnings,
and
4. a few stragglers here and there.
We turned off the extended warnings for many programs in appl/ as
they are nearing the end of their useful lifetime, e.g. rsh, rcp,
popper, ftp and telnet.
Interestingly, glibc's strncmp() macro needed to be worked around
whereas the function calls did not.
We have not yet tried this on 32 bit platforms, so there will be
a few more warnings when we do.
ASN.1 INTEGERs will now compile to C int64_t or uint64_t, depending
on whether the constraint ranges include numbers that cannot be
represented in 32-bit ints and whether they include negative
numbers.
Template backend support included. check-template is now built with
--template, so we know we're testing it.
Tests included.
Highlighs for the compiler is support for CHOICE and in general better
support for tags. This compiler support most of what is needed for
PK-INIT, LDAP, X.509, PKCS-12 and many other protocols.
git-svn-id: svn://svn.h5l.se/heimdal/trunk/heimdal@15617 ec53bebd-3082-4978-b11e-865c3cabbd6b
