asn1: Bare template sup. for SET{...} types

The regular ASN.1 compiler does NOT sort SET { ... } types' members by
tag, though it should.  It cannot because if a field is of an untagged
imported type, then the compiler won't know the field's tag because the
compiler does not read and parse IMPORTed modules.  At least the regular
ASN.1 compiler does handle out-of-order encodings on decode.

The template ASN.1 compiler did not even support SET { ... } types at
all.  With this commit the template ASN.1 compiler does, but still it
does not sort members on encode, and it does not decode out-of-
[definition-]order encodings.

A proper fix to these issues will require run-time sorting of SET
members on encode.  An even better fix will require making the compiler
able to read and parse more than one module in one run, that way it can
know all the things about IMPORTed types that it currently leaves to
run-time.

lib/asn1/Makefile.am

@@ -316,10 +316,8 @@ digest_asn1_files: asn1_compile$(EXEEXT) $(srcdir)/digest.asn1
 kx509_asn1_files: asn1_compile$(EXEEXT) $(srcdir)/kx509.asn1
 	$(ASN1_COMPILE) --one-code-file $(TEMPLATE_OPTION) $(srcdir)/kx509.asn1 kx509_asn1 || (rm -f kx509_asn1_files ; exit 1)
 
-# The template compiler doesn't support SET { ... } types yet, so it can't be
-# used to compile x690sample.asn1
 x690sample_asn1_files: asn1_compile$(EXEEXT) $(srcdir)/x690sample.asn1
-	$(ASN1_COMPILE) --one-code-file $(srcdir)/x690sample.asn1 x690sample_asn1 || (rm -f x690sample_asn1_files ; exit 1)
+	$(ASN1_COMPILE) --one-code-file $(TEMPLATE_OPTION) $(srcdir)/x690sample.asn1 x690sample_asn1 || (rm -f x690sample_asn1_files ; exit 1)
 
 test_template_asn1_files: asn1_compile$(EXEEXT) $(srcdir)/test.asn1
 	$(ASN1_COMPILE) --one-code-file --template --sequence=TESTSeqOf $(srcdir)/test.asn1 test_template_asn1 || (rm -f test_template_asn1_files ; exit 1)

lib/asn1/canthandle.asn1

@@ -2,6 +2,11 @@
 
 CANTHANDLE DEFINITIONS ::= BEGIN
 
+--  Currently the compiler handles SET { ... } types, but does NOT sort
+--  their members as should be done in the case of conforming DER encoders.
+--  The workaround is to sort the members of such types manually in their
+--  definitions.  See X.690, section 10.3, and X.680, section 8.6 for details.
+
 --  Can't handle primitives in SET OF, causing the compiler to crash
 --  Workaround is to define a type that is only an integer and use that
 

lib/asn1/check-gen.c

@@ -1893,8 +1893,17 @@ test_x690sample(void)
 {
     /*
      * Taken from X.690, Appendix A, though sadly it's not specified whether
-     * it's in BER, DER, or CER, but it turns out to be DER since, as you can
-     * see below, we re-encode and get the same encoding back.
+     * it's in BER, DER, or CER, but it is clearly BER and neither DER nor CER
+     * because the tags of the members of the X690SamplePersonnelRecord type
+     * are not canonically sorted in the given sample.
+     *
+     * Our compiler does NOT canonically sort the members of SET { ... } types
+     * so it produces the same encoding after decoding this test vector.  That
+     * is clearly a bug given that we aim to output DER.
+     *
+     * The template compiler doesn't even decode SET { ... } values properly
+     * when their members are not in the same order as defined (but the regular
+     * compiler does).
      */
     X690SamplePersonnelRecord r;
     heim_octet_string os;

lib/asn1/template.c

@@ -269,6 +269,16 @@ _asn1_decode(const struct asn1_template *t, unsigned flags,
 	    int subflags = flags;
             int replace_tag = (t->tt & A1_FLAG_IMPLICIT) && is_tagged(t->ptr);
 
+            /*
+             * XXX If this type (chasing t->ptr through IMPLICIT tags, if this
+             * one is too, till we find a non-TTag) is a [UNIVERSAL SET] type,
+             * then we have to accept fields out of order.  For each field tag
+             * we see we'd have to do a linear search of the SET's template
+             * because it won't be sorted (or we could sort a copy and do a
+             * binary search on that, but these SETs will always be small so it
+             * won't be worthwhile).  We'll need a utility function to do all
+             * of this.
+             */
 	    ret = der_match_tag_and_length(p, len, A1_TAG_CLASS(t->tt),
 					   &dertype, A1_TAG_TAG(t->tt),
 					   &datalen, &l);
@@ -576,6 +586,16 @@ _asn1_encode(const struct asn1_template *t, unsigned char *p, size_t len, const
 	    size_t l, datalen;
             int replace_tag = 0;
 
+            /*
+             * XXX If this type (chasing t->ptr through IMPLICIT tags, if this
+             * one is too) till we find a non-TTag) is a [UNIVERSAL SET] type,
+             * then we have to sort [a copy of] its template by tag, then
+             * encode the SET using that sorted template.  These SETs will
+             * generally be small, so when they are we might want to allocate
+             * the copy on the stack and insertion sort it.  We'll need a
+             * utility function to do all of this.
+             */
+
 	    data = DPOC(data, t->offset);
 
 	    if (t->tt & A1_FLAG_OPTIONAL) {

lib/asn1/x690sample.asn1

@@ -1,6 +1,7 @@
 x690sample DEFINITIONS ::= BEGIN
 
--- This is taken from Appendix A of X.690.
+-- This is taken from Appendix A of X.690.  The same module is used by all
+-- X.690 series specifications of ASN.1 Encoding Rules.
 --
 -- This doesn't exercise every feature, like OPTIONAL, not really DEFAULT, not
 -- EXPLICIT tagging, extensibility markers, etc., but it exercises some hard
@@ -9,6 +10,43 @@ x690sample DEFINITIONS ::= BEGIN
 -- Because we don't yet have an option to add a namespace prefix to generated
 -- symbols, to avoid conflicts with rfc2459's Name we're prefixing the type
 -- names here manually.
+--
+-- WARNING: The encoding rules used for the sample encoding given in Appendix A
+--          of X.690, and used in lib/asn1/check-gen.c, is not specified in
+--          X.690!  It seems very likely that it is neither CER nor DER but BER
+--          because the tags in the X690SamplePersonnelRecord (a SET { ... })
+--          are not in canonical order:
+--
+--      APPL CONS tag 0 = 133 bytes [0]
+--        APPL CONS tag 1 = 16 bytes [1]
+--          UNIV PRIM VisibleString = "John"
+--          UNIV PRIM VisibleString = "P"
+--          UNIV PRIM VisibleString = "Smith"
+--    ->  CONTEXT CONS tag 0 = 10 bytes [0]
+--          UNIV PRIM VisibleString = "Director"
+--    ->  APPL PRIM tag 2 = 1 bytes [2] IMPLICIT content
+--        ...
+--
+--          The canonical ordering of members in SET { ... } types is by tag,
+--          with UNIVERSAL tags before APPLICATION tags, those before CONTEXT,
+--          and those before PRIVATE, and within each class from lowest to
+--          highest numeric tag value.  See X.680, section 8.6, which is
+--          referenced from X.690, section 10.3.
+--
+--          Here we can see that the `title` member should come _after_ the
+--          `number` member, but it does not.
+--
+--          Our test relies on our compiler producing the same test data when
+--          encoding the structure that the given test data decodes to.  That
+--          works here only because our compiler does NOT sort SET { ... }
+--          members as it should (since we always produce DER).
+--
+--          Sorting SET members in the compiler is hard currently because we
+--          don't parse imported modules, so we don't know the tags of imported
+--          types, so we can only sort at run-time, which we don't do.
+--
+--          There is an obvious workaround, however: sort the SET { ... }
+--          definition manually!
 
 X690SamplePersonnelRecord ::= [APPLICATION 0] IMPLICIT SET {
     name X690SampleName,