Created a CHANGELOG file for easy reference.

Compile the generated files with the same extra-strict
settings as the core, in order to detect problems in tests.

Update issue 32
Status: FixedInGit

CHANGELOG

@@ -0,0 +1,37 @@
+nanopb-0.1.6
+ Reorganize the field decoder interface (issue 2)
+ Improve performance in submessage decoding (issue 28)
+ Implement error messages in the decoder side (issue 7)
+ Extended testcases (alltypes test is now complete).
+ Fix some compiler warnings (issues 25, 26, 27, 32).
+
+nanopb-0.1.5
+ Fix bug in decoder with packed arrays (issue 23).
+ Extended testcases.
+ Fix some compiler warnings.
+
+nanopb-0.1.4
+ Add compile-time options for easy-to-use >255 field support.
+ Improve the detection of missing required fields.
+ Added example on how to handle union messages.
+ Fix generator error with .proto without messages.
+ Fix problems that stopped the code from compiling with some compilers.
+ Fix some compiler warnings.
+
+nanopb-0.1.3
+ Refactor the field encoder interface.
+ Improve generator error messages (issue 5)
+ Add descriptor.proto into the #include exclusion list
+ Fix some compiler warnings.
+
+nanopb-0.1.2
+ Make the generator to generate include for other .proto files (issue 4).
+ Fixed generator not working on Windows (issue 3)
+
+nanopb-0.1.1
+ Fixed bug in encoder with 'bytes' fields (issue 1).
+ Fixed a bug in the generator that caused a compiler error on sfixed32 and sfixed64 fields.
+ Extended testcases.
+
+nanopb-0.1.0
+ First stable release.

README

@@ -5,5 +5,7 @@ Homepage: http://kapsi.fi/~jpa/nanopb/
 To compile the library, you'll need these libraries:
 protobuf-compiler python-protobuf libprotobuf-dev
 
+The only runtime dependencies are memset() and memcpy().
+
 To run the tests, run make under the tests folder.
 If it completes without error, everything is fine.

docs/concepts.rst

@@ -50,6 +50,7 @@ There are a few generic rules for callback functions:
 #) Use state to store your own data, such as a file descriptor.
 #) *bytes_written* and *bytes_left* are updated by pb_write and pb_read.
 #) Your callback may be used with substreams. In this case *bytes_left*, *bytes_written* and *max_size* have smaller values than the original stream. Don't use these values to calculate pointers.
+#) Always read or write the full requested length of data. For example, POSIX *recv()* needs the *MSG_WAITALL* parameter to accomplish this.
 
 Output streams
 --------------
@@ -258,18 +259,14 @@ generates this field description array for the structure *Person_PhoneNumber*::
 Return values and error handling
 ================================
 
-Most functions in nanopb return bool: *true* means success, *false* means failure. If this is enough for you, skip this section.
+Most functions in nanopb return bool: *true* means success, *false* means failure. There is also some support for error messages for debugging purposes: the error messages go in *stream->errmsg*.
 
-For simplicity, nanopb doesn't define it's own error codes. This might be added if there is a compelling need for it. You can however deduce something about the error causes:
+The error messages help in guessing what is the underlying cause of the error. The most common error conditions are:
 
 1) Running out of memory. Because everything is allocated from the stack, nanopb can't detect this itself. Encoding or decoding the same type of a message always takes the same amount of stack space. Therefore, if it works once, it works always.
 2) Invalid field description. These are usually stored as constants, so if it works under the debugger, it always does.
-3) IO errors in your own stream callbacks. Because encoding/decoding stops at the first error, you can overwrite the *state* field in the struct and store your own error code there.
-4) Errors that happen in your callback functions. You can use the state field in the callback structure.
+3) IO errors in your own stream callbacks.
+4) Errors that happen in your callback functions.
 5) Exceeding the max_size or bytes_left of a stream.
 6) Exceeding the max_size of a string or array field
-7) Invalid protocol buffers binary message. It's not like you could recover from it anyway, so a simple failure should be enough.
-
-In my opinion, it is enough that 1. and 2. can be resolved using a debugger.
-
-However, you may be interested which of the remaining conditions caused the error. For 3. and 4., you can set and check the state. If you have to detect 5. and 6., you should convert the fields to callback type. Any remaining problem is of type 7.
+7) Invalid protocol buffers binary message.

docs/reference.rst

@@ -6,6 +6,28 @@ Nanopb: API reference
 
 .. contents ::
 
+Compilation options
+===================
+The following options can be specified using -D switch given to the C compiler:
+
+============================  ================================================================================================
+__BIG_ENDIAN__                 Set this if your platform stores integers and floats in big-endian format.
+                               Mixed-endian systems (different layout for ints and floats) are currently not supported.
+NANOPB_INTERNALS               Set this to expose the field encoder functions that are hidden since nanopb-0.1.3.
+PB_MAX_REQUIRED_FIELDS         Maximum number of required fields to check for presence. Default value is 64. Increases stack
+                               usage 1 byte per every 8 fields. Compiler warning will tell if you need this.
+PB_FIELD_16BIT                 Add support for tag numbers > 255 and fields larger than 255 bytes or 255 array entries.
+                               Increases code size 3 bytes per each field. Compiler error will tell if you need this.
+PB_FIELD_32BIT                 Add support for tag numbers > 65535 and fields larger than 65535 bytes or 65535 array entries.
+                               Increases code size 9 bytes per each field. Compiler error will tell if you need this.
+PB_NO_ERRMSG                   Disables the support for error messages; only error information is the true/false return value.
+                               Decreases the code size by a few hundred bytes.
+============================  ================================================================================================
+
+The PB_MAX_REQUIRED_FIELDS, PB_FIELD_16BIT and PB_FIELD_32BIT settings allow raising some datatype limits to suit larger messages.
+Their need is recognized automatically by C-preprocessor #if-directives in the generated .pb.h files. The default setting is to use
+the smallest datatypes (least resources used).
+
 pb.h
 ====
 
@@ -66,7 +88,7 @@ Describes a single structure field with memory position in relation to others. T
 :array_size:    Maximum number of entries in an array, if it is an array type.
 :ptr:           Pointer to default value for optional fields, or to submessage description for PB_LTYPE_SUBMESSAGE.
 
-The *uint8_t* datatypes limit the maximum size of a single item to 255 bytes and arrays to 255 items. Compiler will warn "Initializer too large for type" if the limits are exceeded. The types can be changed to larger ones if necessary.
+The *uint8_t* datatypes limit the maximum size of a single item to 255 bytes and arrays to 255 items. Compiler will give error if the values are too large. The types can be changed to larger ones by defining *PB_FIELD_16BIT*.
 
 pb_bytes_array_t
 ----------------
@@ -283,15 +305,32 @@ Read data from input stream. Always use this function, don't try to call the str
 
 End of file is signalled by *stream->bytes_left* being zero after pb_read returns false.
 
-pb_decode_varint
+pb_decode
+---------
+Read and decode all fields of a structure. Reads until EOF on input stream. ::
+
+    bool pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
+
+:stream:        Input stream to read from.
+:fields:        A field description array. Usually autogenerated.
+:dest_struct:   Pointer to structure where data will be stored.
+:returns:       True on success, false on IO error, on detectable errors in field description, if a field encoder returns false or if a required field is missing.
+
+In Protocol Buffers binary format, EOF is only allowed between fields. If it happens anywhere else, pb_decode will return *false*. If pb_decode returns false, you cannot trust any of the data in the structure.
+
+In addition to EOF, the pb_decode implementation supports terminating a message with a 0 byte. This is compatible with the official Protocol Buffers because 0 is never a valid field tag.
+
+For optional fields, this function applies the default value and sets *has_<field>* to false if the field is not present.
+
+pb_decode_noinit
 ----------------
-Read and decode a varint_ encoded integer. ::
+Same as `pb_decode`_, except does not apply the default values to fields. ::
 
-    bool pb_decode_varint(pb_istream_t *stream, uint64_t *dest);
+    bool pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
 
-:stream:        Input stream to read from. 1-10 bytes will be read.
-:dest:          Storage for the decoded integer. Value is undefined on error.
-:returns:       True on success, false if value exceeds uint64_t range or an IO error happens.
+(parameters are the same as for `pb_decode`_.)
+
+The destination structure should be filled with zeros before calling this function. Doing a *memset* manually can be slightly faster than using `pb_decode`_ if you don't need any default values.
 
 pb_skip_varint
 --------------
@@ -311,67 +350,66 @@ Skip a varint-length-prefixed string. This means skipping a value with wire type
 :stream:        Input stream to read from.
 :returns:       True on success, false on IO error or length exceeding uint32_t.
 
-pb_decode
----------
-Read and decode all fields of a structure. Reads until EOF on input stream. ::
+pb_decode_tag
+-------------
+Decode the tag that comes before field in the protobuf encoding::
 
-    bool pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
+    bool pb_decode_tag(pb_istream_t *stream, pb_wire_type_t *wire_type, int *tag, bool *eof);
 
 :stream:        Input stream to read from.
-:fields:        A field description array. Usually autogenerated.
-:dest_struct:   Pointer to structure where data will be stored.
-:returns:       True on success, false on IO error, on detectable errors in field description, if a field encoder returns false or if a required field is missing.
+:wire_type:     Pointer to variable where to store the wire type of the field.
+:tag:           Pointer to variable where to store the tag of the field.
+:eof:           Pointer to variable where to store end-of-file status.
+:returns:       True on success, false on error or EOF.
 
-In Protocol Buffers binary format, EOF is only allowed between fields. If it happens anywhere else, pb_decode will return *false*. If pb_decode returns false, you cannot trust any of the data in the structure.
+When the message (stream) ends, this function will return false and set *eof* to true. On other
+errors, *eof* will be set to false.
 
-In addition to EOF, the pb_decode implementation supports terminating a message with a 0 byte. This is compatible with the official Protocol Buffers because 0 is never a valid field tag.
-
-For optional fields, this function applies the default value and sets *has_<field>* to false if the field is not present.
-
-Because of memory concerns, the detection of missing required fields is not perfect if the structure contains more than 32 fields.
-
-.. sidebar:: Field decoders
-    
-    The functions with names beginning with *pb_dec_* are called field decoders. Each PB_LTYPE has an own field decoder, which handles translating from Protocol Buffers data to C data.
-
-    Each field decoder reads and decodes a single value. For arrays, the decoder is called repeatedly.
-
-    You can use the decoders from your callbacks. Just be aware that the pb_field_t passed to the callback is not directly compatible 
-    with the *varint* field decoders. Instead, you must create a new pb_field_t structure and set the data_size according to the data type 
-    you pass to *dest*, e.g. *field.data_size = sizeof(int);*. Other fields in the *pb_field_t* don't matter.
-
-    The field decoder interface is a bit messy as a result of the interface required inside the nanopb library.
-    Eventually they may be replaced by separate wrapper functions with a more friendly interface.
-
-pb_dec_varint
+pb_skip_field
 -------------
-Field decoder for PB_LTYPE_VARINT. ::
+Remove the data for a field from the stream, without actually decoding it::
 
-    bool pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+    bool pb_skip_field(pb_istream_t *stream, pb_wire_type_t wire_type);
+
+:stream:        Input stream to read from.
+:wire_type:     Type of field to skip.
+:returns:       True on success, false on IO error.
+
+.. sidebar:: Decoding fields manually
+    
+    The functions with names beginning with *pb_decode_* are used when dealing with callback fields. The typical reason for using callbacks is to have an array of unlimited size. In that case, `pb_decode`_ will call your callback function repeatedly, which can then store the values into e.g. filesystem in the order received in.
+
+    For decoding numeric (including enumerated and boolean) values, use `pb_decode_varint`_, `pb_decode_svarint`_, `pb_decode_fixed32`_ and `pb_decode_fixed64`_. They take a pointer to a 32- or 64-bit C variable, which you may then cast to smaller datatype for storage.
+
+    For decoding strings and bytes fields, the length has already been decoded. You can therefore check the total length in *stream->state* and read the data using `pb_read`_.
+
+    Finally, for decoding submessages in a callback, simply use `pb_decode`_ and pass it the *SubMessage_fields* descriptor array.
+
+pb_decode_varint
+----------------
+Read and decode a varint_ encoded integer. ::
+
+    bool pb_decode_varint(pb_istream_t *stream, uint64_t *dest);
 
 :stream:        Input stream to read from. 1-10 bytes will be read.
-:field:         Field description structure. Only *field->data_size* matters.
-:dest:          Pointer to destination integer. Must have size of *field->data_size* bytes.
-:returns:       True on success, false on IO errors or if `pb_decode_varint`_ fails.
+:dest:          Storage for the decoded integer. Value is undefined on error.
+:returns:       True on success, false if value exceeds uint64_t range or an IO error happens.
 
-This function first calls `pb_decode_varint`_. It then copies the first bytes of the 64-bit result value to *dest*, or on big endian architectures, the last bytes.
+pb_decode_svarint
+-----------------
+Similar to `pb_decode_varint`_, except that it performs zigzag-decoding on the value. This corresponds to the Protocol Buffers *sint32* and *sint64* datatypes. ::
 
-pb_dec_svarint
---------------
-Field decoder for PB_LTYPE_SVARINT. Similar to `pb_dec_varint`_, except that it performs zigzag-decoding on the value. ::
+    bool pb_decode_svarint(pb_istream_t *stream, int64_t *dest);
 
-    bool pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest);
+(parameters are the same as `pb_decode_varint`_)
 
-(parameters are the same as `pb_dec_varint`_)
+pb_decode_fixed32
+-----------------
+Decode a *fixed32*, *sfixed32* or *float* value. ::
 
-pb_dec_fixed32
---------------
-Field decoder for PB_LTYPE_FIXED32. ::
-
-    bool pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest);
+    bool pb_decode_fixed32(pb_istream_t *stream, void *dest);
 
 :stream:        Input stream to read from. 4 bytes will be read.
-:field:         Not used.
 :dest:          Pointer to destination *int32_t*, *uint32_t* or *float*.
 :returns:       True on success, false on IO errors.
 
@@ -379,9 +417,9 @@ This function reads 4 bytes from the input stream.
 On big endian architectures, it then reverses the order of the bytes.
 Finally, it writes the bytes to *dest*.
 
-pb_dec_fixed64
---------------
-Field decoder for PB_LTYPE_FIXED64. ::
+pb_decode_fixed64
+-----------------
+Decode a *fixed64*, *sfixed64* or *double* value. ::
 
     bool pb_dec_fixed(pb_istream_t *stream, const pb_field_t *field, void *dest);
 
@@ -390,53 +428,28 @@ Field decoder for PB_LTYPE_FIXED64. ::
 :dest:          Pointer to destination *int64_t*, *uint64_t* or *double*.
 :returns:       True on success, false on IO errors.
 
-Same as `pb_dec_fixed32`_, except this reads 8 bytes.
+Same as `pb_decode_fixed32`_, except this reads 8 bytes.
 
-pb_dec_bytes
-------------
-Field decoder for PB_LTYPE_BYTES. Reads a length-prefixed block of bytes. ::
+pb_make_string_substream
+------------------------
+Decode the length for a field with wire type *PB_WT_STRING* and create a substream for reading the data. ::
 
-    bool pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest);
+    bool pb_make_string_substream(pb_istream_t *stream, pb_istream_t *substream);
 
-**Note:** This is an internal function that is not useful in decoder callbacks. To read bytes fields in callbacks, use 
-*stream->bytes_left* and `pb_read`_.
+:stream:        Original input stream to read the length and data from.
+:substream:     New substream that has limited length. Filled in by the function.
+:returns:       True on success, false if reading the length fails.
 
-:stream:        Input stream to read from.
-:field:         Field description structure. Only *field->data_size* matters.
-:dest:          Pointer to a structure similar to pb_bytes_array_t.
-:returns:       True on success, false on IO error or if length exceeds the array size.
+This function uses `pb_decode_varint`_ to read an integer from the stream. This is interpreted as a number of bytes, and the substream is set up so that its `bytes_left` is initially the same as the length, and its callback function and state the same as the parent stream.
 
-This function expects a pointer to a structure with a *size_t* field at start, and a variable sized byte array after it. It will deduce the maximum size of the array from *field->data_size*.
+pb_close_string_substream
+-------------------------
+Close the substream created with `pb_make_string_substream`_. ::
 
-pb_dec_string
--------------
-Field decoder for PB_LTYPE_STRING. Reads a length-prefixed string. ::
+    void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream);
 
-    bool pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest);
-
-**Note:** This is an internal function that is not useful in decoder callbacks. To read string fields in callbacks, use 
-*stream->bytes_left* and `pb_read`_.
-
-:stream:        Input stream to read from.
-:field:         Field description structure. Only *field->data_size* matters.
-:dest:          Pointer to a character array of size *field->data_size*.
-:returns:       True on success, false on IO error or if length exceeds the array size.
-
-This function null-terminates the string when successful. On error, the contents of the destination array is undefined.
-
-pb_dec_submessage
------------------
-Field decoder for PB_LTYPE_SUBMESSAGE. Calls `pb_decode`_ to perform the actual decoding. ::
-
-    bool pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest)
-
-**Note:** This is an internal function that is not useful in decoder callbacks. To read submessage fields in callbacks, use 
-`pb_decode`_ directly.
-
-:stream:        Input stream to read from.
-:field:         Field description structure. Only *field->ptr* matters.
-:dest:          Pointer to the destination structure.
-:returns:       True on success, false on IO error or if `pb_decode`_ fails.
-
-The *field->ptr* should be a pointer to *pb_field_t* array describing the submessage.
+:stream:        Original input stream to read the length and data from.
+:substream:     Substream to close
 
+This function copies back the state from the substream to the parent stream.
+It must be called after done with the substream.

example/client.c

@@ -72,7 +72,7 @@ bool listdir(int fd, char *path)
     
     if (!pb_decode(&input, ListFilesResponse_fields, &response))
     {
-        fprintf(stderr, "Decoding failed.\n");
+        fprintf(stderr, "Decode failed: %s\n", PB_GET_ERROR(&input));
         return false;
     }
     

example/server.c

@@ -55,7 +55,7 @@ void handle_connection(int connfd)
     
     if (!pb_decode(&input, ListFilesRequest_fields, &request))
     {
-        printf("Decoding failed.\n");
+        printf("Decode failed: %s\n", PB_GET_ERROR(&input));
         return;
     }
     

example_unions/Makefile

@@ -0,0 +1,17 @@
+CFLAGS=-ansi -Wall -Werror -I .. -g -O0
+DEPS=../pb_decode.c ../pb_decode.h ../pb_encode.c ../pb_encode.h ../pb.h
+
+all: encode decode
+	./encode 1 | ./decode
+	./encode 2 | ./decode
+	./encode 3 | ./decode
+
+clean:
+	rm -f encode unionproto.pb.h unionproto.pb.c
+
+%: %.c $(DEPS) unionproto.pb.h unionproto.pb.c
+	$(CC) $(CFLAGS) -o $@ $< ../pb_decode.c ../pb_encode.c unionproto.pb.c
+
+unionproto.pb.h unionproto.pb.c: unionproto.proto ../generator/nanopb_generator.py
+	protoc -I. -I../generator -I/usr/include -ounionproto.pb $<
+	python ../generator/nanopb_generator.py unionproto.pb

example_unions/decode.c

@@ -0,0 +1,96 @@
+/* This program reads a message from stdin, detects its type and decodes it.
+ */
+ 
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+
+#include <pb_decode.h>
+#include "unionproto.pb.h"
+
+/* This function reads manually the first tag from the stream and finds the
+ * corresponding message type. It doesn't yet decode the actual message.
+ *
+ * Returns a pointer to the MsgType_fields array, as an identifier for the
+ * message type. Returns null if the tag is of unknown type or an error occurs.
+ */
+const pb_field_t* decode_unionmessage_type(pb_istream_t *stream)
+{
+    pb_wire_type_t wire_type;
+    uint32_t tag;
+    bool eof;
+
+    while (pb_decode_tag(stream, &wire_type, &tag, &eof))
+    {
+        if (wire_type == PB_WT_STRING)
+        {
+            const pb_field_t *field;
+            for (field = UnionMessage_fields; field->tag != 0; field++)
+            {
+                if (field->tag == tag && (field->type & PB_LTYPE_SUBMESSAGE))
+                {
+                    /* Found our field. */
+                    return field->ptr;
+                }
+            }
+        }
+        
+        /* Wasn't our field.. */
+        pb_skip_field(stream, wire_type);
+    }
+    
+    return NULL;
+}
+
+bool decode_unionmessage_contents(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
+{
+    pb_istream_t substream;
+    bool status;
+    if (!pb_make_string_substream(stream, &substream))
+        return false;
+    
+    status = pb_decode(&substream, fields, dest_struct);
+    pb_close_string_substream(stream, &substream);
+    return status;
+}
+
+int main()
+{
+    /* Read the data into buffer */
+    uint8_t buffer[512];
+    size_t count = fread(buffer, 1, sizeof(buffer), stdin);
+    pb_istream_t stream = pb_istream_from_buffer(buffer, count);
+    
+    const pb_field_t *type = decode_unionmessage_type(&stream);
+    bool status = false;
+    
+    if (type == MsgType1_fields)
+    {
+        MsgType1 msg = {};
+        status = decode_unionmessage_contents(&stream, MsgType1_fields, &msg);
+        printf("Got MsgType1: %d\n", msg.value);
+    }
+    else if (type == MsgType2_fields)
+    {
+        MsgType2 msg = {};
+        status = decode_unionmessage_contents(&stream, MsgType2_fields, &msg);
+        printf("Got MsgType2: %s\n", msg.value ? "true" : "false");
+    }
+    else if (type == MsgType3_fields)
+    {
+        MsgType3 msg = {};
+        status = decode_unionmessage_contents(&stream, MsgType3_fields, &msg);
+        printf("Got MsgType3: %d %d\n", msg.value1, msg.value2);    
+    }
+    
+    if (!status)
+    {
+        printf("Decode failed: %s\n", PB_GET_ERROR(&stream));
+        return 1;
+    }
+    
+    return 0;
+}
+
+
+

example_unions/encode.c

@@ -0,0 +1,85 @@
+/* This program takes a command line argument and encodes a message in
+ * one of MsgType1, MsgType2 or MsgType3.
+ */
+ 
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+
+#include <pb_encode.h>
+#include "unionproto.pb.h"
+
+/* This function is the core of the union encoding process. It handles
+ * the top-level pb_field_t array manually, in order to encode a correct
+ * field tag before the message. The pointer to MsgType_fields array is
+ * used as an unique identifier for the message type.
+ */
+bool encode_unionmessage(pb_ostream_t *stream, const pb_field_t messagetype[], const void *message)
+{
+    const pb_field_t *field;
+    for (field = UnionMessage_fields; field->tag != 0; field++)
+    {
+        if (field->ptr == messagetype)
+        {
+            /* This is our field, encode the message using it. */
+            if (!pb_encode_tag_for_field(stream, field))
+                return false;
+            
+            return pb_encode_submessage(stream, messagetype, message);
+        }
+    }
+    
+    /* Didn't find the field for messagetype */
+    return false;
+}
+
+int main(int argc, char **argv)
+{
+    if (argc != 2)
+    {
+        fprintf(stderr, "Usage: %s (1|2|3)\n", argv[0]);
+        return 1;
+    }
+    
+    uint8_t buffer[512];
+    pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+    
+    bool status = false;
+    int msgtype = atoi(argv[1]);
+    if (msgtype == 1)
+    {
+        /* Send message of type 1 */
+        MsgType1 msg = {42};
+        status = encode_unionmessage(&stream, MsgType1_fields, &msg);
+    }
+    else if (msgtype == 2)
+    {
+        /* Send message of type 2 */
+        MsgType2 msg = {true};
+        status = encode_unionmessage(&stream, MsgType2_fields, &msg);
+    }
+    else if (msgtype == 3)
+    {
+        /* Send message of type 3 */
+        MsgType3 msg = {3, 1415};
+        status = encode_unionmessage(&stream, MsgType3_fields, &msg);
+    }
+    else
+    {
+        fprintf(stderr, "Unknown message type: %d\n", msgtype);
+        return 2;
+    }
+    
+    if (!status)
+    {
+        fprintf(stderr, "Encoding failed!\n");
+        return 3;
+    }
+    else
+    {
+        fwrite(buffer, 1, stream.bytes_written, stdout);
+        return 0; /* Success */
+    }
+}
+
+

example_unions/unionproto.proto

@@ -0,0 +1,30 @@
+// This is an example of how to handle 'union' style messages
+// with nanopb, without allocating memory for all the message types.
+//
+// There is no official type in Protocol Buffers for describing unions,
+// but they are commonly implemented by filling out exactly one of
+// several optional fields.
+
+message MsgType1
+{
+    required int32 value = 1;
+}
+
+message MsgType2
+{
+    required bool value = 1;
+}
+
+message MsgType3
+{
+    required int32 value1 = 1;
+    required int32 value2 = 2;
+}
+
+message UnionMessage
+{
+    optional MsgType1 msg1 = 1;
+    optional MsgType2 msg2 = 2;
+    optional MsgType3 msg3 = 3;
+}
+

generator/nanopb_generator.py

@@ -217,10 +217,10 @@ class Field:
         prev_field_name is the name of the previous field or None.
         '''
         result = '    {%d, ' % self.tag
-        result += self.htype
+        result += '(pb_type_t) ((int) ' + self.htype
         if self.ltype is not None:
-            result += ' | ' + self.ltype
-        result += ',\n'
+            result += ' | (int) ' + self.ltype
+        result += '),\n'
         
         if prev_field_name is None:
             result += '    offsetof(%s, %s),' % (self.struct_name, self.name)
@@ -251,6 +251,18 @@ class Field:
             result += '\n    &%s_default}' % (self.struct_name + self.name)
         
         return result
+    
+    def largest_field_value(self):
+        '''Determine if this field needs 16bit or 32bit pb_field_t structure to compile properly.
+        Returns numeric value or a C-expression for assert.'''
+        if self.ltype == 'PB_LTYPE_SUBMESSAGE':
+            if self.htype == 'PB_HTYPE_ARRAY':
+                return 'pb_membersize(%s, %s[0])' % (self.struct_name, self.name)
+            else:
+                return 'pb_membersize(%s, %s)' % (self.struct_name, self.name)
+
+        return max(self.tag, self.max_size, self.max_count)        
+
 
 class Message:
     def __init__(self, names, desc):
@@ -343,7 +355,7 @@ def toposort2(data):
     '''
     for k, v in data.items():
         v.discard(k) # Ignore self dependencies
-    extra_items_in_deps = reduce(set.union, data.values()) - set(data.keys())
+    extra_items_in_deps = reduce(set.union, data.values(), set()) - set(data.keys())
     data.update(dict([(item, set()) for item in extra_items_in_deps]))
     while True:
         ordered = set(item for item,dep in data.items() if not dep)
@@ -402,6 +414,51 @@ def generate_header(dependencies, headername, enums, messages):
     for msg in messages:
         yield msg.fields_declaration() + '\n'
     
+    if messages:
+        count_required_fields = lambda m: len([f for f in msg.fields if f.htype == 'PB_HTYPE_REQUIRED'])
+        largest_msg = max(messages, key = count_required_fields)
+        largest_count = count_required_fields(largest_msg)
+        if largest_count > 64:
+            yield '\n/* Check that missing required fields will be properly detected */\n'
+            yield '#if PB_MAX_REQUIRED_FIELDS < %d\n' % largest_count
+            yield '#error Properly detecting missing required fields in %s requires \\\n' % largest_msg.name
+            yield '       setting PB_MAX_REQUIRED_FIELDS to %d or more.\n' % largest_count
+            yield '#endif\n'
+    
+    worst = 0
+    worst_field = ''
+    checks = []
+    for msg in messages:
+        for field in msg.fields:
+            status = field.largest_field_value()
+            if isinstance(status, (str, unicode)):
+                checks.append(status)
+            elif status > worst:
+                worst = status
+                worst_field = str(field.struct_name) + '.' + str(field.name)
+
+    if worst > 255 or checks:
+        yield '\n/* Check that field information fits in pb_field_t */\n'
+        
+        if worst < 65536:
+            yield '#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)\n'
+            if worst > 255:
+                yield '#error Field descriptor for %s is too large. Define PB_FIELD_16BIT to fix this.\n' % worst_field
+            else:
+                assertion = ' && '.join(str(c) + ' < 256' for c in checks)
+                yield 'STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_16BIT)\n' % assertion
+            yield '#endif\n\n'
+        
+        if worst > 65535 or checks:
+            yield '#if !defined(PB_FIELD_32BIT)\n'
+            if worst > 65535:
+                yield '#error Field descriptor for %s is too large. Define PB_FIELD_32BIT to fix this.\n' % worst_field
+            else:
+                assertion = ' && '.join(str(c) + ' < 65536' for c in checks)
+                yield 'STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_32BIT)\n' % assertion
+            yield '#endif\n'
+    
+    # End of header
     yield '\n#endif\n'
 
 def generate_source(headername, enums, messages):

pb.h

@@ -22,6 +22,21 @@
 #define UNUSED(x) (void)(x)
 #endif
 
+/* Compile-time assertion, used for checking compatible compilation options. */
+#ifndef STATIC_ASSERT
+#define STATIC_ASSERT(COND,MSG) typedef char static_assertion_##MSG[(COND)?1:-1];
+#endif
+
+/* Number of required fields to keep track of
+ * (change here or on compiler command line). */
+#ifndef PB_MAX_REQUIRED_FIELDS
+#define PB_MAX_REQUIRED_FIELDS 64
+#endif
+
+#if PB_MAX_REQUIRED_FIELDS < 64
+#error You should not lower PB_MAX_REQUIRED_FIELDS from the default value (64).
+#endif
+
 /* List of possible field types. These are used in the autogenerated code.
  * Least-significant 4 bits tell the scalar type
  * Most-significant 4 bits specify repeated/required/packed etc.
@@ -88,19 +103,37 @@ typedef enum {
 
 /* This structure is used in auto-generated constants
  * to specify struct fields.
- * You can change field sizes here if you need structures
+ * You can change field sizes if you need structures
  * larger than 256 bytes or field tags larger than 256.
  * The compiler should complain if your .proto has such
- * structures ("initializer too large for type").
+ * structures. Fix that by defining PB_FIELD_16BIT or
+ * PB_FIELD_32BIT.
  */
 typedef struct _pb_field_t pb_field_t;
 struct _pb_field_t {
+
+#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)
     uint8_t tag;
     pb_type_t type;
     uint8_t data_offset; /* Offset of field data, relative to previous field. */
     int8_t size_offset; /* Offset of array size or has-boolean, relative to data */
     uint8_t data_size; /* Data size in bytes for a single item */
     uint8_t array_size; /* Maximum number of entries in array */
+#elif defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)
+    uint16_t tag;
+    pb_type_t type;
+    uint8_t data_offset;
+    int8_t size_offset;
+    uint16_t data_size;
+    uint16_t array_size;
+#else
+    uint32_t tag;
+    pb_type_t type;
+    uint8_t data_offset;
+    int8_t size_offset;
+    uint32_t data_size;
+    uint32_t array_size;
+#endif
     
     /* Field definitions for submessage
      * OR default value for all other non-array, non-callback types
@@ -161,7 +194,25 @@ typedef enum {
 #define pb_arraysize(st, m) (pb_membersize(st, m) / pb_membersize(st, m[0]))
 #define pb_delta(st, m1, m2) ((int)offsetof(st, m1) - (int)offsetof(st, m2))
 #define pb_delta_end(st, m1, m2) (offsetof(st, m1) - offsetof(st, m2) - pb_membersize(st, m2))
-#define PB_LAST_FIELD {0,0,0,0}
+#define PB_LAST_FIELD {0,(pb_type_t) 0,0,0,0,0,0}
 
+/* These macros are used for giving out error messages.
+ * They are mostly a debugging aid; the main error information
+ * is the true/false return value from functions.
+ * Some code space can be saved by disabling the error
+ * messages if not used.
+ */
+#ifdef PB_NO_ERRMSG
+#define PB_RETURN_ERROR(stream,msg) return false
+#define PB_GET_ERROR(stream) "(errmsg disabled)"
+#else
+#define PB_RETURN_ERROR(stream,msg) \
+    do {\
+        if ((stream)->errmsg == NULL) \
+            (stream)->errmsg = (msg); \
+        return false; \
+    } while(0)
+#define PB_GET_ERROR(stream) ((stream)->errmsg ? (stream)->errmsg : "(none)")
+#endif
 
 #endif

pb_decode.c

@@ -3,14 +3,15 @@
  * 2011 Petteri Aimonen <jpa@kapsi.fi>
  */
 
-
-#ifdef __GNUC__
-/* Verify that we remember to check all return values for proper error propagation */
-#define checkreturn __attribute__((warn_unused_result))
+/* The warn_unused_result attribute appeared first in gcc-3.4.0 */
+#if !defined(__GNUC__) || ( __GNUC__ < 3) || (__GNUC__ == 3 && __GNUC_MINOR__ < 4)
+    #define checkreturn
 #else
-#define checkreturn
+    /* Verify that we remember to check all return values for proper error propagation */
+    #define checkreturn __attribute__((warn_unused_result))
 #endif
 
+#define NANOPB_INTERNALS
 #include "pb.h"
 #include "pb_decode.h"
 #include <string.h>
@@ -38,10 +39,10 @@ static const pb_decoder_t PB_DECODERS[PB_LTYPES_COUNT] = {
 bool checkreturn pb_read(pb_istream_t *stream, uint8_t *buf, size_t count)
 {
     if (stream->bytes_left < count)
-        return false;
+        PB_RETURN_ERROR(stream, "end-of-stream");
     
     if (!stream->callback(stream, buf, count))
-        return false;
+        PB_RETURN_ERROR(stream, "io error");
     
     stream->bytes_left -= count;
     return true;
@@ -94,7 +95,7 @@ bool checkreturn pb_decode_varint(pb_istream_t *stream, uint64_t *dest)
             return true;
     }
     
-    return false;
+    PB_RETURN_ERROR(stream, "varint overflow");
 }
 
 bool checkreturn pb_skip_varint(pb_istream_t *stream)
@@ -117,12 +118,33 @@ bool checkreturn pb_skip_string(pb_istream_t *stream)
     return pb_read(stream, NULL, length);
 }
 
-/* Currently the wire type related stuff is kept hidden from
- * callbacks. They shouldn't need it. It's better for performance
- * to just assume the correct type and fail safely on corrupt message.
- */
+bool checkreturn pb_decode_tag(pb_istream_t *stream, pb_wire_type_t *wire_type, uint32_t *tag, bool *eof)
+{
+    uint32_t temp;
+    *eof = false;
+    *wire_type = (pb_wire_type_t) 0;
+    *tag = 0;
+    
+    if (!pb_decode_varint32(stream, &temp))
+    {
+        if (stream->bytes_left == 0)
+            *eof = true;
 
-static bool checkreturn skip(pb_istream_t *stream, pb_wire_type_t wire_type)
+        return false;
+    }
+    
+    if (temp == 0)
+    {
+        *eof = true; /* Special feature: allow 0-terminated messages. */
+        return false;
+    }
+    
+    *tag = temp >> 3;
+    *wire_type = (pb_wire_type_t)(temp & 7);
+    return true;
+}
+
+bool checkreturn pb_skip_field(pb_istream_t *stream, pb_wire_type_t wire_type)
 {
     switch (wire_type)
     {
@@ -130,7 +152,7 @@ static bool checkreturn skip(pb_istream_t *stream, pb_wire_type_t wire_type)
         case PB_WT_64BIT: return pb_read(stream, NULL, 8);
         case PB_WT_STRING: return pb_skip_string(stream);
         case PB_WT_32BIT: return pb_read(stream, NULL, 4);
-        default: return false;
+        default: PB_RETURN_ERROR(stream, "invalid wire_type");
     }
 }
 
@@ -160,15 +182,14 @@ static bool checkreturn read_raw_value(pb_istream_t *stream, pb_wire_type_t wire
             *size = 4;
             return pb_read(stream, buf, 4);
         
-        default: return false;
+        default: PB_RETURN_ERROR(stream, "invalid wire_type");
     }
 }
 
 /* Decode string length from stream and return a substream with limited length.
- * Before disposing the substream, remember to copy the substream->state back
- * to stream->state.
+ * Remember to close the substream using pb_close_string_substream().
  */
-static bool checkreturn make_string_substream(pb_istream_t *stream, pb_istream_t *substream)
+bool checkreturn pb_make_string_substream(pb_istream_t *stream, pb_istream_t *substream)
 {
     uint32_t size;
     if (!pb_decode_varint32(stream, &size))
@@ -176,27 +197,34 @@ static bool checkreturn make_string_substream(pb_istream_t *stream, pb_istream_t
     
     *substream = *stream;
     if (substream->bytes_left < size)
-        return false;
+        PB_RETURN_ERROR(stream, "parent stream too short");
     
     substream->bytes_left = size;
     stream->bytes_left -= size;
     return true;
 }
 
+void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream)
+{
+    stream->state = substream->state;
+}
+
 /* Iterator for pb_field_t list */
 typedef struct {
-    const pb_field_t *start;
-    const pb_field_t *current;
-    int field_index;
-    void *dest_struct;
-    void *pData;
-    void *pSize;
+    const pb_field_t *start; /* Start of the pb_field_t array */
+    const pb_field_t *current; /* Current position of the iterator */
+    int field_index; /* Zero-based index of the field. */
+    int required_field_index; /* Zero-based index that counts only the required fields */
+    void *dest_struct; /* Pointer to the destination structure to decode to */
+    void *pData; /* Pointer where to store current field value */
+    void *pSize; /* Pointer where to store the size of current array field */
 } pb_field_iterator_t;
 
 static void pb_field_init(pb_field_iterator_t *iter, const pb_field_t *fields, void *dest_struct)
 {
     iter->start = iter->current = fields;
     iter->field_index = 0;
+    iter->required_field_index = 0;
     iter->pData = (char*)dest_struct + iter->current->data_offset;
     iter->pSize = (char*)iter->pData + iter->current->size_offset;
     iter->dest_struct = dest_struct;
@@ -210,12 +238,16 @@ static bool pb_field_next(pb_field_iterator_t *iter)
     if (PB_HTYPE(iter->current->type) == PB_HTYPE_ARRAY)
         prev_size *= iter->current->array_size;
     
+    if (PB_HTYPE(iter->current->type) == PB_HTYPE_REQUIRED)
+        iter->required_field_index++;
+    
     iter->current++;
     iter->field_index++;
     if (iter->current->tag == 0)
     {
         iter->current = iter->start;
         iter->field_index = 0;
+        iter->required_field_index = 0;
         iter->pData = iter->dest_struct;
         prev_size = 0;
         notwrapped = false;
@@ -226,7 +258,7 @@ static bool pb_field_next(pb_field_iterator_t *iter)
     return notwrapped;
 }
 
-static bool checkreturn pb_field_find(pb_field_iterator_t *iter, int tag)
+static bool checkreturn pb_field_find(pb_field_iterator_t *iter, uint32_t tag)
 {
     int start = iter->field_index;
     
@@ -261,9 +293,10 @@ static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_t
                 && PB_LTYPE(iter->current->type) <= PB_LTYPE_LAST_PACKABLE)
             {
                 /* Packed array */
+                bool status;
                 size_t *size = (size_t*)iter->pSize;
                 pb_istream_t substream;
-                if (!make_string_substream(stream, &substream))
+                if (!pb_make_string_substream(stream, &substream))
                     return false;
                 
                 while (substream.bytes_left && *size < iter->current->array_size)
@@ -273,7 +306,9 @@ static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_t
                         return false;
                     (*size)++;
                 }
-                return (substream.bytes_left == 0);
+                status = (substream.bytes_left == 0);
+                pb_close_string_substream(stream, &substream);
+                return status;
             }
             else
             {
@@ -281,7 +316,7 @@ static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_t
                 size_t *size = (size_t*)iter->pSize;
                 void *pItem = (uint8_t*)iter->pData + iter->current->data_size * (*size);
                 if (*size >= iter->current->array_size)
-                    return false;
+                    PB_RETURN_ERROR(stream, "array overflow");
                 
                 (*size)++;
                 return func(stream, iter->current, pItem);
@@ -292,22 +327,22 @@ static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_t
             pb_callback_t *pCallback = (pb_callback_t*)iter->pData;
             
             if (pCallback->funcs.decode == NULL)
-                return skip(stream, wire_type);
+                return pb_skip_field(stream, wire_type);
             
             if (wire_type == PB_WT_STRING)
             {
                 pb_istream_t substream;
                 
-                if (!make_string_substream(stream, &substream))
+                if (!pb_make_string_substream(stream, &substream))
                     return false;
                 
                 while (substream.bytes_left)
                 {
                     if (!pCallback->funcs.decode(&substream, iter->current, pCallback->arg))
-                        return false;
+                        PB_RETURN_ERROR(stream, "callback failed");
                 }
                 
-                stream->state = substream.state;
+                pb_close_string_substream(stream, &substream);
                 return true;
             }
             else
@@ -329,7 +364,7 @@ static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_t
         }
         
         default:
-            return false;
+            PB_RETURN_ERROR(stream, "invalid field type");
     }
 }
 
@@ -363,7 +398,7 @@ static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_str
         }
         else if (PB_LTYPE(iter.current->type) == PB_LTYPE_SUBMESSAGE)
         {
-            pb_message_set_to_defaults(iter.current->ptr, iter.pData);
+            pb_message_set_to_defaults((const pb_field_t *) iter.current->ptr, iter.pData);
         }
         else if (iter.current->ptr != NULL)
         {
@@ -380,127 +415,165 @@ static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_str
  * Decode all fields *
  *********************/
 
-bool checkreturn pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
+bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
 {
-    uint32_t fields_seen = 0; /* Used to check for required fields */
+    uint8_t fields_seen[(PB_MAX_REQUIRED_FIELDS + 7) / 8] = {0}; /* Used to check for required fields */
     pb_field_iterator_t iter;
-    int i;
-    
-    pb_message_set_to_defaults(fields, dest_struct);
     
     pb_field_init(&iter, fields, dest_struct);
     
     while (stream->bytes_left)
     {
-        uint32_t temp;
-        int tag;
+        uint32_t tag;
         pb_wire_type_t wire_type;
-        if (!pb_decode_varint32(stream, &temp))
+        bool eof;
+        
+        if (!pb_decode_tag(stream, &wire_type, &tag, &eof))
         {
-            if (stream->bytes_left == 0)
-                break; /* It was EOF */
+            if (eof)
+                break;
             else
-                return false; /* It was error */
+                return false;
         }
         
-        if (temp == 0)
-            break; /* Special feature: allow 0-terminated messages. */
-        
-        tag = temp >> 3;
-        wire_type = (pb_wire_type_t)(temp & 7);
-        
         if (!pb_field_find(&iter, tag))
         {
             /* No match found, skip data */
-            if (!skip(stream, wire_type))
+            if (!pb_skip_field(stream, wire_type))
                 return false;
             continue;
         }
         
-        fields_seen |= 1 << (iter.field_index & 31);
+        if (PB_HTYPE(iter.current->type) == PB_HTYPE_REQUIRED
+            && iter.required_field_index < PB_MAX_REQUIRED_FIELDS)
+        {
+            fields_seen[iter.required_field_index >> 3] |= 1 << (iter.required_field_index & 7);
+        }
             
         if (!decode_field(stream, wire_type, &iter))
             return false;
     }
     
-    /* Check that all required fields (mod 31) were present. */
-    for (i = 0; fields[i].tag != 0; i++)
-    {
-        if (PB_HTYPE(fields[i].type) == PB_HTYPE_REQUIRED &&
-            !(fields_seen & (1 << (i & 31))))
+    /* Check that all required fields were present. */
+    pb_field_init(&iter, fields, dest_struct);
+    do {
+        if (PB_HTYPE(iter.current->type) == PB_HTYPE_REQUIRED &&
+            iter.required_field_index < PB_MAX_REQUIRED_FIELDS &&
+            !(fields_seen[iter.required_field_index >> 3] & (1 << (iter.required_field_index & 7))))
         {
-            return false;
+            PB_RETURN_ERROR(stream, "missing required field");
         }
-    }
+    } while (pb_field_next(&iter));
     
     return true;
 }
 
+bool checkreturn pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
+{
+    pb_message_set_to_defaults(fields, dest_struct);
+    return pb_decode_noinit(stream, fields, dest_struct);
+}
+
 /* Field decoders */
 
-/* Copy destsize bytes from src so that values are casted properly.
- * On little endian machine, copy first n bytes of src
- * On big endian machine, copy last n bytes of src
- * srcsize must always be larger than destsize
- */
-static void endian_copy(void *dest, void *src, size_t destsize, size_t srcsize)
+bool pb_decode_svarint(pb_istream_t *stream, int64_t *dest)
 {
-#ifdef __BIG_ENDIAN__
-    memcpy(dest, (char*)src + (srcsize - destsize), destsize);
-#else
-    UNUSED(srcsize);
-    memcpy(dest, src, destsize);
-#endif
+    uint64_t value;
+    if (!pb_decode_varint(stream, &value))
+        return false;
+    
+    if (value & 1)
+        *dest = ~(value >> 1);
+    else
+        *dest = value >> 1;
+    
+    return true;
+}
+
+bool pb_decode_fixed32(pb_istream_t *stream, void *dest)
+{
+    #ifdef __BIG_ENDIAN__
+    uint8_t *bytes = (uint8_t*)dest;
+    uint8_t lebytes[4];
+    
+    if (!pb_read(stream, lebytes, 4))
+        return false;
+    
+    bytes[0] = lebytes[3];
+    bytes[1] = lebytes[2];
+    bytes[2] = lebytes[1];
+    bytes[3] = lebytes[0];
+    return true;
+    #else
+    return pb_read(stream, (uint8_t*)dest, 4);
+    #endif   
+}
+
+bool pb_decode_fixed64(pb_istream_t *stream, void *dest)
+{
+    #ifdef __BIG_ENDIAN__
+    uint8_t *bytes = (uint8_t*)dest;
+    uint8_t lebytes[8];
+    
+    if (!pb_read(stream, lebytes, 8))
+        return false;
+    
+    bytes[0] = lebytes[7];
+    bytes[1] = lebytes[6];
+    bytes[2] = lebytes[5];
+    bytes[3] = lebytes[4];
+    bytes[4] = lebytes[3];
+    bytes[5] = lebytes[2];
+    bytes[6] = lebytes[1];
+    bytes[7] = lebytes[0];
+    return true;
+    #else
+    return pb_read(stream, (uint8_t*)dest, 8);
+    #endif   
 }
 
 bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
-    uint64_t temp;
-    bool status = pb_decode_varint(stream, &temp);
-    endian_copy(dest, &temp, field->data_size, sizeof(temp));
+    uint64_t value;
+    bool status = pb_decode_varint(stream, &value);
+    
+    switch (field->data_size)
+    {
+        case 1: *(uint8_t*)dest = value; break;
+        case 2: *(uint16_t*)dest = value; break;
+        case 4: *(uint32_t*)dest = value; break;
+        case 8: *(uint64_t*)dest = value; break;
+        default: PB_RETURN_ERROR(stream, "invalid data_size");
+    }
+    
     return status;
 }
 
 bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
-    uint64_t temp;
-    bool status = pb_decode_varint(stream, &temp);
-    temp = (temp >> 1) ^ -(int64_t)(temp & 1);
-    endian_copy(dest, &temp, field->data_size, sizeof(temp));
+    int64_t value;
+    bool status = pb_decode_svarint(stream, &value);
+    
+    switch (field->data_size)
+    {
+        case 4: *(int32_t*)dest = value; break;
+        case 8: *(int64_t*)dest = value; break;
+        default: PB_RETURN_ERROR(stream, "invalid data_size");
+    }
+    
     return status;
 }
 
 bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
-#ifdef __BIG_ENDIAN__
-    uint8_t bytes[4] = {0};
-    bool status = pb_read(stream, bytes, 4);
-    if (status) {
-      uint8_t bebytes[4] = {bytes[3], bytes[2], bytes[1], bytes[0]};
-      memcpy(dest, bebytes, 4);
-    }
-    return status;
-#else
     UNUSED(field);
-    return pb_read(stream, (uint8_t*)dest, 4);
-#endif
+    return pb_decode_fixed32(stream, dest);
 }
 
 bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
-#ifdef __BIG_ENDIAN__
-    uint8_t bytes[8] = {0};
-    bool status = pb_read(stream, bytes, 8);
-    if (status) {
-      uint8_t bebytes[8] = {bytes[7], bytes[6], bytes[5], bytes[4], 
-                            bytes[3], bytes[2], bytes[1], bytes[0]};
-      memcpy(dest, bebytes, 8);
-    }
-    return status;
-#else
     UNUSED(field);
-    return pb_read(stream, (uint8_t*)dest, 8);
-#endif
+    return pb_decode_fixed64(stream, dest);
 }
 
 bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest)
@@ -514,7 +587,7 @@ bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, voi
     
     /* Check length, noting the space taken by the size_t header. */
     if (x->size > field->data_size - offsetof(pb_bytes_array_t, bytes))
-        return false;
+        PB_RETURN_ERROR(stream, "bytes overflow");
     
     return pb_read(stream, x->bytes, x->size);
 }
@@ -528,7 +601,7 @@ bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, vo
     
     /* Check length, noting the null terminator */
     if (size + 1 > field->data_size)
-        return false;
+        PB_RETURN_ERROR(stream, "string overflow");
     
     status = pb_read(stream, (uint8_t*)dest, size);
     *((uint8_t*)dest + size) = 0;
@@ -539,14 +612,21 @@ bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field
 {
     bool status;
     pb_istream_t substream;
+    const pb_field_t* submsg_fields = (const pb_field_t*)field->ptr;
     
-    if (!make_string_substream(stream, &substream))
+    if (!pb_make_string_substream(stream, &substream))
         return false;
     
     if (field->ptr == NULL)
-        return false;
+        PB_RETURN_ERROR(stream, "invalid field descriptor");
     
-    status = pb_decode(&substream, (pb_field_t*)field->ptr, dest);
-    stream->state = substream.state;
+    /* New array entries need to be initialized, while required and optional
+     * submessages have already been initialized in the top-level pb_decode. */
+    if (PB_HTYPE(field->type) == PB_HTYPE_ARRAY)
+        status = pb_decode(&substream, submsg_fields, dest);
+    else
+        status = pb_decode_noinit(&substream, submsg_fields, dest);
+    
+    pb_close_string_substream(stream, &substream);
     return status;
 }

pb_decode.h

@@ -3,7 +3,7 @@
 
 /* pb_decode.h: Functions to decode protocol buffers. Depends on pb_decode.c.
  * The main function is pb_decode. You will also need to create an input
- * stream, which is easiest to do with pb_istream_t.
+ * stream, which is easiest to do with pb_istream_from_buffer().
  * 
  * You also need structures and their corresponding pb_field_t descriptions.
  * These are usually generated from .proto-files with a script.
@@ -33,6 +33,10 @@ struct _pb_istream_t
     bool (*callback)(pb_istream_t *stream, uint8_t *buf, size_t count);
     void *state; /* Free field for use by callback implementation */
     size_t bytes_left;
+    
+#ifndef PB_NO_ERRMSG
+    const char *errmsg;
+#endif
 };
 
 pb_istream_t pb_istream_from_buffer(uint8_t *buf, size_t bufsize);
@@ -44,21 +48,49 @@ bool pb_read(pb_istream_t *stream, uint8_t *buf, size_t count);
  */
 bool pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
 
+/* Same as pb_decode, except does not initialize the destination structure
+ * to default values. This is slightly faster if you need no default values
+ * and just do memset(struct, 0, sizeof(struct)) yourself.
+ */
+bool pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
+
 /* --- Helper functions ---
  * You may want to use these from your caller or callbacks.
  */
 
+/* Decode the tag for the next field in the stream. Gives the wire type and
+ * field tag. At end of the message, returns false and sets eof to true. */
+bool pb_decode_tag(pb_istream_t *stream, pb_wire_type_t *wire_type, uint32_t *tag, bool *eof);
+
+/* Skip the field payload data, given the wire type. */
+bool pb_skip_field(pb_istream_t *stream, pb_wire_type_t wire_type);
+
+/* Decode an integer in the varint format. This works for bool, enum, int32,
+ * int64, uint32 and uint64 field types. */
 bool pb_decode_varint(pb_istream_t *stream, uint64_t *dest);
 
-bool pb_skip_varint(pb_istream_t *stream);
-bool pb_skip_string(pb_istream_t *stream);
+/* Decode an integer in the zig-zagged svarint format. This works for sint32
+ * and sint64. */
+bool pb_decode_svarint(pb_istream_t *stream, int64_t *dest);
 
-/* --- Field decoders ---
- * Each decoder takes stream and field description, and a pointer to the field
- * in the destination struct (dest = struct_addr + field->data_offset).
- * For arrays, these functions are called repeatedly.
+/* Decode a fixed32, sfixed32 or float value. You need to pass a pointer to
+ * a 4-byte wide C variable. */
+bool pb_decode_fixed32(pb_istream_t *stream, void *dest);
+
+/* Decode a fixed64, sfixed64 or double value. You need to pass a pointer to
+ * a 8-byte wide C variable. */
+bool pb_decode_fixed64(pb_istream_t *stream, void *dest);
+
+/* Make a limited-length substream for reading a PB_WT_STRING field. */
+bool pb_make_string_substream(pb_istream_t *stream, pb_istream_t *substream);
+void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream);
+
+/* --- Internal functions ---
+ * These functions are not terribly useful for the average library user, but
+ * are exported to make the unit testing and extending nanopb easier.
  */
 
+#ifdef NANOPB_INTERNALS
 bool pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest);
 bool pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest);
 bool pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest);
@@ -68,4 +100,8 @@ bool pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest);
 bool pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest);
 bool pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest);
 
+bool pb_skip_varint(pb_istream_t *stream);
+bool pb_skip_string(pb_istream_t *stream);
+#endif
+
 #endif

pb_encode.c

@@ -8,14 +8,14 @@
 #include "pb_encode.h"
 #include <string.h>
 
-#ifdef __GNUC__
-/* Verify that we remember to check all return values for proper error propagation */
-#define checkreturn __attribute__((warn_unused_result))
+/* The warn_unused_result attribute appeared first in gcc-3.4.0 */
+#if !defined(__GNUC__) || ( __GNUC__ < 3) || (__GNUC__ == 3 && __GNUC_MINOR__ < 4)
+    #define checkreturn
 #else
-#define checkreturn
+    /* Verify that we remember to check all return values for proper error propagation */
+    #define checkreturn __attribute__((warn_unused_result))
 #endif
 
-
 typedef bool (*pb_encoder_t)(pb_ostream_t *stream, const pb_field_t *field, const void *src) checkreturn;
 
 /* --- Function pointers to field encoders ---
@@ -99,7 +99,7 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
         }
         else
         {
-            pb_ostream_t sizestream = {0};
+            pb_ostream_t sizestream = {0,0,0,0};
             p = pData;
             for (i = 0; i < count; i++)
             {
@@ -110,7 +110,7 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
             size = sizestream.bytes_written;
         }
         
-        if (!pb_encode_varint(stream, size))
+        if (!pb_encode_varint(stream, (uint64_t)size))
             return false;
         
         if (stream->callback == NULL)
@@ -168,7 +168,7 @@ bool checkreturn pb_encode(pb_ostream_t *stream, const pb_field_t fields[], cons
                 break;
             
             case PB_HTYPE_OPTIONAL:
-                if (*(bool*)pSize)
+                if (*(const bool*)pSize)
                 {
                     if (!pb_encode_tag_for_field(stream, field))
                         return false;
@@ -179,13 +179,13 @@ bool checkreturn pb_encode(pb_ostream_t *stream, const pb_field_t fields[], cons
                 break;
             
             case PB_HTYPE_ARRAY:
-                if (!encode_array(stream, field, pData, *(size_t*)pSize, func))
+                if (!encode_array(stream, field, pData, *(const size_t*)pSize, func))
                     return false;
                 break;
             
             case PB_HTYPE_CALLBACK:
             {
-                pb_callback_t *callback = (pb_callback_t*)pData;
+                const pb_callback_t *callback = (const pb_callback_t*)pData;
                 if (callback->funcs.encode != NULL)
                 {
                     if (!callback->funcs.encode(stream, field, callback->arg))
@@ -235,29 +235,40 @@ bool checkreturn pb_encode_svarint(pb_ostream_t *stream, int64_t value)
 bool checkreturn pb_encode_fixed32(pb_ostream_t *stream, const void *value)
 {
     #ifdef __BIG_ENDIAN__
-    uint8_t *bytes = value;
-    uint8_t lebytes[4] = {bytes[3], bytes[2], bytes[1], bytes[0]};
+    const uint8_t *bytes = value;
+    uint8_t lebytes[4];
+    lebytes[0] = bytes[3];
+    lebytes[1] = bytes[2];
+    lebytes[2] = bytes[1];
+    lebytes[3] = bytes[0];
     return pb_write(stream, lebytes, 4);
     #else
-    return pb_write(stream, (uint8_t*)value, 4);
+    return pb_write(stream, (const uint8_t*)value, 4);
     #endif
 }
 
 bool checkreturn pb_encode_fixed64(pb_ostream_t *stream, const void *value)
 {
     #ifdef __BIG_ENDIAN__
-    uint8_t *bytes[8] = value;
-    uint8_t lebytes[8] = {bytes[7], bytes[6], bytes[5], bytes[4], 
-                          bytes[3], bytes[2], bytes[1], bytes[0]};
+    const uint8_t *bytes = value;
+    uint8_t lebytes[8];
+    lebytes[0] = bytes[7];
+    lebytes[1] = bytes[6];
+    lebytes[2] = bytes[5];
+    lebytes[3] = bytes[4];
+    lebytes[4] = bytes[3];
+    lebytes[5] = bytes[2];
+    lebytes[6] = bytes[1];
+    lebytes[7] = bytes[0];
     return pb_write(stream, lebytes, 8);
     #else
-    return pb_write(stream, (uint8_t*)value, 8);
+    return pb_write(stream, (const uint8_t*)value, 8);
     #endif
 }
 
 bool checkreturn pb_encode_tag(pb_ostream_t *stream, pb_wire_type_t wiretype, int field_number)
 {
-    int tag = wiretype | (field_number << 3);
+    uint64_t tag = wiretype | (field_number << 3);
     return pb_encode_varint(stream, tag);
 }
 
@@ -294,7 +305,7 @@ bool checkreturn pb_encode_tag_for_field(pb_ostream_t *stream, const pb_field_t
 
 bool checkreturn pb_encode_string(pb_ostream_t *stream, const uint8_t *buffer, size_t size)
 {
-    if (!pb_encode_varint(stream, size))
+    if (!pb_encode_varint(stream, (uint64_t)size))
         return false;
     
     return pb_write(stream, buffer, size);
@@ -303,7 +314,7 @@ bool checkreturn pb_encode_string(pb_ostream_t *stream, const uint8_t *buffer, s
 bool checkreturn pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct)
 {
     /* First calculate the message size using a non-writing substream. */
-    pb_ostream_t substream = {0};
+    pb_ostream_t substream = {0,0,0,0};
     size_t size;
     bool status;
     
@@ -312,7 +323,7 @@ bool checkreturn pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fie
     
     size = substream.bytes_written;
     
-    if (!pb_encode_varint(stream, size))
+    if (!pb_encode_varint(stream, (uint64_t)size))
         return false;
     
     if (stream->callback == NULL)
@@ -347,10 +358,10 @@ bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, co
     
     switch (field->data_size)
     {
-        case 1: value = *(uint8_t*)src; break;
-        case 2: value = *(uint16_t*)src; break;
-        case 4: value = *(uint32_t*)src; break;
-        case 8: value = *(uint64_t*)src; break;
+        case 1: value = *(const uint8_t*)src; break;
+        case 2: value = *(const uint16_t*)src; break;
+        case 4: value = *(const uint32_t*)src; break;
+        case 8: value = *(const uint64_t*)src; break;
         default: return false;
     }
     
@@ -363,8 +374,8 @@ bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, c
     
     switch (field->data_size)
     {
-        case 4: value = *(int32_t*)src; break;
-        case 8: value = *(int64_t*)src; break;
+        case 4: value = *(const int32_t*)src; break;
+        case 8: value = *(const int64_t*)src; break;
         default: return false;
     }
     
@@ -385,7 +396,7 @@ bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, c
 
 bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
-    pb_bytes_array_t *bytes = (pb_bytes_array_t*)src;
+    const pb_bytes_array_t *bytes = (const pb_bytes_array_t*)src;
     UNUSED(field);
     return pb_encode_string(stream, bytes->bytes, bytes->size);
 }
@@ -393,7 +404,7 @@ bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, con
 bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     UNUSED(field);
-    return pb_encode_string(stream, (uint8_t*)src, strlen((char*)src));
+    return pb_encode_string(stream, (const uint8_t*)src, strlen((const char*)src));
 }
 
 bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src)
@@ -401,6 +412,6 @@ bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field
     if (field->ptr == NULL)
         return false;
     
-    return pb_encode_submessage(stream, (pb_field_t*)field->ptr, src);
+    return pb_encode_submessage(stream, (const pb_field_t*)field->ptr, src);
 }
 

tests/Makefile

@@ -1,22 +1,34 @@
 CFLAGS=-ansi -Wall -Werror -I .. -g -O0 --coverage
 LDFLAGS=--coverage
-DEPS=../pb_decode.h ../pb_encode.h ../pb.h person.pb.h callbacks.pb.h unittests.h unittestproto.pb.h alltypes.pb.h
-TESTS=test_decode1 test_encode1 decode_unittests encode_unittests
+DEPS=../pb_decode.h ../pb_encode.h ../pb.h person.pb.h callbacks.pb.h unittests.h unittestproto.pb.h alltypes.pb.h missing_fields.pb.h
+TESTS=test_decode1 test_encode1 decode_unittests encode_unittests test_no_messages
+
+# More strict checks for the core part of nanopb
+CFLAGS_CORE=-pedantic -Wextra -Wcast-qual -Wlogical-op
 
 all: breakpoints $(TESTS) run_unittests
 
 clean:
 	rm -f $(TESTS) person.pb* alltypes.pb* *.o *.gcda *.gcno
 
+%.pb.o: %.pb.c %.pb.h
+	$(CC) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
+
 %.o: %.c
 %.o: %.c $(DEPS)
 	$(CC) $(CFLAGS) -c -o $@ $<
 
 pb_encode.o: ../pb_encode.c $(DEPS)
-	$(CC) $(CFLAGS) -c -o $@ $<
+	$(CC) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
 pb_decode.o: ../pb_decode.c $(DEPS)
-	$(CC) $(CFLAGS) -c -o $@ $<
+	$(CC) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
 
+pb_encode.cxx.o: ../pb_encode.c $(DEPS)
+	$(CXX) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
+pb_decode.cxx.o: ../pb_decode.c $(DEPS)
+	$(CXX) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
+
+test_cxxcompile: pb_encode.cxx.o pb_decode.cxx.o
 test_decode1: test_decode1.o pb_decode.o person.pb.o
 test_decode2: test_decode2.o pb_decode.o person.pb.o
 test_decode3: test_decode3.o pb_decode.o alltypes.pb.o
@@ -25,8 +37,10 @@ test_encode2: test_encode2.o pb_encode.o person.pb.o
 test_encode3: test_encode3.o pb_encode.o alltypes.pb.o
 test_decode_callbacks: test_decode_callbacks.o pb_decode.o callbacks.pb.o
 test_encode_callbacks: test_encode_callbacks.o pb_encode.o callbacks.pb.o
+test_missing_fields: test_missing_fields.o pb_encode.o pb_decode.o missing_fields.pb.o
 decode_unittests: decode_unittests.o pb_decode.o unittestproto.pb.o
 encode_unittests: encode_unittests.o pb_encode.o unittestproto.pb.o
+test_no_messages: no_messages.pb.h no_messages.pb.c no_messages.pb.o
 
 %.pb: %.proto
 	protoc -I. -I../generator -I/usr/include -o$@ $<
@@ -41,7 +55,7 @@ coverage: run_unittests
 	gcov pb_encode.gcda
 	gcov pb_decode.gcda
 
-run_unittests: decode_unittests encode_unittests test_encode1 test_encode2 test_encode3 test_decode1 test_decode2 test_decode3 test_encode_callbacks test_decode_callbacks
+run_unittests: decode_unittests encode_unittests test_cxxcompile test_encode1 test_encode2 test_encode3 test_decode1 test_decode2 test_decode3 test_encode_callbacks test_decode_callbacks test_missing_fields
 	rm -f *.gcda
 	
 	./decode_unittests > /dev/null
@@ -60,7 +74,10 @@ run_unittests: decode_unittests encode_unittests test_encode1 test_encode2 test_
 	"`./test_encode_callbacks | protoc --decode=TestMessage callbacks.proto`" ]
 
 	./test_encode3 | ./test_decode3
-	./test_encode3 | protoc --decode=AllTypes -I. -I../generator -I/usr/include alltypes.proto >/dev/null
+	./test_encode3 1 | ./test_decode3 1
+	./test_encode3 1 | protoc --decode=AllTypes -I. -I../generator -I/usr/include alltypes.proto >/dev/null
+	
+	./test_missing_fields
 
 run_fuzztest: test_decode2
 	bash -c 'I=1; while true; do cat /dev/urandom | ./test_decode2 > /dev/null; I=$$(($$I+1)); echo -en "\r$$I"; done'

tests/alltypes.proto

@@ -1,11 +1,13 @@
 import "nanopb.proto";
 
 message SubMessage {
-    required string substuff1 = 1 [(nanopb).max_size = 16];
-    required int32 substuff2 = 2;
+    required string substuff1 = 1 [(nanopb).max_size = 16, default = "1"];
+    required int32 substuff2 = 2 [default = 2];
+    optional fixed32 substuff3 = 3 [default = 3];
 }
 
 enum MyEnum {
+    Zero = 0;
     First = 1;
     Second = 2;
     Truth = 42;
@@ -33,6 +35,49 @@ message AllTypes {
     required SubMessage req_submsg  = 16;
     required MyEnum     req_enum    = 17;
     
+    
+    repeated int32      rep_int32   = 21 [(nanopb).max_count = 5];
+    repeated int64      rep_int64   = 22 [(nanopb).max_count = 5];
+    repeated uint32     rep_uint32  = 23 [(nanopb).max_count = 5];
+    repeated uint64     rep_uint64  = 24 [(nanopb).max_count = 5];
+    repeated sint32     rep_sint32  = 25 [(nanopb).max_count = 5];
+    repeated sint64     rep_sint64  = 26 [(nanopb).max_count = 5];
+    repeated bool       rep_bool    = 27 [(nanopb).max_count = 5];
+    
+    repeated fixed32    rep_fixed32 = 28 [(nanopb).max_count = 5];
+    repeated sfixed32   rep_sfixed32= 29 [(nanopb).max_count = 5];
+    repeated float      rep_float   = 30 [(nanopb).max_count = 5];
+    
+    repeated fixed64    rep_fixed64 = 31 [(nanopb).max_count = 5];
+    repeated sfixed64   rep_sfixed64= 32 [(nanopb).max_count = 5];
+    repeated double     rep_double  = 33 [(nanopb).max_count = 5];
+    
+    repeated string     rep_string  = 34 [(nanopb).max_size = 16, (nanopb).max_count = 5];
+    repeated bytes      rep_bytes   = 35 [(nanopb).max_size = 16, (nanopb).max_count = 5];
+    repeated SubMessage rep_submsg  = 36 [(nanopb).max_count = 5];
+    repeated MyEnum     rep_enum    = 37 [(nanopb).max_count = 5];
+    
+    optional int32      opt_int32   = 41 [default = 4041];
+    optional int64      opt_int64   = 42 [default = 4042];
+    optional uint32     opt_uint32  = 43 [default = 4043];
+    optional uint64     opt_uint64  = 44 [default = 4044];
+    optional sint32     opt_sint32  = 45 [default = 4045];
+    optional sint64     opt_sint64  = 46 [default = 4046];
+    optional bool       opt_bool    = 47 [default = false];
+    
+    optional fixed32    opt_fixed32 = 48 [default = 4048];
+    optional sfixed32   opt_sfixed32= 49 [default = 4049];
+    optional float      opt_float   = 50 [default = 4050];
+    
+    optional fixed64    opt_fixed64 = 51 [default = 4051];
+    optional sfixed64   opt_sfixed64= 52 [default = 4052];
+    optional double     opt_double  = 53 [default = 4053];
+    
+    optional string     opt_string  = 54 [(nanopb).max_size = 16, default = "4054"];
+    optional bytes      opt_bytes   = 55 [(nanopb).max_size = 16, default = "4055"];
+    optional SubMessage opt_submsg  = 56;
+    optional MyEnum     opt_enum    = 57 [default = Second];
+
     // Just to make sure that the size of the fields has been calculated
     // properly, i.e. otherwise a bug in last field might not be detected.
     required int32      end = 99;

tests/decode_unittests.c

@@ -1,3 +1,5 @@
+#define NANOPB_INTERNALS
+
 #include <stdio.h>
 #include <string.h>
 #include "pb_decode.h"

tests/missing_fields.proto

@@ -0,0 +1,138 @@
+/* Test for one missing field among many */
+
+message AllFields
+{
+    required int32 field1 = 1;
+    required int32 field2 = 2;
+    required int32 field3 = 3;
+    required int32 field4 = 4;
+    required int32 field5 = 5;
+    required int32 field6 = 6;
+    required int32 field7 = 7;
+    required int32 field8 = 8;
+    required int32 field9 = 9;
+    required int32 field10 = 10;
+    required int32 field11 = 11;
+    required int32 field12 = 12;
+    required int32 field13 = 13;
+    required int32 field14 = 14;
+    required int32 field15 = 15;
+    required int32 field16 = 16;
+    required int32 field17 = 17;
+    required int32 field18 = 18;
+    required int32 field19 = 19;
+    required int32 field20 = 20;
+    required int32 field21 = 21;
+    required int32 field22 = 22;
+    required int32 field23 = 23;
+    required int32 field24 = 24;
+    required int32 field25 = 25;
+    required int32 field26 = 26;
+    required int32 field27 = 27;
+    required int32 field28 = 28;
+    required int32 field29 = 29;
+    required int32 field30 = 30;
+    required int32 field31 = 31;
+    required int32 field32 = 32;
+    required int32 field33 = 33;
+    required int32 field34 = 34;
+    required int32 field35 = 35;
+    required int32 field36 = 36;
+    required int32 field37 = 37;
+    required int32 field38 = 38;
+    required int32 field39 = 39;
+    required int32 field40 = 40;
+    required int32 field41 = 41;
+    required int32 field42 = 42;
+    required int32 field43 = 43;
+    required int32 field44 = 44;
+    required int32 field45 = 45;
+    required int32 field46 = 46;
+    required int32 field47 = 47;
+    required int32 field48 = 48;
+    required int32 field49 = 49;
+    required int32 field50 = 50;
+    required int32 field51 = 51;
+    required int32 field52 = 52;
+    required int32 field53 = 53;
+    required int32 field54 = 54;
+    required int32 field55 = 55;
+    required int32 field56 = 56;
+    required int32 field57 = 57;
+    required int32 field58 = 58;
+    required int32 field59 = 59;
+    required int32 field60 = 60;
+    required int32 field61 = 61;
+    required int32 field62 = 62;
+    required int32 field63 = 63;
+    required int32 field64 = 64;
+}
+
+message MissingField
+{
+    required int32 field1 = 1;
+    required int32 field2 = 2;
+    required int32 field3 = 3;
+    required int32 field4 = 4;
+    required int32 field5 = 5;
+    required int32 field6 = 6;
+    required int32 field7 = 7;
+    required int32 field8 = 8;
+    required int32 field9 = 9;
+    required int32 field10 = 10;
+    required int32 field11 = 11;
+    required int32 field12 = 12;
+    required int32 field13 = 13;
+    required int32 field14 = 14;
+    required int32 field15 = 15;
+    required int32 field16 = 16;
+    required int32 field17 = 17;
+    required int32 field18 = 18;
+    required int32 field19 = 19;
+    required int32 field20 = 20;
+    required int32 field21 = 21;
+    required int32 field22 = 22;
+    required int32 field23 = 23;
+    required int32 field24 = 24;
+    required int32 field25 = 25;
+    required int32 field26 = 26;
+    required int32 field27 = 27;
+    required int32 field28 = 28;
+    required int32 field29 = 29;
+    required int32 field30 = 30;
+    required int32 field31 = 31;
+    required int32 field32 = 32;
+    required int32 field33 = 33;
+    required int32 field34 = 34;
+    required int32 field35 = 35;
+    required int32 field36 = 36;
+    required int32 field37 = 37;
+    required int32 field38 = 38;
+    required int32 field39 = 39;
+    required int32 field40 = 40;
+    required int32 field41 = 41;
+    required int32 field42 = 42;
+    required int32 field43 = 43;
+    required int32 field44 = 44;
+    required int32 field45 = 45;
+    required int32 field46 = 46;
+    required int32 field47 = 47;
+    required int32 field48 = 48;
+    required int32 field49 = 49;
+    required int32 field50 = 50;
+    required int32 field51 = 51;
+    required int32 field52 = 52;
+    required int32 field53 = 53;
+    required int32 field54 = 54;
+    required int32 field55 = 55;
+    required int32 field56 = 56;
+    required int32 field57 = 57;
+    required int32 field58 = 58;
+    required int32 field59 = 59;
+    required int32 field60 = 60;
+    required int32 field61 = 61;
+    required int32 field62 = 62;
+/*    required int32 field63 = 63; */
+    required int32 field64 = 64;
+}
+

tests/no_messages.proto

@@ -0,0 +1,7 @@
+/* Test that a file without any messages works. */
+
+enum Test {
+    First = 1;
+}
+
+

tests/test_decode1.c

@@ -69,7 +69,7 @@ int main()
     /* Decode and print out the stuff */
     if (!print_person(&stream))
     {
-        printf("Parsing failed.\n");
+        printf("Parsing failed: %s\n", PB_GET_ERROR(&stream));
         return 1;
     } else {
         return 0;

tests/test_decode2.c

@@ -82,7 +82,7 @@ int main()
     pb_istream_t stream = {&callback, stdin, 10000};
     if (!print_person(&stream))
     {
-        printf("Parsing failed.\n");
+        printf("Parsing failed: %s\n", PB_GET_ERROR(&stream));
         return 1;
     } else {
         return 0;

tests/test_decode3.c

@@ -1,10 +1,11 @@
-/* Tests the decoding of all types. Currently only in the 'required' variety.
+/* Tests the decoding of all types.
  * This is the counterpart of test_encode3.
  * Run e.g. ./test_encode3 | ./test_decode3
  */
 
 #include <stdio.h>
 #include <string.h>
+#include <stdlib.h>
 #include <pb_decode.h>
 #include "alltypes.pb.h"
 
@@ -15,9 +16,12 @@
 
 /* This function is called once from main(), it handles
    the decoding and checks the fields. */
-bool check_alltypes(pb_istream_t *stream)
+bool check_alltypes(pb_istream_t *stream, int mode)
 {
-    AllTypes alltypes = {};
+    AllTypes alltypes;
+    
+    /* Fill with garbage to better detect initialization errors */
+    memset(&alltypes, 0xAA, sizeof(alltypes));
     
     if (!pb_decode(stream, AllTypes_fields, &alltypes))
         return false;
@@ -43,26 +47,146 @@ bool check_alltypes(pb_istream_t *stream)
     TEST(memcmp(alltypes.req_bytes.bytes, "1015", 4) == 0);
     TEST(strcmp(alltypes.req_submsg.substuff1, "1016") == 0);
     TEST(alltypes.req_submsg.substuff2 == 1016);
+    TEST(alltypes.req_submsg.substuff3 == 3);
     TEST(alltypes.req_enum == MyEnum_Truth);
     
+    TEST(alltypes.rep_int32_count == 5 && alltypes.rep_int32[4] == 2001 && alltypes.rep_int32[0] == 0);
+    TEST(alltypes.rep_int64_count == 5 && alltypes.rep_int64[4] == 2002 && alltypes.rep_int64[0] == 0);
+    TEST(alltypes.rep_uint32_count == 5 && alltypes.rep_uint32[4] == 2003 && alltypes.rep_uint32[0] == 0);
+    TEST(alltypes.rep_uint64_count == 5 && alltypes.rep_uint64[4] == 2004 && alltypes.rep_uint64[0] == 0);
+    TEST(alltypes.rep_sint32_count == 5 && alltypes.rep_sint32[4] == 2005 && alltypes.rep_sint32[0] == 0);
+    TEST(alltypes.rep_sint64_count == 5 && alltypes.rep_sint64[4] == 2006 && alltypes.rep_sint64[0] == 0);
+    TEST(alltypes.rep_bool_count == 5 && alltypes.rep_bool[4] == true && alltypes.rep_bool[0] == false);
+    
+    TEST(alltypes.rep_fixed32_count == 5 && alltypes.rep_fixed32[4] == 2008 && alltypes.rep_fixed32[0] == 0);
+    TEST(alltypes.rep_sfixed32_count == 5 && alltypes.rep_sfixed32[4] == 2009 && alltypes.rep_sfixed32[0] == 0);
+    TEST(alltypes.rep_float_count == 5 && alltypes.rep_float[4] == 2010.0f && alltypes.rep_float[0] == 0.0f);
+    
+    TEST(alltypes.rep_fixed64_count == 5 && alltypes.rep_fixed64[4] == 2011 && alltypes.rep_fixed64[0] == 0);
+    TEST(alltypes.rep_sfixed64_count == 5 && alltypes.rep_sfixed64[4] == 2012 && alltypes.rep_sfixed64[0] == 0);
+    TEST(alltypes.rep_double_count == 5 && alltypes.rep_double[4] == 2013.0 && alltypes.rep_double[0] == 0.0);
+    
+    TEST(alltypes.rep_string_count == 5 && strcmp(alltypes.rep_string[4], "2014") == 0 && alltypes.rep_string[0][0] == '\0');
+    TEST(alltypes.rep_bytes_count == 5 && alltypes.rep_bytes[4].size == 4 && alltypes.rep_bytes[0].size == 0);
+    TEST(memcmp(alltypes.rep_bytes[4].bytes, "2015", 4) == 0);
+
+    TEST(alltypes.rep_submsg_count == 5);
+    TEST(strcmp(alltypes.rep_submsg[4].substuff1, "2016") == 0 && alltypes.rep_submsg[0].substuff1[0] == '\0');
+    TEST(alltypes.rep_submsg[4].substuff2 == 2016 && alltypes.rep_submsg[0].substuff2 == 0);
+    TEST(alltypes.rep_submsg[4].substuff3 == 2016 && alltypes.rep_submsg[0].substuff3 == 3);
+    
+    TEST(alltypes.rep_enum_count == 5 && alltypes.rep_enum[4] == MyEnum_Truth && alltypes.rep_enum[0] == MyEnum_Zero);
+    
+    if (mode == 0)
+    {
+        /* Expect default values */
+        TEST(alltypes.has_opt_int32     == false);
+        TEST(alltypes.opt_int32         == 4041);
+        TEST(alltypes.has_opt_int64     == false);
+        TEST(alltypes.opt_int64         == 4042);
+        TEST(alltypes.has_opt_uint32    == false);
+        TEST(alltypes.opt_uint32        == 4043);
+        TEST(alltypes.has_opt_uint64    == false);
+        TEST(alltypes.opt_uint64        == 4044);
+        TEST(alltypes.has_opt_sint32    == false);
+        TEST(alltypes.opt_sint32        == 4045);
+        TEST(alltypes.has_opt_sint64    == false);
+        TEST(alltypes.opt_sint64        == 4046);
+        TEST(alltypes.has_opt_bool      == false);
+        TEST(alltypes.opt_bool          == false);
+        
+        TEST(alltypes.has_opt_fixed32   == false);
+        TEST(alltypes.opt_fixed32       == 4048);
+        TEST(alltypes.has_opt_sfixed32  == false);
+        TEST(alltypes.opt_sfixed32      == 4049);
+        TEST(alltypes.has_opt_float     == false);
+        TEST(alltypes.opt_float         == 4050.0f);
+        
+        TEST(alltypes.has_opt_fixed64   == false);
+        TEST(alltypes.opt_fixed64       == 4051);
+        TEST(alltypes.has_opt_sfixed64  == false);
+        TEST(alltypes.opt_sfixed64      == 4052);
+        TEST(alltypes.has_opt_double    == false);
+        TEST(alltypes.opt_double        == 4053.0);
+        
+        TEST(alltypes.has_opt_string    == false);
+        TEST(strcmp(alltypes.opt_string, "4054") == 0);
+        TEST(alltypes.has_opt_bytes     == false);
+        TEST(alltypes.opt_bytes.size == 4);
+        TEST(memcmp(alltypes.opt_bytes.bytes, "4055", 4) == 0);
+        TEST(alltypes.has_opt_submsg    == false);
+        TEST(strcmp(alltypes.opt_submsg.substuff1, "1") == 0);
+        TEST(alltypes.opt_submsg.substuff2 == 2);
+        TEST(alltypes.opt_submsg.substuff3 == 3);
+        TEST(alltypes.has_opt_enum     == false);
+        TEST(alltypes.opt_enum == MyEnum_Second);
+    }
+    else
+    {
+        /* Expect filled-in values */
+        TEST(alltypes.has_opt_int32     == true);
+        TEST(alltypes.opt_int32         == 3041);
+        TEST(alltypes.has_opt_int64     == true);
+        TEST(alltypes.opt_int64         == 3042);
+        TEST(alltypes.has_opt_uint32    == true);
+        TEST(alltypes.opt_uint32        == 3043);
+        TEST(alltypes.has_opt_uint64    == true);
+        TEST(alltypes.opt_uint64        == 3044);
+        TEST(alltypes.has_opt_sint32    == true);
+        TEST(alltypes.opt_sint32        == 3045);
+        TEST(alltypes.has_opt_sint64    == true);
+        TEST(alltypes.opt_sint64        == 3046);
+        TEST(alltypes.has_opt_bool      == true);
+        TEST(alltypes.opt_bool          == true);
+        
+        TEST(alltypes.has_opt_fixed32   == true);
+        TEST(alltypes.opt_fixed32       == 3048);
+        TEST(alltypes.has_opt_sfixed32  == true);
+        TEST(alltypes.opt_sfixed32      == 3049);
+        TEST(alltypes.has_opt_float     == true);
+        TEST(alltypes.opt_float         == 3050.0f);
+        
+        TEST(alltypes.has_opt_fixed64   == true);
+        TEST(alltypes.opt_fixed64       == 3051);
+        TEST(alltypes.has_opt_sfixed64  == true);
+        TEST(alltypes.opt_sfixed64      == 3052);
+        TEST(alltypes.has_opt_double    == true);
+        TEST(alltypes.opt_double        == 3053.0);
+        
+        TEST(alltypes.has_opt_string    == true);
+        TEST(strcmp(alltypes.opt_string, "3054") == 0);
+        TEST(alltypes.has_opt_bytes     == true);
+        TEST(alltypes.opt_bytes.size == 4);
+        TEST(memcmp(alltypes.opt_bytes.bytes, "3055", 4) == 0);
+        TEST(alltypes.has_opt_submsg    == true);
+        TEST(strcmp(alltypes.opt_submsg.substuff1, "3056") == 0);
+        TEST(alltypes.opt_submsg.substuff2 == 3056);
+        TEST(alltypes.opt_submsg.substuff3 == 3);
+        TEST(alltypes.has_opt_enum      == true);
+        TEST(alltypes.opt_enum == MyEnum_Truth);
+    }
+    
     TEST(alltypes.end == 1099);
     
     return true;
 }
 
-int main()
+int main(int argc, char **argv)
 {
+    /* Whether to expect the optional values or the default values. */
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+    
     /* Read the data into buffer */
-    uint8_t buffer[512];
+    uint8_t buffer[1024];
     size_t count = fread(buffer, 1, sizeof(buffer), stdin);
     
     /* Construct a pb_istream_t for reading from the buffer */
     pb_istream_t stream = pb_istream_from_buffer(buffer, count);
     
     /* Decode and print out the stuff */
-    if (!check_alltypes(&stream))
+    if (!check_alltypes(&stream, mode))
     {
-        printf("Parsing failed.\n");
+        printf("Parsing failed: %s\n", PB_GET_ERROR(&stream));
         return 1;
     } else {
         return 0;

tests/test_decode_callbacks.c

@@ -37,7 +37,7 @@ bool print_int32(pb_istream_t *stream, const pb_field_t *field, void *arg)
 bool print_fixed32(pb_istream_t *stream, const pb_field_t *field, void *arg)
 {
     uint32_t value;
-    if (!pb_dec_fixed32(stream, NULL, &value))
+    if (!pb_decode_fixed32(stream, &value))
         return false;
     
     printf((char*)arg, (long)value);
@@ -47,7 +47,7 @@ bool print_fixed32(pb_istream_t *stream, const pb_field_t *field, void *arg)
 bool print_fixed64(pb_istream_t *stream, const pb_field_t *field, void *arg)
 {
     uint64_t value;
-    if (!pb_dec_fixed64(stream, NULL, &value))
+    if (!pb_decode_fixed64(stream, &value))
         return false;
     
     printf((char*)arg, (long long)value);

tests/test_encode3.c

@@ -1,40 +1,117 @@
 /* Attempts to test all the datatypes supported by ProtoBuf.
- * Currently only tests the 'required' variety.
  */
 
 #include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
 #include <pb_encode.h>
 #include "alltypes.pb.h"
 
-int main()
+int main(int argc, char **argv)
 {
-    /* Initialize the structure with constants */
-    AllTypes alltypes = {
-        1001,
-        1002,
-        1003,
-        1004,
-        1005,
-        1006,
-        true,
-        
-        1008,
-        1009,
-        1010.0f,
-        
-        1011,
-        1012,
-        1013.0,
-        
-        "1014",
-        {4, "1015"},
-        {"1016", 1016},
-        MyEnum_Truth,
-        
-        1099
-    };
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
     
-    uint8_t buffer[512];
+    /* Initialize the structure with constants */
+    AllTypes alltypes = {0};
+    
+    alltypes.req_int32         = 1001;
+    alltypes.req_int64         = 1002;
+    alltypes.req_uint32        = 1003;
+    alltypes.req_uint64        = 1004;
+    alltypes.req_sint32        = 1005;
+    alltypes.req_sint64        = 1006;
+    alltypes.req_bool          = true;
+    
+    alltypes.req_fixed32       = 1008;
+    alltypes.req_sfixed32      = 1009;
+    alltypes.req_float         = 1010.0f;
+    
+    alltypes.req_fixed64       = 1011;
+    alltypes.req_sfixed64      = 1012;
+    alltypes.req_double        = 1013.0;
+    
+    strcpy(alltypes.req_string, "1014");
+    alltypes.req_bytes.size = 4;
+    memcpy(alltypes.req_bytes.bytes, "1015", 4);
+    strcpy(alltypes.req_submsg.substuff1, "1016");
+    alltypes.req_submsg.substuff2 = 1016;
+    alltypes.req_enum = MyEnum_Truth;
+    
+    alltypes.rep_int32_count = 5; alltypes.rep_int32[4] = 2001;
+    alltypes.rep_int64_count = 5; alltypes.rep_int64[4] = 2002;
+    alltypes.rep_uint32_count = 5; alltypes.rep_uint32[4] = 2003;
+    alltypes.rep_uint64_count = 5; alltypes.rep_uint64[4] = 2004;
+    alltypes.rep_sint32_count = 5; alltypes.rep_sint32[4] = 2005;
+    alltypes.rep_sint64_count = 5; alltypes.rep_sint64[4] = 2006;
+    alltypes.rep_bool_count = 5; alltypes.rep_bool[4] = true;
+    
+    alltypes.rep_fixed32_count = 5; alltypes.rep_fixed32[4] = 2008;
+    alltypes.rep_sfixed32_count = 5; alltypes.rep_sfixed32[4] = 2009;
+    alltypes.rep_float_count = 5; alltypes.rep_float[4] = 2010.0f;
+    
+    alltypes.rep_fixed64_count = 5; alltypes.rep_fixed64[4] = 2011;
+    alltypes.rep_sfixed64_count = 5; alltypes.rep_sfixed64[4] = 2012;
+    alltypes.rep_double_count = 5; alltypes.rep_double[4] = 2013.0;
+    
+    alltypes.rep_string_count = 5; strcpy(alltypes.rep_string[4], "2014");
+    alltypes.rep_bytes_count = 5; alltypes.rep_bytes[4].size = 4;
+    memcpy(alltypes.rep_bytes[4].bytes, "2015", 4);
+
+    alltypes.rep_submsg_count = 5;
+    strcpy(alltypes.rep_submsg[4].substuff1, "2016");
+    alltypes.rep_submsg[4].substuff2 = 2016;
+    alltypes.rep_submsg[4].has_substuff3 = true;
+    alltypes.rep_submsg[4].substuff3 = 2016;
+    
+    alltypes.rep_enum_count = 5; alltypes.rep_enum[4] = MyEnum_Truth;
+    
+    if (mode != 0)
+    {
+        /* Fill in values for optional fields */
+        alltypes.has_opt_int32 = true;
+        alltypes.opt_int32         = 3041;
+        alltypes.has_opt_int64 = true;
+        alltypes.opt_int64         = 3042;
+        alltypes.has_opt_uint32 = true;
+        alltypes.opt_uint32        = 3043;
+        alltypes.has_opt_uint64 = true;
+        alltypes.opt_uint64        = 3044;
+        alltypes.has_opt_sint32 = true;
+        alltypes.opt_sint32        = 3045;
+        alltypes.has_opt_sint64 = true;
+        alltypes.opt_sint64        = 3046;
+        alltypes.has_opt_bool = true;
+        alltypes.opt_bool          = true;
+        
+        alltypes.has_opt_fixed32 = true;
+        alltypes.opt_fixed32       = 3048;
+        alltypes.has_opt_sfixed32 = true;
+        alltypes.opt_sfixed32      = 3049;
+        alltypes.has_opt_float = true;
+        alltypes.opt_float         = 3050.0f;
+        
+        alltypes.has_opt_fixed64 = true;
+        alltypes.opt_fixed64       = 3051;
+        alltypes.has_opt_sfixed64 = true;
+        alltypes.opt_sfixed64      = 3052;
+        alltypes.has_opt_double = true;
+        alltypes.opt_double        = 3053.0;
+        
+        alltypes.has_opt_string = true;
+        strcpy(alltypes.opt_string, "3054");
+        alltypes.has_opt_bytes = true;
+        alltypes.opt_bytes.size = 4;
+        memcpy(alltypes.opt_bytes.bytes, "3055", 4);
+        alltypes.has_opt_submsg = true;
+        strcpy(alltypes.opt_submsg.substuff1, "3056");
+        alltypes.opt_submsg.substuff2 = 3056;
+        alltypes.has_opt_enum = true;
+        alltypes.opt_enum = MyEnum_Truth;
+    }
+    
+    alltypes.end = 1099;
+    
+    uint8_t buffer[1024];
     pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
     
     /* Now encode it and check if we succeeded. */
@@ -45,6 +122,7 @@ int main()
     }
     else
     {
+        fprintf(stderr, "Encoding failed!\n");
         return 1; /* Failure */
     }
 }

tests/test_missing_fields.c

@@ -0,0 +1,49 @@
+/* Checks that missing required fields are detected properly */
+
+#include <stdio.h>
+#include <pb_encode.h>
+#include <pb_decode.h>
+#include "missing_fields.pb.h"
+
+int main()
+{
+    uint8_t buffer[512] = {};
+    
+    /* Create a message with one missing field */
+    {
+        MissingField msg = {};
+        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        if (!pb_encode(&stream, MissingField_fields, &msg))
+        {
+            printf("Encode failed.\n");
+            return 1;
+        }
+    }
+
+    /* Test that it decodes properly if we don't require that field */
+    {
+        MissingField msg = {};
+        pb_istream_t stream = pb_istream_from_buffer(buffer, sizeof(buffer));
+        
+        if (!pb_decode(&stream, MissingField_fields, &msg))
+        {
+            printf("Decode failed: %s\n", PB_GET_ERROR(&stream));
+            return 2;
+        }
+    }
+    
+    /* Test that it does *not* decode properly if we require the field */
+    {
+        AllFields msg = {};
+        pb_istream_t stream = pb_istream_from_buffer(buffer, sizeof(buffer));
+        
+        if (pb_decode(&stream, AllFields_fields, &msg))
+        {
+            printf("Decode didn't detect missing field.\n");
+            return 3;
+        }
+    }
+    
+    return 0; /* All ok */
+}
+