Publishing nanopb-0.2.7

Also modified a few tests to be more compatible with coverage information,
so that they use the same pb_encode/decode.c instead of making a copy.

CHANGELOG.txt

@@ -1,3 +1,11 @@
+nanopb-0.2.7 (2014-04-07)
+ Fix bug with default values for extension fields (issue 111)
+ Fix some MISRA-C warnings (issue 91)
+ Implemented optional malloc() support (issue 80)
+ Changed pointer-type bytes field datatype
+ Add a "found" field to pb_extension_t (issue 112)
+ Add convenience function pb_get_encoded_size() (issue 16)
+
 nanopb-0.2.6 (2014-02-15)
  Fix generator error with bytes callback fields (issue 99)
  Fix warnings about large integer constants (issue 102)

docs/reference.rst

@@ -24,10 +24,8 @@ __BIG_ENDIAN__                 Set this if your platform stores integers and
                                floats in big-endian format. Mixed-endian
                                systems (different layout for ints and floats)
                                are currently not supported.
-NANOPB_INTERNALS               Set this to expose the field encoder functions
-                               that are hidden since nanopb-0.1.3. Starting
-                               with nanopb-0.2.4, this flag does nothing. Use
-                               the newer functions that have better interface.
+PB_ENABLE_MALLOC               Set this to enable dynamic allocation support
+                               in the decoder.
 PB_MAX_REQUIRED_FIELDS         Maximum number of required fields to check for
                                presence. Default value is 64. Increases stack
                                usage 1 byte per every 8 fields. Compiler
@@ -77,8 +75,9 @@ max_count                      Allocated number of entries in arrays
                                (*repeated* fields).
 type                           Type of the generated field. Default value
                                is *FT_DEFAULT*, which selects automatically.
-                               You can use *FT_CALLBACK*, *FT_STATIC* or
-                               *FT_IGNORE* to force a callback field, a static
+                               You can use *FT_CALLBACK*, *FT_POINTER*,
+                               *FT_STATIC* or *FT_IGNORE* to force a callback
+                               field, a dynamically allocated field, a static
                                field or to completely ignore the field.
 long_names                     Prefix the enum name to the enum value in
                                definitions, i.e. *EnumName_EnumValue*. Enabled
@@ -417,6 +416,17 @@ Encodes the contents of a structure as a protocol buffers message and writes it
 
 Normally pb_encode simply walks through the fields description array and serializes each field in turn. However, submessages must be serialized twice: first to calculate their size and then to actually write them to output. This causes some constraints for callback fields, which must return the same data on every call.
 
+pb_encode_delimited
+-------------------
+Calculates the length of the message, encodes it as varint and then encodes the message. ::
+
+    bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
+
+(parameters are the same as for `pb_encode`_.)
+
+A common way to indicate the message length in Protocol Buffers is to prefix it with a varint.
+This function does this, and it is compatible with *parseDelimitedFrom* in Google's protobuf library.
+
 .. sidebar:: Encoding fields manually
 
     The functions with names *pb_encode_\** are used when dealing with callback fields. The typical reason for using callbacks is to have an array of unlimited size. In that case, `pb_encode`_ will call your callback function, which in turn will call *pb_encode_\** functions repeatedly to write out values.
@@ -579,6 +589,10 @@ In addition to EOF, the pb_decode implementation supports terminating a message
 
 For optional fields, this function applies the default value and sets *has_<field>* to false if the field is not present.
 
+If *PB_ENABLE_MALLOC* is defined, this function may allocate storage for any pointer type fields.
+In this case, you have to call `pb_release`_ to release the memory after you are done with the message.
+On error return `pb_decode` will release the memory itself.
+
 pb_decode_noinit
 ----------------
 Same as `pb_decode`_, except does not apply the default values to fields. ::
@@ -589,6 +603,35 @@ Same as `pb_decode`_, except does not apply the default values to fields. ::
 
 The destination structure should be filled with zeros before calling this function. Doing a *memset* manually can be slightly faster than using `pb_decode`_ if you don't need any default values.
 
+In addition to decoding a single message, this function can be used to merge two messages, so that
+values from previous message will remain if the new message does not contain a field.
+
+This function *will not* release the message even on error return. If you use *PB_ENABLE_MALLOC*,
+you will need to call `pb_release`_ yourself.
+
+pb_decode_delimited
+-------------------
+Same as `pb_decode`_, except that it first reads a varint with the length of the message. ::
+
+    bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
+
+(parameters are the same as for `pb_decode`_.)
+
+A common method to indicate message size in Protocol Buffers is to prefix it with a varint.
+This function is compatible with *writeDelimitedTo* in the Google's Protocol Buffers library.
+
+pb_release
+----------
+Releases any dynamically allocated fields.
+
+    void pb_release(const pb_field_t fields[], void *dest_struct);
+
+:fields:        A field description array. Usually autogenerated.
+:dest_struct:   Pointer to structure where data will be stored.
+
+This function is only available if *PB_ENABLE_MALLOC* is defined. It will release any
+pointer type fields in the structure and set the pointers to NULL.
+
 pb_skip_varint
 --------------
 Skip a varint_ encoded integer without decoding it. ::

extra/pb_syshdr.h

@@ -53,6 +53,16 @@ typedef int bool;
 
 #endif
 
+/* stdlib.h subset */
+#ifdef PB_ENABLE_MALLOC
+#ifdef HAVE_STDLIB_H
+#include <stdlib.h>
+#else
+void *realloc(void *ptr, size_t size);
+void free(void *ptr);
+#endif
+#endif
+
 /* string.h subset */
 #ifdef HAVE_STRING_H
 #include <string.h>

generator/nanopb_generator.py

@@ -1,7 +1,7 @@
 #!/usr/bin/python
 
 '''Generate header file for nanopb from a ProtoBuf FileDescriptorSet.'''
-nanopb_version = "nanopb-0.2.6"
+nanopb_version = "nanopb-0.2.7"
 
 import sys
 
@@ -246,7 +246,7 @@ class Field:
                 self.ctype = self.struct_name + self.name + 't'
                 self.enc_size = varint_max_size(self.max_size) + self.max_size
             elif self.allocation == 'POINTER':
-                self.ctype = 'pb_bytes_ptr_t'
+                self.ctype = 'pb_bytes_array_t'
         elif desc.type == FieldD.TYPE_MESSAGE:
             self.pbtype = 'MESSAGE'
             self.ctype = self.submsgname = names_from_type_name(desc.type_name)
@@ -266,8 +266,8 @@ class Field:
             if self.pbtype == 'MESSAGE':
                 # Use struct definition, so recursive submessages are possible
                 result += '    struct _%s *%s;' % (self.ctype, self.name)
-            elif self.rules == 'REPEATED' and self.pbtype == 'STRING':
-                # String arrays need to be defined as pointers to pointers
+            elif self.rules == 'REPEATED' and self.pbtype in ['STRING', 'BYTES']:
+                # String/bytes arrays need to be defined as pointers to pointers
                 result += '    %s **%s;' % (self.ctype, self.name)
             else:
                 result += '    %s *%s;' % (self.ctype, self.name)
@@ -351,6 +351,8 @@ class Field:
             result += '0)'
         elif self.pbtype in ['BYTES', 'STRING'] and self.allocation != 'STATIC':
             result += '0)' # Arbitrary size default values not implemented
+        elif self.rules == 'OPTEXT':
+            result += '0)' # Default value for extensions is not implemented
         else:
             result += '&%s_default)' % (self.struct_name + self.name)
         

pb.h

@@ -10,6 +10,9 @@
  * uncommenting the lines, or on the compiler command line.      *
  *****************************************************************/
 
+/* Enable support for dynamically allocated fields */
+/* #define PB_ENABLE_MALLOC 1 */
+
 /* Define this if your CPU architecture is big endian, i.e. it
  * stores the most-significant byte first. */
 /* #define __BIG_ENDIAN__ 1 */
@@ -43,7 +46,7 @@
 
 /* Version of the nanopb library. Just in case you want to check it in
  * your own program. */
-#define NANOPB_VERSION nanopb-0.2.6
+#define NANOPB_VERSION nanopb-0.2.7
 
 /* Include all the system headers needed by nanopb. You will need the
  * definitions of the following:
@@ -63,6 +66,10 @@
 #include <stddef.h>
 #include <stdbool.h>
 #include <string.h>
+
+#ifdef PB_ENABLE_MALLOC
+#include <stdlib.h>
+#endif
 #endif
 
 /* Macro for defining packed structures (compiler dependent).
@@ -234,21 +241,15 @@ STATIC_ASSERT(sizeof(uint64_t) == 8, UINT64_T_WRONG_SIZE)
  * It has the number of bytes in the beginning, and after that an array.
  * Note that actual structs used will have a different length of bytes array.
  */
+#define PB_BYTES_ARRAY_T(n) struct { size_t size; uint8_t bytes[n]; }
+#define PB_BYTES_ARRAY_T_ALLOCSIZE(n) ((size_t)n + offsetof(pb_bytes_array_t, bytes))
+
 struct _pb_bytes_array_t {
     size_t size;
     uint8_t bytes[1];
 };
 typedef struct _pb_bytes_array_t pb_bytes_array_t;
 
-/* Same, except for pointer-type fields. There is no need to variable struct
- * length in this case.
- */
-struct _pb_bytes_ptr_t {
-    size_t size;
-    uint8_t *bytes;
-};
-typedef struct _pb_bytes_ptr_t pb_bytes_ptr_t;
-
 /* This structure is used for giving the callback function.
  * It is stored in the message structure and filled in by the method that
  * calls pb_decode.
@@ -340,8 +341,23 @@ struct _pb_extension_t {
      * If this extension does not match a field, the next handler is
      * automatically called. */
     pb_extension_t *next;
+
+    /* The decoder sets this to true if the extension was found.
+     * Ignored for encoding. */
+    bool found;
 };
 
+/* Memory allocation functions to use. You can define pb_realloc and
+ * pb_free to custom functions if you want. */
+#ifdef PB_ENABLE_MALLOC
+#   ifndef pb_realloc
+#       define pb_realloc(ptr, size) realloc(ptr, size)
+#   endif
+#   ifndef pb_free
+#       define pb_free(ptr) free(ptr)
+#   endif
+#endif
+
 /* These macros are used to declare pb_field_t's in the constant array. */
 /* Size of a structure member, in bytes. */
 #define pb_membersize(st, m) (sizeof ((st*)0)->m)
@@ -484,7 +500,11 @@ struct _pb_extension_t {
  * messages if not used.
  */
 #ifdef PB_NO_ERRMSG
-#define PB_RETURN_ERROR(stream,msg) return false
+#define PB_RETURN_ERROR(stream,msg) \
+    do {\
+        UNUSED(stream); \
+        return false; \
+    } while(0)
 #define PB_GET_ERROR(stream) "(errmsg disabled)"
 #else
 #define PB_RETURN_ERROR(stream,msg) \

pb_decode.c

@@ -13,7 +13,6 @@
     #define checkreturn __attribute__((warn_unused_result))
 #endif
 
-#define NANOPB_INTERNALS
 #include "pb.h"
 #include "pb_decode.h"
 
@@ -130,7 +129,7 @@ bool checkreturn pb_read(pb_istream_t *stream, uint8_t *buf, size_t count)
  * This is an optimization for the varint decoding. */
 static bool checkreturn pb_readbyte(pb_istream_t *stream, uint8_t *buf)
 {
-    if (!stream->bytes_left)
+    if (stream->bytes_left == 0)
         PB_RETURN_ERROR(stream, "end-of-stream");
 
 #ifndef PB_BUFFER_ONLY
@@ -174,7 +173,7 @@ static bool checkreturn pb_decode_varint32(pb_istream_t *stream, uint32_t *dest)
     if (!pb_readbyte(stream, &byte))
         return false;
     
-    if (!(byte & 0x80))
+    if ((byte & 0x80) == 0)
     {
         /* Quick case, 1 byte value */
         result = byte;
@@ -359,6 +358,10 @@ static bool pb_field_next(pb_field_iterator_t *iter)
     {
         prev_size *= iter->pos->array_size;
     }
+    else if (PB_ATYPE(iter->pos->type) == PB_ATYPE_POINTER)
+    {
+        prev_size = sizeof(void*);
+    }
     
     if (iter->pos->tag == 0)
         return false; /* Only happens with empty message types */
@@ -393,7 +396,7 @@ static bool checkreturn pb_field_find(pb_field_iterator_t *iter, uint32_t tag)
         {
             return true;
         }
-        pb_field_next(iter);
+        (void)pb_field_next(iter);
     } while (iter->field_index != start);
     
     return false;
@@ -431,7 +434,7 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
                 if (!pb_make_string_substream(stream, &substream))
                     return false;
                 
-                while (substream.bytes_left && *size < iter->pos->array_size)
+                while (substream.bytes_left > 0 && *size < iter->pos->array_size)
                 {
                     void *pItem = (uint8_t*)iter->pData + iter->pos->data_size * (*size);
                     if (!func(&substream, iter->pos, pItem))
@@ -465,6 +468,136 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
     }
 }
 
+#ifdef PB_ENABLE_MALLOC
+/* Allocate storage for the field and store the pointer at iter->pData.
+ * array_size is the number of entries to reserve in an array. */
+static bool checkreturn allocate_field(pb_istream_t *stream, void *pData, size_t data_size, size_t array_size)
+{    
+    void *ptr = *(void**)pData;
+    size_t size = array_size * data_size;
+    
+    /* Allocate new or expand previous allocation */
+    /* Note: on failure the old pointer will remain in the structure,
+     * the message must be freed by caller also on error return. */
+    ptr = pb_realloc(ptr, size);
+    if (ptr == NULL)
+        PB_RETURN_ERROR(stream, "realloc failed");
+    
+    *(void**)pData = ptr;
+    return true;
+}
+
+/* Clear a newly allocated item in case it contains a pointer, or is a submessage. */
+static void initialize_pointer_field(void *pItem, pb_field_iterator_t *iter)
+{
+    if (PB_LTYPE(iter->pos->type) == PB_LTYPE_STRING ||
+        PB_LTYPE(iter->pos->type) == PB_LTYPE_BYTES)
+    {
+        *(void**)pItem = NULL;
+    }
+    else if (PB_LTYPE(iter->pos->type) == PB_LTYPE_SUBMESSAGE)
+    {
+        pb_message_set_to_defaults((const pb_field_t *) iter->pos->ptr, pItem);
+    }
+}
+#endif
+
+static bool checkreturn decode_pointer_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+{
+#ifndef PB_ENABLE_MALLOC
+    UNUSED(wire_type);
+    UNUSED(iter);
+    PB_RETURN_ERROR(stream, "no malloc support");
+#else
+    pb_type_t type;
+    pb_decoder_t func;
+    
+    type = iter->pos->type;
+    func = PB_DECODERS[PB_LTYPE(type)];
+    
+    switch (PB_HTYPE(type))
+    {
+        case PB_HTYPE_REQUIRED:
+        case PB_HTYPE_OPTIONAL:
+            if (PB_LTYPE(type) == PB_LTYPE_STRING ||
+                PB_LTYPE(type) == PB_LTYPE_BYTES)
+            {
+                return func(stream, iter->pos, iter->pData);
+            }
+            else
+            {
+                if (!allocate_field(stream, iter->pData, iter->pos->data_size, 1))
+                    return false;
+                
+                initialize_pointer_field(*(void**)iter->pData, iter);
+                return func(stream, iter->pos, *(void**)iter->pData);
+            }
+    
+        case PB_HTYPE_REPEATED:
+            if (wire_type == PB_WT_STRING
+                && PB_LTYPE(type) <= PB_LTYPE_LAST_PACKABLE)
+            {
+                /* Packed array, multiple items come in at once. */
+                bool status = true;
+                size_t *size = (size_t*)iter->pSize;
+                size_t allocated_size = *size;
+                void *pItem;
+                pb_istream_t substream;
+                
+                if (!pb_make_string_substream(stream, &substream))
+                    return false;
+                
+                while (substream.bytes_left)
+                {
+                    if (*size + 1 > allocated_size)
+                    {
+                        /* Allocate more storage. This tries to guess the
+                         * number of remaining entries. Round the division
+                         * upwards. */
+                        allocated_size += (substream.bytes_left - 1) / iter->pos->data_size + 1;
+                        
+                        if (!allocate_field(&substream, iter->pData, iter->pos->data_size, allocated_size))
+                        {
+                            status = false;
+                            break;
+                        }
+                    }
+
+                    /* Decode the array entry */
+                    pItem = *(uint8_t**)iter->pData + iter->pos->data_size * (*size);
+                    initialize_pointer_field(pItem, iter);
+                    if (!func(&substream, iter->pos, pItem))
+                    {
+                        status = false;
+                        break;
+                    }
+                    (*size)++;
+                }
+                pb_close_string_substream(stream, &substream);
+                
+                return status;
+            }
+            else
+            {
+                /* Normal repeated field, i.e. only one item at a time. */
+                size_t *size = (size_t*)iter->pSize;
+                void *pItem;
+                
+                (*size)++;
+                if (!allocate_field(stream, iter->pData, iter->pos->data_size, *size))
+                    return false;
+            
+                pItem = *(uint8_t**)iter->pData + iter->pos->data_size * (*size - 1);
+                initialize_pointer_field(pItem, iter);
+                return func(stream, iter->pos, pItem);
+            }
+            
+        default:
+            PB_RETURN_ERROR(stream, "invalid field type");
+    }
+#endif
+}
+
 static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
 {
     pb_callback_t *pCallback = (pb_callback_t*)iter->pData;
@@ -519,6 +652,9 @@ static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_t
         case PB_ATYPE_STATIC:
             return decode_static_field(stream, wire_type, iter);
         
+        case PB_ATYPE_POINTER:
+            return decode_pointer_field(stream, wire_type, iter);
+        
         case PB_ATYPE_CALLBACK:
             return decode_callback_field(stream, wire_type, iter);
         
@@ -534,7 +670,6 @@ static bool checkreturn default_extension_decoder(pb_istream_t *stream,
 {
     const pb_field_t *field = (const pb_field_t*)extension->type->arg;
     pb_field_iterator_t iter;
-    bool dummy;
     
     if (field->tag != tag)
         return true;
@@ -545,7 +680,7 @@ static bool checkreturn default_extension_decoder(pb_istream_t *stream,
     iter.required_field_index = 0;
     iter.dest_struct = extension->dest;
     iter.pData = extension->dest;
-    iter.pSize = &dummy;
+    iter.pSize = &extension->found;
     
     return decode_field(stream, wire_type, &iter);
 }
@@ -558,7 +693,7 @@ static bool checkreturn decode_extension(pb_istream_t *stream,
     pb_extension_t *extension = *(pb_extension_t* const *)iter->pData;
     size_t pos = stream->bytes_left;
     
-    while (extension && pos == stream->bytes_left)
+    while (extension != NULL && pos == stream->bytes_left)
     {
         bool status;
         if (extension->type->decode)
@@ -585,7 +720,7 @@ static bool checkreturn find_extension_field(pb_field_iterator_t *iter)
     do {
         if (PB_LTYPE(iter->pos->type) == PB_LTYPE_EXTENSION)
             return true;
-        pb_field_next(iter);
+        (void)pb_field_next(iter);
     } while (iter->field_index != start);
     
     return false;
@@ -597,45 +732,60 @@ static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_str
     pb_field_iterator_t iter;
     pb_field_init(&iter, fields, dest_struct);
     
-    /* Initialize size/has fields and apply default values */
     do
     {
         pb_type_t type;
         type = iter.pos->type;
     
+        /* Avoid crash on empty message types (zero fields) */
         if (iter.pos->tag == 0)
             continue;
         
         if (PB_ATYPE(type) == PB_ATYPE_STATIC)
         {
-            /* Initialize the size field for optional/repeated fields to 0. */
             if (PB_HTYPE(type) == PB_HTYPE_OPTIONAL)
             {
+                /* Set has_field to false. Still initialize the optional field
+                 * itself also. */
                 *(bool*)iter.pSize = false;
             }
             else if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
             {
+                /* Set array count to 0, no need to initialize contents. */
                 *(size_t*)iter.pSize = 0;
-                continue; /* Array is empty, no need to initialize contents */
+                continue;
             }
             
-            /* Initialize field contents to default value */
             if (PB_LTYPE(iter.pos->type) == PB_LTYPE_SUBMESSAGE)
             {
+                /* Initialize submessage to defaults */
                 pb_message_set_to_defaults((const pb_field_t *) iter.pos->ptr, iter.pData);
             }
             else if (iter.pos->ptr != NULL)
             {
+                /* Initialize to default value */
                 memcpy(iter.pData, iter.pos->ptr, iter.pos->data_size);
             }
             else
             {
+                /* Initialize to zeros */
                 memset(iter.pData, 0, iter.pos->data_size);
             }
         }
+        else if (PB_ATYPE(type) == PB_ATYPE_POINTER)
+        {
+            /* Initialize the pointer to NULL. */
+            *(void**)iter.pData = NULL;
+            
+            /* Initialize array count to 0. */
+            if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
+            {
+                *(size_t*)iter.pSize = 0;
+            }
+        }
         else if (PB_ATYPE(type) == PB_ATYPE_CALLBACK)
         {
-            continue; /* Don't overwrite callback */
+            /* Don't overwrite callback */
         }
     } while (pb_field_next(&iter));
 }
@@ -646,7 +796,7 @@ static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_str
 
 bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
 {
-    uint8_t fields_seen[(PB_MAX_REQUIRED_FIELDS + 7) / 8] = {0}; /* Used to check for required fields */
+    uint8_t fields_seen[(PB_MAX_REQUIRED_FIELDS + 7) / 8] = {0, 0, 0, 0, 0, 0, 0, 0};
     uint32_t extension_range_start = 0;
     pb_field_iterator_t iter;
     
@@ -722,7 +872,7 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
         } while (pb_field_next(&iter));
         
         /* Fixup if last field was also required. */
-        if (PB_HTYPE(last_type) == PB_HTYPE_REQUIRED && iter.pos->tag)
+        if (PB_HTYPE(last_type) == PB_HTYPE_REQUIRED && iter.pos->tag != 0)
             req_field_count++;
         
         /* Check the whole bytes */
@@ -742,8 +892,16 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
 
 bool checkreturn pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
 {
+    bool status;
     pb_message_set_to_defaults(fields, dest_struct);
-    return pb_decode_noinit(stream, fields, dest_struct);
+    status = pb_decode_noinit(stream, fields, dest_struct);
+    
+#ifdef PB_ENABLE_MALLOC
+    if (!status)
+        pb_release(fields, dest_struct);
+#endif
+    
+    return status;
 }
 
 bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
@@ -759,6 +917,62 @@ bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *
     return status;
 }
 
+#ifdef PB_ENABLE_MALLOC
+void pb_release(const pb_field_t fields[], void *dest_struct)
+{
+    pb_field_iterator_t iter;
+    pb_field_init(&iter, fields, dest_struct);
+    
+    do
+    {
+        pb_type_t type;
+        type = iter.pos->type;
+    
+        /* Avoid crash on empty message types (zero fields) */
+        if (iter.pos->tag == 0)
+            continue;
+        
+        if (PB_ATYPE(type) == PB_ATYPE_POINTER)
+        {
+            if (PB_HTYPE(type) == PB_HTYPE_REPEATED &&
+                (PB_LTYPE(type) == PB_LTYPE_STRING ||
+                 PB_LTYPE(type) == PB_LTYPE_BYTES))
+            {
+                /* Release entries in repeated string or bytes array */
+                void **pItem = *(void***)iter.pData;
+                size_t count = *(size_t*)iter.pSize;
+                while (count--)
+                {
+                    pb_free(*pItem);
+                    *pItem++ = NULL;
+                }
+            }
+            else if (PB_LTYPE(type) == PB_LTYPE_SUBMESSAGE)
+            {
+                /* Release fields in submessages */
+                void *pItem = *(void**)iter.pData;
+                size_t count = (pItem ? 1 : 0);
+                
+                if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
+                {
+                    count = *(size_t*)iter.pSize;   
+                }
+                
+                while (count--)
+                {
+                    pb_release((const pb_field_t*)iter.pos->ptr, pItem);
+                    pItem = (uint8_t*)pItem + iter.pos->data_size;
+                }
+            }
+            
+            /* Release main item */
+            pb_free(*(void**)iter.pData);
+            *(void**)iter.pData = NULL;
+        }
+    } while (pb_field_next(&iter));
+}
+#endif
+
 /* Field decoders */
 
 bool pb_decode_svarint(pb_istream_t *stream, int64_t *dest)
@@ -817,7 +1031,7 @@ bool pb_decode_fixed64(pb_istream_t *stream, void *dest)
     #endif   
 }
 
-bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     uint64_t value;
     if (!pb_decode_varint(stream, &value))
@@ -835,7 +1049,7 @@ bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, vo
     return true;
 }
 
-bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     uint64_t value;
     if (!pb_decode_varint(stream, &value))
@@ -851,7 +1065,7 @@ bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, v
     return true;
 }
 
-bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     int64_t value;
     if (!pb_decode_svarint(stream, &value))
@@ -867,51 +1081,80 @@ bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, v
     return true;
 }
 
-bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     UNUSED(field);
     return pb_decode_fixed32(stream, dest);
 }
 
-bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     UNUSED(field);
     return pb_decode_fixed64(stream, dest);
 }
 
-bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest)
-{
-    pb_bytes_array_t *x = (pb_bytes_array_t*)dest;
-    
-    uint32_t temp;
-    if (!pb_decode_varint32(stream, &temp))
-        return false;
-    x->size = temp;
-    
-    /* Check length, noting the space taken by the size_t header. */
-    if (x->size > field->data_size - offsetof(pb_bytes_array_t, bytes))
-        PB_RETURN_ERROR(stream, "bytes overflow");
-    
-    return pb_read(stream, x->bytes, x->size);
-}
-
-bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     uint32_t size;
+    pb_bytes_array_t *bdest;
+    
+    if (!pb_decode_varint32(stream, &size))
+        return false;
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+    {
+#ifndef PB_ENABLE_MALLOC
+        PB_RETURN_ERROR(stream, "no malloc support");
+#else
+        if (!allocate_field(stream, dest, PB_BYTES_ARRAY_T_ALLOCSIZE(size), 1))
+            return false;
+        bdest = *(pb_bytes_array_t**)dest;
+#endif
+    }
+    else
+    {
+        if (PB_BYTES_ARRAY_T_ALLOCSIZE(size) > field->data_size)
+            PB_RETURN_ERROR(stream, "bytes overflow");
+        bdest = (pb_bytes_array_t*)dest;
+    }
+    
+    bdest->size = size;
+    return pb_read(stream, bdest->bytes, size);
+}
+
+static bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest)
+{
+    uint32_t size;
+    size_t alloc_size;
     bool status;
     if (!pb_decode_varint32(stream, &size))
         return false;
     
-    /* Check length, noting the null terminator */
-    if (size + 1 > field->data_size)
-        PB_RETURN_ERROR(stream, "string overflow");
+    /* Space for null terminator */
+    alloc_size = size + 1;
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+    {
+#ifndef PB_ENABLE_MALLOC
+        PB_RETURN_ERROR(stream, "no malloc support");
+#else
+        if (!allocate_field(stream, dest, alloc_size, 1))
+            return false;
+        dest = *(void**)dest;
+#endif
+    }
+    else
+    {
+        if (alloc_size > field->data_size)
+            PB_RETURN_ERROR(stream, "string overflow");
+    }
     
     status = pb_read(stream, (uint8_t*)dest, size);
     *((uint8_t*)dest + size) = 0;
     return status;
 }
 
-bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest)
+static bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     bool status;
     pb_istream_t substream;

pb_decode.h

@@ -73,6 +73,9 @@ bool pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struc
  *
  * This can also be used for 'merging' two messages, i.e. update only the
  * fields that exist in the new message.
+ *
+ * Note: If this function returns with an error, it will not release any
+ * dynamically allocated fields. You will need to call pb_release() yourself.
  */
 bool pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
 
@@ -82,6 +85,14 @@ bool pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *des
  */
 bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
 
+#ifdef PB_ENABLE_MALLOC
+/* Release any allocated pointer fields. If you use dynamic allocation, you should
+ * call this for any successfully decoded message when you are done with it. If
+ * pb_decode() returns with an error, the message is already released.
+ */
+void pb_release(const pb_field_t fields[], void *dest_struct);
+#endif
+
 
 /**************************************
  * Functions for manipulating streams *

pb_encode.c

@@ -3,7 +3,6 @@
  * 2011 Petteri Aimonen <jpa@kapsi.fi>
  */
 
-#define NANOPB_INTERNALS
 #include "pb.h"
 #include "pb_encode.h"
 
@@ -174,11 +173,12 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
                 return false;
 
             /* Normally the data is stored directly in the array entries, but
-             * for pointer-type string fields, the array entries are actually
-             * string pointers. So we have to dereference once more to get to
-             * the character data. */
+             * for pointer-type string and bytes fields, the array entries are
+             * actually pointers themselves also. So we have to dereference once
+             * more to get to the actual data. */
             if (PB_ATYPE(field->type) == PB_ATYPE_POINTER &&
-                PB_LTYPE(field->type) == PB_LTYPE_STRING)
+                (PB_LTYPE(field->type) == PB_LTYPE_STRING ||
+                 PB_LTYPE(field->type) == PB_LTYPE_BYTES))
             {
                 if (!func(stream, field, *(const void* const*)p))
                     return false;      
@@ -378,6 +378,17 @@ bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const
     return pb_encode_submessage(stream, fields, src_struct);
 }
 
+bool pb_get_encoded_size(size_t *size, const pb_field_t fields[], const void *src_struct)
+{
+    pb_ostream_t stream = PB_OSTREAM_SIZING;
+    
+    if (!pb_encode(&stream, fields, src_struct))
+        return false;
+    
+    *size = stream.bytes_written;
+    return true;
+}
+
 /********************
  * Helper functions *
  ********************/
@@ -404,9 +415,9 @@ bool checkreturn pb_encode_svarint(pb_ostream_t *stream, int64_t value)
 {
     uint64_t zigzagged;
     if (value < 0)
-        zigzagged = (uint64_t)(~(value << 1));
+        zigzagged = ~((uint64_t)value << 1);
     else
-        zigzagged = (uint64_t)(value << 1);
+        zigzagged = (uint64_t)value << 1;
     
     return pb_encode_varint(stream, zigzagged);
 }
@@ -447,7 +458,7 @@ bool checkreturn pb_encode_fixed64(pb_ostream_t *stream, const void *value)
 
 bool checkreturn pb_encode_tag(pb_ostream_t *stream, pb_wire_type_t wiretype, uint32_t field_number)
 {
-    uint64_t tag = wiretype | (field_number << 3);
+    uint64_t tag = ((uint64_t)field_number << 3) | wiretype;
     return pb_encode_varint(stream, tag);
 }
 
@@ -543,7 +554,7 @@ bool checkreturn pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fie
 
 /* Field encoders */
 
-bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     int64_t value = 0;
     
@@ -561,7 +572,7 @@ bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, co
     return pb_encode_varint(stream, (uint64_t)value);
 }
 
-bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     uint64_t value = 0;
     
@@ -575,7 +586,7 @@ bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, c
     return pb_encode_varint(stream, value);
 }
 
-bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     int64_t value = 0;
     
@@ -589,36 +600,38 @@ bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, c
     return pb_encode_svarint(stream, value);
 }
 
-bool checkreturn pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     UNUSED(field);
     return pb_encode_fixed64(stream, src);
 }
 
-bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     UNUSED(field);
     return pb_encode_fixed32(stream, src);
 }
 
-bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
-    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+    const pb_bytes_array_t *bytes = (const pb_bytes_array_t*)src;
+    
+    if (src == NULL)
     {
-        const pb_bytes_ptr_t *bytes = (const pb_bytes_ptr_t*)src;
-        return pb_encode_string(stream, bytes->bytes, bytes->size);
+        /* Threat null pointer as an empty bytes field */
+        return pb_encode_string(stream, NULL, 0);
     }
-    else
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_STATIC &&
+        PB_BYTES_ARRAY_T_ALLOCSIZE(bytes->size) > field->data_size)
     {
-        const pb_bytes_array_t *bytes = (const pb_bytes_array_t*)src;
-        if (bytes->size + offsetof(pb_bytes_array_t, bytes) > field->data_size)
-            PB_RETURN_ERROR(stream, "bytes size exceeded");
-
-        return pb_encode_string(stream, bytes->bytes, bytes->size);
+        PB_RETURN_ERROR(stream, "bytes size exceeded");
     }
+    
+    return pb_encode_string(stream, bytes->bytes, bytes->size);
 }
 
-bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     /* strnlen() is not always available, so just use a loop */
     size_t size = 0;
@@ -628,16 +641,23 @@ bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, co
     if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
         max_size = (size_t)-1;
 
-    while (size < max_size && *p != '\0')
+    if (src == NULL)
     {
-        size++;
-        p++;
+        size = 0; /* Threat null pointer as an empty string */
+    }
+    else
+    {
+        while (size < max_size && *p != '\0')
+        {
+            size++;
+            p++;
+        }
     }
 
     return pb_encode_string(stream, (const uint8_t*)src, size);
 }
 
-bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+static bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     if (field->ptr == NULL)
         PB_RETURN_ERROR(stream, "invalid field descriptor");

pb_encode.h

@@ -71,6 +71,10 @@ bool pb_encode(pb_ostream_t *stream, const pb_field_t fields[], const void *src_
  */
 bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
 
+/* Encode the message to get the size of the encoded data, but do not store
+ * the data. */
+bool pb_get_encoded_size(size_t *size, const pb_field_t fields[], const void *src_struct);
+
 /**************************************
  * Functions for manipulating streams *
  **************************************/

tests/Makefile

@@ -4,3 +4,18 @@ all:
 clean:
 	scons -c
 
+coverage:
+	rm -rf build coverage
+
+	# LCOV does not like the newer gcov format
+	scons CC=gcc-4.6 CXX=gcc-4.6
+
+	# We are only interested in pb_encode.o and pb_decode.o
+	find build -name '*.gcda' -and \! \( -name '*pb_encode*' -or -name '*pb_decode*' \) -exec rm '{}' \;
+
+	# Collect the data
+	mkdir build/coverage
+	lcov --base-directory . --directory build/ --gcov-tool gcov-4.6 -c -o build/coverage/nanopb.info
+
+	# Generate HTML
+	genhtml -o build/coverage build/coverage/nanopb.info

tests/SConstruct

@@ -50,6 +50,7 @@ if not env.GetOption('clean'):
     stdint = conf.CheckCHeader('stdint.h')
     stddef = conf.CheckCHeader('stddef.h')
     string = conf.CheckCHeader('string.h')
+    stdlib = conf.CheckCHeader('stdlib.h')
     if not stdbool or not stdint or not stddef or not string:
         conf.env.Append(CPPDEFINES = {'PB_SYSTEM_HEADER': '\\"pb_syshdr.h\\"'})
         conf.env.Append(CPPPATH = "#../extra")
@@ -58,7 +59,8 @@ if not env.GetOption('clean'):
         if stdint: conf.env.Append(CPPDEFINES = {'HAVE_STDINT_H': 1})
         if stddef: conf.env.Append(CPPDEFINES = {'HAVE_STDDEF_H': 1})
         if string: conf.env.Append(CPPDEFINES = {'HAVE_STRING_H': 1})
-        
+        if stdlib: conf.env.Append(CPPDEFINES = {'HAVE_STDLIB_H': 1})
+    
     # Check if we can use pkg-config to find protobuf include path
     status, output = conf.TryAction('pkg-config protobuf --variable=includedir > $TARGET')
     if status:
@@ -70,7 +72,7 @@ if not env.GetOption('clean'):
     if 'gcc' in env['CC']:
         if conf.CheckLib('mudflap'):
             conf.env.Append(CCFLAGS = '-fmudflap')
-            conf.env.Append(LINKFLAGS = '-lmudflap -fmudflap')
+            conf.env.Append(LINKFLAGS = '-fmudflap')
     
     # Check if we can use extra strict warning flags (only with GCC)
     extra = '-Wcast-qual -Wlogical-op -Wconversion'
@@ -89,9 +91,9 @@ if 'gcc' in env['CC']:
     # GNU Compiler Collection
     
     # Debug info, warnings as errors
-    env.Append(CFLAGS = '-ansi -pedantic -g -O0 -Wall -Werror --coverage -fstack-protector-all')
+    env.Append(CFLAGS = '-ansi -pedantic -g -O0 -Wall -Werror -fprofile-arcs -ftest-coverage -fstack-protector-all')
     env.Append(CORECFLAGS = '-Wextra')
-    env.Append(LINKFLAGS = '--coverage')
+    env.Append(LINKFLAGS = '-g --coverage')
     
     # We currently need uint64_t anyway, even though ANSI C90 otherwise..
     env.Append(CFLAGS = '-Wno-long-long')

tests/alltypes_pointer/SConscript

@@ -3,21 +3,46 @@
 
 Import("env")
 
+# We need our own pb_decode.o for the malloc support
+env = env.Clone()
+env.Append(CPPDEFINES = {'PB_ENABLE_MALLOC': 1});
+
+# Disable libmudflap, because it will confuse valgrind
+# and other memory leak detection tools.
+if '-fmudflap' in env["CCFLAGS"]:
+    env["CCFLAGS"].remove("-fmudflap")
+    env["LINKFLAGS"].remove("-fmudflap")
+    env["LIBS"].remove("mudflap")
+
+strict = env.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_with_malloc.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_with_malloc.o", "$NANOPB/pb_encode.c")
+
 c = Copy("$TARGET", "$SOURCE")
 env.Command("alltypes.proto", "#alltypes/alltypes.proto", c)
 
 env.NanopbProto(["alltypes", "alltypes.options"])
-enc = env.Program(["encode_alltypes_pointer.c", "alltypes.pb.c", "$COMMON/pb_encode.o"])
+enc = env.Program(["encode_alltypes_pointer.c", "alltypes.pb.c", "pb_encode_with_malloc.o"])
+dec = env.Program(["decode_alltypes_pointer.c", "alltypes.pb.c", "pb_decode_with_malloc.o"])
 
-refdec = "$BUILD/alltypes/decode_alltypes$PROGSUFFIX"
-
-# Encode and compare results
+# Encode and compare results to non-pointer alltypes test case
 env.RunTest(enc)
-env.RunTest("decode_alltypes.output", [refdec, "encode_alltypes_pointer.output"])
 env.Compare(["encode_alltypes_pointer.output", "$BUILD/alltypes/encode_alltypes.output"])
 
+# Decode (under valgrind if available)
+valgrind = env.WhereIs('valgrind')
+kwargs = {}
+if valgrind:
+    kwargs['COMMAND'] = valgrind
+    kwargs['ARGS'] = ["-q", dec[0].abspath]
+
+env.RunTest("decode_alltypes.output", [dec, "encode_alltypes_pointer.output"], **kwargs)
+
 # Do the same thing with the optional fields present
 env.RunTest("optionals.output", enc, ARGS = ['1'])
-env.RunTest("optionals.decout", [refdec, "optionals.output"], ARGS = ['1'])
 env.Compare(["optionals.output", "$BUILD/alltypes/optionals.output"])
 
+kwargs['ARGS'] = kwargs.get('ARGS', []) + ['1']
+env.RunTest("optionals.decout", [dec, "optionals.output"], **kwargs)
+

tests/alltypes_pointer/decode_alltypes_pointer.c

@@ -0,0 +1,173 @@
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <pb_decode.h>
+#include "alltypes.pb.h"
+#include "test_helpers.h"
+
+#define TEST(x) if (!(x)) { \
+    fprintf(stderr, "Test " #x " failed.\n"); \
+    status = false; \
+    }
+
+/* This function is called once from main(), it handles
+   the decoding and checks the fields. */
+bool check_alltypes(pb_istream_t *stream, int mode)
+{
+    bool status = true;
+    AllTypes alltypes;
+    
+    /* Fill with garbage to better detect initialization errors */
+    memset(&alltypes, 0xAA, sizeof(alltypes));
+    
+    if (!pb_decode(stream, AllTypes_fields, &alltypes))
+        return false;
+    
+    TEST(alltypes.req_int32     && *alltypes.req_int32         == -1001);
+    TEST(alltypes.req_int64     && *alltypes.req_int64         == -1002);
+    TEST(alltypes.req_uint32    && *alltypes.req_uint32        == 1003);
+    TEST(alltypes.req_uint64    && *alltypes.req_uint64        == 1004);
+    TEST(alltypes.req_sint32    && *alltypes.req_sint32        == -1005);
+    TEST(alltypes.req_sint64    && *alltypes.req_sint64        == -1006);
+    TEST(alltypes.req_bool      && *alltypes.req_bool          == true);
+    
+    TEST(alltypes.req_fixed32   && *alltypes.req_fixed32       == 1008);
+    TEST(alltypes.req_sfixed32  && *alltypes.req_sfixed32      == -1009);
+    TEST(alltypes.req_float     && *alltypes.req_float         == 1010.0f);
+    
+    TEST(alltypes.req_fixed64   && *alltypes.req_fixed64       == 1011);
+    TEST(alltypes.req_sfixed64  && *alltypes.req_sfixed64      == -1012);
+    TEST(alltypes.req_double    && *alltypes.req_double        == 1013.0f);
+    
+    TEST(alltypes.req_string    && strcmp(alltypes.req_string, "1014") == 0);
+    TEST(alltypes.req_bytes     && alltypes.req_bytes->size == 4);
+    TEST(alltypes.req_bytes     && memcmp(&alltypes.req_bytes->bytes, "1015", 4) == 0);
+    TEST(alltypes.req_submsg    && alltypes.req_submsg->substuff1
+                                && strcmp(alltypes.req_submsg->substuff1, "1016") == 0);
+    TEST(alltypes.req_submsg    && alltypes.req_submsg->substuff2
+                                && *alltypes.req_submsg->substuff2 == 1016);
+    TEST(*alltypes.req_enum == MyEnum_Truth);
+
+    TEST(alltypes.rep_int32_count == 5 && alltypes.rep_int32[4] == -2001 && alltypes.rep_int32[0] == 0);
+    TEST(alltypes.rep_int64_count == 5 && alltypes.rep_int64[4] == -2002 && alltypes.rep_int64[0] == 0);
+    TEST(alltypes.rep_uint32_count == 5 && alltypes.rep_uint32[4] == 2003 && alltypes.rep_uint32[0] == 0);
+    TEST(alltypes.rep_uint64_count == 5 && alltypes.rep_uint64[4] == 2004 && alltypes.rep_uint64[0] == 0);
+    TEST(alltypes.rep_sint32_count == 5 && alltypes.rep_sint32[4] == -2005 && alltypes.rep_sint32[0] == 0);
+    TEST(alltypes.rep_sint64_count == 5 && alltypes.rep_sint64[4] == -2006 && alltypes.rep_sint64[0] == 0);
+    TEST(alltypes.rep_bool_count == 5 && alltypes.rep_bool[4] == true && alltypes.rep_bool[0] == false);
+    
+    TEST(alltypes.rep_fixed32_count == 5 && alltypes.rep_fixed32[4] == 2008 && alltypes.rep_fixed32[0] == 0);
+    TEST(alltypes.rep_sfixed32_count == 5 && alltypes.rep_sfixed32[4] == -2009 && alltypes.rep_sfixed32[0] == 0);
+    TEST(alltypes.rep_float_count == 5 && alltypes.rep_float[4] == 2010.0f && alltypes.rep_float[0] == 0.0f);
+    
+    TEST(alltypes.rep_fixed64_count == 5 && alltypes.rep_fixed64[4] == 2011 && alltypes.rep_fixed64[0] == 0);
+    TEST(alltypes.rep_sfixed64_count == 5 && alltypes.rep_sfixed64[4] == -2012 && alltypes.rep_sfixed64[0] == 0);
+    TEST(alltypes.rep_double_count == 5 && alltypes.rep_double[4] == 2013.0 && alltypes.rep_double[0] == 0.0);
+    
+    TEST(alltypes.rep_string_count == 5 && strcmp(alltypes.rep_string[4], "2014") == 0 && alltypes.rep_string[0][0] == '\0');
+    TEST(alltypes.rep_bytes_count == 5 && alltypes.rep_bytes[4]->size == 4 && alltypes.rep_bytes[0]->size == 0);
+    TEST(memcmp(&alltypes.rep_bytes[4]->bytes, "2015", 4) == 0);
+
+    TEST(alltypes.rep_submsg_count == 5);
+    TEST(strcmp(alltypes.rep_submsg[4].substuff1, "2016") == 0 && alltypes.rep_submsg[0].substuff1[0] == '\0');
+    TEST(*alltypes.rep_submsg[4].substuff2 == 2016 && *alltypes.rep_submsg[0].substuff2 == 0);
+    TEST(*alltypes.rep_submsg[4].substuff3 == 2016 && alltypes.rep_submsg[0].substuff3 == NULL);
+    
+    TEST(alltypes.rep_enum_count == 5 && alltypes.rep_enum[4] == MyEnum_Truth && alltypes.rep_enum[0] == MyEnum_Zero);
+    TEST(alltypes.rep_emptymsg_count == 5);
+
+    if (mode == 0)
+    {
+        /* Expect that optional values are not present */
+        TEST(alltypes.opt_int32         == NULL);
+        TEST(alltypes.opt_int64         == NULL);
+        TEST(alltypes.opt_uint32        == NULL);
+        TEST(alltypes.opt_uint64        == NULL);
+        TEST(alltypes.opt_sint32        == NULL);
+        TEST(alltypes.opt_sint64        == NULL);
+        TEST(alltypes.opt_bool          == NULL);
+        
+        TEST(alltypes.opt_fixed32       == NULL);
+        TEST(alltypes.opt_sfixed32      == NULL);
+        TEST(alltypes.opt_float         == NULL);
+        TEST(alltypes.opt_fixed64       == NULL);
+        TEST(alltypes.opt_sfixed64      == NULL);
+        TEST(alltypes.opt_double        == NULL);
+        
+        TEST(alltypes.opt_string        == NULL);
+        TEST(alltypes.opt_bytes         == NULL);
+        TEST(alltypes.opt_submsg        == NULL);
+        TEST(alltypes.opt_enum          == NULL);
+    }
+    else
+    {
+        /* Expect filled-in values */
+        TEST(alltypes.opt_int32 && *alltypes.opt_int32      == 3041);
+        TEST(alltypes.opt_int64 && *alltypes.opt_int64      == 3042);
+        TEST(alltypes.opt_uint32 && *alltypes.opt_uint32    == 3043);
+        TEST(alltypes.opt_uint64 && *alltypes.opt_uint64    == 3044);
+        TEST(alltypes.opt_sint32 && *alltypes.opt_sint32    == 3045);
+        TEST(alltypes.opt_sint64 && *alltypes.opt_sint64    == 3046);
+        TEST(alltypes.opt_bool && *alltypes.opt_bool        == true);
+        
+        TEST(alltypes.opt_fixed32 && *alltypes.opt_fixed32  == 3048);
+        TEST(alltypes.opt_sfixed32 && *alltypes.opt_sfixed32== 3049);
+        TEST(alltypes.opt_float && *alltypes.opt_float      == 3050.0f);
+        TEST(alltypes.opt_fixed64 && *alltypes.opt_fixed64  == 3051);
+        TEST(alltypes.opt_sfixed64 && *alltypes.opt_sfixed64== 3052);
+        TEST(alltypes.opt_double && *alltypes.opt_double    == 3053.0);
+        
+        TEST(alltypes.opt_string && strcmp(alltypes.opt_string, "3054") == 0);
+        TEST(alltypes.opt_bytes && alltypes.opt_bytes->size == 4);
+        TEST(alltypes.opt_bytes && memcmp(&alltypes.opt_bytes->bytes, "3055", 4) == 0);
+        TEST(alltypes.opt_submsg && strcmp(alltypes.opt_submsg->substuff1, "3056") == 0);
+        TEST(alltypes.opt_submsg && *alltypes.opt_submsg->substuff2 == 3056);
+        TEST(alltypes.opt_enum && *alltypes.opt_enum == MyEnum_Truth);
+        TEST(alltypes.opt_emptymsg);
+    }
+    
+    TEST(alltypes.req_limits->int32_min && *alltypes.req_limits->int32_min   == INT32_MIN);
+    TEST(alltypes.req_limits->int32_max && *alltypes.req_limits->int32_max   == INT32_MAX);
+    TEST(alltypes.req_limits->uint32_min && *alltypes.req_limits->uint32_min == 0);
+    TEST(alltypes.req_limits->uint32_max && *alltypes.req_limits->uint32_max == UINT32_MAX);
+    TEST(alltypes.req_limits->int64_min && *alltypes.req_limits->int64_min   == INT64_MIN);
+    TEST(alltypes.req_limits->int64_max && *alltypes.req_limits->int64_max   == INT64_MAX);
+    TEST(alltypes.req_limits->uint64_min && *alltypes.req_limits->uint64_min == 0);
+    TEST(alltypes.req_limits->uint64_max && *alltypes.req_limits->uint64_max == UINT64_MAX);
+    TEST(alltypes.req_limits->enum_min && *alltypes.req_limits->enum_min     == HugeEnum_Negative);
+    TEST(alltypes.req_limits->enum_max && *alltypes.req_limits->enum_max     == HugeEnum_Positive);
+    
+    TEST(alltypes.end && *alltypes.end == 1099);
+
+    pb_release(AllTypes_fields, &alltypes);
+
+    return status;
+}
+
+int main(int argc, char **argv)
+{
+    uint8_t buffer[1024];
+    size_t count;
+    pb_istream_t stream;
+    
+    /* Whether to expect the optional values or the default values. */
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+    
+    /* Read the data into buffer */
+    SET_BINARY_MODE(stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
+    
+    /* Construct a pb_istream_t for reading from the buffer */
+    stream = pb_istream_from_buffer(buffer, count);
+    
+    /* Decode and verify the message */
+    if (!check_alltypes(&stream, mode))
+    {
+        fprintf(stderr, "Test failed: %s\n", PB_GET_ERROR(&stream));
+        return 1;
+    }
+    else
+    {
+        return 0;
+    }
+}

tests/alltypes_pointer/encode_alltypes_pointer.c

@@ -27,7 +27,7 @@ int main(int argc, char **argv)
     int64_t     req_sfixed64      = -1012;
     double      req_double        = 1013.0;
     char*       req_string        = "1014";
-    pb_bytes_ptr_t req_bytes      = {4, (uint8_t*)"1015"};
+    PB_BYTES_ARRAY_T(4) req_bytes = {4, {'1', '0', '1', '5'}};
     static int32_t req_substuff   = 1016;
     SubMessage  req_submsg        = {"1016", &req_substuff};
     MyEnum      req_enum          = MyEnum_Truth;
@@ -50,7 +50,8 @@ int main(int argc, char **argv)
     int64_t     rep_sfixed64[5]   = {0, 0, 0, 0, -2012};
     double      rep_double[5]     = {0, 0, 0, 0, 2013.0f};
     char*       rep_string[5]     = {"", "", "", "", "2014"};
-    pb_bytes_ptr_t rep_bytes[5]   = {{0,0}, {0,0}, {0,0}, {0,0}, {4, (uint8_t*)"2015"}};
+    static PB_BYTES_ARRAY_T(4) rep_bytes_4 = {4, {'2', '0', '1', '5'}};
+    pb_bytes_array_t *rep_bytes[5]= {NULL, NULL, NULL, NULL, (pb_bytes_array_t*)&rep_bytes_4};
     static int32_t rep_sub2zero   = 0;
     static int32_t rep_substuff2  = 2016;
     static uint32_t rep_substuff3 = 2016;
@@ -77,7 +78,7 @@ int main(int argc, char **argv)
     int64_t     opt_sfixed64      = 3052;
     double      opt_double        = 3053.0;
     char*       opt_string        = "3054";
-    pb_bytes_ptr_t opt_bytes      = {4, (uint8_t*)"3055"};
+    PB_BYTES_ARRAY_T(4) opt_bytes = {4, {'3', '0', '5', '5'}};
     static int32_t opt_substuff   = 3056;
     SubMessage  opt_submsg        = {"3056", &opt_substuff};
     MyEnum      opt_enum          = MyEnum_Truth;
@@ -117,7 +118,7 @@ int main(int argc, char **argv)
     alltypes.req_sfixed64      = &req_sfixed64;
     alltypes.req_double        = &req_double;
     alltypes.req_string        = req_string;
-    alltypes.req_bytes         = &req_bytes;
+    alltypes.req_bytes         = (pb_bytes_array_t*)&req_bytes;
     alltypes.req_submsg        = &req_submsg;
     alltypes.req_enum          = &req_enum;
     alltypes.req_emptymsg      = &req_emptymsg;
@@ -159,7 +160,7 @@ int main(int argc, char **argv)
         alltypes.opt_sfixed64      = &opt_sfixed64;
         alltypes.opt_double        = &opt_double;
         alltypes.opt_string        = opt_string;
-        alltypes.opt_bytes         = &opt_bytes;
+        alltypes.opt_bytes         = (pb_bytes_array_t*)&opt_bytes;
         alltypes.opt_submsg        = &opt_submsg;
         alltypes.opt_enum          = &opt_enum;
         alltypes.opt_emptymsg      = &opt_emptymsg;

tests/buffer_only/SConscript

@@ -4,8 +4,6 @@ Import("env")
 
 # Take copy of the files for custom build.
 c = Copy("$TARGET", "$SOURCE")
-env.Command("pb_encode.c", "#../pb_encode.c", c)
-env.Command("pb_decode.c", "#../pb_decode.c", c)
 env.Command("alltypes.pb.h", "$BUILD/alltypes/alltypes.pb.h", c)
 env.Command("alltypes.pb.c", "$BUILD/alltypes/alltypes.pb.c", c)
 env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
@@ -15,9 +13,15 @@ env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)
 opts = env.Clone()
 opts.Append(CPPDEFINES = {'PB_BUFFER_ONLY': 1})
 
+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_bufonly.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_bufonly.o", "$NANOPB/pb_encode.c")
+
 # Now build and run the test normally.
-enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
-dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_bufonly.o"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_bufonly.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])

tests/encode_unittests/encode_unittests.c

@@ -281,6 +281,15 @@ int main()
         TEST(WRITES(pb_encode_delimited(&s, IntegerContainer_fields, &msg),
                     "\x09\x0A\x07\x0A\x05\x01\x02\x03\x04\x05"))
     }
+
+    {
+        IntegerContainer msg = {{5, {1,2,3,4,5}}};
+        size_t size;
+        
+        COMMENT("Test pb_get_encoded_size.")
+        TEST(pb_get_encoded_size(&size, IntegerContainer_fields, &msg) &&
+             size == 9);
+    }
     
     {
         uint8_t buffer[10];

tests/extensions/decode_extensions.c

@@ -49,7 +49,9 @@ int main(int argc, char **argv)
     }
 
     /* Check that the extensions decoded properly */
+    TEST(ext1.found)
     TEST(extensionfield1 == 12345)
+    TEST(ext2.found)
     TEST(strcmp(extensionfield2.test1, "test") == 0)
     TEST(extensionfield2.test2 == 54321)
     

tests/extensions/extensions.proto

@@ -1,7 +1,7 @@
 import 'alltypes.proto';
 
 extend AllTypes {
-    optional int32 AllTypes_extensionfield1 = 255;
+    optional int32 AllTypes_extensionfield1 = 255 [default = 5];
 }
 
 message ExtensionMessage {

tests/field_size_16/SConscript

@@ -5,8 +5,6 @@ Import("env")
 
 # Take copy of the files for custom build.
 c = Copy("$TARGET", "$SOURCE")
-env.Command("pb_encode.c", "#../pb_encode.c", c)
-env.Command("pb_decode.c", "#../pb_decode.c", c)
 env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
 env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)
 
@@ -16,9 +14,15 @@ env.NanopbProto(["alltypes", "alltypes.options"])
 opts = env.Clone()
 opts.Append(CPPDEFINES = {'PB_FIELD_16BIT': 1})
 
+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_fields16.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_fields16.o", "$NANOPB/pb_encode.c")
+
 # Now build and run the test normally.
-enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
-dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_fields16.o"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_fields16.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])

tests/field_size_32/SConscript

@@ -5,8 +5,6 @@ Import("env")
 
 # Take copy of the files for custom build.
 c = Copy("$TARGET", "$SOURCE")
-env.Command("pb_encode.c", "#../pb_encode.c", c)
-env.Command("pb_decode.c", "#../pb_decode.c", c)
 env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
 env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)
 
@@ -16,9 +14,15 @@ env.NanopbProto(["alltypes", "alltypes.options"])
 opts = env.Clone()
 opts.Append(CPPDEFINES = {'PB_FIELD_32BIT': 1})
 
+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_fields32.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_fields32.o", "$NANOPB/pb_encode.c")
+
 # Now build and run the test normally.
-enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
-dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_fields32.o"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_fields32.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])

tests/no_errmsg/SConscript

@@ -4,8 +4,6 @@ Import("env")
 
 # Take copy of the files for custom build.
 c = Copy("$TARGET", "$SOURCE")
-env.Command("pb_encode.c", "#../pb_encode.c", c)
-env.Command("pb_decode.c", "#../pb_decode.c", c)
 env.Command("alltypes.pb.h", "$BUILD/alltypes/alltypes.pb.h", c)
 env.Command("alltypes.pb.c", "$BUILD/alltypes/alltypes.pb.c", c)
 env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
@@ -15,9 +13,15 @@ env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)
 opts = env.Clone()
 opts.Append(CPPDEFINES = {'PB_NO_ERRMSG': 1})
 
+# Build new version of core
+strict = opts.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_noerr.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_noerr.o", "$NANOPB/pb_encode.c")
+
 # Now build and run the test normally.
-enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
-dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode_noerr.o"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode_noerr.o"])
 
 env.RunTest(enc)
 env.RunTest([dec, "encode_alltypes.output"])

tests/site_scons/site_init.py

@@ -19,19 +19,24 @@ def add_nanopb_builders(env):
         else:
             infile = None
         
-        args = [str(source[0])]
+        if env.has_key("COMMAND"):
+            args = [env["COMMAND"]]
+        else:
+            args = [str(source[0])]
+        
         if env.has_key('ARGS'):
             args.extend(env['ARGS'])
         
+        print 'Command line: ' + str(args)
         pipe = subprocess.Popen(args,
                                 stdin = infile,
                                 stdout = open(str(target[0]), 'w'),
                                 stderr = sys.stderr)
         result = pipe.wait()
         if result == 0:
-            print '\033[32m[ OK ]\033[0m   Ran ' + str(source[0])
+            print '\033[32m[ OK ]\033[0m   Ran ' + args[0]
         else:
-            print '\033[31m[FAIL]\033[0m   Program ' + str(source[0]) + ' returned ' + str(result)
+            print '\033[31m[FAIL]\033[0m   Program ' + args[0] + ' returned ' + str(result)
         return result
         
     run_test_builder = Builder(action = run_test,

tests/splint/SConscript

@@ -0,0 +1,13 @@
+# Check the nanopb core using splint
+
+Import('env')
+
+p = env.WhereIs('splint')
+
+if p:
+    env.Command('pb_decode.splint', '$NANOPB/pb_decode.c',
+        'splint -f splint/splint.rc $SOURCE 2> $TARGET')
+
+    env.Command('pb_encode.splint', '$NANOPB/pb_encode.c',
+        'splint -f splint/splint.rc $SOURCE 2> $TARGET')
+

tests/splint/splint.rc

@@ -0,0 +1,37 @@
++checks
++partial
++matchanyintegral
++strictlib
+-isoreserved		# to be fixed in 0.3
+-nullassign
+-predboolint
+-predboolptr
++ptrnegate
+-switchloopbreak
++ignoresigns
+-infloopsuncon
+-type
+
+# splint's memory checks don't quite work without annotations
+-mustfreeonly
+-compmempass
+-nullret
+-observertrans
+-statictrans
+-compdestroy
+-nullpass
+-nullstate
+-compdef
+-usereleased
+-temptrans
+-dependenttrans
+-kepttrans
+-branchstate
+-immediatetrans
+
+# These tests give false positives, compiler typically has
+# better warnings for these.
+-noret
+-noeffect
+-usedef
+